CCPEM Pipeliner
Guide

CCP-EM Jobs: Validation and Map Processing

CCP-EM jobs for model validation and scoring, and map processing


Overview

This page covers the CCP-EM job types in ccpem-pipeliner that perform atomic model validation, scoring, and map processing. These jobs let you assess model quality (geometry, map fit, glycan conformation, sequence agreement, rigid-body decomposition), compute confidence maps and difference maps, sharpen or denoise reconstructions, and manipulate map geometry — all as discrete, reproducible pipeline steps. Because no CCP-EM job supports continuation, each run creates a new job with a new sequential number.


Usage

All CCP-EM jobs follow the same pattern as any other pipeliner job: import PipelinerProject from pipeliner.api.manage_project, set job options via job.joboptions["key"].value = ..., and call proj.run_job() or proj.schedule_job(). Because none of these jobs set can_continue = True, calling get_continuation_job() on them raises NotImplementedError; use run_job(overwrite=...) to redo a step.

Job options are always set on the .value property of the entry in job.joboptions. Multiple-input options (e.g. input_models) hold a List[str] and serialise their values with ::: as the separator in run.job and star files.

Availability of each job depends on whether its required external programs are on the process PATH at the time the job object is created. If a required executable is missing, the job loads normally with job.jobinfo.is_available == False and job.jobinfo.unavailable_message populated; it can still be scheduled and will fail at execution time.


Model validation and scoring

pipeliner.atomic_model_validation — Atomic model validation (ModelValidate)

Runs multiple validation metrics against an atomic model and optional map:

Option keyLabelTypeNotes
input_modelInput modelInputNode (required).cif, .pdb, .ent
quick_evalQuick evaluation?BooleanRuns only Molprobity clashes and TEMPy SMOC when True; deactivates FSC, global, FDR, CheckMySequence
run_molprobityMolprobity (Geometry statistics)BooleanRequires molprobity.molprobity on PATH
run_fscServalcat FSC (Model-map FSC)BooleanDeactivated when quick_eval=True
run_globTEMPy global (Real space map fit)BooleanRequires TEMPy.scores; deactivated when quick_eval=True
run_smocTEMPy SMOC (Per-residue map fit)BooleanRequires TEMPy.smoc
run_fdrFDR backbone (Per-residue backbone trace)BooleanDeactivated when quick_eval=True
run_checkmysequenceCheckMySequence (Sequence agreement)BooleanDeactivated when quick_eval=True
input_mapInput mapInputNode.mrc, .ccp4, .map; required for global/SMOC/FDR/CheckMySequence metrics
resolutionResolutionFloatÅ; required when run_glob=True or run_smoc=True
input_halfmap1Input halfmap 1InputNodeFor model-map FSC
input_halfmap2Input halfmap 2InputNodeFor model-map FSC
input_maskInput map maskInputNode.mrc, .ccp4, .map; used by the TEMPy global score and, when use_mask_fsc=True, the model-map FSC
use_mask_fscUse input mask for FSC?Boolean—
input_contourInput contour threshold?BooleanUse contour for masking when no mask is provided
contourlevelContour LevelFloatDefault -100
input_fdrmapInput FDR confidence mapInputNodePre-calculated confidence map
input_sequenceInput sample sequenceInputNodeFor CheckMySequence

Required external program: gemmi. Additional programs are checked when the job options are validated (for example at scheduling), depending on which metrics are enabled (molprobity.molprobity, TEMPy.scores, TEMPy.smoc, checkmysequence, and the FDRcontrol script when no FDR map is given). servalcat is not checked.

Output directory: ModelValidation/


molprobity.atomic_model_validation.geometry — Molprobity (ValidateGeometry)

Assesses atomic model stereo-chemistry: clashes, Ramachandran angles and Z-scores, side-chain rotamers, Cβ deviations, cis-peptides. Up to three models can be compared in one job.

Option keyLabelTypeNotes
input_modelsInput modelsMultiInputNode (required).cif, .pdb; maximum 3
run_globalComprehensive assessment?BooleanFull molprobity run; requires molprobity.molprobity
run_clashscoreClashscore?BooleanRequires molprobity.clashscore; deactivated when run_global=True
run_ramaRamalyze?BooleanRequires molprobity.ramalyze; deactivated when run_global=True
run_rotaRotalyze?BooleanRequires molprobity.rotalyze; deactivated when run_global=True
resolutionResolutionFloatOptional; used to report resolution-range percentiles

At least one run mode must be selected, or the job raises a validation error.

Required external program: molprobity.molprobity (CCP4 ≥ 8.0.015). Input .cif files are automatically converted to .pdb via gemmi convert --shorten before being passed to Molprobity.

Outputs (vary by mode): per-model *_molprobity.out (NODE_LOGFILE), *_clashscore.json, *_ramalyze.json, *_rotalyze.json (NODE_EVALUATIONMETRIC). CIF inputs also produce a converted .pdb (NODE_ATOMCOORDS).

Output directory: Molprobity/


tempy.atomic_model_validation.map_fit — TEMPy Scores (TEMPyMapFit)

Computes global map-fit scores (CCC, MI, model overlap) and per-residue SMOC scores. Up to three models can be assessed against one map.

Option keyLabelTypeNotes
input_modelsInput modelsMultiInputNode (required).cif, .pdb; maximum 3
input_mapInput mapInputNode (required).mrc, .ccp4, .map
resolutionResolutionFloat (required)Å
run_global_onlyCalculate global scores only?BooleanDeactivated when run_smoc_only=True
run_smoc_onlyCalculate SMOC score only?BooleanDeactivated when run_global_only=True
input_maskInput map maskInputNodePreferred over contour; deactivated when run_smoc_only=True
input_contourInput contour threshold?BooleanDeactivated when mask is supplied or run_smoc_only=True
contourlevelContour LevelFloatDefault -9999; used when input_contour=True
calc_contourAuto estimate contour?BooleanEstimates contour from map σ

When global scores are enabled, one of mask, contour, or auto-contour must be provided, or the job raises a validation error.

Required external programs: TEMPy.smoc and TEMPy.scores (pip install biotempy, requires biotempy ≥ 2.2.0a3).

Outputs (vary by mode): per-model-map globscores_<modelid>_vs_<mapid>.csv (NODE_EVALUATIONMETRIC) and <modelid>_<mapid>_smoc_score.json (NODE_EVALUATIONMETRIC).

Output directory: TEMPyScores/


privateer.atomic_model_validation.glycans — Privateer (PrivateerValidate)

Evaluates and improves atomic structures of carbohydrates and N-glycans.

Option keyLabelTypeNotes
input_modelInput modelInputNode (required).cif, .pdb
input_mapInput mapInputNode (required).mrc, .ccp4, .map
resolutionResolutionFloat (required)Å
mask_radiusMask radiusFloatDefault 2.5; hard min 1.0
expression_system_modeValidate against:MultipleChoiceundefined (default), bacterial, fungal, yeast, plant, insect, mammalian, human
glytoucanCheck in Glytoucan/Glyconnect?BooleanDefault True
closestmatchFind closest match?BooleanDeactivated when glytoucan=False
allpermutationsGenerate all permutations for search?BooleanDeactivated when glytoucan=False or closestmatch=False; expensive for O-glycans
undefinedsugarUndefined sugar present?BooleanEnables custom sugar group
undefinedsugar_codeAnalyse sugar with codeStringPDB 3-letter code; required when undefinedsugar=True
input_anomerSugar anomerMultipleChoicealpha (default), beta
input_handednessSugar handednessMultipleChoice-D- (default), -L-
input_ring_conformationRing conformationMultipleChoicepyranose (default), furanose
input_conformation_pyranosePyranose conformationMultipleChoice38 options, default 4c1
input_conformation_furanoseFuranose conformationMultipleChoice20 options, default 3t2
ring_oxygenAtom code of anomeric OxygenStringDefault O5
ring_C1 … ring_C5Atom code of ring C1 …StringDefaults C1–C5
ncpusNumber of coresIntDefault 1; increase to parallelise

Required external program: privateer.

Output directory: Privateer/


dpiscore.interface_validation — DPI-Score (DPIscore)

Assesses protein-protein interface quality using a pre-trained graph neural network.

Option keyLabelTypeNotes
input_structureInput structureInputNode (required).pdb, .cif, .mmcif, .ent
model_pathPretrained modelString (required)Path to pre-trained model directory
batch_sizeBatch size for inferenceIntDefault 64; use powers of 2
min_num_residuesMinimum interface sizeIntDefault 10; hard min 1
max_neighbours_distMaximum distance for interface detectionFloatDefault 7.0 Å; hard min 1.0
min_neighbours_chainsMinimum chain lengthIntDefault 30; hard min 1
use_gpuUse GPU acceleration?BooleanDefault False
gpu_idsWhich GPUs to use:StringDeactivated when use_gpu=False

Required external program: dpi-inference.

Outputs: results/dataset_<model_basename>.csv (NODE_EVALUATIONMETRIC), colour_model_by_dpiscore_prob.json (NODE_VISUALISATIONSCRIPT).

Output directory: DPIscore/


confidencemap.map_analysis — Confidence Map (ConfidenceMap)

Thresholds cryo-EM density maps by False Discovery Rate (FDR) control.

Option keyLabelTypeNotes
input_mapInput mapInputNode (required).mrc only; raw or unmasked
window_sizeWindow sizeIntPixels; if blank, calculated as 5% of map size or 10, whichever is greater
noise_box_coordx/y/zNoise box coordinate X/Y/ZIntAll three required together if any one is set
apixPixel sizeFloatÅ; if blank, read from map header
meanMapMean mapInputNodeOptional 3D map of noise means
varianceMapVariance mapInputNodeOptional 3D map of noise variance
locResMapLocal resolution mapInputNodeOptional local resolution map
methodMethodMultipleChoiceFDR-BY, FDR-BH (default), FWER-Holm, FWER-Hochberg
test_procTest procedureMultipleChoiceLeft-sided, Right-sided (default), Two-sided
lowPassFilterLow-pass filterFloatOptional; resolution in Å for pre-filter
ecdfUse ECDF?BooleanDefault False

Required external programs: ccpem-python and the FDRcontrol.py script located relative to it.

Outputs: <mapid>_confidenceMap.mrc (NODE_MASK3D), diag_image.pdf (NODE_LOGFILE).

Output directory: ConfidenceMap/


ribfind.atomic_model_split.rigid_body — RIBFIND (RigidBodySplit)

Clusters secondary structures into rigid bodies using RIBFIND2. Protein and RNA components are processed separately.

Option keyLabelTypeNotes
input_modelInput modelInputNode (required).cif, .pdb
structure_contentStructure contentMultipleChoiceprotein (default), RNA, protein+RNA

Required external programs: ribfind (version 2.x required) and gemmi. When structure_content includes RNA, rnaview must be on PATH. When protein is included, mkdssp (CCP4) must be on PATH. These secondary-program checks are performed at validation time, not at job creation.

Outputs: ribfind.dssp (NODE_PROCESSDATA, protein mode), <model>_no_wat_lig.pdb.xml (NODE_PROCESSDATA, RNA mode), plus rigid-body .txt files (NODE_RIGIDBODIES) and a ribfind_<modelid>_outputs.json (NODE_RESTRAINTS) discovered after the run.

Output directory: RIBFIND/


proshade.map_analysis.symmetry — ProShade Symmetry (ProShadeJobSym)

Detects point-group symmetry, symmetry axes, and symmetry elements from a map.

Option keyLabelTypeNotes
input_mapInput mapInputNode (required).mrc
point_symExpected symmetryStringe.g. C3, D2, T, O; forces search to prefer this group if found

Required external program: proshade. Also requires the CCPEM share directory (PIPELINER_CCPEM_SHARE_DIR); if not found, the job loads with is_available=False.

Outputs: proshade_symmetry.json (NODE_LOGFILE).

Output directory: ProShade/


proshade.map_utilities.overlay — ProShade Overlay (ProShadeJobOverlay)

Finds the rotation and translation to optimally overlay one map or model on another.

Option keyLabelTypeNotes
static_ismapStatic structure is map?BooleanDefault True; if False, use static_model
static_mapStatic mapInputNode.mrc, .map, .ccp4; required when static_ismap=True
static_modelStatic atomic modelInputNode.ent, .pdb; required when static_ismap=False
moving_ismapMoving structure is map?BooleanDefault True
moving_mapMoving mapInputNodeRequired when moving_ismap=True
moving_modelMoving atomic modelInputNodeRequired when moving_ismap=False
add_space / extra_spaceAdd extra space around structure? / Extra space around structureBoolean / Floatextra_space default 0.0; required when add_space=True

Required external program: proshade. Also requires CCPEM share directory.

Output directory: ProShade/


emdb_validation_analysis.map_model_validation — EMDB Validation Analysis (MapModelValidate)

Runs the EMDB validation analysis tool (va) on an input map and model, reporting map and model metrics.

Registered entry point: emdb_validation_analysis.map_model_validation → pipeliner.jobs.other.emdb_validation_analysis_job:MapModelValidate

Output directory: EMDB_validation/


Map processing and utilities

locscale.postprocess.local_sharpening — LocScale (LocalSharpen)

Reference-based local amplitude scaling to improve map contrast.

Option keyLabelTypeNotes
input_halfmap1Input halfmap 1InputNode (required).mrc, .ccp4, .map; tagged halfmap
input_halfmap2Input halfmap 2InputNode (required).mrc, .ccp4, .map; tagged halfmap
methodMethodMultipleChoiceFeature enhance (default), Model-free, Hybrid, Model-based
input_modelInput modelInputNode.cif, .pdb; required for Model-based and Hybrid
ligand_cifLigand dictionary (cif)InputNode.cif; for new ligands not in CCP4 monomer library
sym_groupSymmetry groupStringDefault C1; deactivated for Model-based
use_gpuUse GPU acceleration?BooleanDefault True; deactivated for Model-based and Hybrid
gpu_idsWhich GPUs to use:StringSpace-separated; required when GPU enabled for non-model-based modes
ncpusNumber of coresIntDefault 1; only active for Feature enhance
use_advancedUse advanced options?BooleanDefault False
locscale_windowLocScale window sizeIntPixels; advanced only
resolutionResolutionFloatÅ; advanced only
input_maskInput maskInputNode.mrc; advanced only
fdr_windowFDR window sizeIntAdvanced only; deactivated when mask is provided
use_low_contextUse low context model for EMmerNet?BooleanAdvanced only; deactivated for Model-based/Hybrid
mc_cyclesMonte-carlo cyclesIntDefault 15; advanced, Feature enhance only
batch_sizeBatch sizeIntDefault 8; advanced, non-model-based
strideStrideIntDefault 16; advanced, non-model-based
model_pathCustom trained model:ExternalFile.h5, .hdf5; advanced, non-model-based
pseudomodel_cyclesPseudomodal cyclesIntDefault 0; Hybrid only
skip_brefineSkip B-factor refinement?BooleanDefault False; Model-based only
brefinement_cyclesB-factor refinement cyclesIntDefault 10; deactivated when skip_brefine=True

MPI-enabled (use nr_mpi and mpi_command); MPI options are deactivated for Feature enhance mode.

Required external program: locscale.

Outputs: locscale_output.mrc (NODE_DENSITYMAP), processing_files/locscale.log (NODE_LOGFILE), and colour_output_by_pVDDT.json (NODE_VISUALISATIONSCRIPT, Feature enhance only).

Output directory: LocScale/


lafter.postprocess.local_denoising — LAFTER (LafterJob)

Local de-noising of cryo-EM maps using a local agreement filtering algorithm applied to half-map pairs.

Option keyLabelTypeNotes
half_map_1Half map 1InputNode (required).mrc, .map
half_map_2Half map 2InputNode (required).mrc, .map
maskInput maskInputNode.mrc, .map; MRC mode 2; required if no particle diameter or diameter ≤ 20
particle_diameterParticle diameterFloatVoxels; required when no mask; must be > 20
sharpAdditional sharpeningBooleanDefault False
fscFSC cut-off thresholdFloatDefault 0.143; range 0.1–1.0
oversampleOver-sample output?BooleanDefault False; doubles output file size
overfittingCompensate for overfittingBooleanDefault False

Required external program: lafter.

Outputs: LAFTER_filtered.mrc (NODE_DENSITYMAP).

Output directory: Lafter/


servalcat.difference_map — Servalcat Difference Map (DifferenceMap)

Calculates a weighted Fo-Fc difference map from half-maps and an atomic model.

Option keyLabelTypeNotes
input_modelInput modelInputNode (required).cif, .pdb
input_ligandInput ligandInputNode.cif; ligand dictionary
resolutionResolutionFloat (required)Å
input_half_map1Input map 1InputNode (required)Half map 1
input_half_map2Input map 2InputNode (required)Half map 2
input_maskInput maskInputNode.mrc, .map
mask_radiusMask radiusFloatDefault 25; hard min 0; used when no mask provided; required when input_mask is empty

Required external program: servalcat. Also uses gemmi sf2map to convert the output MTZ to MRC.

Outputs: diffmap.mtz (NODE_STRUCTUREFACTORS), diffmap.mrc (NODE_DENSITYMAP).

Output directory: ServalcatDifferenceMap/


servalcat.mask_creation.mask_from_model — Servalcat Mask from Model (ServalcatMaskFromModel)

Generates a mask from an atomic model using a reference map for grid parameters.

Option keyLabelTypeNotes
input_modelInput modelInputNode (required).cif, .mmcif, .pdb
input_mapReference mapInputNode (required).mrc, .map; used for grid size and cell parameters only
radiusMask radius (Å)Float (required)Default 4.0; hard min 0
soft_edgeSoft edge radius (Å)FloatDefault 4.0; hard min 0
atom_selectionAtom selectionStringGemmi selection syntax; e.g. //A/10-30

Required external programs: servalcat (≥ 0.4.88 / CCP4 ≥ 9.0.005) and gemmi.

Outputs: mask_from_model.mrc (NODE_MASK3D).

Output directory: MaskCreate/


gemmi.atomic_model_utilities.model_to_map — Gemmi Model to Map (GemmiModelToMap)

Calculates a density map from an atomic model using gemmi sfcalc.

Option keyLabelTypeNotes
input_modelInput modelInputNode (required).cif, .pdb
resolutionResolutionFloat (required)Default 10.0 Å
reference_mapReference mapInputNode.mrc, .map; optional; extracts grid dimensions
ref_alignedReference map aligned?BooleanDefault True; deactivated when reference_map is empty
cell_x / cell_y / cell_zCell x/y/zFloatOptional cell dimensions in Å; all three required together

No external programs required (uses the bundled gemmi Python library and a job script).

Outputs: gemmi_<model_stem>.mrc (NODE_DENSITYMAP).

Output directory: GemmiModelToMap/


ccpem_utils.map_utilities.crop_pad_mask_map — Crop, Pad or Mask Map (CropPadMap)

Applies a mask, crops, or pads maps using the CCP-EM Utils library. Accepts multiple input maps.

Option keyLabelTypeNotes
input_mapMap to processMultiInputNode (required).mrc, .map, .ccp4
crop_mapCrop map?BooleanDefault False; mutually exclusive with pad_map
pad_mapPad map?BooleanDefault False; mutually exclusive with crop_map
apply_maskApply input mask?BooleanDefault False; mutually exclusive with threshold_map
threshold_mapThreshold map?BooleanDefault False; mutually exclusive with apply_mask
crop_modeCrop based onMultipleChoicedimensions (default), input_mask, input_contour; deactivated when crop_map=False
input_maskInput maskInputNode.mrc, .ccp4, .map; required when apply_mask=True or crop_mode=input_mask
thr_maskThreshold mask atFloatDefault 0; deactivated for contour/dimensions modes
contourlevelContour LevelFloatDefault -100; required when thresholding or crop_mode=input_contour
croppad_x/y/zCrop/Pad along X/Y/ZIntDefault 0; used in dimensions crop or pad mode
add_paddingAdd paddingIntDefault 10; used with mask/contour crop modes
out_cubicOutput cubic map?BooleanDefault False; deactivated for dimensions crop
fill_paddingFill padding withMultipleChoiceminimum (default), zero, input_value; deactivated when pad_map=False
fill_valuePadding fill valueFloatDefault 0.0; required when fill_padding=input_value

No external programs required.

Output node type matches the input node type (density map or mask). Output filenames are derived from input names with suffixes reflecting the operation (e.g. _masked_cropped.mrc).

Output directory: CropPadMaskMap/


ccpem_utils.map_utilities.shift_origin_nstart — Map Shift (MapShift)

Sets the map origin or nstart header record based on a reference map or explicit coordinates. Processes multiple maps and masks in one job.

Option keyLabelTypeNotes
input_mapsInput associated mapsMultiInputNode (required).mrc, .ccp4, .map
input_masksInput associated masksMultiInputNode.mrc, .ccp4, .map; optional
origin_or_nstartHeader record to setMultipleChoiceorigin (default), nstart, both
reference_mapInput reference mapInputNodeReference to copy header values from
origin_x/y/zOrigin coordinate along X/Y/ZFloatDefault 0; used when no reference and origin is being set
nstart_x/y/zNstart coordinate along X/Y/ZIntDefault 0; used when no reference and nstart is being set

No external programs required.

Outputs: per-input <mapid>_shifted.mrc preserving the input node type (NODE_DENSITYMAP or NODE_MASK3D).

Output directory: MapShift/


Examples

Example 1 — Run Molprobity geometry assessment on a refined model

import os
from pipeliner.api.manage_project import PipelinerProject
from pipeliner.api.api_utils import write_default_jobstar, edit_jobstar

os.chdir("/data/my_project")
proj = PipelinerProject()

# Generate a default job.star template
template = write_default_jobstar("molprobity.atomic_model_validation.geometry")

# Edit parameters
params = {
    "input_models": "ServalcatRefine/job005/refined.mmcif",
    "run_global": "Yes",
    "resolution": "3.2",
}
edited = edit_jobstar(template, params, "molprobity_run.star")

# Run the job
job = proj.run_job(edited, run_in_foreground=True)

Example 2 — Schedule an atomic model validation job and wait for it to finish

import os
from pipeliner.api.manage_project import PipelinerProject
from pipeliner.job_factory import new_job_of_type

os.chdir("/data/my_project")
proj = PipelinerProject()

job = new_job_of_type("pipeliner.atomic_model_validation")
job.joboptions["input_model"].value = "ServalcatRefine/job005/refined.mmcif"
job.joboptions["input_map"].value = "PostProcess/job008/postprocess.mrc"
job.joboptions["input_halfmap1"].value = "Refine3D/job007/run_half1_class001_unfil.mrc"
job.joboptions["input_halfmap2"].value = "Refine3D/job007/run_half2_class001_unfil.mrc"
job.joboptions["resolution"].value = "3.2"
job.joboptions["run_molprobity"].value = "Yes"
job.joboptions["run_smoc"].value = "Yes"
job.joboptions["run_fsc"].value = "Yes"

# schedule_job takes a job file or a params dict, not a PipelinerJob,
# so write the job's options to a job.star file first
job.write_jobstar("", output_fn="model_validation_job.star")
scheduled = proj.schedule_job("model_validation_job.star")
proj.run_scheduled_job(scheduled, run_in_foreground=True)
final_status = proj.wait_for_job_to_finish(scheduled, error_on_fail=True)

Example 3 — Run LAFTER denoising on half-maps from a prior refinement job

import os
from pipeliner.api.manage_project import PipelinerProject
from pipeliner.api.api_utils import write_default_jobstar, edit_jobstar

os.chdir("/data/my_project")
proj = PipelinerProject()

template = write_default_jobstar("lafter.postprocess.local_denoising")
params = {
    "half_map_1": "Refine3D/job007/run_half1_class001_unfil.mrc",
    "half_map_2": "Refine3D/job007/run_half2_class001_unfil.mrc",
    "mask": "MaskCreate/job009/mask.mrc",
    "fsc": "0.143",
}
edited = edit_jobstar(template, params, "lafter_run.star")
job = proj.run_job(edited, run_in_foreground=True)