RELION Jobs: Import and Preprocessing

This page documents the RELION job types available in ccpem-pipeliner for the import and preprocessing stages of a single-particle analysis (SPA) workflow. It covers every registered job from raw movie import through motion correction, CTF estimation, manual and automated particle picking, particle extraction, and subset selection — including all helical variants. Each section describes the job's purpose, its exact process name (entry-point key), the key input and output nodes, and the job options you need to set when constructing pipelines programmatically or editing run.job files.

Before using these jobs you need:

• Python ≥ 3.9 installed in your environment
• ccpem-pipeliner installed (see the installation section)
• RELION available on your PATH — the pipeliner wraps RELION executables for most of the jobs on this page; jobs whose required executables are missing load normally but with job.jobinfo.is_available == False
• CTFFIND 4.1 (ctffind, or the path set by PIPELINER_CTFFIND_EXECUTABLE) for relion.ctffind.ctffind4
• Gctf (Gctf, or PIPELINER_GCTF_EXECUTABLE) for relion.ctffind.gctf
• MotionCor2 (MotionCor2, or PIPELINER_MOTIONCOR2_EXECUTABLE) for relion.motioncorr.motioncor2
• Topaz (optional, PIPELINER_TOPAZ_EXECUTABLE) for Topaz autopick jobs
• An MPI runtime (e.g. OpenMPI) if you want to run jobs with nr_mpi > 1
• A project directory initialised by pipeliner --new_project or an existing RELION project

Install the package from PyPI into any Python 3.9+ environment:

pip install ccpem-pipeliner

Verify the installation and check which external programs are found:

pipeliner.check_setup

If the pipeliner console script is not found after installation, force-reinstall to restore entry points:

pip install --force-reinstall ccpem-pipeliner

The settings file at ~/.config/ccpem/pipeliner/ccpem-pipeliner-settings.json (Linux) or ~/Library/Application Support/ccpem/pipeliner/ccpem-pipeliner-settings.json (macOS) controls defaults for all jobs on this page. Every setting can also be overridden by an environment variable; PIPELINER_* takes precedence over RELION_* fallbacks.

Job-type availability is checked each time get_job_types() or new_job_of_type() is called. A job whose required executables cannot be found on PATH loads normally with job.jobinfo.is_available == False and job.jobinfo.unavailable_message populated with the reason.

All job types on this page are imported from the ccpem_pipeliner.jobs entry-point group and manipulated through the PipelinerProject API or the pipeliner CLI.

Importing the API

from pipeliner.api.manage_project import PipelinerProject
from pipeliner.job_factory import new_job_of_type, get_job_types
from pipeliner.api.api_utils import write_default_runjob, write_default_jobstar, edit_jobstar

Opening or creating a project

Always cd into your project directory first. PipelinerProject operates on the current working directory and is not a context manager.

import os
os.chdir('/data/my_project')
proj = PipelinerProject(make_new_project=True)  # creates default_pipeline.star

Generating a default parameter file

Use write_default_runjob to produce an editable run.job file for any job type:

pipeliner --default_runjob relion.import.movies
# writes relion_import_movies_run.job

From Python:

from pipeliner.api.api_utils import write_default_runjob
write_default_runjob('relion.import.movies')
# writes relion_import_movies_run.job in the current directory

Running a job from a dict

Pass a dict with _rlnJobTypeLabel plus any option keys you want to override. Option values are strings in job_default_parameters_dict() output but the API also accepts typed values.

job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.import.movies',
    'fn_in_raw': 'Movies/*.tif',
    'angpix': '1.06',
    'kV': '300',
    'Cs': '2.7',
    'Q0': '0.1',
    'is_multiframe': 'Yes',
})
print(job.output_dir)  # e.g. Import/job001/

Running a job from a run.job file

# Edit the generated file first, then:
pipeliner --run_job relion_import_movies_run.job

Continuing a job

After a successful run, a continue_job.star file is written in the job directory. Edit it, then:

pipeliner --continue_job Import/job001/

Checking job availability

from pipeliner.job_factory import new_job_of_type
job = new_job_of_type('relion.ctffind.ctffind4')
print(job.jobinfo.is_available)        # True or False
print(job.jobinfo.unavailable_message) # None or explanation string

Setting job options programmatically

Job options are stored in job.joboptions and values are set via the .value property:

job = new_job_of_type('relion.motioncorr.own')
job.joboptions['input_star_mics'].value = 'Import/job001/movies.star'
job.joboptions['dose_per_frame'].value = '1.2'
job.joboptions['nr_mpi'].value = 4

Example 1 — Import movies and run motion correction in sequence

This example shows a minimal two-job pipeline: import multi-frame movies, then run RELION's own motion correction implementation.

import os
from pipeliner.api.manage_project import PipelinerProject

os.chdir('/data/my_project')
proj = PipelinerProject(make_new_project=True)

# Step 1: Import movies
import_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.import.movies',
    'fn_in_raw': 'Movies/*.tif',
    'angpix': '1.06',
    'kV': '300',
    'Cs': '2.7',
    'Q0': '0.1',
    'is_multiframe': 'Yes',
    'optics_group_name': 'opticsGroup1',
})
# Output node: Import/job001/movies.star

# Step 2: Motion correction with RELION's own implementation
mc_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.motioncorr.own',
    'input_star_mics': import_job.output_dir + 'movies.star',
    'dose_per_frame': '1.2',
    'patch_x': '5',
    'patch_y': '5',
    'nr_mpi': '4',
    'nr_threads': '4',
})
# Output node: MotionCorr/job002/corrected_micrographs.star

Example 2 — CTF estimation with CTFFIND4

ctf_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.ctffind.ctffind4',
    'input_star_mics': mc_job.output_dir + 'corrected_micrographs.star',
    'box': '512',
    'resmin': '30',
    'resmax': '5',
    'dfmin': '5000',
    'dfmax': '50000',
    'dfstep': '500',
    'dast': '100',
    'nr_mpi': '4',
})
# Output node: CtfFind/job003/micrographs_ctf.star

Example 3 — LoG autopick followed by particle extraction

# Autopick with Laplacian-of-Gaussian
pick_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.autopick.log',
    'fn_input_autopick': ctf_job.output_dir + 'micrographs_ctf.star',
    'nr_mpi': '4',
})
# Output node: AutoPick/job004/autopick.star

# Extract particles
extract_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.extract',
    'star_mics': ctf_job.output_dir + 'micrographs_ctf.star',
    'coords_suffix': pick_job.output_dir + 'autopick.star',
    'extract_size': '256',
    'do_norm': 'Yes',
    'do_invert': 'Yes',
    'do_float16': 'Yes',
    'nr_mpi': '4',
})
# Output node: Extract/job005/particles.star

Example 4 — Generate a default run.job file from the CLI and inspect it

pipeliner --default_runjob relion.ctffind.ctffind4
# Writes: relion_ctffind_ctffind4_run.job

The generated file uses label == value format, for example:

job_type == relion.ctffind.ctffind4
is_continue == False
is_tomo == False
Input micrographs STAR file: == 
Estimate phase shifts? == No
...

Edit this file and submit it:

pipeliner --run_job relion_ctffind_ctffind4_run.job

Example 5 — Interactive selection job to pick particles from 2D classification

select_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.select.interactive',
    'fn_model': 'Class2D/job006/run_it025_optimiser.star',
})
# Output: Select/job007/particles.star and class_averages.star

Example 6 — Join multiple micrograph STAR files

from pipeliner.job_factory import new_job_of_type

join_job = proj.run_job({
    '_rlnJobTypeLabel': 'relion.joinstar.micrographs',
    'fn_mics': 'MotionCorr/job002/corrected_micrographs.star:::MotionCorr/job008/corrected_micrographs.star',
})
# Output: JoinStar/jobNNN/join_mics.star
# Note: multiple values for MultiInputNodeJobOption use ':::' as separator

Example 7 — Schedule a job without running it immediately

scheduled = proj.schedule_job({
    '_rlnJobTypeLabel': 'relion.motioncorr.own',
    'input_star_mics': 'Import/job001/movies.star',
    'dose_per_frame': '1.0',
})
print(scheduled.output_dir)  # e.g. MotionCorr/job002/

Job loads with is_available=False

Symptom: job.jobinfo.is_available is False; job.jobinfo.unavailable_message contains "executable '<name>' not found".

Cause: The required external program (e.g. ctffind, Gctf, MotionCor2) is not on your PATH.

Fix: Either add the program to your PATH before starting Python, or set the appropriate environment variable (PIPELINER_CTFFIND_EXECUTABLE, PIPELINER_GCTF_EXECUTABLE, PIPELINER_MOTIONCOR2_EXECUTABLE, etc.) to the full path of the executable. Run pipeliner.check_setup to see which programs are found.

ValueError: Parameter <name> is not found in jobtype <type>

Symptom: run_job() raises ValueError when passing a dict.

Cause: A key in your parameters dict does not match any registered option name for that job type. Common mistakes include using 'voltage' instead of 'kV', 'ampl_contrast' instead of 'Q0', 'fn_in' instead of 'input_star_mics' for motion correction, or 'fn_in_raw' for motion correction (that key belongs to relion.import.movies).

Fix: Check the exact option key names in the job type reference below. Use pipeliner --default_runjob <jobtype> to see all labels and their corresponding option keys, or call write_default_jobstar('<jobtype>') and inspect the generated STAR file.

RuntimeError: Cannot find job type <type>, used in job file <file>

Symptom: pipeliner --run_job myfile.run.job fails with this error.

Cause: The job_type == line in your run.job file contains a job type that is not registered. This can happen if you mistyped the process name or if the plugin providing that job type is not installed.

Fix: Verify the exact process name with pipeliner --available_jobs. Use the full dot-separated name (e.g. relion.motioncorr.own, not MotionCorr or own).

RuntimeError('Job does not contain label: <label>')

Symptom: Editing a run.job file manually and re-running it raises this error.

Cause: You changed the text of an option label in the run.job file. The format is <exact label text> == <value> and the label must match character-for-character what the job type registered. For example, the import movies input is Raw input files: — changing it to Input files: will fail.

Fix: Use pipeliner --default_runjob <jobtype> to get a fresh file with the correct labels. Only change the value after ==, never the label before it.

RuntimeError("No '==' entry on JobOption line: <line>")

Symptom: run.job parsing fails with this message.

Cause: A line in your run.job file is missing the == separator.

Fix: Ensure every option line follows the format <label> == <value>. The three header lines (job_type == ..., is_continue == ..., is_tomo == ...) are also required and must use ==.

ValueError: File does not appear to be a valid job.star file from edit_jobstar()

Symptom: Calling edit_jobstar() with a run.job file raises this error.

Cause: edit_jobstar() expects a STAR-format job.star file (produced by write_default_jobstar()), not a run.job file. These two formats are incompatible.

Fix: Use write_default_jobstar('<jobtype>') to produce the correct template for edit_jobstar(). Use write_default_runjob('<jobtype>') if you intend to pass the file directly to pipeliner --run_job.

Queue submission fails with 'Error reading template submission script in: <path>'

Symptom: A job with do_queue set to Yes fails immediately at submission time.

Cause: The qsub_template setting is empty (its default) or the path it points to does not exist. When queue submission is enabled, a template script file is mandatory.

Fix: Set PIPELINER_QSUB_TEMPLATE to the path of a valid cluster submission script, or edit qsub_template in ccpem-pipeliner-settings.json. See the settings documentation for the supported XXX...XXX placeholders.

PipelinerProject raises FileNotFoundError on instantiation

Symptom: PipelinerProject() raises FileNotFoundError: Could not find a project to open in directory ....

Cause: There is no default_pipeline.star or .CCPEM_pipeliner_project file in the current working directory.

Fix: Either cd into your existing project directory before instantiating PipelinerProject, or pass make_new_project=True to create a new project in the current directory.

relion.select.* jobs raise NotImplementedError when continued

Symptom: Calling proj.continue_job('Select/jobNNN/') raises NotImplementedError.

Cause: All relion.select.* job types have can_continue=False. Select jobs do not support continuation.

Fix: Re-run the select job as a new job (with pipeliner --run_job or proj.run_job()), adjusting the parameters as needed.

relion.joinstar.* job fails with column mismatch error

Symptom: The join job fails at execution time with a message about mismatched columns.

Cause: The input STAR files being joined do not all have the same blocks, column names, and column order. relion.joinstar.tomograms also validates at scheduling time that at least 2 input files are present.

Fix: Ensure all input STAR files come from the same type of upstream job (e.g. all from MotionCorr jobs) and were produced by the same version of RELION. Inspect the files with a STAR file viewer to confirm they share the same loop structure.