CCPEM Pipeliner
Guide

Python API: Running and Scheduling Jobs

PipelinerProject methods for running, scheduling, continuing and aborting jobs


Overview

PipelinerProject is your primary handle for running, scheduling, continuing, and aborting jobs in a pipeliner project. Import it with from pipeliner.api.manage_project import PipelinerProject, instantiate it in the project directory, then call its methods to drive the full job lifecycle — from creating a new job and launching it immediately, through scheduling a batch of jobs for later execution, to continuing a completed job or aborting one that is still running. This page provides a complete API reference for every execution-related method on PipelinerProject.


Usage

All methods operate on the current working directory, so change into the project root before constructing PipelinerProject.

import os
os.chdir('/path/to/my/project')

from pipeliner.api.manage_project import PipelinerProject
proj = PipelinerProject()          # open an existing project
# or, to create a new one:
proj = PipelinerProject(make_new_project=True)

PipelinerProject is not a context manager; instantiate it with a plain assignment and call methods directly.


Supplying job parameters

Most execution methods accept a job definition as one of:

  • A parameters dict keyed by job-option key (not label) with _rlnJobTypeLabel identifying the job type.
  • A path to a run.job or job.star file (any filename ending in .job or .star).
  • A PipelinerJob object (accepted by run_job() only; not accepted by schedule_job()).

The helper utilities write_default_runjob() and write_default_jobstar() from pipeliner.api.api_utils generate starter files; job_default_parameters_dict() (also from pipeliner.api.api_utils) returns a ready-to-edit parameters dict.

from pipeliner.api.api_utils import (
    write_default_runjob,
    write_default_jobstar,
    edit_jobstar,
    job_default_parameters_dict,
)

# Option A — run.job route (labels)
runjob = write_default_runjob('relion.import.movies')  # writes relion_import_movies_run.job
# Edit the file manually, then:
job = proj.run_job(runjob)

# Option B — job.star route (keys)
jobstar = write_default_jobstar('relion.import.movies')  # writes relion_import_movies_job.star
new_star = edit_jobstar(jobstar, {'fn_in_raw': 'Movies/*.tiff', 'kV': '300'}, 'my_import.star')
job = proj.run_job(new_star)

# Option C — dict route (keys)
params = job_default_parameters_dict('relion.import.movies')
params['fn_in_raw'] = 'Movies/*.tiff'
params['kV'] = '300'
job = proj.run_job(params)

Note: dict keys are job-option keys (e.g. fn_in_raw, kV, Q0), never GUI labels like 'Raw input files:'. Only run.job files use the label form.


Job numbering

Jobs are numbered project-wide (Import/job001/, MotionCorr/job002/, …). run_job() always creates a new sequential number. To replace an existing job in place, pass overwrite=<full_job_name>. To resume a job without changing its number, use continue_job().


Foreground vs background execution

By default (run_in_foreground=False) run_job() launches the job in a detached background process and returns immediately. If you need to wait for the job to finish before launching the next one — common in automation scripts — either pass run_in_foreground=True (local jobs only; raises ValueError when the job is set to queue submission) or chain with wait_for_job_to_finish().

# Foreground — blocks until done
job = proj.run_job(params, run_in_foreground=True)

# Background + explicit wait
job = proj.run_job(params)
status = proj.wait_for_job_to_finish(job)

Examples

Open or create a project

import os
os.chdir('/data/em/my_project')

from pipeliner.api.manage_project import PipelinerProject

# Open an existing project
proj = PipelinerProject()

# Create a new project in the current directory
proj = PipelinerProject(make_new_project=True,
                        project_name='Tutorial dataset',
                        description='SPA of apoferritin')

run_job(jobinput, overwrite=None, alias=None, ignore_invalid_joboptions=False, run_in_foreground=False) -> PipelinerJob

Create and immediately run a new job. Returns a PipelinerJob whose output_dir attribute holds the job directory (e.g. 'Import/job001/').

Parameters

ParameterTypeDescription
jobinputdict | str | PipelinerJobJob parameters as a dict, a path to a run.job/job.star file, or a PipelinerJob object
overwritestr | NoneFull job name to delete and replace (must be the same job type; no downstream consumers)
aliasstr | NoneHuman-readable alias for the job
ignore_invalid_joboptionsboolRun the job even if its job options fail validation (default False)
run_in_foregroundboolBlock until the job finishes (default False); incompatible with queue submission

Returns PipelinerJob – the newly created and launched job.

Raises

  • ValueError – if input files are missing (including a queue submission script that does not exist), the job type is unknown (in a dict or in a .job/.star file), a parameter key is invalid, the job options are invalid, or run_in_foreground=True with do_queue=Yes.
  • RuntimeError – if the job cannot be launched, for example because the queue submission template can no longer be read when the job starts; the job is marked Failed.
from pipeliner.api.api_utils import job_default_parameters_dict

# Build a parameter dict for importing movies
params = job_default_parameters_dict('relion.import.movies')
params['fn_in_raw'] = 'Movies/*.tiff'
params['kV'] = '300'
params['Cs'] = '2.7'
params['Q0'] = '0.1'
params['angpix'] = '0.832'

import_job = proj.run_job(params, run_in_foreground=True)
print(import_job.output_dir)  # e.g. 'Import/job001/'

Run from a run.job file:

from pipeliner.api.api_utils import write_default_runjob, edit_jobstar

# Generate a default run.job file, then run it
runjob_file = write_default_runjob('relion.ctffind.ctffind4')
# Edit the file manually or programmatically, then:
ctf_job = proj.run_job(runjob_file, run_in_foreground=True)

Overwrite an existing job:

# 'Import/job001/' must be the same type and have no downstream consumers
new_job = proj.run_job(params, overwrite='Import/job001/')

schedule_job(job_input, comment=None, alias=None) -> PipelinerJob

Register a job in the pipeline graph without running it. The job receives a Scheduled status and its run.job/job.star files are written to the job directory. Returns a PipelinerJob with an output_dir attribute; the job has no name attribute.

schedule_job() accepts a dict or file path only — it does not accept a PipelinerJob object.

Parameters

ParameterTypeDescription
job_inputdict | strParameters dict or path to a run.job/job.star file
commentstr | NoneOptional comment to attach to the job
aliasstr | NoneHuman-readable alias

Returns PipelinerJob

params = job_default_parameters_dict('relion.motioncorr.own')
params['input_star_mics'] = 'Import/job001/movies.star'
params['patch_x'] = '5'
params['patch_y'] = '5'

motioncorr_job = proj.schedule_job(params, alias='motioncorr_pass1')
print(motioncorr_job.output_dir)  # e.g. 'MotionCorr/job002/'

run_scheduled_job(job, run_in_foreground=False) -> PipelinerJob

Run a job that has already been scheduled. Accepts a PipelinerJob instance (as returned by schedule_job()), not a string or dict. Raises ValueError if the job's status is not Scheduled.

motioncorr_job = proj.schedule_job(params)
# ... optionally inspect or edit the job directory ...
ran_job = proj.run_scheduled_job(motioncorr_job, run_in_foreground=True)

continue_job(job_to_continue, comment=None, run_in_foreground=False) -> PipelinerJob

Resume a previously run job. Accepts a full job name string (e.g. 'MotionCorr/job002/'). Does not check the can_continue attribute; call get_continuation_job() first if you need that guard. The job directory must contain a PIPELINER_JOB_EXIT_SUCCESS file (the job has Succeeded); otherwise FileNotFoundError is raised.

For optimiser-based jobs (Class2D, Class3D, InitialModel, Refine3D, Multibody) you must manually set fn_cont in continue_job.star to the desired _optimiser.star file before calling this method; otherwise it raises ValueError. get_continuation_job() populates fn_cont automatically for Class2D, Class3D, InitialModel and Refine3D jobs (not MultiBody), on the job object it returns; it does not edit continue_job.star.

# continue_job.star was written at scheduling time;
# edit it to set fn_cont for optimiser jobs before calling this.
resume = proj.continue_job('MotionCorr/job002/', run_in_foreground=True)

get_continuation_job(job_name) -> PipelinerJob

Return a PipelinerJob prepared for continuation, with its options updated in memory via continuation_joboption_updates() (which populates fn_cont for Class2D, Class3D, InitialModel and Refine3D jobs). It does not write or modify continue_job.star. Enforces can_continue — raises NotImplementedError if the job type does not support continuation, and ValueError if the job status is not Succeeded.

Use this method when you need to inspect or modify continuation options before actually running the continuation.

cont_job = proj.get_continuation_job('Refine3D/job005/')
# Inspect options, then run:
result = proj.run_job(cont_job)

schedule_continue_job(job_to_continue, params_dict=None, comments=None) -> PipelinerJob

Schedule a continuation without running it immediately. Returns a PipelinerJob. Does not check can_continue. Keys in params_dict must be options present in the job's continue_job.star (options that can be changed on continuation); any other key raises ValueError.

sched_cont = proj.schedule_continue_job(
    'MotionCorr/job002/',
    params_dict={'nr_threads': '8'},
)
proj.run_scheduled_job(sched_cont, run_in_foreground=True)

run_schedule(fn_sched, job_ids, nr_repeat=1, minutes_wait=0, minutes_wait_before=0, seconds_wait_after=5) -> str

Run a named list of scheduled jobs sequentially. Returns the schedule name fn_sched. Raises ValueError if a schedule with that name is already running.

Parameters

ParameterTypeDefaultDescription
fn_schedstr—Schedule name / identifier
job_idsList[str]—Ordered list of job output directories to run
nr_repeatint1Number of times to repeat the schedule
minutes_waitint0Minutes to wait between repeats
minutes_wait_beforeint0Minutes to wait before starting
seconds_wait_afterint5Seconds to wait after each job finishes

Note: The parameter names are fn_sched and job_ids. Passing name= or jobs= raises TypeError.

import_job  = proj.schedule_job(import_params)
mcorr_job   = proj.schedule_job(mcorr_params)
ctf_job     = proj.schedule_job(ctf_params)

schedule_name = proj.run_schedule(
    fn_sched='preprocessing',
    job_ids=[
        import_job.output_dir,
        mcorr_job.output_dir,
        ctf_job.output_dir,
    ],
)

stop_schedule(schedule_name) -> bool

Kill the running schedule immediately and request an abort of the currently executing job (via abort_job()). Returns True on success, False if no running schedule with that name is found.

ok = proj.stop_schedule('preprocessing')

abort_job(job_name) -> None

Write the abort-request trigger file for a Running job. Does not wait for the job to stop. Raises ValueError if the job name is not found; raises RuntimeError if the job is not currently Running.

The exact job name (e.g. 'MotionCorr/job002/') is required — short forms are not resolved.

proj.abort_job('MotionCorr/job002/')

wait_for_job_to_finish(job, ping=1.0, timeout=86400.0, error_on_fail=False, error_on_abort=False) -> str

Poll until a job reaches a terminal state ('Succeeded', 'Failed', or 'Aborted') and return that status string. Accepts a full job name string or a PipelinerJob instance.

Raises RuntimeError on timeout, or when error_on_fail=True and the job fails, or when error_on_abort=True and the job is aborted.

import_job = proj.run_job(params)            # returns immediately
status = proj.wait_for_job_to_finish(
    import_job,
    timeout=3600.0,
    error_on_fail=True,
)
# status is 'Succeeded' or 'Aborted' here; a failed job raised RuntimeError

parse_procname(in_proc, search_trash=False) -> str

Resolve an ambiguous job reference to a full job name. Accepts full names ('Import/job001/'), short forms ('job001', '1'), and full aliases ('Import/my_alias/'). Use this before passing short forms to any other PipelinerProject method.

full = proj.parse_procname('3')   # -> e.g. 'CtfFind/job003/'
full = proj.parse_procname('job002')  # -> e.g. 'MotionCorr/job002/'

Full pipeline automation example

import os
os.chdir('/data/em/apoferritin')

from pipeliner.api.manage_project import PipelinerProject
from pipeliner.api.api_utils import job_default_parameters_dict

proj = PipelinerProject(make_new_project=True, project_name='Apoferritin SPA')

# 1. Import movies
imp_params = job_default_parameters_dict('relion.import.movies')
imp_params['fn_in_raw'] = 'Movies/*.tiff'
imp_params['kV'] = '300'
imp_params['Cs'] = '2.7'
imp_params['Q0'] = '0.1'
imp_params['angpix'] = '0.832'
import_job = proj.run_job(imp_params, run_in_foreground=True)

# 2. Motion correction
mc_params = job_default_parameters_dict('relion.motioncorr.own')
mc_params['input_star_mics'] = import_job.output_dir + 'movies.star'
mc_job = proj.run_job(mc_params, run_in_foreground=True)

# 3. CTF estimation
ctf_params = job_default_parameters_dict('relion.ctffind.ctffind4')
ctf_params['input_star_mics'] = mc_job.output_dir + 'corrected_micrographs.star'
ctf_job = proj.run_job(ctf_params, run_in_foreground=True)

print('Preprocessing complete.')
print('CTF job directory:', ctf_job.output_dir)