CCPEM Pipeliner
Concept

Alternatives

How pipeliner compares with shell scripts, the RELION GUI and general-purpose workflow engines


Overview

This page explains how CCPEM-Pipeliner relates to the alternatives you might already be using: shell scripts, the RELION GUI, and general-purpose workflow engines such as Nextflow or Snakemake. Understanding where pipeliner sits relative to these tools helps you decide when to use it directly, when to layer it on top of an existing system, and what trade-offs to expect.


Content

Shell scripts

The most common alternative to pipeliner for cryoEM automation is a collection of shell scripts that call RELION, CTFFIND, MotionCor2, and similar tools in sequence. Shell scripts are immediately familiar and require no additional dependencies, but they have several structural weaknesses that pipeliner addresses.

What shell scripts lack:

  • Provenance tracking. A shell script that runs ten programs leaves no structured record of which parameters produced which output. Pipeliner stores every job's parameters in job.star and run.job files inside the job directory, and records the full dependency graph in default_pipeline.star.
  • Status accounting. When a shell script fails mid-way, you typically re-read the logs to find out which step failed and which outputs are trustworthy. Pipeliner writes sentinel files (PIPELINER_JOB_EXIT_SUCCESS, PIPELINER_JOB_EXIT_FAILED, PIPELINER_JOB_EXIT_ABORTED) into each job directory so the state of every job is always machine-readable.
  • GUI compatibility. Shell-script outputs are not automatically viewable in the Doppio GUI or the RELION GUI. Because pipeliner writes its pipeline graph as a STAR file (default_pipeline.star) that both GUIs understand, you can inspect, continue, and visualise any pipeliner-managed project in either interface without extra conversion steps.
  • Reproducibility artefacts. Reproducing a shell-script run requires passing the exact script and environment to a colleague. Pipeliner's pipeliner.make_project_tutorial command can read a completed project and write a dated .rst tutorial document capturing all parameter choices up to a specified terminal job.

Where shell scripts still fit: For one-off processing steps, very simple linear workflows, or situations where you control only a single program, the overhead of a pipeliner project may not be worthwhile. Pipeliner is most valuable when you need reproducibility, status tracking, or GUI integration across multiple heterogeneous programs.


The RELION GUI

Pipeliner is the business-logic layer that underpins the CCP-EM software suite (v2), and it is deliberately designed to be compatible with RELION's project format. The RELION GUI and pipeliner share the same pipeline STAR file (default_pipeline.star) and the same job directory layout.

Key differences:

ConcernRELION GUICCPEM-Pipeliner
InterfaceGraphical, interactiveCLI and Python API
AutomationManual job-by-job submissionScripted pipelines via PipelinerProject or pipeliner --run_schedule
Headless operationRequires a displayFully headless
Job typesRELION built-insRELION built-ins plus any registered plugin (entry-point group ccpem_pipeliner.jobs)
ExtensibilityNot extensible by third partiesThird-party jobs registered as Python entry-point plugins
Deposition supportNoneOneDep and EMPIAR deposition jobs built-in

Interoperability, not replacement: Pipeliner and the RELION GUI are complementary. You can start a project in the RELION GUI and then open it with PipelinerProject() (pipeliner detects the existing default_pipeline.star and treats it as a pipeliner project). Conversely, projects created headlessly with pipeliner can be opened in the RELION GUI or the Doppio GUI at any point — the pipeline file format is shared.

When you open a RELION 3.x pipeline file in pipeliner with write access, pipeliner converts it in place to the current format and saves the original as a .old backup. Opening a RELION 3.x file in read-only mode raises a RuntimeError because conversion cannot be performed without writing. RELION 4 pipeline files are read as-is, with node type names mapped in memory, so opening the project (for example with PipelinerProject()) leaves the file unchanged. However, any writable session on the pipeline, such as scheduling, running or deleting a job, saves it in pipeliner format when the session closes. The RELION 4 file is then rewritten in place without a .old backup, so keep a copy if you need the original.


General-purpose workflow engines (Nextflow, Snakemake, and others)

Tools such as Nextflow, Snakemake, CWL runners, and Luigi are designed to express arbitrary directed acyclic graphs of computational tasks, handle cluster submission, and provide provenance. They solve a genuinely overlapping problem with pipeliner, but at a different layer of abstraction.

Where general-purpose engines excel:

  • Heterogeneous compute environments (cloud, HPC, local) with portable workflow descriptions.
  • Workflows that span many different scientific domains beyond cryoEM.
  • Large communities, extensive documentation, and integration with container registries.

Where pipeliner provides value that general-purpose engines do not:

  • CryoEM domain awareness. Pipeliner understands cryoEM data types (micrographs, particle stacks, tilt series, maps, atomic models) as first-class objects. Node types such as ParticleGroupMetadata and MicrographGroupMetadata are part of the type system, not just arbitrary file names. This lets the GUI display the right viewer for each output automatically.
  • RELION/Doppio GUI compatibility. No general-purpose workflow engine produces output that the RELION or Doppio GUI can open natively. Pipeliner does so as a core property, not as an add-on.
  • Integrated job parameter management. Pipeliner tracks parameter files (job.star, run.job), continuation state (continue_job.star), and status sentinel files as part of its core model. In a general-purpose engine you would need to build this tracking yourself.
  • Built-in deposition support. The pipeliner.deposition.onedep and EMPIAR deposition tools are domain-specific features with no equivalent in generic workflow engines.
  • Plugin system. New job types are registered via the ccpem_pipeliner.jobs Python entry-point group and discovered automatically — no configuration file editing or engine restart required.

Using pipeliner alongside a workflow engine: Facility automation pipelines (such as the on-the-fly processing system at eBIC/DLS) commonly use pipeliner as the cryoEM-aware inner layer and wrap it with a higher-level orchestration system. The PipelinerProject Python API and the pipeliner CLI are designed to be called from scripts, making this layered architecture straightforward. For example, an outer Nextflow pipeline can call pipeliner --run_job to run a motion-correction job and then poll job status by checking for PIPELINER_JOB_EXIT_SUCCESS in the job directory, while leaving all parameter tracking and GUI compatibility to pipeliner.


Deciding which tool to use

Use shell scripts when:

  • You are running a single program or a very short, one-off sequence.
  • You do not need GUI integration or structured provenance.

Use the RELION GUI when:

  • You prefer interactive, point-and-click parameter tuning.
  • You are processing a single dataset and do not need automation.

Use a general-purpose workflow engine when:

  • Your workflow spans multiple scientific domains or compute environments.
  • Portability and reuse across institutions with different software stacks is a priority.

Use CCPEM-Pipeliner when:

  • You need headless, scripted, or on-the-fly cryoEM processing.
  • You want results viewable in the RELION GUI or Doppio without manual conversion.
  • You are building facility automation or integrating cryoEM processing into a larger system.
  • You need structured provenance, status tracking, archiving, and deposition support in a single library.
  • You want to extend the job catalogue with custom job types without modifying the core package.

Examples

Check whether an existing RELION project can be opened by pipeliner

If you have a project created by the RELION GUI, navigate into the project directory and verify that pipeliner can read the pipeline file:

cd /path/to/my_relion_project
python -c "
from pipeliner.project_graph import ProjectGraph
with ProjectGraph(read_only=True) as pg:
    print(f'Jobs in pipeline: {len(pg.process_list)}')
"

Open an existing RELION project with the Python API

Pipeliner detects default_pipeline.star automatically when you instantiate PipelinerProject in the project directory. No --new_project flag or extra setup is needed:

import os
from pipeliner.api.manage_project import PipelinerProject

os.chdir("/path/to/my_relion_project")
proj = PipelinerProject()  # opens the existing project; does not create a new one

Schedule and run a job from a script (replacing a shell-script step)

The following replaces a shell-script call to relion_run_motioncorr with a pipeliner-managed job that writes provenance, tracks status, and produces GUI-compatible output:

import os
from pipeliner.api.manage_project import PipelinerProject
from pipeliner.api.api_utils import job_default_parameters_dict

os.chdir("/path/to/project")
proj = PipelinerProject(make_new_project=True)

# Get default parameters for the motion-correction job type
params = job_default_parameters_dict("relion.motioncorr.own")

# Set required input and key parameters (keys, not labels)
params["input_star_mics"] = "Import/job001/movies.star"
params["fn_gain_ref"] = "gain.mrc"
params["patch_x"] = "5"
params["patch_y"] = "5"

# Run the job; pipeliner assigns a sequential job number and writes sentinel files
job = proj.run_job(params, run_in_foreground=True)
print(job.output_dir)  # e.g. MotionCorr/job002/

Call pipeliner from an outer workflow script (facility automation pattern)

An outer shell script (or Nextflow process) can invoke pipeliner headlessly and poll for completion by checking the sentinel file:

# Generate a default run.job file for import
pipeliner --default_runjob relion.import.movies
# (edit relion_import_movies_run.job as needed, then run it)
pipeliner --run_job relion_import_movies_run.job

# Poll for completion (the outer script decides how long to wait)
while [ ! -f Import/job001/PIPELINER_JOB_EXIT_SUCCESS ]; do
    sleep 10
done
echo "Import job succeeded"

Generate a reproducibility document after processing

After completing a dataset, produce a dated .rst tutorial that captures every parameter choice up to the final refinement job:

pipeliner.make_project_tutorial --job Refine3D/job007/

Related concepts
  • Project — Understand the project directory layout, the default_pipeline.star graph file, and the .CCPEM_pipeliner_project marker that pipeliner writes when it opens or creates a project.
  • Job — How individual processing steps are represented, parameterised, and tracked with status sentinel files.
  • Pipeline — The directed graph of jobs and data nodes that makes reproducible, end-to-end processing possible.
  • Job plugin entry-point (ccpem_pipeliner.jobs) — How to extend pipeliner with custom job types that appear alongside built-in jobs, without modifying the core package.
  • Doppio GUI — The CCP-EM graphical interface where you can open, inspect, and continue any pipeliner project without the command line.
  • Queue submission — How pipeliner integrates with HPC cluster schedulers (SLURM, SGE) so that the same job definitions work both locally and on a cluster, controlled per-job via the do_queue option.