CCPEM Pipeliner
Concept

Problem Being Solved

Why a coordination layer for cryo-EM processing exists and what pipeliner automates


Overview

CryoEM data processing involves dozens of heterogeneous programs — motion correction, CTF estimation, particle picking, classification, refinement, model building — each producing outputs that feed into the next step. Coordinating these programs by hand is error-prone, non-reproducible, and difficult to automate at scale. This page explains the coordination problem that CCPEM-Pipeliner solves and the design choices it makes to solve it.


Content

The problem: coordinating heterogeneous processing programs

A typical single-particle analysis (SPA) workflow involves at least eight to twelve discrete processing steps. Each step is a separate program — often from a different software suite — with its own input format, output format, configuration flags, and failure modes. Running these steps manually creates several compounding problems:

Fragile data handoff. The output of one program must be located and passed as input to the next. A misspelled path, a missing file, or a changed directory structure silently breaks the chain. There is no central record of which file came from which step.

No reproducibility. When a project is shared or revisited months later, the exact parameters used at each step are typically scattered across shell scripts, notebook cells, or memory. Reproducing or auditing a result requires reconstructing this history from scratch.

No automation at scale. Facilities such as eBIC/DLS run processing pipelines on data as it arrives from the microscope. Launching each step manually is not feasible. The steps must be triggered automatically when their inputs are ready, and failures must be detected and reported without human supervision.

Interoperability friction. Biologists want to inspect intermediate results, adjust parameters, and continue processing — often using a GUI such as RELION or Doppio. A purely scripted pipeline that produces no compatible project structure forces users to choose between automation and interactivity.

How pipeliner addresses the coordination problem

Pipeliner introduces a project directory as the single unit of coordination. Every processing step (a job) runs inside its own subdirectory within this project. A pipeline graph stored as a STAR-format file (default_pipeline.star) records the entire directed graph of jobs and the data files (nodes) connecting them. This graph is the authoritative record of what ran, in what order, with what outputs.

Jobs, nodes, and the pipeline graph

Each job is a typed processing step with declared input nodes, output nodes, and configurable parameters. When a job is scheduled, pipeliner registers its expected output nodes in the pipeline graph and creates empty placeholder files under .Nodes/ to mark them. When the job completes, the actual output files appear at those paths. Downstream jobs reference these paths as their inputs — the graph enforces the contract.

Job status is tracked via sentinel files written into the job directory:

Sentinel fileMeaning
PIPELINER_JOB_EXIT_SUCCESSJob finished successfully
PIPELINER_JOB_EXIT_FAILEDJob failed
PIPELINER_JOB_EXIT_ABORTEDJob was aborted by the user

A job with none of these files is either Scheduled or Running. This file-based status mechanism means project state survives process restarts and is readable by external tools such as facility scripts. The RELION GUI does not read these files (it looks for its own RELION_JOB_EXIT_* files); it sees each job's status through default_pipeline.star, where pipeliner also records it.

Reproducibility by construction

Because pipeliner writes every job's parameters into a job.star file and a run.job file before execution, the exact configuration of every step is preserved on disk. The pipeline graph records which outputs fed which inputs. Together, these make it possible to regenerate a tutorial document for any completed workflow using pipeliner.make_project_tutorial, or to produce a structured metadata report with the pipeliner --metadata_report flag.

Headless and scripted operation

Pipeliner is designed to run without a GUI. The pipeliner CLI and the PipelinerProject Python API (imported from pipeliner.api.manage_project) expose the same operations: running jobs, continuing them, cleaning up intermediate files, archiving results, and generating reports. This makes it straightforward to embed pipeliner in facility automation scripts that respond to incoming data without human intervention.

GUI compatibility

Projects created by pipeliner are fully compatible with the RELION GUI and the Doppio graphical interface. The default_pipeline.star file is a standard STAR-format pipeline graph that both tools understand. Users can start a pipeline headlessly, inspect results in Doppio, adjust parameters, and continue — without any format conversion.

Extensibility

New processing steps can be added as Python classes and registered with pipeliner through the ccpem_pipeliner.jobs entry-point group. Third-party packages can register job types without modifying the core package. Job types are discovered dynamically each time they are needed, so installing a plugin package is sufficient to make it appear alongside built-in jobs.

What pipeliner does not do

Pipeliner is the business-logic layer — it orchestrates, tracks, and coordinates. It does not replace the underlying processing programs (RELION, MotionCor2, CTFFIND, crYOLO, and others). Those programs must be installed and accessible on the system path. The pipeliner.check_setup command verifies that pipeliner can find the programs it needs and reports which job types are unavailable due to missing executables.


Examples

Inspect what pipeliner tracks about a completed project

This snippet opens an existing project in read-only mode and prints every job name, type, and status recorded in the pipeline graph. It shows how the pipeline graph is the single source of truth for workflow state.

import os
os.chdir("/path/to/my/project")

from pipeliner.project_graph import ProjectGraph

with ProjectGraph(name="default", read_only=True) as pipeline:
    for proc in pipeline.process_list:
        print(proc.name, proc.type, proc.status)

Check which job types are available on the current system

Because pipeliner separates coordination from execution, you can ask it which job types have their required executables installed before attempting to run anything.

from pipeliner.job_factory import get_job_types

for job in get_job_types():
    status = "available" if job.jobinfo.is_available else "unavailable"
    print(f"{job.PROCESS_NAME}: {status}")
    if not job.jobinfo.is_available:
        print(f"  Reason: {job.jobinfo.unavailable_message}")

Verify the installation and available programs from the command line

The pipeliner.check_setup command prints four sections — python version, available job types, pipeliner console scripts, and a final report — giving a complete picture of what the coordination layer can and cannot drive on the current system.

pipeliner.check_setup

Open an existing RELION project in pipeliner

Because projects are just directories with a default_pipeline.star file, any RELION project can be used with pipeliner immediately. Change into the project directory and open it:

import os
os.chdir("/path/to/relion/project")

from pipeliner.api.manage_project import PipelinerProject

# Opens the existing pipeline; does not create a new one
proj = PipelinerProject(pipeline_name="default")

After this, all PipelinerProject methods are available: you can schedule new jobs, inspect existing ones, run cleanup, or generate reports — all tracked in the same graph that RELION wrote.

Create a new project and run the first job programmatically

This shows the minimal pattern for starting a pipeline from scratch. The coordination layer creates the project directory structure, assigns job numbers sequentially across the whole project, and records the import job's output nodes so they are available as inputs to the next step.

import os
os.makedirs("/path/to/new/project", exist_ok=True)
os.chdir("/path/to/new/project")

from pipeliner.api.manage_project import PipelinerProject
from pipeliner.api.api_utils import job_default_parameters_dict

proj = PipelinerProject(
    pipeline_name="default",
    project_name="My first pipeline",
    make_new_project=True,
)

# Get the default parameters for a movie import job
params = job_default_parameters_dict("relion.import.movies")

# Override values that differ from the defaults
params["fn_in_raw"] = "Movies/*.tiff"  # must be a path inside the project
params["kV"] = "300"
params["Cs"] = "2.7"
params["Q0"] = "0.1"
params["angpix"] = "0.83"

# Input paths must be relative to the project directory: an absolute path such as
# "/data/raw_movies/*.tiff" fails validation and schedule_job raises ValueError.
# Copy or symlink the raw movies into the project first (e.g. into Movies/).

# Schedule the job (registers it in the graph without running)
job = proj.schedule_job(params)
print(job.output_dir)  # e.g. Import/job001/

Related concepts
  • Project structure — How the project directory is laid out, what default_pipeline.star contains, and how sentinel files record job status.
  • Running a processing job — How to schedule and execute a job using the CLI or the PipelinerProject Python API.
  • Building an automated workflow script — How to chain jobs into a reproducible end-to-end pipeline using run_schedule and the PipelinerProject API.
  • Writing and registering a custom job plugin — How to create a new job type as a Python class and register it under the ccpem_pipeliner.jobs entry-point group.
  • Viewing results in Doppio or RELION GUI — How projects created by pipeliner are compatible with graphical interfaces for interactive inspection and continuation.
  • Install and verify pipeliner — How to install ccpem-pipeliner with pip and run pipeliner.check_setup to confirm that required external programs are available.