CCPEM Pipeliner
Runbook

Troubleshooting

Diagnosing setup problems, failed or stuck jobs, locked projects and settings that don't take effect


Objective

This runbook helps you diagnose and resolve the most common operational problems with ccpem-pipeliner: broken installations, failed or stuck jobs, locked projects, and environment settings that appear to have no effect.


Scope

This runbook covers:

  • Verifying and repairing a pipeliner installation
  • Diagnosing why a job failed, is stuck, or shows the wrong status
  • Unlocking a project whose .relion_lock directory was not cleaned up
  • Recovering a corrupted or missing default_pipeline.star
  • Troubleshooting environment variables and settings that do not take effect
  • Identifying unavailable job types due to missing external executables

Out of scope:

  • Tuning cryoEM processing parameters (box size, resolution, etc.)
  • Cluster scheduler configuration (SLURM, SGE)
  • Installation of external programs (RELION, CTFFIND, MotionCor2, etc.)
  • Doppio GUI issues unrelated to the pipeliner project files

Prerequisites

Before using this runbook you need:

  • ccpem-pipeliner installed in a Python ≥ 3.9 environment (pip install ccpem-pipeliner)
  • Shell access to the machine or HPC login node where the project lives
  • Read/write access to the project directory
  • The project directory as your current working directory for all project-level commands (exceptions are noted per step)
  • Optionally: RELION, an MPI runtime, and any other external programs that the jobs you are debugging require

Steps

1. Verify the installation

Run the built-in setup check:

pipeliner.check_setup

This prints four sections:

  1. Python version check — confirms Python ≥ 3.9.
  2. Available jobtypes check — for each registered job type, shows Success (with the resolved executable path) or Unavailable (with '<command>' missing from system path for each missing program).
  3. Pipeliner setup check — verifies that all pipeliner console scripts (pipeliner, pipeliner.bfactor_plot, pipeliner.update_metadata_schema, pipeliner.recover_project, pipeliner.check_setup, pipeliner.regenerate_results) are findable on PATH.
  4. Final setup report — summarises whether the installation is fully working or lists problems found.

If one or more pipeliner console scripts show ERROR: reinstall the package to restore entry points:

pip install --force-reinstall ccpem-pipeliner

Then re-run pipeliner.check_setup.

If job types are listed as Unavailable: the relevant external program is not on PATH. Either install the program, add its directory to PIPELINER_ADDITIONAL_PROGRAM_PATHS, or confirm that any environment setup scripts listed in PIPELINER_PATH_TO_SOURCE_FILES are correct. Note that PIPELINER_ADDITIONAL_PROGRAM_PATHS and PIPELINER_PATH_TO_SOURCE_FILES are applied once at import time, so you must restart the Python process (or re-invoke the CLI) after changing them.


2. Diagnose a failed job

a. Identify the failure status.

Check which sentinel file exists in the job directory:

  • PIPELINER_JOB_EXIT_FAILED — the job ran but exited with a non-zero status or an internal error.
  • PIPELINER_JOB_EXIT_ABORTED — the job was aborted (either by pipeliner --abort_job <job_name> or the abort trigger file).
  • PIPELINER_JOB_EXIT_SUCCESS — the job succeeded (if you see this alongside a failed report, the pipeline status may be out of sync; see step 5).
  • No sentinel file, and no PIPELINER_JOB_EXIT_SUCCESS — the job may still be running, or it crashed before writing any status.

b. Read the job logs.

In the job directory (e.g. MotionCorr/job002/):

  • run.out — standard output from the job runner, including timestamped progress lines.
  • run.err — standard error, including tracebacks for unexpected errors.
  • note.txt — the exact command(s) that were executed.

Read these files with your preferred text viewer:

cat MotionCorr/job002/run.err
cat MotionCorr/job002/note.txt

c. Check job availability.

If run.err shows that the program was not found, or if the job was never attempted, confirm the job type is available:

pipeliner --check_jobs

This prints each available job type with its resolved executable path, and an Unavailable jobs section that lists the programs of each unavailable job type: a program that was found is shown as * <command>: with its path: <path> on the next line, a missing program as - <command> followed by ERROR: '<command>' not found in system path, and any other reason the job is unavailable as - <reason>. A job type that is unavailable will still be schedulable but will fail at execution time when its command is invoked.

d. Check input files.

If no job directory was created at all, the most likely cause is missing input files. Pipeliner validates input files before creating the job directory; run_job raises ValueError('Job cannot be run because: ...') and no output is written. (A job that was scheduled but never started has a job directory with an empty run.out and run.err, created at scheduling time, and no sentinel file.) Confirm that all upstream jobs succeeded and their output nodes exist.


3. Diagnose a stuck or seemingly running job

A job appears stuck if it has no PIPELINER_JOB_EXIT_* sentinel file and the process is no longer active.

a. Confirm the process is not actually running.

Check whether a job runner process is still active for that job directory (use ps, top, or your cluster scheduler's job query command). If nothing is running, the job crashed without writing a sentinel file.

b. Manually reset the job status.

Use --set_status to mark the job as failed so you can rerun it:

pipeliner --set_status MotionCorr/job002/ failed

Valid status values for --set_status are substrings of: success, failed, aborted, scheduled, running. (finished is not valid and will raise ValueError.)

Important: A job whose status is currently Running cannot be set to aborted via --set_status (this raises ValueError); use pipeliner --abort_job <job_name> to abort a job that is actively running, or, if it is definitely not running, set it to failed or success first. Setting a job to the status it already has also raises ValueError.

--set_status automatically removes any existing PIPELINER_JOB_EXIT_* files and writes the new status sentinel (for success, failed and aborted; no sentinel is written for scheduled or running). Do not manually create or delete these files.

c. Check for a stale lock.

If pipeliner commands fail with a message about not being able to acquire the lock, the project lock directory .relion_lock may have been left behind by a crashed process. Use pipeliner.recover_project to remove it (see step 4).


4. Recover a locked or corrupted project

Use pipeliner.recover_project when default_pipeline.star has become inconsistent, corrupted, or the .relion_lock directory has been left in place after a crash.

Run from the project root directory:

pipeliner.recover_project

By default, the tool prompts Perform recovery (Y/N)? before making changes. To skip the prompt:

pipeliner.recover_project --assume-yes

What it does:

  1. Confirms default_pipeline.star exists (exits with No pipeline file found… if it is completely missing — see note below).
  2. Removes .relion_lock if present.
  3. Finds the per-job backup copy of default_pipeline.star in the highest-numbered job directory that has one and copies it back to the project root.
  4. Warns if any jobs were scheduled after the restored backup (those jobs will be lost from the restored pipeline).

The tool is mostly silent on success; it prints only warnings and errors. To confirm recovery succeeded:

  • default_pipeline.star should have a recent modification time (the time the recovery ran) and the same contents as the backup source.
  • .relion_lock should no longer exist.

Important: pipeliner.recover_project requires that default_pipeline.star already exists in the project root. If the file is completely missing, the tool exits immediately and cannot restore from per-job backups. In that case you must manually copy a per-job backup:

# Find the most recent per-job backup
ls -lt */job*/default_pipeline.star | head -5
# Copy the most recent one
cp MotionCorr/job002/default_pipeline.star default_pipeline.star

If the restored pipeline file is readable but is missing nodes, also try:

pipeliner.update_nodes

This adds any expected nodes that are missing from the pipeline file, which is useful when converting or importing a project started in RELION. It has to read the pipeline file first, so it cannot repair a corrupted or unreadable one.


5. Fix a job whose status does not match reality

If the pipeline graph shows a job as Running, Scheduled, or Succeeded when it is not, use --set_status to correct it:

pipeliner --set_status Import/job001/ success
pipeliner --set_status CtfFind/job003/ failed
pipeliner --set_status Class2D/job005/ scheduled

Note the argument order: job name first, then status. Short forms (e.g. job001 or 1) and full names (e.g. Import/job001/) are both accepted by the CLI.

From the Python API, use capitalized status strings:

from pipeliner.api.manage_project import PipelinerProject

proj = PipelinerProject()
proj.update_job_status('Import/job001/', 'Succeeded')  # capital S

Valid values for update_job_status are: Succeeded, Failed, Aborted, Running, Scheduled.


6. Diagnose queue submission failures

If a scheduled job is immediately marked Failed when it is started with queue submission enabled, the most likely cause is a missing submission script template. (With pipeliner --run_job, a missing or empty template is caught before the job is created: the command fails with ValueError: Job cannot be run because: File <path> not found and no job is added.)

a. Check the current template setting.

The template path is resolved from (in priority order):

  1. PIPELINER_QSUB_TEMPLATE environment variable
  2. RELION_QSUB_TEMPLATE environment variable
  3. "qsub_template" key in ccpem-pipeliner-settings.json

This setting only supplies the default for a new job's Standard submission script: (qsubscript) option; a job uses the value stored in its own run.job or job.star file. If that file does not exist when a scheduled job is started, the job will be marked Failed with cause Error reading template submission script in: <path> before any job runner starts. The job's run.out and run.err stay empty.

b. Verify the template and queue settings.

Check your settings file:

cat ~/.config/ccpem/pipeliner/ccpem-pipeliner-settings.json

(On macOS: ~/Library/Application Support/ccpem/pipeliner/ccpem-pipeliner-settings.json)

Ensure qsub_template points to a valid, readable file, qsub_command is set (default: qsub), and queue_name is correct (default: openmpi).

c. Disable queue submission for local testing.

Set PIPELINER_QUEUE_USE to a value other than yes, true, y, or 1 (case-insensitive), or edit the job's do_queue option to No in its run.job or job.star file before running.

export PIPELINER_QUEUE_USE=false

Note: PIPELINER_QUEUE_USE sets the default do_queue value for newly created jobs only. Jobs that already have do_queue == Yes in their parameter files will still be submitted to the queue regardless of this environment variable.

d. Check for unrecognised placeholders.

If the template contains placeholder variables other than the nine fixed ones (XXXnameXXX, XXXmpinodesXXX, XXXcoresXXX, XXXdedicatedXXX, XXXthreadsXXX, XXXqueueXXX, XXXcommandXXX, XXXerrfileXXX, XXXoutfileXXX) and the numbered extra placeholders (XXXextra1XXX through XXXextraNXXX), pipeliner will emit a RuntimeWarning. Misspelled known placeholders also trigger a warning. These warnings are printed to the standard error of the command or Python process that started the job, not to the job's run.err.


7. Fix environment settings that do not take effect

Understanding when settings are applied:

  • PIPELINER_PATH_TO_SOURCE_FILES and PIPELINER_ADDITIONAL_PROGRAM_PATHS are applied once at import time (when pipeliner is first imported in a Python process). Changing these after import has no effect; you must restart the process or re-invoke the CLI.
  • All other settings (queue_name, qsub_template, qsub_command, mpirun_command, etc.) are re-read from environment variables on every getter call and can be overridden at any time without restarting.
  • The settings file (ccpem-pipeliner-settings.json) is read once on first access within a process. Edits to the file after first access are not picked up until the process restarts.

If a setting via environment variable is not working:

  1. Confirm the variable is exported in the same shell session running pipeliner (echo $PIPELINER_QUEUE_USE).
  2. Confirm you are using the correct variable name. Most settings accept both a PIPELINER_* name and a RELION_* fallback name, and PIPELINER_* takes precedence. Some have only a PIPELINER_* name (for example PIPELINER_MPIRUN_COMMAND, PIPELINER_ADDITIONAL_PROGRAM_PATHS and PIPELINER_PATH_TO_SOURCE_FILES), and the scratch directory is set only with RELION_SCRATCH_DIR, which has no PIPELINER_-prefixed equivalent.
  3. For PIPELINER_ADDITIONAL_PROGRAM_PATHS and PIPELINER_PATH_TO_SOURCE_FILES, restart your shell session or Python process after setting the variable.

If the settings file cannot be written:

Pipeliner logs a WARNING of the form CCP-EM Pipeliner could not create a settings file at <path>. Settings can still be set using environment variables. and continues with built-in defaults. You can still override all settings using environment variables.

On an unsupported platform:

Pipeliner logs Unable to set up pipeliner python environment at import and raises RuntimeError when a setting is first accessed. Only Linux and macOS are supported.


8. Resolve a run.job or job.star parsing error

An option silently keeps its default value — You edited a run.job file and changed an option label, or removed the == separator from a line. Reading a run.job file never raises an error for this: the line is ignored, the option keeps its default value, and only a warning <file>; no value for `<label>` using default value: <default> is logged. Labels must match exactly those produced by pipeliner --default_runjob <jobtype>. Re-generate the file:

pipeliner --default_runjob relion.motioncorr.own

This writes relion_motioncorr_own_run.job in the current directory. Edit the values (after ==) but never the labels (before ==).

Ensure every non-header line uses the format <option label> == <value>.

ValueError: File <file> is not a valid STAR file. — You passed a run.job file (which uses label == value format) to edit_jobstar(), which expects a STAR-format job.star file. Use write_default_jobstar(<jobtype>) to generate a STAR template instead.

ValueError: JobOption <name> not in the STAR file — You passed an unrecognised parameter key to edit_jobstar(). Parameter keys must match the job's registered option names (e.g. input_star_mics, kV, Q0).

ValueError: Job type '<type>' not found; conversion failed — The job_type == line in a run.job file references an unregistered job type. Use the exact entry-point process name (e.g. relion.motioncorr.own), not an abbreviated form.


9. Resolve ValueError when calling run_job with a dict

ValueError: Job type '<type>' not found — The _rlnJobTypeLabel value in your dict is not a registered entry-point name. Use pipeliner --available_jobs to list registered types and ensure you use the exact name.

ValueError: Parameter <name> is not found in jobtype <type> — Your dict contains a key that is not a valid option for that job type. Remove or correct the key. Note that unknown parameter keys raise ValueError regardless of the ignore_invalid_joboptions setting.

ValueError: Cannot create a job from this dict; key '_rlnJobTypeLabel' is missing — Your dict is missing the _rlnJobTypeLabel key. Add it:

params = {
    '_rlnJobTypeLabel': 'relion.import.movies',
    'fn_in_raw': 'Movies/*.tiff',
    'kV': '300',
    'Cs': '2.7',
    'Q0': '0.1',
    'angpix': '1.06',
}

Values do not have to be strings; numbers are accepted too (for example 'kV': 300).


10. Recover from a with PipelinerProject() error

PipelinerProject is not a context manager and does not support the with statement. Instantiate it directly:

from pipeliner.api.manage_project import PipelinerProject

proj = PipelinerProject()  # correct
proj.run_job(params)       # call methods directly

If you see FileNotFoundError on instantiation, either you are not in a project directory or default_pipeline.star is missing. Change to the project root. (pipeliner.recover_project cannot help here, because it exits when default_pipeline.star is missing; copy a per-job backup manually as described in step 4.)

If you see ValueError: A pipeliner project has been found in this directory, but the expected pipeline file ... is missing, the marker file .CCPEM_pipeliner_project exists but default_pipeline.star does not. Restore default_pipeline.star from a per-job backup (see step 4).


Verification

After completing the relevant steps, confirm success by checking the following observable outcomes:

  1. Installation — pipeliner.check_setup prints SUCCESS: The pipeliner and all job types appear to be correctly installed and ready to use in the Final setup report section. All pipeliner console scripts show Success.

  2. Failed job diagnosed and restarted — The job's sentinel file changes to PIPELINER_JOB_EXIT_SUCCESS after rerunning. run.out in the job directory shows a completion line. The job appears with status Succeeded when you inspect the pipeline.

  3. Stuck job recovered — pipeliner --set_status <job> failed completes without error. No stale PIPELINER_JOB_EXIT_* file from the previous status remains. You can now rerun the job.

  4. Project recovery — default_pipeline.star exists and has been updated (check modification time). .relion_lock does not exist. PipelinerProject() instantiates without error from the project directory. Existing succeeded jobs appear in the pipeline.

  5. Queue submission fixed — Running a job with queue submission enabled produces a submitted cluster job (visible via your scheduler's query command) and a run_submit.script in the job directory, rather than run_job stopping with ValueError: Job cannot be run because: ... not found (no job created) or the job being marked Failed with Error starting job.

  6. Settings taking effect — The desired behaviour is observed (e.g. jobs route to the correct queue, MPI count defaults are applied). For path-based settings, pipeliner.check_setup shows the expected executable paths.


Rollback

Each time a job is scheduled, run or continued, pipeliner copies default_pipeline.star (with that job already added) into the job's own directory. Other operations, such as --set_status, deleting or cleaning up jobs and pipeliner.recover_project, write no backup. Use these per-job backups to undo changes:

  1. Undo pipeliner.recover_project: The recovery script overwrites default_pipeline.star. If the overwrite made things worse, copy the backup from the specific job directory you trust:

    cp MotionCorr/job002/default_pipeline.star default_pipeline.star
    
  2. Undo --set_status: Set the status back to what it was:

    pipeliner --set_status MotionCorr/job002/ <previous_status>
    
  3. Undo a deleted lock directory: If you removed .relion_lock while another pipeliner process was legitimately running (e.g. a long job), that process may write a corrupt pipeline file on exit. Stop any legitimate pipeliner processes before removing the lock.

  4. Undo accidentally overwritten settings file: The settings file is plain JSON at ~/.config/ccpem/pipeliner/ccpem-pipeliner-settings.json (Linux) or ~/Library/Application Support/ccpem/pipeliner/ccpem-pipeliner-settings.json (macOS). Restore from a backup if you have one, or delete the file so pipeliner regenerates it with defaults on next run.

  5. General project rollback: If the pipeline graph cannot be recovered from per-job backups, restore the entire project directory from your site's backup system.


Escalation

If this runbook does not resolve your problem, collect the following information before seeking help:

  1. Full output of pipeliner.check_setup (copy the complete terminal output).
  2. Python version: python --version
  3. Installed package version: pip show ccpem-pipeliner
  4. Operating system and version: uname -a (Linux) or sw_vers (macOS)
  5. The full error message and traceback from the failed command or Python session.
  6. Contents of run.out and run.err from the affected job directory (if a job failed).
  7. Relevant environment variables: env | grep -E 'PIPELINER_|RELION_'
  8. The job type involved (e.g., relion.motioncorr.own) and how it was invoked (CLI flag or Python API call with code snippet).

Contact channels:

  • CCP-EM mailing list: https://www.jiscmail.ac.uk/CCPEM — suitable for usage questions and community discussion.
  • CCP-EM direct support: ccpem@stfc.ac.uk — for bugs or issues not resolved by the mailing list.
  • Issue tracker / merge requests: File an issue or merge request in the public repository if you have identified a reproducible bug or have a fix.

When contacting support, include all eight items listed above and describe the steps you have already taken from this runbook.