📖
ccpem-pipeliner
CCPEM-Pipeliner is a Python library and command-line tool that lets structural biologists design, run, and automate cryoEM image-processing workflows by chaining discrete processing jobs into reproducible pipelines.
Intended Audience
Who ccpem-pipeliner is for: facility automation engineers, software integrators and job-plugin authors
Concept
Problem Being Solved
Why a coordination layer for cryo-EM processing exists and what pipeliner automates
Concept
Alternatives
How pipeliner compares with shell scripts, the RELION GUI and general-purpose workflow engines
Concept
Key Concepts
Glossary of the core ideas: projects, jobs, nodes, the pipeline graph, job status and plugins
Concept
Getting Started
From a fresh install to your first project and job
Tutorial+2 sub-pages
Python API (PipelinerProject)
Driving projects from Python with PipelinerProject: running, scheduling and managing jobs, cleanup, reports and depositions
API reference+3 sub-pages
Usage Patterns
Common ways to use pipeliner from the command line and from Python scripts
Guide
Examples
Worked end-to-end examples of building and running workflows with the pipeliner CLI and Python API
Guide
Troubleshooting
Diagnosing setup problems, failed or stuck jobs, locked projects and settings that don't take effect
Runbook
Development Guide
Setting up a development environment, running tests and linters, and building the docs
Guide
Pipeliner CLI Reference
Every flag of the pipeliner command, grouped by task
Guide+1 sub-page
Projects and the Pipeline Graph
What a project directory contains and how jobs and nodes form the pipeline graph
Guide
Running, Scheduling and Aborting Jobs
The job lifecycle: scheduling, running, continuing, aborting, status files and logs
Guide
Queue Submission and MPI
Running jobs on HPC clusters with queue templates, and configuring MPI
Guide
Writing a Job Plugin
Writing a new job type as a PipelinerJob subclass and registering it as an entry point
Guide
Job Catalogue
How job types are named and discovered, with links to each category of jobs
Guide+9 sub-pages
Cleanup and Archiving
Reclaiming disk space with cleanup and trash, and preserving projects as archives
Guide
Depositions (OneDep/EMDB and EMPIAR)
Preparing EMDB (OneDep) and EMPIAR depositions from a project
Guide
RELION Compatibility
How pipeliner projects stay interchangeable with RELION
Guide