Architecture

This page describes the internal architecture of Trilio Share Protection (codename Abacá): how its components are arranged, how control and data flow between them, and the reasoning behind the structural decisions that shape everything from your API calls to your restore guarantees. Understanding this architecture helps you predict system behavior, interpret job states, design your bucket layout, and debug failures — all as a consumer of the REST API or as an operator deploying and managing the service.

abaca-api

The single entry point for all product operations. It runs as stateless REST pods on OpenShift, authenticates every request through Keystone middleware, and enforces access control via oslo.policy RBAC. It is registered in the Keystone service catalog as share-backup. Its job is to validate your request, record the desired state in the database, and — when a target is fully enrolled — cast a run_job message to the conductor over RabbitMQ. The API itself never touches secrets, never mounts shares, and never calls Kopia; it is a thin, horizontally scalable gate.

abaca-conductor

The orchestration brain. It owns the job lifecycle from the moment a run_job message arrives until the job reaches a terminal state (available or error). It drives the state-machine walk, resolves network and snapshot strategies, manages the worker fleet (boot, heartbeat tracking, capacity accounting, drain, and reap), runs the cron scheduler, and executes three background reconciliation sweeps: orphan-queued job recovery, dead-worker reaping, and backup expiry synchronization with Kopia. It also schedules and owns repository maintenance jobs. The conductor coordinates everything but touches no plaintext data.

abaca-worker-agent

The agent running inside each worker VM. Workers are RHEL virtual machines (in FIPS mode) that live in the abaca service project and boot into the tenant's own network scope so they can reach both the Manila share and the tenant's S3 endpoint. On cloud-init boot, the agent registers with the conductor and begins heartbeating. For each dispatched job, the agent: attaches the worker to the share network, obtains a read-only snapshot of the share data, fetches S3 credentials and the Kopia repository password from Barbican into process memory only, runs Kopia as a subprocess, streams client-side encrypted and deduplicated data into the tenant's S3 bucket, and tears everything down in reverse. Secrets are passed to Kopia exclusively via environment variables and are discarded when the process exits — nothing sensitive is ever written to the worker's disk or logged.

Kopia

The open-source backup engine baked into each worker VM. Kopia performs incremental-forever, deduplicated, client-side-encrypted snapshots. Every repository is created with FIPS-approved algorithms (AES256-GCM-HMAC-SHA256 encryption, HMAC-SHA256-128 block hash). From a restore perspective every snapshot is a full — content deduplication happens at the block level inside Kopia's content-addressed store, so there is no incremental chain to reconstruct. The Kopia binary is version-pinned and compiled at image-build time; it is never pulled at runtime, which is critical for airgap deployments.

abaca-conductor (scheduler thread)

A SchedulerLoop daemon thread inside the conductor process parses each protection policy's cron expression (via croniter), compares it against the last-run watermark, and enqueues backup jobs when the window comes due. Scheduling is conductor-native rather than delegated to Kopia so that per-backend concurrency caps, AZ affinity, and uniform job observability are preserved for both scheduled and on-demand backups.

MySQL / MariaDB

Holds bookkeeping and metadata only: targets, policies, backup records, restore records, jobs, workers, share capability probe results, and usage meters. It contains no secrets, no file paths, and no repository internals. Losing this database does not destroy backup data — the Kopia repositories in your S3 bucket remain independently restorable and can be used to rebuild the index via the rebuild-from-repository procedure.

RabbitMQ

The dedicated message broker used for job dispatch between the conductor and the worker fleet. Quorum queues are used for durability. Messages carry Barbican hrefs and Keystone trust IDs — never secret material. The conductor casts run_job messages; the worker agent receives run_kopia, mount, and umount RPCs on the abaca-worker topic.

Barbican

The OpenStack Key Manager service that holds your S3 access key, S3 secret key, and Kopia repository password as tenant-owned secrets. Abacá never stores these secrets itself — the database holds only the Barbican href references. The worker redeems a Keystone trust to pull secrets into memory at job execution time; they are discarded when the job ends.

python-abacaclient

A python-openstackclient plugin and thin Python SDK. It lets you manage share protection resources from the OpenStack CLI using openstack share protection … commands and lets you call the API from Python code.

abaca-dashboard

A Horizon plugin that adds Share Protection panels to the OpenStack web UI. It gives tenants a graphical view of their targets, policies, backups, restores, and usage, and gives operators a fleet and coverage overview. It is a thin client that consumes only the public REST API — it has no privileged internal paths.

abaca-manage

A server-side management CLI used by operators to perform database migrations, inspect configuration, and carry out administrative tasks that do not go through the REST API.

abaca-dev

A developer and operator CLI that automates enrollment, worker VM provisioning, and live data-path probing against a real OpenStack environment, and generates findings reports for fidelity and capability validation.

The following traces a complete on-demand backup from your API call to encrypted data landing in your S3 bucket.

1. You authenticate and submit the request. You call POST /v1/backups with a Keystone token. abaca-api validates the token, checks oslo.policy RBAC, creates a Backup row and a Job row (state: queued) in MySQL, and — because your target is enrolled — casts a run_job(job_id) message to the conductor over RabbitMQ.

2. The conductor picks up the job. abaca-conductor receives the cast. It resolves the backup job to backup_dispatch.py, stamps executed_by=conductor:<hostname> for audit, then redeems your Keystone trust (stored as a href in the database) to open a trust-scoped session. Using that session it fetches your S3 access_key, secret_key, and Kopia repository_password from Barbican into conductor memory. Nothing hits disk.

3. Share metadata is gathered. The conductor calls Manila to retrieve the share's protocol, share network ID, export locations, and snapshot capabilities. This is projected into a ShareAttributes bundle that the strategy selector uses.

4. A worker is fleet-picked. WorkerFleet.pick_worker_for_job counts non-terminal jobs on every ACTIVE worker against each worker's capacity_slots. It picks the least-loaded worker with a free slot (tie-broken on freshest heartbeat). If the pool is saturated, the job remains queued and the reconciliation loop will re-cast it after the deadline.

5. Strategies are resolved. Two independent dimensions are resolved from the share attributes:

• Network strategy: if your share has share_network_id (DHSS=true), the attach-to-share-network strategy is selected; a Neutron port is hot-plugged into your share network and the worker's IP is granted access via a Manila access rule. For DHSS=false backends, static-reachability is used and only the access rule is toggled per job.
• Snapshot strategy: the conductor inspects your share type's capabilities and selects the safest available method — .snapshot directory (snapdir), mountable snapshot, clone-from-snapshot, or live share (flagged crash_consistent_not_point_in_time if no snapshot support exists).

6. The job state machine walks forward. The conductor drives the job through states, each persisted to MySQL so you can poll GET /v1/jobs/{job_id} for live progress:

7. The backup is available. The job reaches state available. Your Backup record is visible at GET /v1/backups/{backup_id} and in the backups list. The encrypted, deduplicated snapshot now lives permanently in your Kopia repository inside your S3 bucket.

8. Background: retention and maintenance. The reconciliation loop's _sweep_expire_backups periodically fleet-picks a worker, calls kopia snapshot list --json against your repository, and marks any Backup DB rows whose kopia_snapshot_id no longer appears as status=expired. Separately, _sweep_target_maintenance enqueues a MAINTENANCE job when the last successful one is older than maintenance_interval_seconds (default: weekly). That job runs kopia maintenance to reclaim orphaned content blobs left after retention pruning.

Restore follows the same shape in reverse: POST /v1/restores creates a job, the conductor assembles a writable NetworkContext and SnapshotContext, fleet-picks a worker, the worker fetches secrets from Barbican, connects to your Kopia repository, and calls kopia restore with the requested snapshot ID and destination path. By default, files are written to a new Manila share; in-place restore requires force=true.

Workers are VMs, not pods

Worker VMs are independent RHEL virtual machines rather than Kubernetes pods, and the conductor is the only orchestrator for them. Per-job work — Neutron port hot-plugging, kernel NFS mounts, FIPS mode at the OS level — is node-level and gains nothing from a pod abstraction. Adding a second orchestration layer (a Kubernetes cluster in the data plane) would introduce cluster operations complexity without benefit. Workers are stateless and replaceable: a new image triggers a conductor-driven fleet rotation, which is also the CVE turnaround mechanism.

Kopia is a baked-in binary, not a container

Kopia is compiled into the worker VM image at build time. Runtime image pulls are incompatible with per-job egress pinning (each job's S3 traffic must route through a pinned NIC/route to the tenant's bucket), and in airgap RHOSO deployments there is no guarantee of public internet reachability from tenant networks. Baking the binary in also means immutable image replacement — not in-place upgrades — is the update model.

One executor path in production

Every kopia-invoking job type (enrollment, backup, restore) dispatches over RabbitMQ to a worker via WorkerRPCExecutor. An EphemeralContainerExecutor (Kopia in a throwaway container on the conductor host) exists for Docker Compose development environments that have no worker fleet, but it is not on the production hot path and is not paired with RpcWorkerMounter. This deliberate single path means the production code is always exercised, not a special-cased shortcut.

The database is bookkeeping, not the source of truth

MySQL holds job records, policy schedules, backup manifests, and usage meters — but the Kopia repositories in your S3 bucket are the authoritative record of your backup data. This separation means losing the database is a loss of convenience and indexing, not a loss of backup data. The rebuild-from-repository procedure re-derives the database view by enumerating Kopia sources and snapshot manifests. This is a required, periodically exercised runbook item.

No key escrow; secrets travel via environment only

Encryption keys never leave your Barbican secret store except into the worker's process memory during a job. Secrets are passed to Kopia subprocesses exclusively as environment variables (KOPIA_PASSWORD, AWS_*), never as command-line arguments (which would appear in /proc/*/cmdline) and never written to disk. The repository password is generated by the worker at enrollment time, stored immediately in your Barbican, and never held by the control plane. This means even a fully compromised Abacá control plane cannot decrypt your backup data.

Scheduling is conductor-native, not delegated to Kopia

The conductor's SchedulerLoop parses cron expressions and enqueues backup jobs rather than relying on Kopia's built-in snapshot scheduling. This preserves three capabilities that Kopia's client-side scheduler cannot provide: per-backend concurrent-job caps, AZ affinity for worker placement, and a single unified jobs table that covers scheduled, on-demand, retry, maintenance, and enrollment jobs with identical observability.

Bucket-to-repository is a strict 1:1 mapping

Each backup target maps to exactly one S3 bucket, and each bucket holds exactly one Kopia repository. This makes the repository the deduplication domain: identical data across a tenant's shares within one repository is stored once, but deduplication never crosses tenant boundaries. Multiple targets per tenant are supported for separating data classes (e.g., object-locked production vs. development), but each additional repository incurs its own maintenance and verification overhead, and reassigning a share from one target to another requires a full re-ingest.

S3 object lock for immutability, not a separate system

Immutability is enforced via S3 object lock (WORM), which must be enabled at bucket creation time. This means even a compromised S3 credential cannot silently delete backups within the retention window. Object lock capability is detected and recorded during enrollment preflight; lifecycle expiration rules on the bucket are a hard failure during enrollment (tenant_action_required) because they would silently undermine retention.

Shared worker pool means workers are not per-tenant

Workers live in the abaca service project and pick up jobs from any tenant. This is efficient (no idle per-tenant fleets) and simpler to operate, but it means the plaintext of one tenant's data and another tenant's data coexist in the same Nova project's network at the same time. Isolation is logical (per-job network attachment, per-job Barbican credential fetch, no disk persistence) rather than physical. If your security posture requires physical per-tenant compute isolation, this shared-pool model is not appropriate without additional controls.

No application-consistent backups in the current release

All backups are crash-consistent: they capture the on-disk state at a single point in time, equivalent to a sudden power loss. Filesystem structures are intact and recoverable, but in-flight application writes that had not been flushed to disk may be missing. Application-consistent backups via tenant quiesce hooks are explicitly out of scope for the current release. If your workload requires application-level consistency (for example, databases that need a clean shutdown checkpoint), you must arrange quiescing independently.

Adding a second backup target means a full re-ingest

If you reassign a share from one backup target to another — for example, to move it into an object-locked bucket — the share's backup history in the old repository is not migrated. Old backups remain restorable from the old target until retention expires them, but the new target starts with a full baseline ingest. Plan target layout before your first backup to avoid this.

The EphemeralContainerExecutor is not production-supported

The Docker Compose development mode executor (Kopia in a throwaway container on the conductor) exists to let you evaluate the system locally without booting worker VMs. It is not paired with RpcWorkerMounter, it does not exercise the real production dispatch path, and it is not supported for production use. Local evaluation findings from this mode may not accurately reflect behavior against a real OpenStack environment.

Manila event-driven reconciliation is not yet wired

Detection of deleted shares via the OpenStack notification bus is aspirational and not implemented in the current release. Share deletion and other Manila lifecycle events are detected only through periodic reconciliation sweeps, not in real time.

kopia snapshot verify is not yet scheduled

Repository verification (kopia snapshot verify) is tracked as a planned job type but is not yet scheduled by the reconciliation loop. Operators cannot currently rely on automated data-integrity verification against live repositories.

Maintenance cadence is deployment-scoped, not per-policy

Repository maintenance (the job that reclaims orphaned content blobs after retention pruning) runs on a single deployment-level interval (maintenance_interval_seconds, default weekly) shared across all targets. You cannot set a finer maintenance cadence for a single high-churn target without changing the global setting. Overdue maintenance is surfaced as a target health warning, but during the gap retention is not enforced at the storage level even if backup records are marked expired in the database.

Restoring to the original share requires an explicit flag

In-place restore (writing recovered data back onto the source share rather than onto a new share) requires force=true in your restore request. This is intentional: overwriting live data is destructive and irreversible. If you omit the flag, the restore always targets a new share, which is the safe default but may require you to manually swap share mount points in your application.