Architecture

This page describes the architecture of Trilio Share Protection for OpenStack (Abacá): how its components are structured, how they relate to each other, and why the system is designed the way it is. Understanding this architecture helps you reason about failure modes, plan deployments, and integrate with the API. The central design choice is a strict separation between a long-lived, stateless control plane running on OpenShift and short-lived worker VMs running inside tenant network scopes — a separation that keeps plaintext data and encryption keys out of the control plane entirely.

abaca-api The public REST API, versioned at /v1. It runs as stateless pods in the abaca OpenShift namespace, authenticated by Keystone via keystonemiddleware, and enforced by oslo.policy. Every tenant and operator action — creating policies, requesting backups, triggering restores, querying jobs — goes through this service. The API writes desired state to the database and casts fire-and-forget RPC messages to the conductor; it never moves data itself.

abaca-conductor The orchestration brain, also running as pods in the abaca namespace. The conductor owns three responsibilities: driving the job state machine from queued through to available or error; scheduling policy-driven backups from cron expressions; and managing the worker VM fleet (booting, draining, retiring, and reconciling stale or orphaned jobs). The conductor communicates with workers by publishing leased commands to the database — workers claim those commands over outbound HTTPS. The conductor also runs periodic reconciliation sweeps (fleet reap, expiry, catalogue sync, maintenance, usage sampling) to keep the system consistent without operator intervention.

abaca-worker-agent A process baked into every worker VM image. After first boot, cloud-init starts the agent, which registers with the conductor over HTTPS, then enters a long-poll loop claiming leased commands. For each command the agent mounts Manila shares using NFS utilities, invokes the Kopia binary to create or restore snapshots, and reports the result back to the conductor. Credentials (S3 keys, Kopia repository password) are received at claim time, held only in process memory for the duration of the job, and never written to disk or logged.

abaca-manage The operator CLI for database schema migrations (abaca-manage db_sync), Domain registration, and disaster-recovery operations such as rebuild-from-repository. You run this on the control plane host or as a Kubernetes Job during deployment and upgrades.

python-abacaclient (openstack share protection …) The python-openstackclient plugin that adds openstack share protection subcommands. This is the primary command-line interface for tenants and operators. The same package provides the thin HTTP SDK used by scripts and automation.

abaca-dashboard A Horizon plugin that surfaces Share Protection panels (targets, policies, backups, restores, usage) in the OpenStack dashboard, plus an Admin panel for fleet and coverage status. Installed into the existing Horizon pod's Python venv; optional but recommended for tenant self-service workflows.

abaca-dev A developer and operator CLI used during bring-up and testing to enroll targets, boot worker VMs, and run data-path probes against real OpenStack environments. Not part of the production tenant workflow.

Worker VM image A pre-baked RHEL virtual machine image containing Kopia, NFS utilities (nfs-common), and the abaca-worker-agent. Built with virt-customize against a base RHEL cloud image and uploaded to Glance with the property abaca_worker_image=1. The image gate (abaca-dev worker-boot refuses images without this property) prevents accidentally booting an unprepared OS image into the worker role. All runtime dependencies are baked in — worker VMs must not fetch packages from the internet at boot (the airgap invariant).

MySQL / MariaDB (Galera) Abacá's dedicated relational database. It holds only bookkeeping metadata: targets, policies, job records, worker fleet state, snapshot manifests (as pointers), and the leased command queue. Backup data itself never touches this database — it lives in S3. This means losing the database loses convenience and indexing, but not the actual backups; the rebuild-from-repository operation reconstructs the catalog from the Kopia repositories in S3.

RabbitMQ (dedicated broker) Abacá's dedicated oslo.messaging broker, used exclusively for API-to-conductor RPC. Messages carry Barbican hrefs and trust IDs — never the secret values themselves. This broker is separate from the platform's shared RabbitMQ. Worker VMs are not on this bus; they use outbound HTTPS instead.

Kopia The open-source backup engine that runs as a subprocess inside worker VMs. Kopia creates encrypted, deduplicated, content-addressed snapshots of Manila share data in S3 repositories. Every repository is initialized with a FIPS crypto profile (AES-256-GCM-HMAC-SHA256 encryption, HMAC-SHA256-128 block hash, PBKDF2 key derivation). The Kopia binary is version-pinned and baked into the worker image.

Barbican Abacá stores S3 credentials and Kopia repository passwords as Barbican secret references (hrefs). The actual secret values never appear in the Abacá database or in RabbitMQ messages. Secrets are redeemed at job dispatch time by the conductor, which presents them to the worker at command-claim time. The tenant can revoke the Keystone trust that grants the conductor access at any time.

The following traces a policy-driven backup from request to stored data, illustrating how each component participates.

1. Policy creation and scheduling A tenant calls openstack share protection policy create (or the equivalent REST call to POST /v1/policies). The API validates the request, checks that a registered bucket exists for the tenant's project under the chosen template, and writes a Policy row to MySQL. The scheduler loop inside the conductor reads enabled policies every [conductor] scheduler_interval seconds, evaluates each policy's cron expression against the current time, and enqueues a Backup row plus a Job row in MySQL for any policy that is due. It then casts run_job(job_id) to the conductor over RabbitMQ.

2. Job dispatch The conductor receives the RPC cast and begins driving the job state machine. First it redeems the target's S3 credentials and the Kopia repository password from Barbican, using the Keystone trust the tenant granted at enrollment time. It reads the Manila share's metadata (export locations, snapshot capabilities, share network ID). It then calls WorkerFleet.pick_worker_for_job, which counts non-terminal jobs already assigned to each ACTIVE worker and selects one whose running job count is below its capacity_slots limit.

3. State machine execution The conductor walks the job through its states by publishing leased commands to the worker_commands table in MySQL:

• queued → provisioning_network — The conductor publishes a mount command. For DHSS=true shares (dynamic share servers), it hot-plugs a Neutron port into the worker VM's network interface so the worker can reach the share's export IP. For DHSS=false shares with static reachability, this step is a no-op.
• provisioning_network → provisioning_source — The conductor publishes commands to prepare a consistent read-only source view: creating a Manila snapshot and granting the worker's IP an access rule, or reading the share's .snapshot directory, depending on the snapshot access strategy selected for this backend.
• provisioning_source → connecting_repository — The conductor publishes a run_kopia command for kopia repository connect. At claim time, the worker listener asks the conductor over RPC to materialise the S3 credentials and Kopia password; those values are returned in the claim response body and held only in the worker's process memory.
• connecting_repository → transferring — The conductor publishes a run_kopia command for kopia snapshot create, with the source identity pinned to share-<uuid>@manila rather than the worker's hostname. This ensures a snapshot taken by any worker is attributed to the share, not the ephemeral VM.
• transferring → finalizing — The conductor parses the snapshot manifest returned by Kopia, records the snapshot ID and size statistics on the Backup row, and marks the backup available.
• finalizing → releasing → available — Cleanup hooks run in reverse order: repository disconnect, source unmount and access-rule removal, network port detach. Each hook is idempotent.

If any stage fails, the state machine transitions to error, runs the same cleanup chain in reverse, and records the error code and category (operator_action_required or tenant_action_required) on the job row.

4. Data storage The worker streams client-side encrypted, deduplicated objects into the tenant's S3 bucket. The encryption key (the Kopia repository password) never leaves Barbican except transiently in memory. The Kopia repository — not the Abacá database — is the authoritative record of what backup data exists. Abacá's Backup table rows are pointers and metadata; if the database is lost, abaca-manage rebuild-from-repository can reconstruct it by reading the Kopia repositories directly.

5. Restore A restore follows the same state machine in reverse for the data path. The conductor picks a worker, connects it to the target repository, and runs kopia restore either into a new Manila share (new_share mode, the default) or back over the original share (in_place mode, requires force=true). Sub-path and single-file restores are supported by passing a sub_path to the restore operation.

Control plane on OpenShift, data plane on tenant VMs The control plane (API and conductor) runs as OpenShift pods, benefiting from Kubernetes health management, rolling updates, and horizontal scaling. Data movement is deliberately excluded from the control plane: plaintext bytes and encryption keys would then transit pods that may share nodes, logs, and storage with unrelated workloads. Worker VMs run inside the tenant's own network scope, so they can reach the tenant's Manila share exports and S3 endpoints directly without the control plane being on that path. This is the same architectural pattern as Octavia's amphora model.

Workers claim commands over outbound HTTPS, not inbound RPC Early versions used oslo.messaging (AMQP) for the conductor-to-worker transport. That required giving each worker VM a RabbitMQ credential and making the broker reachable from tenant networks — on RHOSO this meant a NodePort, a pinned Neutron port, and a private CA. A compromised worker VM holding a broker credential could publish messages to the bus the entire control plane runs on. The leased-command protocol replaces this: the conductor writes a command row to MySQL; the worker claims it over outbound HTTPS to the worker listener; credentials are materialised by the conductor at claim time and returned in the response body, never stored in the command row. The worker's only outbound address is the listener URL, and its only credential is a bearer token scoped to itself.

Secrets in Barbican, references in the database Abacá's database contains Barbican hrefs (references) but never the secret values themselves. S3 credentials and Kopia repository passwords are redeemed from Barbican at job dispatch time using the Keystone trust the tenant granted. A tenant can revoke the trust at any time, immediately revoking Abacá's ability to access their data. This means a database breach exposes only references, not usable credentials.

Kopia repository as the source of truth MySQL holds bookkeeping metadata — job records, policy configuration, snapshot manifests as pointers. The encrypted, deduplicated backup data lives exclusively in the Kopia repositories in the tenant's S3 bucket. This separation means a database failure or data loss is a loss of convenience and indexing, not a loss of backup data. The rebuild-from-repository disaster-recovery operation reconstructs the database catalog by reading the Kopia repositories directly.

FIPS crypto profile on repository creation Every Kopia repository is initialised with a fixed crypto profile: AES-256-GCM-HMAC-SHA256 encryption, HMAC-SHA256-128 block hash, and PBKDF2 key derivation. This profile is applied at repository create time and is immutable afterward. If the running Kopia binary does not support PBKDF2, the system records a fips_kdf_gap flag on the repository's capabilities so operators can identify repositories that fall back to scrypt (Kopia's compile-time default, not FIPS-approved).

Per-Domain service projects Each Keystone Domain gets its own service project where that Domain's worker VMs, S3 buckets, and Barbican secrets physically reside. This scopes billing, Nova quota, and network isolation to the Domain rather than sharing a single operator project across all tenants. Workers are assigned to serve a specific project (per design chapter 15) so a worker that mounts one project's shares in plaintext cannot be assigned to another project's jobs.

Shared vs. per-project bucket schemes A BackupTargetTemplate can use either a shared scheme (one S3 bucket and one Kopia repository for all projects in the Domain, enabling cross-project deduplication) or a per_project scheme (one bucket and repository per project, stronger isolation). The scheme is immutable after template creation because changing it would silently re-point existing policies at a different repository and orphan their backups. Under the shared scheme, the domain must nominate a key-custody project for the Kopia password, because there is no single owning project whose Barbican can hold the key.

Scheduler inside the conductor, not delegated to Kopia Abacá implements its own cron-based scheduler rather than using Kopia's built-in scheduling. This allows per-share concurrency caps, availability-zone affinity, uniform job observability (every scheduled backup is a Job row with the same state machine as on-demand backups), and independent control of cadence vs. retention.

Long-lived worker VMs vs. truly ephemeral workers Worker VMs are long-lived and serve multiple concurrent jobs (the capacity_slots setting), rather than being booted per-job and destroyed immediately. This trades VM boot latency for a standing fleet: a freshly queued backup can be dispatched in seconds rather than waiting for cloud-init. The cost is that secrets held in memory for one job coexist in the same process as secrets for another job on the same worker. The security boundary is enforced by the conductor's slot accounting and per-project worker assignment (a worker assigned to project A will not be handed project B's jobs), but the isolation is process-level, not VM-level.

No Kopia-level idempotency key for snapshot create The leased-command protocol is at-least-once: a lease that expires is re-published, and a worker that dies mid-transfer will have its command re-claimed by another worker. For mount and umount commands this is safe. For kopia snapshot create it is not: a duplicate invocation writes a second snapshot, spending a retention slot and potentially evicting an older restore point. For this reason, automatic retries of backup jobs are limited to phases before transferring (queued, provisioning_network, provisioning_source, connecting_repository), where no snapshot can yet have been written. Retries after the transfer phase began require operator intervention.

One Kopia repository per S3 bucket Abacá enforces a strict 1:1 mapping between S3 buckets and Kopia repositories. This simplifies credential management, preflight validation, and Object Lock configuration. The trade-off is that the repository is the deduplication domain: two shares in different repositories cannot deduplicate against each other. Under per_project bucket scheme, each project has its own repository and therefore its own deduplication scope. If cross-project deduplication matters more than isolation, use the shared scheme — but be aware that a single outlier policy with long retention locks all data in the shared repository for its full retention period, not just its own snapshots.

S3 Object Lock is required for true immutability Object Lock must be enabled on a bucket at creation time; it cannot be added afterward. Abacá's preflight conformance check treats a bucket without Object Lock as a warning or failure (depending on the check's fatal flag). If your S3 provider or existing bucket configuration does not support Object Lock, backups remain encrypted and deduplicated but are not write-once — a compromised S3 credential could delete backup data within the retention window.

Database loss is recoverable but costly Losing the MySQL database does not lose backup data, but it loses all job history, policy configuration, and the backup catalog index. Reconstructing the catalog with abaca-manage rebuild-from-repository requires reading every Kopia repository, which is proportional to the number of registered targets and the number of snapshots in each. Policy configuration (schedules, retention rules) cannot be recovered from the repositories and must be re-entered manually.

Horizon plugin is optional The abaca-dashboard Horizon plugin is installed separately into the Horizon pod's Python venv. Environments that do not use Horizon, or that prefer CLI-only workflows, do not need to install it. All functionality is also available through openstack share protection CLI commands and the REST API.

Development environment limitations The deploy/docker-compose.dev.yml stack runs abaca-api and abaca-conductor in --noauth mode with SQLite and in-memory fakes. The kopia.executor = ephemeral_container setting (a dev-only override) runs Kopia in a throwaway container on the conductor host instead of dispatching to a worker VM. This path is not supported in production: it briefly holds tenant credentials on the conductor host, it has no worker fleet, and it does not exercise the worker-listener or leased-command protocol. The Kolla-Ansible reference environment (deploy/kolla/) targets development and reference use only.