Architecture

This page describes the internal architecture of Trilio Site Recovery for OpenStack — how its components are structured, how they communicate, and how a disaster recovery operation moves through the system from a tenant API call to storage promotion on a remote site. Understanding the architecture helps you make informed decisions about network topology, role assignments, and storage configuration before deploying the service, and helps you diagnose problems when something goes wrong.

protector-api

A WSGI REST service built on Paste and Routes, running on port 8788 on each site. It is the only entry point for tenant and operator requests — from the openstack dr CLI, the Horizon panel, or direct HTTP calls. The API validates your Keystone token (X-Auth-Token), enforces role-based access control, and dispatches long-running work to the engine over RPC. It does not execute DR workflows itself.

Key facts for API consumers:

• All resource endpoints require the /v1/{tenant_id}/ prefix. A URL omitting /v1/ returns HTTP 404.
• The version discovery endpoint GET / requires no authentication and returns the supported microversion range.
• The /healthz liveness endpoint requires no authentication and returns {"status": "ok", "checks": {"database": "ok"}} when healthy, or HTTP 503 when the database is unavailable.
• Microversioning is opt-in: omitting the OpenStack-API-Version: protector <version> header defaults to microversion 1.0. Only values outside the 1.0–1.1 range return HTTP 400.
• DR actions (failover, failback, test failover, test cleanup) are all triggered via POST /v1/{tenant_id}/protection-groups/{pg_id}/action with an action body — there are no separate sub-paths per action.

protector-engine

The orchestration service that executes multi-step DR workflows. It runs as a separate process on each site, consuming RPC messages from protector-api via RabbitMQ (oslo.messaging). The engine is the only component that communicates cross-site: it calls Nova, Cinder, and Neutron on both sites using Keystone trusts, and calls the storage driver to manage replication on the FlashArray.

The engine also runs a background scheduler that creates periodic recovery points for ActiveDR Protection Groups, and recovers orphaned operations (those left in running or pending state after a crash) on startup.

openstack dr (CLI plugin)

A python-openstackclient plugin installed from the triliositerecovery package (located in the client/ subdirectory of the repository). All tenant and operator commands follow the pattern openstack dr <resource> <action>. Installing the server package does not install the CLI — you must install it separately.

The CLI authenticates against Keystone to obtain a token and then calls protector-api. It never contacts the engine or the storage array directly.

Horizon Site Recovery panel

A dashboard plugin that provides a browser-based interface for managing Protection Groups, viewing replication health, selecting recovery points, and tracking operation progress step by step. It communicates with protector-api using the same REST API as the CLI.

Pure Storage FlashArray driver

A pluggable storage driver inside protector-engine that communicates directly with Pure Storage FlashArray arrays via the py-pure-client SDK. It manages Protection Group replication (async snapshot-based) and ActiveDR pod-based replication, takes snapshots, and promotes replicated volumes during failover.

Activated by setting storage_driver = pure in the [storage] section of protector.conf. The default value is mock, which is suitable only for development and testing — production deployments must set storage_driver = pure.

Two replication modes are supported:

protector-manage

A database management CLI binary used to create or migrate the protector MariaDB/MySQL schema. The canonical command is protector-manage db sync. In Juju deployments, the charm runs this automatically when relations are satisfied — you do not need to invoke it manually.

Consistency Group (Cinder)

A Cinder Consistency Group is automatically created and managed by the service for each Protection Group. It groups all volumes belonging to a Protection Group so they are snapshotted and replicated together, guaranteeing crash-consistent recovery. Consistency Group details are returned as part of the Protection Group API response — there is no separate endpoint.

Keystone trust

Rather than storing tenant credentials, the engine uses OpenStack Keystone trusts to act on a tenant's behalf on the remote site. Trusts are created automatically during the first DR operation for a tenant and reused for all subsequent operations. Both dr_operator and dr_site_admin roles must be pre-created on both sites — the service looks them up and grants them; it does not create them.

The following traces a planned failover from the moment you issue the CLI command to the moment your VMs are running on the secondary site.

1. Authentication

You run:

openstack dr failover <pg-id> --failover-type planned

The CLI authenticates against your local Keystone (port 5000) and obtains an X-Auth-Token. This token scopes to your project and carries the dr_operator role required for failover.

2. API request

The CLI sends:

POST /v1/{tenant_id}/protection-groups/{pg_id}/action
X-Auth-Token: <token>
Content-Type: application/json

{"failover": {"failover_type": "planned"}}

protector-api validates the token with Keystone, checks that your project has a tenant mapping to the secondary site, verifies the Protection Group exists and is in active state, and creates a DR Operation record with status pending.

3. RPC dispatch

The API sends a synchronous RPC call (execute_failover) over oslo.messaging to protector-engine via RabbitMQ. The engine immediately returns the operation object (with its ID) to the API, which responds HTTP 202 to the CLI:

{
  "message": "Failover initiated",
  "status": "pending",
  "protection_group_id": "<pg-id>",
  "operation_id": "<op-id>",
  "operation": { ... }
}

4. Async execution in the engine

The engine spawns a background thread and begins the multi-step failover workflow:

1. Pre-flight validation — Checks that every network, security group, and flavor used by member VMs exists on the secondary site. If any are missing, the operation fails before anything is shut down.
2. Trust creation — Creates (or reuses) Keystone trusts for the tenant on both sites so the engine can act on the tenant's behalf after the original token expires.
3. Transitional status — Sets the Protection Group status to failing_over and syncs this status to the secondary site via POST /v1/{tenant_id}/protection-groups/{pg_id}/metadata/sync on the secondary protector-api.
4. Graceful shutdown — For a planned failover, the engine calls Nova on the primary site (using the Keystone trust) to shut down each VM in boot-order sequence (lowest boot order first; ties are parallelised).
5. Final snapshot — The Pure Storage driver calls the FlashArray API to take a final consistent snapshot of the Protection Group's replication group, ensuring zero data loss.
6. Volume promotion — The Pure Storage driver calls the secondary FlashArray to promote the replicated volumes, making them writable on the secondary site.
7. VM recreation — The engine calls Nova and Cinder on the secondary site to recreate each VM, attaching the promoted volumes. Network and flavor mappings are applied if configured.
8. Replication reversal — The Pure Storage driver reconfigures replication to run from secondary to primary so failback is possible.
9. Status update — The Protection Group is set to failed_over. The engine syncs this status to the peer site.

5. Polling the operation

While the engine works, you can poll the operation:

openstack dr operation show <op-id>

Or directly via the API:

GET /v1/{tenant_id}/operations/{op-id}
X-Auth-Token: <token>

The operation record tracks status (pending → running → completed or failed), progress (0–100), and a step-by-step result log. If you passed --wait to the CLI, the client polls this endpoint automatically and blocks until completion.

6. Cross-site metadata sync

At several points during the workflow, the primary engine calls the secondary site's protector-api on port 8788 to synchronise Protection Group state and tenant mappings. This is why cross-site connectivity on port 8788 must be bidirectional — one-way connectivity leaves both sites with inconsistent views of Protection Group status.

7. Replication health monitoring

At any time, independent of an active operation, you can check replication health:

openstack dr replication health <pg-id>

This calls GET /v1/{tenant_id}/protection-groups/{pg_id}/replication-health, which the engine resolves by querying the FlashArray via the Pure Storage driver. The response includes sync state, lag, RPO compliance, the failover_validation.can_failover boolean, and available recovery points.

Standard OpenStack service pattern

protector-api and protector-engine follow the same split used by Nova (nova-api / nova-conductor / nova-compute) and Cinder (cinder-api / cinder-volume). The API tier is stateless and horizontally scalable; the engine tier holds orchestration state in the local database. This means you can run multiple protector-api workers behind a load balancer without coordination, while each site runs a single protector-engine that owns its local database.

Oslo.messaging RPC over direct HTTP

The API dispatches work to the engine via oslo.messaging (RabbitMQ) rather than a direct HTTP call. This decouples the API response time from the duration of the DR operation: the API returns HTTP 202 immediately with an operation ID, and the engine works asynchronously. The trade-off is that RabbitMQ becomes a required infrastructure dependency and must be healthy for any DR operation to start.

Keystone trusts instead of stored credentials

The engine needs to call OpenStack APIs on the remote site on a tenant's behalf, potentially long after the tenant's original token has expired (a failover can take minutes). Storing the tenant's password would be a security liability. Keystone trusts solve this: the tenant delegates a scoped, role-restricted token to the protector service account at the start of the first DR operation, and the engine uses that delegation for all subsequent cross-site calls. The tenant's password is never stored or transmitted to the remote site.

Protection Group as the unit of replication

Cinder's native replication operates at the storage-backend level and requires administrator intervention. The Protection Group abstraction lifts replication to the tenant level: a named collection of VMs that are replicated and failed over together. This lets tenants manage their own DR without cloud administrator involvement, and lets the service guarantee crash-consistency across multiple volumes by grouping them in a Cinder Consistency Group.

Pluggable storage driver

All array-level operations are isolated behind the StorageDriver abstract base class (protector/engine/storage/storage_driver.py). The engine calls create_storage_group, failover, get_replication_health, and so on without knowing which array vendor is underneath. The currently shipped driver targets Pure Storage FlashArray (storage_driver = pure). The mock driver (the default) provides an in-memory implementation for development and testing that requires no real storage hardware.

Independent per-site databases with explicit sync

Each site runs its own MariaDB/MySQL database. There is no multi-master or bidirectional database replication between sites. State is propagated explicitly through metadata sync operations (POST /v1/{tenant_id}/protection-groups/{pg_id}/metadata/sync) triggered by the engine during DR workflows. This keeps each site fully functional even when the cross-site link is down, at the cost of requiring explicit sync to keep both sides consistent after an operation.

Juju charm as the deployment model

The service is packaged as a machine charm (trilio-site-recovery-for-openstack) that manages systemd services via the trilio-site-recovery Debian package. The charm renders protector.conf from relation data and Juju config values, runs schema migration, and registers with Keystone automatically when all relations are satisfied. This means you must never edit protector.conf directly — the charm overwrites manual edits on every config-changed event. All configuration must be applied via juju config.

Cinder Consistency Groups are deprecated upstream

The service uses Cinder Consistency Groups (consistency_group) to guarantee crash-consistent snapshots across multiple volumes. Upstream Cinder deprecated Consistency Groups in the Ocata release in favour of Generic Volume Groups. The service relies on Consistency Groups because Generic Volume Groups do not yet support the replication semantics required. If your Cinder deployment has removed Consistency Group support, you cannot use Trilio Site Recovery.

RPO is bounded by replication interval for async mode

With replication_type: async, recovery points are periodic snapshots. The minimum achievable RPO equals the configured snapshot_interval_seconds (default: 300 seconds). In an unplanned failover, you accept data loss up to the age of the most recent snapshot. If near-zero data loss is required, use replication_type: active_dr, which requires Pure Storage ActiveDR-capable arrays running Purity//FA 6.0 or later.

ActiveCluster synchronous replication is not a shipping capability

Zero-RPO synchronous replication via Pure Storage ActiveCluster stretched pods is not implemented in the current release. Do not plan DR architectures that depend on it. The two shipping replication modes are async (periodic snapshots) and ActiveDR (continuous async, near-zero RPO).

Cross-site port 8788 must be bidirectional

Metadata sync and tenant mapping propagation require the primary engine to reach the secondary protector-api on port 8788, and the secondary engine to reach the primary protector-api on port 8788. Firewall rules that allow traffic in only one direction cause validation failures on remote site operations and leave Protection Group status inconsistent between sites. Plan your network topology accordingly.

The openstackclient snap is incompatible with the CLI plugin

The triliositerecovery CLI plugin installs as a python-openstackclient entry point. The snap-packaged openstackclient uses an isolated Python environment that cannot load external plugins. You must install python-openstackclient via pip, not via snap.

Each site's database is an independent failure domain

Because each site has its own database with no automatic cross-site replication, a failure that corrupts the primary site's database does not corrupt the secondary. However, it also means that a Protection Group's history (past operations, metadata) on the primary is unavailable after a failover until explicitly synced. Operations left in running or pending state when the engine crashes are recovered on restart, but if the database itself is lost, those records are gone.

Kolla and OpenShift paths are not validated for production

The repository contains deployment scripts for Kolla (deploy/kolla/) and OpenShift (deploy/openshift/) for internal development use. These paths are not validated for customer deployments during the tech preview. The only supported deployment method is the Juju charm. Do not use the Kolla or OpenShift paths for production workloads.

All VMs must boot from Cinder volumes

Instances that boot from ephemeral storage (a Nova-local disk) cannot be added to a Protection Group. The service rejects them at member-add time with InstanceNotCinderBoot (HTTP 422). Additionally, every Cinder volume attached to a protected VM must use a volume type with replication_enabled='<is> True' set as an extra-spec. Volumes using non-replicated types are rejected with VolumeTypeNotReplicated (HTTP 422). You can pre-check eligibility without side effects using POST /v1/{tenant_id}/protection-groups/{pg_id}/members/validate or openstack dr protection group member validate.