What This Service Does
The service's role in the OpenStack ecosystem and the problem it solves
Trilio Site Recovery for OpenStack is a native disaster recovery service that fills the gap between Cinder's admin-only storage replication and the tenant-facing, application-aware DR that production OpenStack environments require. This page explains what problem the service solves, how its core abstractions work, where it sits in the OpenStack ecosystem, and what it can and cannot do — giving you the foundation to evaluate it, deploy it, and reason about its behavior during a real incident.
The gap in native OpenStack DR
OpenStack provides strong intra-site resilience: Nova schedules across hosts, Cinder spreads volumes across backends, Neutron routes around failed links. None of that helps when an entire site goes dark. Cinder does expose volume replication at the storage-backend level, and Pure Storage FlashArray performs that replication reliably — but that capability sits below the OpenStack control plane in ways that limit its usefulness as a DR tool:
- Cinder's replication actions are admin-only. A tenant cannot trigger or monitor replication for their own volumes.
- Volume replication is application-unaware. It has no concept of which VMs depend on which volumes, or what order they need to boot in.
- Nobody automates the actual recovery. Recreating VM definitions, importing replicated volumes, rewiring networking, and assigning floating IPs is a high-stress manual exercise during an outage.
- There is no self-service DR testing. Without a way to verify that a failover would succeed, DR plans remain theoretical until an actual incident.
Trilio Site Recovery addresses all four of these gaps by wrapping Pure Storage replication in a tenant-facing control plane that integrates with OpenStack as a first-class service.
How the service fits into OpenStack
Trilio Site Recovery registers in the Keystone service catalog under the service type disaster-recovery (with an additional alias of protector) and enforces access control through oslo.policy, using the same role-based model as every other OpenStack service. Users interact with it through the openstack dr CLI subcommand family, the versioned REST API on port 8788, the Horizon dashboard panel, or the triliositerecovery Python client library.
The service runs as two cooperating processes:
- protector-api serves the versioned REST API, validates Keystone tokens, enforces RBAC policy, and dispatches DR operation requests to the engine over RabbitMQ RPC.
- protector-engine executes all DR workflows — failover, failback, test failover, metadata synchronization, and storage replication management — as a background RPC service.
All service state — sites, protection groups, consistency groups, members, volumes, DR operations, replication policies, tenant mappings, and resource mappings — is stored in a MariaDB or MySQL database. Schema is managed by Alembic migrations, with the current head at migration 025_add_member_removing.
The service account that protector-api uses in Keystone is named protector. It is registered with two Keystone roles: dr_operator (delegated by every per-tenant trust the service creates, enabling cross-site operations on behalf of tenants) and dr_site_admin (which gates cloud-wide administrative operations like site registration and tenant mapping). Both roles must exist in Keystone on both sites before the service can function.
The protection group abstraction
The central concept is the protection group: a named set of Nova virtual machines whose Cinder volumes are replicated together as a crash-consistent unit. When you add a VM to a protection group, all of its Cinder volumes are enrolled so that every disk in the group snapshots at the same instant, producing a coherent recovery point across the entire application tier rather than a collection of individually-consistent volumes that may not be mutually consistent.
A protection group names exactly two sites — a primary site and a secondary site, both explicitly specified at creation time. The protection group tracks which site is currently primary (current_primary_site_id), which flips on failover and flips back on failback.
Protection groups progress through a defined lifecycle:
CREATING → ACTIVE → FAILING_OVER → FAILED_OVER → FAILING_BACK → ACTIVE
\→ ERROR (on failure)
- ACTIVE — normal state; replication is running from the primary site to the secondary site.
- FAILED_OVER — workloads are running on the secondary site; the primary is read-only.
- FAILING_OVER / FAILING_BACK — transient states during DR operations; no new operations can start while in these states.
- ERROR — an operation failed; the group may need manual intervention or reconciliation.
Each VM in a protection group carries an integer boot order that controls startup sequencing during failover — your database tier starts before your application tier attempts to connect. This maps directly to the concept of VM priority groups in VMware Site Recovery Manager.
Replication modes
The replication mode is selected per protection group at creation time with --replication-type. Two modes are generally available:
Async (periodic snapshot replication)
The default mode. The engine creates a Pure Protection Group on the source array, enrolls the consistency group's volumes in it, and replicates scheduled snapshots to the target array. Every replicated snapshot is a selectable recovery point — you can target any available snapshot during failover or test failover, not just the most recent. RPO is bounded by the configured snapshot_interval_seconds (default 300 seconds) plus transfer time.
--volume-type is required when creating a protection group with --replication-type async. The volume type must have replication_enabled=True and replication_type=async on both sites.
ActiveDR (continuous asynchronous replication)
A continuous-asynchronous mode using Pure pod replica links. Two single-array pods — one on the source array, one on the target — are joined by a directional pod replica link that streams writes continuously. Typical RPO is sub-30 seconds, bounded by WAN bandwidth and write rate. Recovery points are supported: snapshots scheduled on the promoted pod ride the replica stream, with retention enforced automatically. The configured RPO default is 900 seconds.
ActiveDR requires Purity//FA 6.0 or later and an existing replication-type array connection between the two FlashArrays. --volume-type is not required for active_dr — it is the one replication type exempt from that requirement, enforced both client-side by the CLI and server-side by the API.
ActiveDR is not synchronous replication. It does not use a stretched pod. Zero-RPO synchronous replication via Pure ActiveCluster is on the roadmap but is not a currently supported replication type in Trilio Site Recovery.
Mock driver
An in-memory storage backend that simulates replication without requiring physical arrays. It stores state under /var/lib/protector and is intended for initial deployment validation and hardware-free evaluation. It is the default storage driver. Do not use it in production.
DR operations
All DR operations are recorded in the database as operation records with a type, status, and progress percentage (0–100). The service emits no oslo.messaging notifications; observability is entirely through polling: openstack dr replication health, openstack dr replication audit, openstack dr recovery point list, and the operation records themselves.
Failover
Moves workloads from the primary site to the secondary site. Two modes:
- Planned failover — orderly: VMs are shut down gracefully on the primary, a final snapshot is taken for zero data loss, the snapshot is replicated to the secondary, volumes are promoted and imported there, and VMs are recreated. On partial failure, all changes are rolled back.
- Unplanned failover — emergency: no graceful shutdown (the primary may be unreachable), promotes the most recent available recovery point. On partial failure, continues with VMs that succeeded.
After failover, recovered VMs are created with the name <original-name>-failover. The original instance name is preserved in the member record, so the suffix does not compound if you later run failback. Note that anything outside OpenStack keyed on the original name — DNS entries, monitoring, runbooks — will need to be updated.
Failover supports optional network and flavor mapping (--network-mapping source-net-name=target-net-id, --flavor-mapping source-flavor-id=target-flavor-id). The separator is =, not :. Without explicit mappings, networks and flavors are resolved by name on the target site.
Failback
Reverses the failover: syncs data accumulated during the failover window back to the original primary, then recreates VMs there as <original-name>-restored. Failback waits for each DR volume's Nova attachment to release (up to 120 seconds per volume) before unmanaging it, to avoid a race where Cinder rejects the unmanage request because the volume is still attached. This wait is serial over volumes — a site whose detaches lag will show a visibly slower failback, which is the wait working correctly, not a hang.
Test failover
Clones a recovery point, boots isolated VMs on the secondary site (optionally with an --instance-prefix), and leaves production completely untouched. This is the non-disruptive DR test that auditors require. When you are done, openstack dr test failover cleanup removes the test VMs. Test failover operations do not alter the protection group's status.
Reconcile
Repairs a protection group left in an inconsistent state by an interrupted failover or failback. Reconciliation inspects storage, VMs, and operation history to fix the protection group status, relink discovered VMs, restart SHUTOFF workloads on the writable site, and synchronize the corrected state to the peer. It works even when the primary site is unreachable. Each reconcile is recorded in the database as an operation with operation_type=reconcile and appears in openstack dr operation list output.
Site-level failover
In addition to per-protection-group failover, the service supports site-level failover: openstack dr site failover <site> (or openstack dr failover to this site) fails over every protection group associated with a site in a single operation. The result is recorded as a site operation, visible via openstack dr site operation list and openstack dr site operation show.
Cross-site authentication
The service authenticates cross-site operations through Keystone Trusts — scoped, revocable delegations — rather than shared credentials or stored passwords. When a user first interacts with the DR service, a trust is transparently created where the user (trustor) delegates roles to the protector service account (trustee). This trust allows the engine to act on the user's behalf on the remote site without requiring standing access.
The roles delegated by each trust default to member and dr_operator. member is required for Nova and Cinder API calls; dr_operator satisfies Protector's own cross-site sync policy. Both roles must already exist in Keystone — trust creation fails at the first DR operation if either is missing.
Sessions and trust IDs are cached per (site, tenant) to avoid redundant Keystone round-trips within long-running operations like failover. The cache is invalidated on authentication failure so the next call retries fresh.
Tenant ID translation
Project UUIDs differ between independent OpenStack deployments (each Keystone assigns its own UUIDs). The service uses tenant mappings to translate a local project ID to its counterpart on the peer site. Without a tenant mapping, the engine assumes the same project UUID exists on the peer site — a fallback that holds for shared or federated Keystone deployments but essentially never for two independent deployments, which is exactly the architecture this product is designed for. If you are running two independent OpenStack clusters, configure tenant mappings before attempting any DR operation.
What the service does not do
- No estimated completion time for operations. The service tracks a progress percentage (0–100) and exposes it in
openstack dr operation show, but it does not compute or expose an ETA or projected finish time for failover, failback, or test failover. - No oslo.messaging notifications. The service does not publish lifecycle events. Monitoring must be done by polling the health and operation APIs.
- No LUKS-encrypted volume support. Adding a member whose boot or data volume is LUKS-encrypted raises
InvalidInputand the member is rejected. Cinder's manage API cannot associate Barbican keys on the DR site, so an encrypted volume could never be recovered. Use array-side encryption (Pure data-at-rest) instead. - No ActiveCluster (synchronous) replication. Zero-RPO synchronous replication via Pure ActiveCluster is on the roadmap but not yet supported. The two supported replication types are
asyncandactive_dr. - No standalone Alembic invocation. Database migrations are run via
protector-manage db sync, not by invoking Alembic directly. On Juju, use thedb-synccharm action.
Check which sites are registered
openstack dr site list
Create a protection group with async replication
openstack dr protection group create my-app-pg \
--primary-site <primary-site-id> \
--secondary-site <secondary-site-id> \
--replication-type async \
--volume-type replicated-type
Create a protection group with ActiveDR (no volume type required)
openstack dr protection group create my-app-pg-adr \
--primary-site <primary-site-id> \
--secondary-site <secondary-site-id> \
--replication-type active_dr
Add a VM to a protection group
openstack dr protection group member add <pg-id> \
--instance <vm-id> \
--boot-order 1
Check replication health before a failover window
The --probe flag asks the recovery site whether the protection group's networks, security groups, and flavors resolve there in the project it recovers into. It does not test the storage backend or replication lag.
openstack dr replication health <pg-id> --probe
Execute a planned failover with network mapping
The separator between source and destination in --network-mapping is =, not :.
openstack dr failover <pg-id> \
--failover-type planned \
--network-mapping my-app-network=dr-site-network-id \
--wait
Run a non-disruptive test failover
openstack dr test failover <pg-id> \
--auto-network \
--recovery-point <snapshot-suffix>
Clean up after a test failover
openstack dr test failover cleanup <pg-id>
Reconcile an inconsistent protection group
Use this when a failover or failback was interrupted and the group is in an inconsistent state. This command works even when the primary site is unreachable.
openstack dr protection group reconcile <pg-id>
Track an operation's progress
openstack dr operation show <operation-id>
Fail over all protection groups for a site (site-level failover)
openstack dr site failover <site-id> --failover-type unplanned
- Deployment overview — How protector-api and protector-engine are deployed via Juju, Kolla-Ansible, or OpenShift manifests, and how they relate to MySQL and RabbitMQ.
- Configuration reference — All
protector.confoptions by section, including[DEFAULT],[pure_storage],[engine], and[service_credentials], with the requirement to setlocal_sitebefore the service can participate in any DR operation. - RBAC and policy — The
dr_operatoranddr_site_adminroles, the default oslo.policy rules, and how to override them viapolicy.yaml. Note thatdr_site_adminis cloud-wide regardless of which project it is granted on and belongs to a cloud operator on the admin project. - Protection group lifecycle — Detailed state machine for protection group and member status transitions, including the
REMOVINGmember state for ActiveDR volume relinks. - Failover and failback workflows — Step-by-step engine behavior during planned and unplanned failover, failback with volume detach waiting, and the naming convention for recovered instances (
-failoverand-restoredsuffixes). - Reconciliation — When and how to use
openstack dr protection group reconcileto repair a protection group left inconsistent by an interrupted operation. - Pure Storage driver configuration — How to configure
[pure_storage]with primary and secondary array credentials, theprimary_site_name/secondary_site_namepairing invariant, and theadditional_arraysoption for multi-array deployments. - Tenant mappings — Why tenant mappings are needed for independent OpenStack clusters, what happens when they are absent (same-UUID fallback), and how to create and validate them.
- Observability — How to monitor the service through
openstack dr replication health,openstack dr replication audit,openstack dr recovery point list, and operation records, given that the service emits no oslo.messaging notifications. - Upgrade procedures — The
protector-manage db preflightrequirement before migration022_add_reconcile_operation_typefor Kolla and bare-metal operators, and theupgrade_preflight.pyscript obtained from the target release tag.