Trilio Site Recovery for OpenStackDisaster Recovery Solution for OpenStack Virtual Machines
Docs/Trilio Site Recovery for OpenStack/Problem Being Solved
Concept

Problem Being Solved

What this service provides

Overview

OpenStack has no built-in mechanism for tenants to protect and recover their own workloads across sites — Cinder's native volume replication operates at the storage-backend level and requires administrator intervention at every step. Trilio Site Recovery closes this gap by giving tenants a self-service disaster recovery control plane built on top of Pure Storage FlashArray replication, coordinated through standard OpenStack identity and APIs. This page explains the problem Trilio Site Recovery solves, the architectural gap it fills, and the core abstractions it introduces so you can evaluate whether the service fits your recovery requirements.

Content

The gap in native OpenStack disaster recovery

OpenStack is designed for high availability within a single region: Nova spreads instances across compute hosts, Cinder distributes volumes across backends, and Neutron routes around failed links. None of this helps when an entire site becomes unavailable. For cross-site recovery, OpenStack exposes Cinder's backend-level replication capability — but that capability has three significant limitations for tenants:

  • Admin-only control. Triggering or monitoring Cinder replication requires cloud administrator privileges. A tenant cannot initiate failover or check replication status for their own volumes without opening a support ticket.
  • No application awareness. Cinder replication operates on individual volumes in isolation. It has no concept of which VMs depend on which volumes, what order services need to start in, or which volumes must be crash-consistent with each other.
  • No automated recovery. Even with replicated volumes on a secondary site, recreating VM definitions, reattaching volumes, and reconnecting networking is a manual, error-prone process that must be performed under the pressure of an active outage.
  • No non-disruptive DR testing. There is no native way to verify that a failover would succeed without actually triggering one and disrupting production workloads.

Trilio Site Recovery addresses all four of these problems through a tenant-facing API and a set of abstractions that sit above the storage layer.

What Trilio Site Recovery provides

Trilio Site Recovery runs as two OpenStack services — protector-api and protector-engine — deployed on each site. Tenants interact through the openstack dr CLI, a Horizon dashboard panel, or the REST API directly. Administrators do not need to be involved in day-to-day DR operations.

The Protection Group abstraction

The central concept is the Protection Group: a named collection of Nova VMs that are replicated and failed over together as a single unit. Every Protection Group has exactly one primary site, one secondary site, and one Cinder Consistency Group that ensures all attached volumes are snapshotted at the same instant, producing a crash-consistent recovery point across the entire application tier.

For a VM to be eligible for protection, two conditions must be met:

  1. The instance must boot from a Cinder volume, not ephemeral storage. Instances using ephemeral boot disks are rejected at member-add time with InstanceNotCinderBoot.
  2. All Cinder volumes attached to the instance must use a volume type with replication_enabled='<is> True' set as an extra-spec. Volumes using non-replicated types are rejected with VolumeTypeNotReplicated.

These constraints exist because the service drives replication at the storage layer: it can only protect volumes it can replicate, and it can only recreate VMs whose boot disks are recoverable on the secondary site.

Self-service DR operations

Once VMs are added to a Protection Group, tenants can perform the following operations without administrator involvement:

  • Planned failover — Gracefully shuts down VMs on the primary site, takes a final snapshot for zero data loss, promotes replicated volumes on the secondary site, and recreates the VMs there. Triggered via openstack dr failover <pg-id> --failover-type planned.
  • Unplanned failover — Skips graceful shutdown (the primary site may be unreachable) and promotes the most recent available recovery point. Accepts data loss up to the configured RPO. Triggered via openstack dr failover <pg-id> --failover-type unplanned.
  • Failback — Returns workloads to the original primary site after a failover: reverses replication direction, synchronizes data accumulated during the failover window, and recreates VMs on the primary. Triggered via openstack dr failback <pg-id>.
  • Test failover — Clones a recovery point on the secondary site, boots isolated VMs with a test- prefix on a separate network, and leaves production completely untouched. This is the non-disruptive DR validation that auditors require. Triggered via openstack dr test failover <pg-id>.
  • Point-in-time recovery — Every replicated snapshot is a selectable recovery point. During failover or test failover you can target a specific point in time rather than the latest copy — critical for recovering from ransomware or accidental data corruption. Recovery points are listed via openstack dr recovery point list <pg-id>.

Boot order and application sequencing

Each VM in a Protection Group has an integer boot order value that controls the sequence in which VMs are started or stopped during failover and failback. VMs with the same boot order are processed in parallel; lower-numbered tiers complete before higher-numbered tiers begin. This gives you the same sequencing control that VMware SRM's VM priority groups provide — your database tier is ACTIVE before your application tier attempts to connect.

Boot order is set when adding a member:

openstack dr protection group member add <pg-id> --instance <vm-id> --boot-order <n>

and can be updated later with:

openstack dr protection group member set <pg-id> --member <member-id> --boot-order <n>

Cross-site authentication without shared credentials

Cross-site operations authenticate through Keystone Trusts: a scoped, revocable delegation mechanism built into OpenStack identity. When a tenant performs their first DR operation, the engine automatically creates a trust where the tenant (trustor) delegates the member and dr_operator roles to the protector service account (trustee) on the remote site. The engine then acts on the tenant's behalf on both sites using these trusts — no passwords are shared across sites, and trusts are revocable by the tenant at any time.

This is a meaningful distinction from native Cinder replication, where a cloud administrator must hold credentials on both sites and be involved in every recovery operation.

Replication modes

Trilio Site Recovery supports two replication modes, both backed by Pure Storage FlashArray:

Async (periodic snapshot replication) is the default. The engine creates a Protection Group on the primary FlashArray, configures a replication target on the secondary array, and takes scheduled snapshots at a configured interval. Each replicated snapshot becomes a selectable recovery point. The RPO is bounded by the snapshot interval. This mode works on any supported Purity//FA version.

ActiveDR (continuous async replication) uses Pure Storage FlashArray pod replica links to continuously stream writes from a source pod to a target pod, providing near-zero RPO without full synchronous replication. This mode requires Purity//FA 6.0 or later and is configured by setting replication_type: active_dr on the Protection Group. Use ActiveDR when your recovery objectives require minimal data loss.

Important: Synchronous replication via Pure Storage ActiveCluster is not a current capability of Trilio Site Recovery. Do not configure or plan for zero-RPO ActiveCluster failover — the storage feature exists on the array, but the DR orchestration layer for it is not yet implemented. Async and ActiveDR are the two supported modes.

Operation tracking

Every DR operation — failover, failback, test failover, test cleanup, and sync — is tracked as a DR Operation with a unique ID, a progress percentage from 0 to 100, and a step-by-step record of what completed or failed. You can query any operation at any time:

openstack dr operation show <operation-id>

or via the API at GET /v1/{tenant_id}/operations/{id}. Operations persist after completion, giving you an audit trail of every DR action taken against a Protection Group.

Replication health monitoring

The service provides a live replication health report for each Protection Group, including sync state, replication lag in seconds and bytes, RPO compliance, failover readiness, and the number of available recovery points on the secondary site. The failover_validation object within the health report includes a can_failover boolean that is the programmatic check for whether a Protection Group is ready to fail over:

openstack dr replication health <pg-id>

or via GET /v1/{tenant_id}/protection-groups/{pg-id}/replication-health.

Protection Group lifecycle

A Protection Group moves through a defined set of states:

CREATING → ACTIVE → FAILING_OVER → FAILED_OVER → FAILING_BACK → ACTIVE
                ↘ ERROR (on failure)
  • ACTIVE — Normal operating state. Replication is running from primary to secondary.
  • FAILED_OVER — Workloads are running on the secondary site. The primary is read-only or unavailable.
  • FAILING_OVER / FAILING_BACK — Transient states during DR operations. Most management operations are blocked while a group is in transition.
  • ERROR — An operation failed and left the group in an inconsistent state. Use openstack dr protection group reconcile <pg-id> to recover.

What the service does not do

Understanding the boundaries of the service helps you design your recovery architecture correctly:

  • Networks and subnets are never created on the secondary site. They are matched by name. If a network by the required name does not exist on the secondary site, a failover that depends on it will block. Run openstack dr protection group sync <pg-id> after adding members to see which resources have been mapped and which are missing.
  • Security groups are adopted by name. Whether missing security groups are created automatically depends on the [engine] auto_create_security_groups configuration option. By default this is false and missing security groups are reported rather than created.
  • The service does not manage Cinder volume type configuration. You must create volume types with replication_enabled='<is> True' in the Cinder extra-specs on both sites before adding VMs to a Protection Group. Volume type UUIDs differ between OpenStack deployments; the service resolves them by name on the secondary site.
  • Tenant project IDs differ between OpenStack deployments. Each Keystone assigns independent UUIDs. You must create a Tenant Mapping that translates your local project ID to the corresponding project ID on the secondary site before cross-site operations can succeed. Manage these with openstack dr tenant mapping create and related commands.
Examples

Check replication health before a planned failover

Before executing a failover, confirm the Protection Group is in a healthy state and ready to fail over:

openstack dr replication health <pg-id>

Expected output (abbreviated):

+----------------------+----------------------------------------+
| Field                | Value                                  |
+----------------------+----------------------------------------+
| protection_group_id  | a1b2c3d4-...                           |
| protection_group_name| web-tier-pg                            |
| can_failover         | True                                   |
| sync_state           | in_sync                                |
| lag_seconds          | 47                                     |
| rpo_compliant        | True                                   |
| recovery_points      | 12                                     |
+----------------------+----------------------------------------+

The can_failover: True field is the programmatic signal that the Protection Group is ready. If it is False, the blockers field in the response explains what must be resolved first.

Validate a VM before adding it to a Protection Group

Use the validate command to pre-check eligibility without side effects:

openstack dr protection group member validate <pg-id> --instance <vm-id>

Expected output when the VM is eligible:

+------------------------+---------------------------+
| Field                  | Value                     |
+------------------------+---------------------------+
| valid                  | Yes                       |
| instance_id            | e5f6a7b8-...              |
| instance_name          | db-primary                |
| protection_group_id    | a1b2c3d4-...              |
| protection_group_name  | web-tier-pg               |
| errors                 | None                      |
| warnings               | None                      |
+------------------------+---------------------------+

Expected output when the VM uses ephemeral storage:

+------------------------+----------------------------------------------+
| Field                  | Value                                        |
+------------------------+----------------------------------------------+
| valid                  | No                                           |
| errors                 | [InstanceNotCinderBoot] Instance e5f6a7b8-...|
|                        | is not a Cinder boot instance. All protected |
|                        | VMs must boot from a Cinder volume.          |
+------------------------+----------------------------------------------+

Run a non-disruptive test failover

A test failover clones recovery point data onto the secondary site and boots isolated VMs on a separate network, leaving production untouched:

openstack dr test failover <pg-id> --auto-network

Either --auto-network or --network-mapping is required. The --auto-network flag instructs the service to select an appropriate isolated network on the secondary site automatically.

To target a specific recovery point (for example, a snapshot taken before a suspected ransomware event):

openstack dr test failover <pg-id> --auto-network --recovery-point <recovery-point-name>

The operation returns immediately with an operation ID. Poll progress with:

openstack dr operation show <operation-id>

When the test is complete, clean up the test VMs and cloned volumes:

openstack dr test failover cleanup <pg-id>

To clean up a specific test operation rather than the most recent one:

openstack dr test failover cleanup <pg-id> --operation <operation-id>

Execute a planned failover

openstack dr failover <pg-id> --failover-type planned

The API returns HTTP 202 Accepted immediately. The initial operation status is pending. Use --wait to block until the operation completes:

openstack dr failover <pg-id> --failover-type planned --wait

For an unplanned failover when the primary site is unreachable:

openstack dr failover <pg-id> --failover-type unplanned

Monitor an operation via the API

All DR operations are queryable at any time. First, obtain a Keystone token:

export TOKEN=$(openstack token issue -f value -c id)
export TENANT_ID=$(openstack token issue -f value -c project_id)

List recent operations:

curl -s -H "X-Auth-Token: $TOKEN" \
  http://<protector-api-host>:8788/v1/$TENANT_ID/operations

Fetch a specific operation:

curl -s -H "X-Auth-Token: $TOKEN" \
  http://<protector-api-host>:8788/v1/$TENANT_ID/operations/<operation-id>

Expected response shape:

{
  "operation": {
    "id": "op-uuid",
    "type": "failover",
    "status": "completed",
    "progress": 100,
    "protection_group_id": "pg-uuid",
    "steps": [
      {"name": "quiesce_vms", "status": "completed"},
      {"name": "take_final_snapshot", "status": "completed"},
      {"name": "promote_volumes", "status": "completed"},
      {"name": "recreate_vms", "status": "completed"}
    ]
  }
}
Related concepts
  • Sites — Before creating a Protection Group, both the primary and secondary OpenStack deployments must be registered as Sites. Each site has its own Keystone, Nova, Cinder, and Neutron. See the site registration workflow for details on openstack dr site create.
  • Tenant Mappings — Because each OpenStack site assigns independent project UUIDs, you must create a Tenant Mapping that links your local project ID to the corresponding project on the secondary site. Without a Tenant Mapping, cross-site trust creation fails. See the tenant mapping commands (openstack dr tenant mapping create) for setup instructions.
  • Replication Policy — The Pure Storage connection details and RPO settings attached to a Protection Group (FlashArray URLs, API tokens, Protection Group name, replication interval, and target RPO). These are configured via POST /v1/{tenant_id}/protection-groups/{pg_id}/policy after the Protection Group is created.
  • Keystone Trusts — The identity delegation mechanism the service uses to act on a tenant's behalf on the remote site. Trusts are created automatically on first use; the dr_operator and member Keystone roles must exist on both sites before any cross-site operation can succeed.
  • Cinder Consistency Groups — The Cinder construct, automatically created and managed by the service, that groups all volumes belonging to a Protection Group so they are snapshotted together. You do not manage Consistency Groups directly; they are embedded in the Protection Group API response.
  • Recovery Points — Point-in-time snapshots of a Protection Group's volumes available on the secondary site. Use openstack dr recovery point list <pg-id> to see available recovery points and select one during failover or test failover.
  • Replication Health — The live health report for a Protection Group's replication link, including RPO compliance and failover readiness. See openstack dr replication health <pg-id> and GET /v1/{tenant_id}/protection-groups/{pg_id}/replication-health.
  • DR Operations — Every long-running action is tracked as a DR Operation with step-level progress. See openstack dr operation show <id> and GET /v1/{tenant_id}/operations/{id}.