Key Concepts

This page defines the core domain concepts and API design patterns that underpin Trilio Site Recovery for OpenStack. Understanding these terms will help you navigate the REST API, interpret CLI output, and reason about how Protection Groups, Consistency Groups, replication, and DR operations fit together. Every concept listed here has a direct representation in the API — either as a resource, a field on a resource, or a tracked operation.

Sites

A Site is a registered OpenStack deployment — either the primary location where workloads normally run, or the secondary (DR) location they fail over to. Each site has its own independent Keystone, Nova, Cinder, and Neutron. Sites are registered in the service with a dedicated service account so the engine can operate across both without storing tenant credentials.

Sites are identified by UUID and registered via openstack dr site create. Each site record stores the Keystone auth_url, optional region_name, service account credentials, and an optional explicit protector_api_url for when the protector-api endpoint differs from the Keystone host. Before any cross-site operation can succeed, port 8788 must be reachable bidirectionally between sites — one-way connectivity leaves tenant mappings inconsistent and causes validation failures.

Service credentials on each site (service_username, service_password, service_project_name, service_user_domain_name, service_project_domain_name) are set via openstack dr site set and validated against the remote Keystone at save time.

Protection Groups

A Protection Group (PG) is the core unit of disaster recovery — a named collection of Nova VMs that are replicated and failed over together as a single group. Every Protection Group has exactly one primary site, one secondary site, and one embedded Consistency Group.

Protection Groups move through a well-defined lifecycle:

CREATING → ACTIVE → FAILING_OVER → FAILED_OVER → FAILING_BACK → ACTIVE
                 ↘ ERROR

• ACTIVE — Normal state; replication runs from primary to secondary.
• FAILED_OVER — Workloads are running on the secondary site; primary volumes are read-only.
• FAILING_OVER / FAILING_BACK — Transient states during DR operations.
• ERROR — A DR operation failed; use openstack dr protection group reconcile to recover.

Consistency Group data is embedded in the Protection Group API response. There is no separate /consistency-group sub-resource.

Membership requirements

Every Nova instance added to a Protection Group must satisfy two conditions:

1. Cinder-backed boot volume — instances booting from ephemeral storage are rejected with InstanceNotCinderBoot (HTTP 422).
2. Replicated volume type — all attached Cinder volumes must use a volume type with replication_enabled='<is> True' set as an extra-spec. Volumes without this extra-spec are rejected with VolumeTypeNotReplicated (HTTP 422).

Use openstack dr protection group member validate (or POST /v1/{tenant_id}/protection-groups/{pg_id}/members/validate) to pre-check a VM's eligibility without side effects. The endpoint returns HTTP 200 with a structured errors array.

Consistency Groups

A Consistency Group (CG) is a Cinder construct, automatically created and managed by the service, that groups all volumes belonging to a Protection Group so they are snapshotted and replicated together. This guarantees crash-consistent recovery across multiple volumes — all volumes in the group share the same recovery point.

Consistency Group details (including primary_cg_id, secondary_cg_id, volume_type_id, backend_name, and status) are returned as part of the Protection Group API response at GET /v1/{tenant_id}/protection-groups/{id}. The service uses upstream Cinder Consistency Groups (consistency_group API). Note that upstream Cinder deprecated CGs in favour of Generic Volume Groups starting in Ocata; the service manages this transparently.

Replication Policy

A Replication Policy stores the Pure Storage connection details and RPO settings for a Protection Group. It includes:

• Primary and secondary FlashArray management IPs and API tokens
• The Pure Storage Protection Group name (pure_pg_name) used on the array
• Snapshot interval (seconds)
• Target RPO (seconds)
• Retention settings (retention_all_for_hours, retention_per_day, retention_days)

Policies are managed at POST /v1/{tenant_id}/protection-groups/{pg_id}/policy and GET /v1/{tenant_id}/protection-groups/{pg_id}/policy. A Protection Group cannot execute failover or failback until a policy is attached.

Replication Type

The Replication Type controls how storage data is mirrored from the primary to the secondary FlashArray:

You set the replication type at Protection Group creation time via --replication-type <async|active_dr>. ActiveDR is a fully implemented and shipping capability configured with replication_type: active_dr. Synchronous zero-RPO replication (ActiveCluster) is not a shipping capability.

For async, the storage driver creates a Pure Storage Protection Group on the primary array, sets the replication target to the secondary array, and schedules periodic snapshots. For active_dr, the driver creates a source pod on the primary array and a target pod on the secondary, establishes a directional replica link, and uses continuous replication instead of periodic snapshots.

Recovery Points

A Recovery Point is a point-in-time snapshot of a Protection Group's volumes available on the secondary site. During failover or test failover, you can select a specific recovery point to restore data to a particular moment — useful for recovering from ransomware or accidental data corruption.

Recovery points are listed via openstack dr recovery point list <protection-group> or GET /v1/{tenant_id}/protection-groups/{id}/recovery-points. On-demand recovery points can be created with openstack dr recovery point create <protection-group> or POST /v1/{tenant_id}/protection-groups/{id}/recovery-points.

Retention of recovery points is governed by three config options in [pure_storage]: retention_all_for_hours (keep all snapshots for this initial window), retention_per_day (snapshots to retain per day after the initial window), and retention_days (total days of retention).

RPO and RTO

RPO (Recovery Point Objective) is the maximum acceptable age of data on the secondary site, expressed in minutes. For async replication, the RPO equals the configured snapshot_interval_seconds. An unplanned failover may lose up to one replication interval of data. For active_dr, the RPO is near zero.

RTO (Recovery Time Objective) is the maximum acceptable time from a disaster declaration to workloads running again on the secondary site. RTO is driven by the time required to promote storage volumes on the secondary array, recreate VMs via Nova, and wait for them to reach ACTIVE state.

The replication health endpoint (GET /v1/{tenant_id}/protection-groups/{pg_id}/replication-health) reports current lag in seconds, whether the replication link is RPO-compliant, and how many recovery points are available — giving you the inputs to evaluate both RPO and RTO readiness before a real event.

Failover

Failover moves a Protection Group's workloads from the primary site to the secondary site. It is triggered via openstack dr failover <pg-id> or POST /v1/{tenant_id}/protection-groups/{pg_id}/action with the appropriate action body.

Two modes are supported (specified with --failover-type):

• Planned — Gracefully shuts down VMs on the primary, takes a final snapshot for zero data loss, then promotes volumes and recreates VMs on the secondary. Use this for scheduled maintenance windows.
• Unplanned — Skips VM shutdown and accepts data loss up to the RPO. Use this when the primary site is unreachable.

The Protection Group transitions through FAILING_OVER and lands in FAILED_OVER on success. The initial action response returns HTTP 202 with a pending operation that you can poll via openstack dr operation show <id> or GET /v1/{tenant_id}/operations/{id}.

Failback

Failback returns a Protection Group's workloads to the original primary site after a failover. It is triggered via openstack dr failback <pg-id> or the same POST /v1/{tenant_id}/protection-groups/{pg_id}/action endpoint.

Failback includes reversing the replication direction, syncing data accumulated on the secondary during the failover period, and recreating VMs on the primary site. The Protection Group transitions through FAILING_BACK and returns to ACTIVE on success.

Test Failover

A Test Failover is a non-disruptive DR validation that clones replicated volumes from a specific recovery point on the secondary site and boots isolated test VMs — prefixed with test- — on a separate network. Production workloads are completely untouched.

Triggered via openstack dr test failover <pg-id> (requires either --auto-network or --network-mapping) or POST /v1/{tenant_id}/protection-groups/{pg_id}/action.

After validation, clean up test resources with openstack dr test failover cleanup <protection-group>. The required positional argument is the Protection Group ID or name, not an operation ID. Use the optional --operation <id> flag to target a specific test failover operation (defaults to the most recent for that PG), and --force to force cleanup even if some resources fail to delete.

DR Operations

A DR Operation is a tracked, long-running task — failover, failback, test failover, test cleanup, or sync — with a unique UUID, a status (pending, running, completed, failed, rolling_back), a progress percentage (0–100), and a detailed record of which steps completed or failed.

Operations are queryable at any time:

• CLI: openstack dr operation show <id> or openstack dr operation list
• API: GET /v1/{tenant_id}/operations/{id} or GET /v1/{tenant_id}/operations

In-progress operations can be cancelled with openstack dr operation cancel <operation>.

Important constraints:

• There is no DELETE route for operations — only GET routes are registered.
• The initial response from a failover, failback, or test failover action returns HTTP 202 with status: pending and an operation_id field. The status transitions to running once the engine picks up the task.
• There is no /dr-operations or /tasks endpoint — only /v1/{tenant_id}/operations.

Replication Health

Replication Health is a live report of the replication link's status for a Protection Group. It includes:

• Sync state and whether the link is connected
• Lag in seconds and bytes
• Whether the current lag is within the configured RPO target
• Failover readiness (can_failover boolean, plus warnings and blockers)
• Number of available recovery points on the secondary site

Accessed via openstack dr replication health <pg-id> or GET /v1/{tenant_id}/protection-groups/{pg_id}/replication-health. Response fields are nested under replication_health.health_report.*, with sibling keys protection_group_id, protection_group_name, failover_validation, and storage_config. Use replication_health.failover_validation.can_failover as the programmatic readiness check before initiating a failover.

There is no /replication-status, /health, or /replication-metrics sub-resource — the replication health endpoint is the only path.

Keystone Trusts

A Keystone Trust is an OpenStack identity mechanism the service uses to act on a tenant's behalf on the remote site without storing the tenant's password. Trusts are created automatically on the tenant's first DR operation and reused for all subsequent operations.

The engine creates trusts by delegating the roles listed in [service_credentials] default_trust_roles (default: member,dr_operator). Both dr_operator and dr_site_admin roles must be pre-created on both sites before any cross-site operation — the service looks them up and grants them rather than creating them. If a role is missing, trust creation fails with a message naming the absent role.

Trusts are scoped to the tenant's project and never expose the tenant's credentials across sites.

Service Accounts

A Service Account is a dedicated OpenStack user (named protector by convention, created automatically by the Juju charm via the identity-credentials relation) that authenticates cross-site API calls. Service account credentials are stored per-site in the sites table and are never exposed to tenants. The Keystone service type registered for the service is disaster-recovery.

DR users require the member and dr_operator roles on their project. The protector service account requires member and dr_operator on each DR-enabled project on each site. Site-level administrators require member and dr_site_admin on a separate account.

Tenant Mappings

A Tenant Mapping is a stored translation between a project UUID on the primary site and the corresponding project UUID on the secondary site. This is required because each site has an independent Keystone that assigns different UUIDs to what is logically the same tenant.

Full CRUD is available via openstack dr tenant mapping create/list/show/delete/validate or the /v1/{tenant_id}/tenant-mappings endpoints. Tenant mappings created locally are propagated to the remote site automatically when port 8788 is reachable bidirectionally; without it, mappings exist only locally and cross-site operations fail validation.

Boot Order

Boot Order is an integer assigned to each VM in a Protection Group that controls the sequence in which VMs are started or stopped during failover and failback. VMs with the same boot order value are processed in parallel; lower-numbered tiers complete before higher-numbered tiers begin.

Boot order is set with openstack dr protection group member set <protection-group> --member <member-id> --boot-order <n>.

Metadata Sync Status

Metadata Sync Status describes the synchronisation state of a Protection Group's metadata between primary and secondary sites. Each site maintains its own copy of the Protection Group metadata; changes are propagated via explicit sync operations.

Synchronisation is triggered with openstack dr metadata sync <protection-group> or openstack dr protection group sync. The current sync state is accessible at GET /v1/{tenant_id}/protection-groups/{pg_id}/metadata/status. There is no /sync-status sub-resource.

Storage Driver

The Storage Driver is a pluggable software component inside the engine that handles all array-level replication operations — creating storage groups, adding volumes, taking snapshots, and promoting replicated data — without exposing vendor-specific details to the rest of the service.

The driver is selected via storage_driver in the [storage] section of protector.conf:

Production deployments must set storage_driver = pure. The mock default is suitable only for development and testing. The short name registered in the driver registry is pure — not pure_flasharray.

The pure driver is configured in [pure_storage] with primary_management_ip, primary_api_token, secondary_management_ip, and secondary_api_token. Multi-array configurations use additional [pure_array:<name>] subsections listed in additional_arrays.

Site-Level Operations

A Site Operation tracks a bulk failover triggered at the site level rather than the Protection Group level. The openstack dr site failover <site> command (requiring the dr_site_admin role) initiates failover of all Protection Groups at a site simultaneously. The complementary openstack dr failover to this site command fails over all remote PGs to the local site.

Site operations are tracked separately from PG-level operations and are queryable via openstack dr site operation list and openstack dr site operation show <operation>, or via GET /v1/{tenant_id}/site-operations and GET /v1/{tenant_id}/site-operations/{id}. Final status is one of completed, partial_success, or failed.

API Design Patterns

Versioning

All versioned resource endpoints require the /v1/{tenant_id}/ prefix. URLs omitting /v1/ return 404. The version discovery endpoint GET / (port 8788) requires no authentication and returns:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://<host>:8788/v1/"}]
    }
  ]
}

Include the optional microversion header on requests to access version 1.1 features:

OpenStack-API-Version: protector 1.1

When the header is omitted, the API defaults to microversion 1.0 and processes the request normally. A malformed value or a version outside the 1.0–1.1 range returns HTTP 400 Bad Request.

Authentication

All authenticated API calls require an X-Auth-Token header obtained from Keystone on port 5000. The protector-api itself has no authentication endpoint.

Asynchronous actions

Failover, failback, and test failover are triggered via POST /v1/{tenant_id}/protection-groups/{pg_id}/action with an action body. All three return HTTP 202 Accepted with a response body containing operation_id, status: pending, protection_group_id, message, and an operation object. Poll the operation via GET /v1/{tenant_id}/operations/{operation_id} or pass --wait to the CLI to block until completion.

Error responses

Error responses include code, message, and type (the exception class name). Microversion rejection errors from the version middleware include only code and message. Key HTTP status codes to handle:

Liveness

The /healthz endpoint (no authentication required) returns HTTP 200 with {"status": "ok", "checks": {"database": "ok"}} when healthy, or HTTP 503 when the database is unavailable.

Check version discovery

Verify the API is reachable and confirm the maximum supported microversion:

curl http://protector-api.example.com:8788/

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://protector-api.example.com:8788/v1/"}]
    }
  ]
}

List registered sites

openstack dr site list

+--------------------------------------+----------+---------+-------------------------------+
| ID                                   | Name     | Type    | Auth URL                      |
+--------------------------------------+----------+---------+-------------------------------+
| a1b2c3d4-...                         | site-lon | primary | https://keystone-lon:5000/v3  |
| e5f6g7h8-...                         | site-fra | secondary | https://keystone-fra:5000/v3 |
+--------------------------------------+----------+---------+-------------------------------+

Pre-validate a VM before adding it to a Protection Group

Use the validate endpoint to check eligibility without side effects:

openstack dr protection group member validate my-pg --instance a1b2c3d4-5678-...

Or via the API:

curl -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"instance_id": "a1b2c3d4-5678-..."}' \
  http://protector-api.example.com:8788/v1/$OS_PROJECT_ID/protection-groups/pg-uuid/members/validate

{
  "valid": false,
  "instance_id": "a1b2c3d4-5678-...",
  "instance_name": "web-server-1",
  "protection_group_id": "pg-uuid",
  "protection_group_name": "my-pg",
  "errors": [
    {
      "code": "VOLUME_TYPE_NOT_REPLICATED",
      "message": "Volume type 'standard' does not support replication. Set replication_enabled='<is> True' on the volume type."
    }
  ],
  "warnings": []
}

Trigger a planned failover and poll for completion

openstack dr failover my-pg --failover-type planned --wait

Or via the API:

# Trigger
curl -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"failover": {"failover_type": "planned"}}' \
  http://protector-api.example.com:8788/v1/$OS_PROJECT_ID/protection-groups/pg-uuid/action

{
  "message": "Failover initiated",
  "status": "pending",
  "protection_group_id": "pg-uuid",
  "operation_id": "op-uuid",
  "operation": {"id": "op-uuid", "status": "pending", "progress": 0}
}

# Poll
curl -H "X-Auth-Token: $OS_TOKEN" \
  http://protector-api.example.com:8788/v1/$OS_PROJECT_ID/operations/op-uuid

{
  "operation": {
    "id": "op-uuid",
    "status": "completed",
    "progress": 100,
    "operation_type": "failover"
  }
}

Check replication health before failover

openstack dr replication health my-pg

Or via the API:

curl -H "X-Auth-Token: $OS_TOKEN" \
  http://protector-api.example.com:8788/v1/$OS_PROJECT_ID/protection-groups/pg-uuid/replication-health

{
  "replication_health": {
    "protection_group_id": "pg-uuid",
    "protection_group_name": "my-pg",
    "health_report": {
      "replication_type": "async",
      "link": {"status": "healthy", "connected": true},
      "lag": {"lag_seconds": 42, "rpo_compliant": true, "rpo_target_seconds": 900},
      "recovery_points": {"total_count": 12}
    },
    "failover_validation": {
      "can_failover": true,
      "warnings": [],
      "blockers": [],
      "message": "Protection group is ready for failover"
    }
  }
}

Run a test failover and clean up

# Run test failover using automatic network selection
openstack dr test failover my-pg --auto-network

# ... validate your DR environment ...

# Clean up test resources for the most recent test failover
openstack dr test failover cleanup my-pg

# Or clean up a specific test failover operation
openstack dr test failover cleanup my-pg --operation op-uuid

Create a tenant mapping between sites

openstack dr tenant mapping create \
  --local-tenant $OS_PROJECT_ID \
  --remote-site site-fra-uuid \
  --remote-tenant remote-project-uuid

Then validate the mapping:

openstack dr tenant mapping validate <mapping-id>

• Installation guide — How to deploy protector-api and protector-engine using the Juju charm on Charmed OpenStack, including charm relation requirements and configuration options.
• API reference — Complete endpoint reference for all /v1/{tenant_id}/ resources, including request and response schemas, supported microversions, and error codes.
• CLI reference — Full command listing for the openstack dr plugin, including all flags and positional arguments.
• Configuring Pure Storage replication — How to set up [pure_storage] options, configure a Replication Policy, and choose between async and active_dr replication types.
• Roles and RBAC — How to pre-create the dr_operator and dr_site_admin Keystone roles, grant them to users and the protector service account, and understand how policy rules gate each API endpoint.
• Monitoring replication health — How to integrate the replication health endpoint into observability pipelines and interpret the failover_validation.can_failover signal for automated runbooks.