REST API

This page documents the Trilio Site Recovery for OpenStack REST API (v1). The API runs on port 8788, validates Keystone tokens, enforces RBAC through oslo.policy, and dispatches DR operation requests to the protector-engine over RabbitMQ RPC. You interact with it directly via HTTP, through the openstack dr CLI plugin, or through the Python client library (triliositerecovery). All endpoints are versioned under /v1/{tenant_id}, where {tenant_id} is your Keystone project UUID. Operations that modify protection group state return HTTP 202 Accepted; the operation completes asynchronously and you poll the operation record for progress.

Before using the API:

• Two registered OpenStack sites — each with its own Keystone, Nova, Cinder, and Neutron. Both must run OpenStack Victoria or later.
• Keystone roles — dr_operator and dr_site_admin created on both sites. Absence of either causes trust creation to fail at the first DR operation.
• Service registered in Keystone — the disaster-recovery service type and its public/internal/admin endpoints in the catalogue. The Juju register-service action or the Kolla/OpenShift install script handles this.
• Cinder policy relaxations on both sites — five rules required:
  • volume_extension:volume_manage
  • volume_extension:volume_unmanage
  • volume_extension:services:index
  • volume_extension:type_get
  • volume_extension:type_get_all All five must allow rule:admin_or_owner. Omitting type_get or type_get_all causes 403 errors during protection group creation and volume management on the recovery site.
• Cinder volume type with replication_enabled=True and replication_type=async on both sites.
• Network connectivity between sites on port 8788 (protector API), 5000 (Keystone), 8774 (Nova), and 8776 (Cinder).
• python3-triliositerecovery installed on any client machine where you use the CLI or Python library.
• A valid Keystone token — all API requests (except GET /v1/{tenant_id}/sites/local) must carry an X-Auth-Token header.

The API is served by the protector-api service. Install and start it using one of the three supported deployment paths.

Juju (Charmed OpenStack)

# 1. Add the Trilio apt repository
cat > /etc/apt/sources.list.d/trilio-site-recovery.list <<'EOF'
deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /
EOF

# 2. Deploy the charm from Charmhub
juju deploy trilio-site-recovery-for-openstack \
  --channel latest/stable \
  --config local-site=site-a \
  --config storage-driver=pure \
  --config pure-primary-management-ip=10.0.0.1 \
  --config pure-primary-api-token=<token> \
  --config pure-secondary-management-ip=10.0.1.1 \
  --config pure-secondary-api-token=<token> \
  --config pure-primary-site-name=site-a \
  --config pure-secondary-site-name=site-b

# 3. Relate to MySQL, RabbitMQ, and Keystone
juju relate trilio-site-recovery-for-openstack:shared-db mysql:shared-db
juju relate trilio-site-recovery-for-openstack:amqp rabbitmq-server:amqp
juju relate trilio-site-recovery-for-openstack:identity-credentials keystone:identity-credentials

# 4. Run database migration and service registration
juju run trilio-site-recovery-for-openstack/0 db-sync
juju run trilio-site-recovery-for-openstack/0 register-service

# 5. Install the CLI on your client machine
apt-get install -y python3-triliositerecovery

Important: After upgrading to a new charm revision, the unit enters BLOCKED state with message set local-site, pure-primary-site-name, pure-secondary-site-name until those config keys are confirmed. This is expected — set them explicitly even if they were set before the upgrade.

Kolla-Ansible

# Run the install script (manages container lifecycle)
bash deploy/kolla/install.sh

The script deploys the protector_api and protector_engine containers using the protector:latest image and runs database migrations automatically.

OpenShift (RHOSO18)

# Run the installer — it applies all twelve manifests in the correct order,
# creates init Jobs for database/rabbitmq/keystone setup, and waits for each
bash deploy/openshift/scripts/install.sh \
  --namespace trilio-site-recovery \
  --image docker.io/xyzmd/trilio-site-recovery-openstack:dev

Do not apply manifests individually. The installer gates each init Job on oc wait --for=condition=complete before proceeding; skipping this order leaves pods that cannot authenticate tokens, reach the database, or dispatch RPC.

The service reads /etc/protector/protector.conf, written by the Juju charm or Kolla install script. Do not edit it by hand on charm-managed deployments — the charm rewrites it on every hook. Use juju config for charm-modelled keys, and config-flags for any key the charm does not expose directly.

Key options by section

[DEFAULT]

local_site is mandatory. The service cannot participate in any DR operation until it knows which registered site it is.

[api]

[engine]

[pure_storage]

Array-to-site pairing: The driver resolves which array to use by matching the protection group's primary site name against primary_site_name. The pairing must be consistent — primary_site_name must name the site whose array is at primary_management_ip. An identical [pure_storage] section works unchanged on both nodes.

[service_credentials]

[storage]

Reaching keys the charm does not model

The charm exposes 47 named options. For any key not among them (such as engine.cross_site_timeout), use config-flags:

juju config trilio-site-recovery config-flags="engine.cross_site_timeout=60"

Do not set a key via config-flags that the charm template already renders — this causes DuplicateSectionError.

All API requests require a Keystone token in the X-Auth-Token header and a Content-Type: application/json header for requests with a body. The base URL is http://<protector-api-host>:8788/v1/<project-id>.

Authentication pattern

export OS_TOKEN=$(openstack token issue -f value -c id)
export PROTECTOR_URL="http://protector.example.com:8788/v1/${OS_PROJECT_ID}"

Or use the openstack dr CLI, which reads your standard OS_* environment variables and handles token acquisition automatically.

RBAC summary

dr_site_admin carries no per-project restriction — a holder on any project can perform cloud-wide operations. Grant it only to cloud operators on the admin project.

Resource-mapping separator

When passing network, subnet, or flavor mappings on the CLI, use = as the separator between source and destination, never ::

openstack dr failover my-pg \
  --network-mapping src-net-id=dst-net-id \
  --flavor-mapping src-flavor-id=dst-flavor-id

A colon-separated value is accepted by the argument parser as a single opaque string, so the mapping silently does not apply. The supported mapping resource types are network, subnet, security_group, and flavor.

--volume-type is required for async replication

When creating a protection group with --replication-type async (the default), you must supply --volume-type. Omitting it returns HTTP 400:

openstack dr protection group create my-pg \
  --primary-site site-a \
  --secondary-site site-b \
  --replication-type async \
  --volume-type cinder-replicated-type

Polling operations

Most DR actions (failover, failback, test failover, member add/remove) return HTTP 202 with an operation_id. Poll the operation until status is completed or failed:

openstack dr operation show <operation-id>

The operation record includes a progress integer (0–100) and, on completion, instances_created, volumes_managed, and any warnings or errors. There is no estimated completion time — the service does not compute one.

Observability

The service emits no oslo.messaging notifications. Monitor DR state by polling:

• openstack dr replication health <pg> — replication status per volume
• openstack dr replication audit — volumes replicating without a protection group
• openstack dr recovery point list <pg> — available recovery points
• openstack dr operation list — all DR operations and their progress

The --probe flag on dr replication health asks the recovery site whether the group's networks, security groups, and flavors resolve there as the project it recovers into. It does not check the storage backend or replication lag. Run it before a failover window to catch resource-resolution problems early.

Recovered instance naming

Failover creates recovered instances as <original-name>-failover. Failback creates them as <original-name>-restored. Names do not compound over a round trip — the member row keeps the original instance name — but workloads never return to their original name after a round trip. Update DNS, monitoring, and CMDB entries accordingly.

LUKS-encrypted volumes

Adding a member whose volume is LUKS-encrypted raises InvalidInput (HTTP 400). Cinder's manage API cannot associate Barbican keys on the DR site. Use array-side encryption (Pure Storage data-at-rest encryption) instead.

List registered DR sites

openstack dr site list

Register a DR site

openstack dr site create site-b \
  --auth-url https://keystone-site-b.example.com:5000/v3 \
  --region RegionB \
  --service-username protector \
  --service-password <password> \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default

Create a protection group

openstack dr protection group create prod-pg \
  --primary-site site-a \
  --secondary-site site-b \
  --replication-type async \
  --volume-type cinder-replicated-type

--volume-type is required when --replication-type is async (the default). Omitting it returns HTTP 400.

Add a VM to a protection group

openstack dr protection group member add prod-pg \
  --instance <instance-uuid> \
  --boot-order 10

Lower --boot-order values start first and stop last during failover and failback.

Validate a VM before adding it

openstack dr protection group member validate prod-pg \
  --instance <instance-uuid>

This runs all eligibility checks without modifying the protection group.

Synchronise metadata to the secondary site

openstack dr protection group sync prod-pg

Create a resource mapping (network)

# Via CLI
openstack dr protection group mapping create prod-pg \
  --resource-type network \
  --primary-resource-id <primary-net-uuid> \
  --secondary-resource-id <secondary-net-uuid>

# Via REST API
curl -s -X POST \
  -H "X-Auth-Token: ${OS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "mapping": {
      "resource_type": "network",
      "primary_resource_id": "<primary-net-uuid>",
      "secondary_resource_id": "<secondary-net-uuid>"
    }
  }' \
  "${PROTECTOR_URL}/protection-groups/<pg-id>/mappings"

Auto-map resources

# REST API
curl -s -X POST \
  -H "X-Auth-Token: ${OS_TOKEN}" \
  "${PROTECTOR_URL}/protection-groups/<pg-id>/mappings/auto"

Check replication health (with recovery-site probe)

# Basic check
openstack dr replication health prod-pg

# With --probe: asks the recovery site whether networks, security groups,
# and flavors resolve there. Slower but catches resource-resolution problems.
openstack dr replication health prod-pg --probe

List recovery points

openstack dr recovery point list prod-pg --limit 10

Execute a planned failover

# CLI
openstack dr failover prod-pg \
  --failover-type planned \
  --network-mapping <primary-net-id>=<secondary-net-id> \
  --wait

# REST API
curl -s -X POST \
  -H "X-Auth-Token: ${OS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "failover": {
      "failover_type": "planned",
      "network_mapping": {
        "<primary-net-id>": "<secondary-net-id>"
      },
      "vm_creation_params": {
        "availability_zone": "nova-az-2"
      }
    }
  }' \
  "${PROTECTOR_URL}/protection-groups/<pg-id>/action"

The response is HTTP 202 with an operation_id. Poll with:

openstack dr operation show <operation-id>

Execute a test failover

openstack dr test failover prod-pg \
  --network-mapping <primary-net-id>=<test-net-id> \
  --availability-zone nova-az-test \
  --instance-prefix test- \
  --wait

Test failover clones replicated data to the secondary site and boots isolated test VMs without affecting production. Clean up afterwards:

openstack dr test failover cleanup prod-pg --operation <operation-id>

Execute failback

openstack dr failback prod-pg \
  --failback-type planned \
  --network-mapping <secondary-net-id>=<primary-net-id> \
  --wait

Failback waits up to 120 seconds per volume for Nova to release Cinder attachments before unmanaging them. A visibly slower failback on a site with lagging detaches is the wait working correctly, not a hang.

Site-level failover (all protection groups at a site)

# CLI (requires dr_site_admin role)
openstack dr site failover site-a --failover-type unplanned --yes

# REST API
curl -s -X POST \
  -H "X-Auth-Token: ${OS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"failover": {"failover_type": "unplanned"}}' \
  "${PROTECTOR_URL}/sites/<site-id>/action"

The /sites/{id}/action endpoint accepts only failover or failback as the top-level key.

Create a tenant mapping

openstack dr tenant mapping create \
  --local-tenant <local-project-uuid> \
  --remote-site site-b \
  --remote-tenant <remote-project-uuid> \
  --description "Production DR tenant pair"

Without a tenant mapping the engine assumes the same project UUID on the peer site. That holds for a shared Keystone and essentially never for two independent deployments. The error surfaces during a later cross-site operation rather than at mapping time.

Python client example

from keystoneauth1 import session as ks_session
from keystoneauth1.identity import v3 as ks_identity
from triliositerecovery.v1.client import Client

auth = ks_identity.Password(
    auth_url="https://keystone.example.com:5000/v3",
    username="druser",
    password="secret",
    project_name="myproject",
    user_domain_name="Default",
    project_domain_name="Default",
)
sess = ks_session.Session(auth=auth)

client = Client(
    session=sess,
    endpoint="http://protector.example.com:8788",
    region_name="RegionOne",
)

# List protection groups — returns a list of plain dicts
pgs = client.protection_groups.list()
for pg in pgs:
    print(pg['id'], pg['name'], pg['status'])  # use pg['key'], not pg.key

# Trigger a failover — pg_id is positional
operation = client.protection_groups.failover(
    pg['id'],
    failover_type='planned',
    network_mapping={'src-net-id': 'dst-net-id'},  # dict, not 'src=dst' string
    vm_creation_params={'availability_zone': 'nova-az-2'},
)
print(operation['operation_id'])

# Clean up a test failover
client.protection_groups.cleanup_test_failover(pg['id'])

Python client conventions:

1. Every manager returns parsed JSON — plain dicts, not objects. Write pg['id'], not pg.id.
2. The resource ID is always the first positional argument (pg_id). Passing it as a keyword argument fails with a missing-argument error.
3. Mappings are dicts ({'src': 'dst'}), not 'src=dst' strings. The = separator is argparse-only.
4. endpoint is required. Omitting it produces request URLs beginning with None/ and fails at the first call.

Usage report (admin)

# JSON format
openstack dr usage report --period monthly --format json

# PDF download
openstack dr usage report --period quarterly --format pdf --output report.pdf

Use a consistent diagnostic pattern: check the operation record first (openstack dr operation show <id>), then service logs, then connectivity.

Juju logs: juju debug-log --include trilio-site-recovery-for-openstack

Kolla logs: docker logs protector_api / docker logs protector_engine

OpenShift logs: oc -n trilio-site-recovery logs -l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api and ...component=engine. Both label selectors are required — there is no bare app label on any manifest. On OpenShift, stdout is the only log destination; there is no log file to fall back to.

HTTP 400 — volume_type_id is required for async replication

Symptom: POST /v1/{tenant_id}/protection-groups returns 400.

Cause: --volume-type was omitted when creating a protection group with --replication-type async.

Fix: Always supply --volume-type <cinder-replicated-type> for async replication.

HTTP 403 — Policy doesn't allow volume_extension:volume_manage

Symptom: Failover starts successfully (storage work completes), then fails partway through with a 403 from Cinder.

Cause: The Cinder policy on one or both sites does not allow volume_manage, volume_unmanage, services:index, type_get, or type_get_all for non-admin users. All five rules must be set to rule:admin_or_owner. The type_get and type_get_all rules are read-only and open on some Cinder versions, which is why the service deploys and reports healthy but fails on the first real failover.

Fix: Apply all five Cinder policy relaxations on both sites:

"volume_extension:volume_manage": "rule:admin_or_owner"
"volume_extension:volume_unmanage": "rule:admin_or_owner"
"volume_extension:services:index": "rule:admin_or_owner"
"volume_extension:type_get": "rule:admin_or_owner"
"volume_extension:type_get_all": "rule:admin_or_owner"

HTTP 409 — InvalidState

Symptom: An action on a protection group returns HTTP 409 with InvalidState.

Cause: The protection group is not in a state that allows the requested operation (for example, attempting a failover on a group that is already failed_over, or modifying membership on a group that is not on its primary site).

Fix: Check the group's current status with openstack dr protection group show <pg>. Use openstack dr protection group set <pg> --reset-to-primary or --reset-to-secondary with --status to reset state after a partial failure, then retry.

HTTP 409 — ResourceConflict on site or protection group create

Symptom: Create returns 409 with message The request conflicts with an existing resource. A name may still be held by a deleted resource.

Cause: A unique-constraint violation in the database — a site or protection group with that name already exists, possibly soft-deleted.

Fix: Choose a different name, or use openstack dr site delete --force to remove the conflicting record before recreating.

HTTP 422 — InvalidInput when adding a member

Symptom: POST /v1/{tenant_id}/protection-groups/{pg_id}/members returns 422 or 400 with InvalidInput.

Likely causes:

• The instance boots from an ephemeral disk, not a Cinder volume (InstanceNotCinderBoot).
• The instance has no bootable volume (NoBootableVolumeFound).
• The instance's volume is LUKS-encrypted — encrypted volumes cannot be protected. Use array-side encryption instead.
• The instance is already protected in another protection group (InstanceAlreadyProtected).

Fix: Use openstack dr protection group member validate <pg> --instance <id> to get a detailed validation report before adding.

Unit in BLOCKED state after charm upgrade

Symptom: After upgrading the trilio-site-recovery-for-openstack charm, the unit reports BLOCKED with: set local-site, pure-primary-site-name, pure-secondary-site-name.

Cause: This is expected behaviour. The charm does not infer which site it is (the inference was wrong on the recovery site, where failover runs). The config keys must be set explicitly after every upgrade.

Fix:

juju config trilio-site-recovery-for-openstack \
  local-site=site-a \
  pure-primary-site-name=site-a \
  pure-secondary-site-name=site-b

Trust creation fails naming a missing role

Symptom: The first DR operation (failover, failback, or test failover) fails with an error naming a missing Keystone role.

Cause: dr_operator or member does not exist in Keystone on one or both sites. Trust creation fails immediately if any role in default_trust_roles is missing rather than skipping it.

Fix: Create both roles on both sites before any trust is created:

openstack role create dr_operator
openstack role create dr_site_admin

--probe shows missing networks but the network exists

Symptom: openstack dr replication health <pg> --probe reports a network as missing on the recovery site, but you can see it in openstack network list there.

Cause: The --probe flag queries the recovery site's Neutron through the same trust-scoped session that a real recovery uses. A network that exists under a different project on the recovery site is correctly reported as missing from the project the protection group recovers into.

Fix: Ensure the network on the recovery site belongs to the correct project, or create an explicit resource mapping (openstack dr protection group mapping create) to map the primary network to the correct secondary network.

Operation source_site and target_site columns show names, not IDs

This is correct behaviour as of the current release. Both columns show the site name, falling back to the site ID if the name cannot be resolved. If you see UUIDs instead of names, the site relationship could not be loaded (for example, if the site row was deleted after the operation was created).

Database schema mismatch on startup

Symptom: The service fails to start, or reads of a table fail after an upgrade.

Cause: Database migrations have not been applied. The current schema head is 021_add_member_status_reason. Migrations must run before the new code starts.

Fix (Juju):

juju run trilio-site-recovery-for-openstack/0 db-sync

Fix (Kolla / manual):

protector-manage db sync

Do not invoke alembic directly or use python3 -m protector.db.migration upgrade head — neither is supported. The protector-manage db sync command is the only supported migration path.

Failback is slow (volumes still attached)

Symptom: Failback takes longer than expected; logs show repeated attachment checks.

Cause: Failback waits up to 120 seconds per volume for Nova to release Cinder attachments after deleting the DR instance. Nova releases the attachment asynchronously. This is the wait working correctly, not a hang. A volume still attached at the deadline is unmanaged anyway and logged — failback is not aborted.

Fix: No action needed if failback eventually completes. If volumes end up in error_deleting state, check that Nova has fully deleted the DR instance before attempting manual cleanup.

Cross-site calls time out on a slow link

Symptom: Operations fail with connection timeouts to the peer site.

Cause: The default cross_site_timeout of 30 seconds is too short for the inter-site network latency.

Fix (Juju):

juju config trilio-site-recovery config-flags="engine.cross_site_timeout=60"

Do not use --cross-site-timeout or --engine-cross-site-timeout — neither option exists on the charm and both produce an unknown-option error.