Components

This page describes the internal architecture of Trilio Site Recovery for OpenStack: the two service processes that make up the product, how they communicate, what each one does, and how they relate to the database and external systems. Understanding this architecture helps you diagnose failures, tune deployments, and reason about what happens during a DR operation. The service is split into two cooperating processes — protector-api and protector-engine — that communicate over RabbitMQ RPC, share a MariaDB/MySQL database, and delegate storage operations to a pluggable backend driver.

This page assumes you are familiar with the following:

• Basic OpenStack service architecture (WSGI API process, background worker, oslo.messaging)
• The Trilio Site Recovery deployment model for your target platform (Juju, Kolla-Ansible, or OpenShift)
• The openstack dr CLI installed from the python3-triliositerecovery Debian package

Before the service can participate in any DR operation:

• Both DR sites must be registered and reachable on port 8788 (protector API), 5000 (Keystone), 8774 (Nova), and 8776 (Cinder)
• The local_site option in the [DEFAULT] section of /etc/protector/protector.conf must be set to the name of the DR site this instance belongs to
• The dr_operator and dr_site_admin Keystone roles must exist on both sites
• The MariaDB/MySQL protector database must exist and be at schema head 021_add_member_status_reason

The service is not installed in isolation — each deployment tool installs and wires up both the API and engine together. Choose the path that matches your platform.

Juju (Charmed OpenStack)

The trilio-site-recovery Juju charm installs and manages both service processes on Ubuntu Jammy or Noble machines.

1. Configure the Trilio apt repository. The charm reads its package source from the trilio-pkg-source config key, which defaults to the Trilio apt repository:

   Copy

   deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /
   

   This is written to /etc/apt/sources.list.d/trilio-site-recovery.list on each managed machine.

2. Deploy the charm:

   Copy

   juju deploy trilio-site-recovery-for-openstack --channel latest/stable
   

3. Add the required relations:

   Copy

   juju integrate trilio-site-recovery-for-openstack:shared-db mysql-innodb-cluster:db-router
   juju integrate trilio-site-recovery-for-openstack:amqp rabbitmq-server:amqp
   juju integrate trilio-site-recovery-for-openstack:identity-credentials keystone:identity-credentials
   

   Note: The charm's relation endpoint names are shared-db, amqp, and identity-credentials — use these exact names.

4. Set the required configuration (the unit will remain in BLOCKED state until these are provided):

   Copy

   juju config trilio-site-recovery-for-openstack local-site=site-primary
   

   If you are using Pure Storage, also set:

   Copy

   juju config trilio-site-recovery-for-openstack \
     pure-primary-site-name=site-primary \
     pure-secondary-site-name=site-secondary
   

5. Run the database migration:

   Copy

   juju run trilio-site-recovery-for-openstack/leader db-sync
   

6. Register the service in Keystone:

   Copy

   juju run trilio-site-recovery-for-openstack/leader register-service
   

7. (Optional) Install the Horizon dashboard plugin:

   Copy

   juju deploy trilio-site-recovery-for-openstack-dashboard
   juju integrate trilio-site-recovery-for-openstack-dashboard:dashboard-plugin openstack-dashboard:dashboard-plugin
   juju run trilio-site-recovery-for-openstack-dashboard/leader install-plugin
   

Kolla-Ansible

The Kolla install script deploys the protector_api and protector_engine containers using the protector:latest Docker image alongside a Kolla-managed OpenStack deployment.

1. Run the installer from the deploy directory:

   Copy

   bash deploy/kolla/install.sh
   

   The script creates and starts both containers, runs the database migration, and registers the service in Keystone. It polls /healthz on port 8788 to confirm the API is serving before returning.

2. Set local_site in /etc/protector/protector.conf before starting the service if the installer did not set it for your deployment.

OpenShift (RHOSO18)

On Red Hat OpenStack Services on OpenShift, the service is deployed by the deploy/openshift/scripts/install.sh script, which applies all manifests in the correct order including init Jobs for database creation, RabbitMQ setup, Keystone registration, and schema migration.

1. Run the install script:

   Copy

   bash deploy/openshift/scripts/install.sh
   

   The script creates the trilio-site-recovery namespace, applies the ConfigMap (which contains the full protector.conf), adds credentials to the shared trilio-openstack-secret in the trilio-openstack namespace, runs four init Jobs with completion gates, then deploys the trilio-site-recovery-api and trilio-site-recovery-engine Deployments.

   Important: Do not apply the manifests from deploy/openshift/manifests/ individually. The init Jobs — job-db-init, job-rabbitmq-init, job-keystone-init, and job-db-sync — must complete in order before the Deployments are applied. The script handles this sequencing.

2. Verify both Deployments are Running:

   Copy

   oc -n trilio-site-recovery get pods \
     -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api'
   
   oc -n trilio-site-recovery get pods \
     -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'
   

Installing the CLI separately

On any machine that runs openstack dr commands but is not managed by the charm, install the client package from the Trilio apt repository:

1. Add the repository:

   Copy

   echo "deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /" \
     | sudo tee /etc/apt/sources.list.d/trilio-site-recovery.list
   sudo apt-get update
   

2. Install the client package:

   Copy

   sudo apt-get install python3-triliositerecovery
   

3. To upgrade the CLI independently (the charm's upgrade-packages action upgrades the service unit only):

   Copy

   sudo apt-get install --only-upgrade python3-triliositerecovery
   

Configuration is managed by oslo.config. The file is written to /etc/protector/protector.conf by the Juju charm or Kolla install script. Do not edit this file by hand on charm-managed deployments — the charm rewrites it on every hook run. Use juju config for options the charm models, and config-flags for anything else.

[DEFAULT]

[api]

[engine]

To set cross_site_timeout on a Juju-managed deployment (the charm has no dedicated option for it):

juju config trilio-site-recovery-for-openstack \
  config-flags="engine.cross_site_timeout=60"

[pure_storage]

Required when storage_driver = pure. Both nodes in a deployment can use an identical [pure_storage] section — the driver resolves which array to talk to by matching the protection group's primary site name against primary_site_name and secondary_site_name.

Pairing invariant: primary_site_name must name the site whose array is at primary_management_ip, and secondary_site_name must name the site whose array is at secondary_management_ip. Swapping the names causes failover to read volumes from the wrong array.

[mock_storage]

Used when use_mock_storage = true. State is stored under /var/lib/protector.

[storage]

[service_credentials]

Controls the Keystone trust delegation the engine uses when performing cross-site operations on behalf of tenants.

[keystone_authtoken]

[database]

Annotated sample configuration

[DEFAULT]
# REQUIRED: the registered name of this DR site
local_site = site-primary

# Set to 'pure' for production, 'mock' for evaluation without hardware
use_mock_storage = false

[api]
bind_host = 0.0.0.0
bind_port = 8788
workers = 4

[engine]
topic = protector-engine
recovery_point_interval_minutes = 60
recovery_point_max_count = 20
recovery_point_ttl_hours = 24
auto_create_security_groups = false
cross_site_timeout = 30

[database]
connection = mysql+pymysql://protector:PASSWORD@db-host/protector

[keystone_authtoken]
auth_url = https://keystone.primary.example.com:5000
www_authenticate_uri = https://keystone.primary.example.com:5000

[service_credentials]
username = protector
project_name = service
# Do not change this; both roles must exist in Keystone
default_trust_roles = member,dr_operator

[storage]
storage_driver = pure

[pure_storage]
primary_management_ip = 10.10.1.10
# primary_api_token contains a secret — inspect with grep, not section anchors
primary_api_token = T-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
secondary_management_ip = 10.20.1.10
secondary_api_token = T-yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
primary_site_name = site-primary
secondary_site_name = site-secondary
default_rpo_seconds = 900
snapshot_interval_seconds = 300
verify_ssl = false

To inspect a specific option on a live deployment without exposing secrets:

grep '^local_site' /etc/protector/protector.conf
grep '^default_trust_roles' /etc/protector/protector.conf
grep '^storage_driver' /etc/protector/protector.conf

Service health

The API exposes an unauthenticated health endpoint at /healthz on port 8788. It returns HTTP 200 when the API can serve requests and HTTP 503 when it cannot (for example, when the database is unreachable). This endpoint is used by the Kolla container HEALTHCHECK, the installer, and cross-site peer checks.

curl -s -o /dev/null -w "%{http_code}" http://protector-api.primary.example.com:8788/healthz

You can also run the charm health check action:

juju run trilio-site-recovery-for-openstack/leader health-check

Checking service status

The observability surface for Trilio Site Recovery is polled, not event-driven. The service does not emit oslo.messaging notifications. Monitor it using the CLI commands below.

Replication health for a specific protection group:

openstack dr replication health my-protection-group

Add --probe to ask the recovery site whether the group's networks, security groups, and flavors resolve there as the project it recovers into. This talks to the recovery site's Neutron and Nova through a trust-scoped session — the same session a real recovery would use — and is slower than the plain command. Run it before a planned failover window:

openstack dr replication health my-protection-group --probe

Audit replication across all protection groups:

openstack dr replication audit

List available recovery points:

openstack dr recovery point list my-protection-group

List recent DR operations:

openstack dr operation list
openstack dr operation list --protection-group my-protection-group
openstack dr operation list --status failed

Show details of a specific operation:

openstack dr operation show OPERATION-ID

The operation record includes a progress field (0–100) and the Source Site and Target Site by name. There is no estimated completion time — the service does not compute one.

Database migration

The supported way to run schema migrations is protector-manage db sync. It reads /etc/protector/protector.conf by default.

On Juju deployments:

juju run trilio-site-recovery-for-openstack/leader db-sync

On Kolla and manual deployments:

protector-manage db sync
# Or with an explicit config path:
protector-manage --config-file /etc/protector/protector.conf db sync

Migrations must run before starting a new version of the service. The current schema head is 021_add_member_status_reason.

To check the current schema revision:

protector-manage db current

Log access

Juju and Kolla deployments write logs to /var/log/protector/. View them with:

tail -f /var/log/protector/protector-api.log
tail -f /var/log/protector/protector-engine.log

Or via Juju:

juju debug-log --include trilio-site-recovery-for-openstack

OpenShift deployments log to stdout only. Use the label selectors to target the correct pods:

# API pods
oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api'

# Engine pods
oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Python client library

The python3-triliositerecovery package includes the triliositerecovery Python client library for programmatic access. Import it from triliositerecovery.v1.client:

from keystoneauth1 import identity, session
from keystoneauth1.adapter import Adapter
from triliositerecovery.v1.client import Client

auth = identity.v3.Password(
    auth_url='https://keystone.primary.example.com:5000/v3',
    username='myuser',
    password='mypassword',
    project_name='myproject',
    user_domain_name='Default',
    project_domain_name='Default',
)
ks_session = session.Session(auth=auth)
adapter = Adapter(
    session=ks_session,
    service_type='disaster-recovery',
    endpoint_override='http://protector-api.primary.example.com:8788',
    region_name='RegionOne',
)
client = Client(session=adapter, endpoint='http://protector-api.primary.example.com:8788')

Key conventions when using the client:

• Return shape: Every manager method returns parsed JSON — plain Python dicts and lists. Use site['id'], not site.id.
• Resource ID is positional: Pass the protection group ID as the first positional argument, not as a keyword. For example: client.protection_groups.failover(pg_id, failover_type='planned'), not client.protection_groups.failover(protection_group_id=pg_id).
• Mappings are dicts: Pass network and flavor mappings as Python dicts (network_mapping={'src-net-id': 'dst-net-id'}), not as 'src=dst' strings. The src=dst format is argparse-only and is parsed into a dict by the CLI before it reaches the library.
• endpoint is required: Omitting it causes all request URLs to begin with None/ and fail at the first call.

List all registered DR sites

openstack dr site list

Register a secondary DR site

openstack dr site create site-secondary \
  --auth-url https://keystone.secondary.example.com:5000 \
  --region RegionTwo \
  --service-username protector \
  --service-password SECRET \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default

Create a protection group

Both --primary-site and --secondary-site are required — the secondary site is never inferred. When --replication-type async is used (the default), --volume-type is also required:

openstack dr protection group create my-app-pg \
  --primary-site site-primary \
  --secondary-site site-secondary \
  --replication-type async \
  --volume-type ceph-replicated

Add a VM to a protection group

openstack dr protection group member add my-app-pg \
  --instance db-server-uuid \
  --boot-order 1

openstack dr protection group member add my-app-pg \
  --instance app-server-uuid \
  --boot-order 2

Check replication health before a failover window

The plain command checks locally cached replication state:

openstack dr replication health my-app-pg

Add --probe to verify that networks, security groups, and flavors resolve on the recovery site as the project the group recovers into:

openstack dr replication health my-app-pg --probe

Create an on-demand recovery point

openstack dr recovery point create my-app-pg --suffix pre-upgrade

Execute a test failover

Test failover is non-disruptive and must be run from the secondary site. It clones replicated data and boots test VMs in isolation:

openstack dr test failover my-app-pg \
  --network-mapping src-net-id=dst-net-id \
  --instance-prefix test- \
  --wait

Note: The separator in --network-mapping is =, not :. A colon-separated value is silently accepted by argparse as one string and the mapping does not apply.

Clean up after a test failover

openstack dr test failover cleanup my-app-pg

Execute a planned failover

openstack dr failover my-app-pg \
  --failover-type planned \
  --network-mapping src-net-id=dst-net-id \
  --flavor-mapping src-flavor-id=dst-flavor-id \
  --wait

After a successful failover, recovered instances are named <original-name>-failover.

Execute an unplanned failover

openstack dr failover my-app-pg \
  --failover-type unplanned \
  --force \
  --wait

Fail back to the primary site

openstack dr failback my-app-pg \
  --network-mapping dst-net-id=src-net-id \
  --wait

After a successful failback, instances on the primary site are named <original-name>-restored.

Monitor a running operation

openstack dr operation list --protection-group my-app-pg --status running
openstack dr operation show OPERATION-ID

The progress field reports 0–100. There is no estimated completion time.

Run the replication audit

openstack dr replication audit

Create a tenant mapping (admin operation)

Tenant mappings are required when primary and secondary sites use independent Keystone deployments that assign different UUIDs to the same project. Without a mapping, the engine assumes the project UUID is identical on the peer site.

openstack dr tenant mapping create \
  --local-tenant PRIMARY-PROJECT-UUID \
  --remote-site site-secondary \
  --remote-tenant SECONDARY-PROJECT-UUID

Execute a site-level failover (cloud admin operation)

Site-level failover requires the dr_site_admin role. It fails over all protection groups at the named site:

openstack dr site failover site-primary --failover-type unplanned

Alternatively, from the recovery site:

openstack dr failover to this site --failover-type unplanned

Python client: list protection groups and check health

from keystoneauth1 import identity, session
from keystoneauth1.adapter import Adapter
from triliositerecovery.v1.client import Client

auth = identity.v3.Password(
    auth_url='https://keystone.primary.example.com:5000/v3',
    username='myuser',
    password='mypassword',
    project_name='myproject',
    user_domain_name='Default',
    project_domain_name='Default',
)
ks_session = session.Session(auth=auth)
adapter = Adapter(
    session=ks_session,
    service_type='disaster-recovery',
    endpoint_override='http://protector-api.primary.example.com:8788',
    region_name='RegionOne',
)
client = Client(
    session=adapter,
    endpoint='http://protector-api.primary.example.com:8788',
)

# Managers return lists of plain dicts — use dict indexing, not attribute access
groups = client.protection_groups.list()
for pg in groups:
    health = client.protection_groups.replication_health(pg['id'])
    print(pg['name'], health)

# cleanup_test_failover is the correct method name
client.protection_groups.cleanup_test_failover(pg_id)

Run database migration directly

# On Juju
juju run trilio-site-recovery-for-openstack/leader db-sync

# On Kolla or manual deployments
protector-manage db sync

# With an explicit config path
protector-manage --config-file /etc/protector/protector.conf db sync

# Check current schema revision
protector-manage db current

The unit is in BLOCKED state after charm deployment or upgrade

Symptom: juju status shows the trilio-site-recovery-for-openstack unit in BLOCKED with the message set local-site, pure-primary-site-name, pure-secondary-site-name.

Cause: The charm requires local-site to be configured before it can start the service. If you are using Pure Storage, pure-primary-site-name and pure-secondary-site-name must also be set. This is expected behaviour after a fresh deploy or upgrade — it is not a failed deployment.

Fix:

juju config trilio-site-recovery-for-openstack local-site=site-primary
# If using Pure Storage:
juju config trilio-site-recovery-for-openstack \
  pure-primary-site-name=site-primary \
  pure-secondary-site-name=site-secondary

Failover fails with HTTP 403 "Policy doesn't allow volume_extension:volume_manage"

Symptom: The service deploys and reports healthy. The first failover attempt stops partway through with a Cinder 403 error referencing volume_extension:volume_manage or a similar extension.

Cause: Cinder's default policy restricts the volume manage, unmanage, and related APIs to admin. The service runs as a tenant through a Keystone trust, so it is blocked. All five Cinder policy rules must be relaxed on both sites:

"volume_extension:volume_manage": "rule:admin_or_owner"
"volume_extension:volume_unmanage": "rule:admin_or_owner"
"volume_extension:services:index": "rule:admin_or_owner"
"volume_extension:type_get": "rule:admin_or_owner"
"volume_extension:type_get_all": "rule:admin_or_owner"

Fix: Apply these rules to the Cinder policy on both the primary and secondary sites. On Juju-managed Cinder, use the policyd-override resource (not policyd):

juju attach-resource cinder policyd-override=./cinder-policy-override.yaml

DR operations fail with "Trust-based authentication failed" or "No trust found for tenant"

Symptom: Operations that require cross-site access fail with TrustAuthFailed (HTTP 401) or TrustNotFoundForTenant (HTTP 404).

Cause: The most common causes are:

• The dr_operator role does not exist in Keystone on one or both sites. Trust creation fails naming the missing role rather than skipping it.
• default_trust_roles has been changed to omit member, removing the Nova and Cinder access the trust needs.
• No tenant mapping exists for a deployment with independent Keystone instances, so the engine is scoping its trust to a project UUID that does not exist on the peer site.

Fix:

1. Verify both roles exist on both sites:

   Copy

   openstack role list | grep -E 'dr_operator|dr_site_admin'
   

2. Check default_trust_roles:

   Copy

   grep '^default_trust_roles' /etc/protector/protector.conf
   

   The value must be member,dr_operator.

3. If your sites use independent Keystone deployments with different project UUIDs, create a tenant mapping:

   Copy

   openstack dr tenant mapping create \
     --local-tenant PRIMARY-PROJECT-UUID \
     --remote-site site-secondary \
     --remote-tenant SECONDARY-PROJECT-UUID
   

openstack dr commands fail with "unrecognized arguments: --site-type"

Symptom: A script or documentation example calling openstack dr site create with a --site-type flag fails.

Cause: The --site-type flag was removed. A site is registered by name only; primary and secondary are roles of a protection group, not a site property.

Fix: Remove --site-type from the command. The correct form is:

openstack dr site create site-secondary --auth-url https://keystone.secondary.example.com:5000

Failover reports "2 of 2 volumes missing or destroyed"

Symptom: Failover completes with an error indicating all volumes are missing or destroyed on the target site, even though replication appeared healthy.

Cause: The [pure_storage] section on this node has primary_site_name and secondary_site_name swapped relative to the primary_management_ip and secondary_management_ip they are paired with. The driver resolves which array to read volumes from by matching the protection group's primary site name against the configured site names. A mismatch causes it to read from the wrong array.

Fix: Verify that primary_site_name names the site whose array is at primary_management_ip, and secondary_site_name names the site at secondary_management_ip. An identical [pure_storage] section works correctly on both nodes:

grep '^primary_site_name\|^secondary_site_name\|^primary_management_ip\|^secondary_management_ip' \
  /etc/protector/protector.conf

The service reports HTTP 409 instead of HTTP 400 on invalid state transitions

Symptom: An attempt to perform a DR operation on a protection group in the wrong state (for example, failing over a group that is already failed_over) returns HTTP 409, not 400.

Cause: InvalidState is defined as HTTP 409 on all endpoints, including /action. This is correct and intentional.

Operations left in running or pending state after engine restart

Symptom: After an engine process restart, some operations remain in running or pending state and the protection group appears stuck.

Cause: When the engine restarts, it recovers orphaned operations from the previous run. It uses the checkpoint phase saved in the operation's result_data to determine the correct terminal state for both the operation and the protection group.

Fix: The engine handles this automatically on startup. If a group remains stuck after the engine has fully started, use the reconcile command to inspect live infrastructure and reset state:

openstack dr protection group reconcile my-app-pg

No logs visible on an OpenShift deployment

Symptom: oc logs returns "No resources found" or no output.

Cause: The pod label selector is wrong. There is no bare app label on any pod in the trilio-site-recovery namespace. Both app.kubernetes.io/name and app.kubernetes.io/component are required.

Fix: Use the correct label selectors:

# API logs
oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api'

# Engine logs
oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

On OpenShift, the service logs to stdout only. There is no /var/log/protector/ directory in the containers.

Adding a member fails with "Instance is not a Cinder boot instance"

Symptom: openstack dr protection group member add fails with InstanceNotCinderBoot (HTTP 422).

Cause: All VMs in a protection group must boot from a Cinder volume. Instances that boot from ephemeral or image-backed storage cannot be replicated by this service.

Fix: Migrate the instance to boot from a Cinder volume before adding it to a protection group. LUKS-encrypted volumes are also unsupported — use array-side encryption (Pure data-at-rest) instead.

config-flags setting is silently ignored or causes DuplicateSectionError

Symptom: A setting applied via config-flags has no effect, or the service fails to start with a DuplicateSectionError.

Cause: If config-flags names a config key that the charm template already renders (in a section the template also writes), the charm appends a second copy of that section and configparser raises DuplicateSectionError. Keys that the charm does not model — such as engine.cross_site_timeout — are safe to set this way.

Fix: Only use config-flags for keys the charm does not have a dedicated option for. Check the list of 47 charm config options with juju config trilio-site-recovery-for-openstack before adding a flag. For options the charm already models, use the corresponding charm config key instead.