Deployment Guide

This guide walks you through deploying Trilio Site Recovery for OpenStack to a production environment using the Juju charm — the only validated, customer-facing deployment method for the tech preview. You will install and configure the protector-api and protector-engine services on each OpenStack site, wire them to your existing Keystone, RabbitMQ, and MySQL infrastructure, and verify that the services are healthy before proceeding to register sites and configure Protection Groups. The Juju charm manages all systemd services, renders protector.conf from relation data, migrates the database schema, and registers with Keystone automatically once all required relations are satisfied.

Before deploying, ensure the following are in place on both sites:

OpenStack requirements

• OpenStack Victoria or later
• Nova, Cinder (with a volume type carrying replication_enabled='<is> True'), Keystone, and Neutron operational on each site
• Pure Storage FlashArray with an async replication connection between sites

Infrastructure

• MariaDB or MySQL ≥ 10.3 (shared with the cloud or dedicated — see topologies below)
• RabbitMQ with oslo.messaging support
• python-openstackclient installed via pip (not the snap — the snap is incompatible with the triliositerecovery CLI plugin)
• Juju 2.9 or 3.x with access to the target Charmed OpenStack model

Keystone roles — the service never creates roles, only looks them up. Both roles must exist on both sites before deployment:

openstack role create dr_operator
openstack role create dr_site_admin

Trust creation fails with Role <name> does not exist on site <site> if either role is absent.

DR user grants — on every project that will use DR:

# For each DR-enabled project on each site
openstack role add --project <project> --user <dr-user> member
openstack role add --project <project> --user <dr-user> dr_operator

The protector service account also requires member and dr_operator on each DR-enabled project on each site (not just the service project).

Network connectivity — bidirectional between sites on all of the following ports:

Port 8788 must be bidirectional. One-way connectivity causes tenant mapping sync failures and leaves the system in an inconsistent state.

Charm build artefact

bash deploy/charms/build.sh

The built .charm file is required before running the deploy script.

Infrastructure charm revision pins — newer revisions of these charms ship wheelhouses that fail to install on Ubuntu Jammy:

• mysql-innodb-cluster: use revision 164 (not 179)
• mysql-router: use revision 1154 (not 1178)
• rabbitmq-server: use revision 295

Installation

Perform every step on both your primary site and your secondary site unless a step is explicitly marked as single-site.

Option A — Kolla-style Docker containers

This is the recommended path for operators running a Kolla-managed OpenStack cluster. protector-api and protector-engine run as Docker containers alongside the rest of the Kolla stack.

Step 1 — Add Trilio variables to your Kolla inventory

In your Kolla globals file (typically /etc/kolla/globals.yml), enable the Trilio Site Recovery services and supply your Pure Storage details:

enable_trilio_site_recovery: "yes"
trilio_protector_api_port: 8788
trilio_storage_driver: "pure"

Step 2 — Configure protector.conf

Kolla renders /etc/protector/protector.conf from your globals and passwords files. Verify that the following sections and keys will be present after rendering (adjust values to match your environment):

[DEFAULT]
host = site-a-controller

[api]
bind_host = 0.0.0.0
bind_port = 8788
workers = 4

[database]
connection = mysql+pymysql://protector:PROTECTOR_PASS@DB_HOST:3306/protector

[storage]
storage_driver = pure

[pure_storage]
primary_management_ip = 10.0.1.10
primary_api_token = YOUR_PRIMARY_TOKEN
secondary_management_ip = 10.0.2.10
secondary_api_token = YOUR_SECONDARY_TOKEN
verify_ssl = false
default_rpo_seconds = 900
snapshot_interval_seconds = 300

Important: storage_driver defaults to mock, which is for development only. Production deployments must set storage_driver = pure.

Step 3 — Deploy with Kolla-style Docker containers

kolla-ansible deploy --tags trilio-site-recovery

Run this against both sites (update your -i inventory path as appropriate).

Step 4 — Run the database migration

After containers are up, initialise the schema on each site:

docker exec protector_api protector-manage db upgrade

Step 5 — Verify services are running

docker ps --filter name=protector_api
docker ps --filter name=protector_engine

Both containers should show status Up. Check the version endpoint to confirm the API is ready:

curl -s http://SITE_API_HOST:8788/

Expected response:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://SITE_API_HOST:8788/v1"}]
    }
  ]
}

Step 6 — Register the Keystone service catalogue entry

On each site, register the service and its endpoints:

openstack service create --name protector \
  --description "Trilio Site Recovery" \
  disaster-recovery

openstack endpoint create --region RegionOne \
  protector public http://EXTERNAL_HOST:8788/v1/%\(tenant_id\)s

openstack endpoint create --region RegionOne \
  protector internal http://INTERNAL_HOST:8788/v1/%\(tenant_id\)s

openstack endpoint create --region RegionOne \
  protector admin http://INTERNAL_HOST:8788/v1/%\(tenant_id\)s

Option B — Juju Charms

The charm installs the trilio-site-recovery Debian package from your Trilio apt repository, renders /etc/protector/protector.conf from relation data, and supervises protector-api and protector-engine as systemd services. There is no separate migration or registration step: once the mandatory relations are satisfied the charm creates and migrates the schema, registers the Keystone service and endpoints, opens port 8788 and starts both services.

Juju version. Every command below is Juju 2.9 syntax, which is what charmed Caracal clouds most often run. On Juju 3.x, juju run-action <unit> <action> --wait becomes juju run <unit> <action>, and juju run --unit becomes juju exec --unit. Check juju --version first: on 2.9 juju run means "execute a command" and rejects an action name with ERROR You must specify a target, either through --all, --machine, --application or --unit — an error that never mentions actions.

Step 1 — Deploy the charm

juju deploy trilio-site-recovery-for-openstack trilio-site-recovery \
    --num-units 1 --to lxd:<MACHINE_ID> \
    --config openstack-origin=cloud:jammy-caracal

The Charmhub name is trilio-site-recovery-for-openstack. The second argument is an application alias, which keeps the shorter name for every later command. Deploying trilio-site-recovery on its own fails with ERROR resolving error: The Charm with the given name was not found in the Store.

No --channel is needed; the default resolves to stable.

Placement (--to) is not optional. Without it Juju asks the cloud for a new machine, which on MAAS means provisioning bare metal. Run juju machines first and pick one with capacity. Use a container on a machine (lxd:1), not the machine itself (1): setting openstack-origin runs apt-get dist-upgrade on whatever machine the unit lands on, which is not something to do to a control node by accident.

trilio-pkg-source can usually be omitted, since the charm already defaults to the Trilio repository. It is needed only when that repository requires a read token:

juju config trilio-site-recovery \
    trilio-pkg-source="deb [trusted=yes] https://<TOKEN>@apt.fury.io/<REPO>/ /"

Set openstack-origin to match the cloud, read from juju config keystone openstack-origin rather than copied from this page. openstack-origin=distro is also supported and needs no Ubuntu Cloud Archive mirror, which matters for air-gapped sites.

Step 2 — Choose a topology and add relations

Site Recovery needs a database and a message broker, and can use the cloud's own or its own. The charm is identical either way; the topology is purely which applications you relate to. It is expected to hold for the life of the deployment, because a mysql-router subordinate relates to exactly one cluster.

Both topologies need a mysql-router subordinate. This is not optional:

juju deploy mysql-router trilio-site-recovery-mysql-router \
    --channel 8.0/stable --revision 1154

Do not relate shared-db directly to the Galera cluster. mysql-innodb-cluster does expose a shared-db/mysql-shared endpoint, so juju add-relation trilio-site-recovery:shared-db mysql-innodb-cluster:shared-db is accepted, creates the database and user, and stops the unit reporting shared-db as missing — and then the deployment silently never starts. The direct relation hands out one fixed cluster member rather than routing to whichever member is currently primary; that member reports super_read_only=1, and schema creation dies with pymysql.err.OperationalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement'). All juju status shows is blocked services not running: protector-api, protector-engine, which points nowhere near the database. Every other charmed OpenStack service routes through a mysql-router subordinate for exactly this reason.

Shared topology — five relations:

juju add-relation trilio-site-recovery:shared-db \
                  trilio-site-recovery-mysql-router:shared-db
juju add-relation trilio-site-recovery-mysql-router:db-router \
                  mysql-innodb-cluster:db-router
juju add-relation trilio-site-recovery:amqp rabbitmq-server:amqp
juju add-relation trilio-site-recovery:identity-credentials \
                  keystone:identity-credentials
juju add-relation trilio-site-recovery:certificates vault:certificates

Dedicated topology — deploy its own database and broker first:

juju deploy mysql-innodb-cluster trilio-mysql \
    --channel 8.0/stable --revision 164 \
    --num-units 3 --to lxd:1,lxd:2,lxd:3

juju deploy rabbitmq-server trilio-rabbitmq \
    --channel 3.9/stable --revision 295 \
    --num-units 1 --to lxd:0

Then the same five relations with trilio-mysql:db-router and trilio-rabbitmq:amqp substituted, plus a sixth giving the dedicated Galera its certificate:

juju add-relation trilio-mysql:certificates vault:certificates

Three Galera units forming a quorum is by far the longest step — allow roughly 25 minutes for a dedicated deployment, most of it that cluster. trilio-mysql sits in maintenance and Site Recovery stays blocked on missing relations: shared-db throughout; neither is a failure. The cluster is ready when every unit reports Cluster is ONLINE with exactly one member in Mode: R/W.

Pin the revisions. mysql-innodb-cluster 179 and mysql-router 1178 both ship wheelhouses that fail to install on jammy — missing typing-extensions and tomli respectively for Python earlier than 3.11 — and the install hook dies. Match whatever revisions the cloud's own Galera and routers already run.

Two relations are deliberately absent from the lists above:

• identity-service must not be related. Keystone only creates catalogue entries for names in its built-in valid_services table, which has no protector. The charm marks identity-service optional and prefers identity-credentials, which alone registers 2 services, 6 endpoints, the protector user and the dr_site_admin role correctly.
• certificates is effectively required, not optional, against a TLS Keystone, which is the charmed default. Without it the unit blocks with keystone is TLS but no CA is available: relate to vault:certificates, set ssl-ca, or set verify-ssl=false. From release 1.0.7 this matters more, because TLS verification is genuinely enforced rather than silently disabled.

Optional: juju add-relation trilio-site-recovery:memcache memcached:cache gives keystonemiddleware a token cache.

Step 3 — Configure the storage backend

The pure-* options are inert while storage-driver remains at its mock default, so all four values have to move together:

juju config trilio-site-recovery \
    storage-driver=pure \
    use-mock-storage=false \
    use-mock-cinder=false \
    pure-primary-management-ip=<fa1-ip> \
    pure-primary-api-token=<fa1-token> \
    pure-secondary-management-ip=<fa2-ip> \
    pure-secondary-api-token=<fa2-token>

With storage-driver=pure the charm also installs python3-pure-client, the Pure SDK, from the same trilio-pkg-source. If it cannot be installed the unit blocks with python3-pure-client is not installed; the pure driver cannot work. That is deliberate: without the SDK both services run and the API answers while the engine does no storage work at all. The block clears by itself once the repository carries the package, because the next config-changed retries the install.

Leave the defaults for a hardware-free evaluation; the mock driver performs no real storage operations.

Step 4 — Verify the deployment

juju status trilio-site-recovery --relations
juju run-action trilio-site-recovery/leader health-check --wait

Expect active/idle, with healthy: "True" and both protector-api and protector-engine active.

The API address is the unit's address, not the machine's. Take it from the Units table of juju status, or from the catalogue with openstack endpoint list --service protector:

curl -s http://<unit-ip>:8788/ | python3 -m json.tool

Then confirm the service is usable rather than merely running. Install the CLI plugin as described below, then:

openstack dr site list

An empty result is the expected pass: with no sites registered it prints nothing and exits 0, which means the plugin loaded, Keystone authenticated, the catalogue resolved the endpoint and the API answered. Errors are loud.

Step 5 — Optional: the Horizon dashboard plugin

The Site Recovery panels in Horizon come from a separate subordinate charm that co-locates on the openstack-dashboard unit:

juju deploy trilio-site-recovery-for-openstack-dashboard \
    trilio-site-recovery-dashboard --channel latest/stable

juju add-relation trilio-site-recovery-dashboard:dashboard-plugin \
                  openstack-dashboard:dashboard-plugin

Being a subordinate it takes no --num-units and no placement; Juju places one unit on each dashboard unit. It reaches active reporting a panel count, for example Unit is ready (4 panels). It holds no credentials and needs no relation to the service, because the panels resolve the API from the Keystone catalogue using the logged-in user's own token. Skip it if the openstack dr CLI is enough.

Actions

Address actions to <app>/leader. Juju never reuses unit numbers, so a hard-coded <app>/0 fails with couldn't queue one action: "unit-<app>-0" not found after any remove-and-redeploy.

Upgrading

Do not re-run the installation steps. A package-only build is picked up with one action, which restarts the services and runs the migrations only if the version actually moved:

juju run-action trilio-site-recovery/leader upgrade-packages --wait

version=<x.y.z> installs an exact version and permits downgrades, which is the rollback path. Packages are per unit, so run it on each unit of a multi-unit deployment, and separately on trilio-site-recovery-dashboard if the plugin is deployed.

A new charm revision is ordinary Juju:

juju refresh trilio-site-recovery --channel latest/stable

Neither disturbs the Keystone catalogue. Juju fires upgrade-charm for a refresh and stop only when a unit goes away, and it is the stop hook that deregisters. Config, schema, DR history and relations all survive.

Removing

juju remove-application trilio-site-recovery
juju remove-application trilio-site-recovery-mysql-router

Add juju remove-application trilio-site-recovery-dashboard only if the plugin was deployed, and trilio-mysql and trilio-rabbitmq only on the dedicated topology. Removal deregisters the Keystone service and endpoints; the protector user and the dr_site_admin role are deliberately left in place.

On the shared topology two things survive that affect a reinstall. The schema stays on the cloud's Galera unless you run purge-database before removing the application. And the mysql-router registration stays in the cluster's metadata, which will block the next deployment onto the same address with a router instance named '...' has been previously configured — the unit then sits at MySQL Router not yet bootstrapped while Site Recovery reports missing relations: shared-db, pointing nowhere near the cause. Clear it after the removals, from the Juju client, against the member reporting Mode: R/W:

PW=$(juju run --unit <galera>/leader "leader-get cluster-password" | tr -d '\r\n')
juju ssh <galera>/<rw-unit> -- "sudo mysqlsh --uri clusteruser:'$PW'@<rw-member-ip> \
    --cluster -e \"cluster.remove_router_metadata('<address>::<router-name>')\""

Use clusteruser over the member's own IP, not root@localhost, and the snake_case remove_router_metadata. leader-get is a Juju hook tool and does not exist inside juju ssh, which is why the password is read on the client. Success prints nothing, so verify by querying mysql_innodb_cluster_metadata.routers. Redeploying the router under a different application name avoids the write entirely. On the dedicated topology none of this applies: the schema and the metadata row are destroyed along with trilio-mysql.

Option C — Red Hat OpenStack Services on OpenShift (RHOSO18)

For RHOSO18 environments, protector-api and protector-engine are deployed as Kubernetes Deployments. You can use raw manifests (kubectl) or install via OLM.

Path C1 — kubectl with raw manifests

Step 1 — Create the namespace

kubectl create namespace trilio-site-recovery

Step 2 — Apply RBAC and service account

kubectl apply -f deploy/openshift/manifests/serviceaccount.yaml

This creates the trilio-site-recovery ServiceAccount, Role, and RoleBinding in the trilio-site-recovery namespace.

Step 3 — Create the ConfigMap for protector.conf

Create a ConfigMap holding your site's protector.conf. The minimal required sections are shown below — adjust all values for your environment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: protector-config
  namespace: trilio-site-recovery
data:
  protector.conf: |
    [DEFAULT]
    host = rhoso-site-a

    [api]
    bind_host = 0.0.0.0
    bind_port = 8788
    workers = 4

    [database]
    connection = mysql+pymysql://protector:PROTECTOR_PASS@DB_HOST:3306/protector

    [storage]
    storage_driver = pure

    [pure_storage]
    primary_management_ip = 10.0.1.10
    primary_api_token = YOUR_PRIMARY_TOKEN
    secondary_management_ip = 10.0.2.10
    secondary_api_token = YOUR_SECONDARY_TOKEN
    verify_ssl = false

kubectl apply -f protector-configmap.yaml

Step 4 — Run the database init Job

Apply the DB initialisation Job (schema creation via protector-manage db upgrade):

kubectl apply -f deploy/openshift/manifests/job-keystone-init.yaml
kubectl wait --for=condition=complete job/protector-db-init \
  -n trilio-site-recovery --timeout=120s

Step 5 — Deploy the API and engine

kubectl apply -f deploy/openshift/manifests/

This applies all manifests in the directory, including the Deployments for protector-api and protector-engine.

Step 6 — Expose the API service

kubectl apply -f deploy/openshift/manifests/service.yaml

This creates the trilio-site-recovery-api ClusterIP Service on port 8788 and an OpenShift Route with TLS edge termination.

Step 7 — Verify pods are running

kubectl get pods -n trilio-site-recovery
kubectl logs -n trilio-site-recovery deployment/protector-api
kubectl logs -n trilio-site-recovery deployment/protector-engine

Path C2 — OLM operator bundle

# Install from the operator bundle directory
kubectl apply -f deploy/openshift/bundle/
kubectl apply -f deploy/openshift/operator/

The OLM subscription will manage upgrades through the operator lifecycle. Consult the bundle's metadata/annotations.yaml for the correct channel name.

Installing the CLI plugin

The openstack dr CLI is a separate package, python3-triliositerecovery. Installing the server package does not install it, and it has to live in the same Python environment as the OpenStack client.

On a Juju deployment there is nothing to do. The charm installs it alongside the service (install-client defaults to true), which is why the verification in Option B Step 4 works on the unit with no setup.

On any other node — an operator workstation, a jump host — install it from the same Trilio apt repository. It does not pull in the service:

apt-get install python3-openstackclient python3-triliositerecovery

It will not work through the openstackclients snap. Snaps are confined and ship their own Python, so a plugin installed as a Debian package is invisible to them. The only symptom is openstack: 'dr site list' is not an openstack command, which gives no hint of the cause. Check with command -v openstack: if it reports /snap/bin/openstack, install the Debian client as above and call /usr/bin/openstack explicitly, since /snap/bin usually comes first in PATH. Measured on one host with both installed, /usr/bin/openstack registered 39 dr subcommands and /snap/bin/openstack registered none.

From a source checkout, for development only:

pip install -e ./client

Verify the plugin loaded:

openstack dr --help

All configuration is managed through juju config. Never edit /etc/protector/protector.conf directly — the charm renders it from relation data and config values, and overwrites any manual edits on every config-changed event.

Configuration option names use hyphens when passed to juju config (e.g. pure-primary-management-ip), which the charm translates to underscore-separated keys in protector.conf.

Core API options ([api] section)

Storage driver ([storage] and [pure_storage] sections)

The storage driver is the most critical production setting:

Production requirement: storage-driver defaults to mock. You must set storage-driver=pure and configure all pure-* options before the service can replicate real workloads.

Snapshot retention ([pure_storage] section)

These options control how many recovery points are available on the secondary site:

Set these in protector.conf via config-flags (see below), for example:

juju config trilio-site-recovery config-flags="pure_storage:retention_all_for_hours=48,pure_storage:retention_days=14"

Engine options ([engine] section)

Keystone and identity

Use identity-credentials, not identity-service. The identity-service relation only works if Keystone recognizes protector in its valid_services table, which is not the default. The identity-credentials relation is preferred and works out of the box.

TLS

For cross-site TLS verification in protector.conf, the [service_credentials] section supports:

[service_credentials]
cafile = /etc/protector/ca.pem   # PEM CA bundle for cross-site calls
insecure = false                  # Set true only for dev/test

Arbitrary config overrides

The config-flags charm option lets you set any protector.conf option not exposed as a dedicated charm config. Use section:key=value format with a colon separator (not a dot — dot separators fail silently):

juju config trilio-site-recovery \
  config-flags="engine:cross_site_timeout=60,DEFAULT:rpc_conn_pool_size=50"

OpenStack origin

For air-gapped or offline sites, set:

juju config trilio-site-recovery openstack-origin=distro

For Ubuntu Cloud Archive pockets (match the OpenStack release of the cloud being protected):

juju config trilio-site-recovery openstack-origin=cloud:jammy-caracal

Name resolution for cross-site connectivity

If the two sites cannot resolve each other's hostnames via DNS, use extra-hosts:

juju config trilio-site-recovery extra-hosts="
10.80.65.60  keystone.site-a.example.com
10.32.65.166 keystone.site-b.example.com"

This injects entries into /etc/hosts on the unit. Prefer fixing DNS at the infrastructure layer (MAAS upstream forwarder) for production.

Once the services are deployed and healthy, the primary workflow is:

1. Register both sites
2. Create tenant mappings
3. Create a Protection Group
4. Add VMs to the Protection Group
5. Monitor replication health
6. Execute failover, failback, or test failover as needed

All operations use the openstack dr CLI. Source your OpenStack RC file before running any openstack dr commands.

Register sites

Register the primary site (run from either site):

openstack dr site create site-a \
  --site-type PRIMARY \
  --auth-url https://keystone.site-a.example.com:5000 \
  --region RegionOne \
  --service-username protector \
  --service-password <protector-service-password> \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default \
  --service-user-id <protector-user-id-on-site-a>

Register the secondary site:

openstack dr site create site-b \
  --site-type SECONDARY \
  --auth-url https://keystone.site-b.example.com:5000 \
  --region RegionOne \
  --service-username protector \
  --service-password <protector-service-password-on-site-b> \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default \
  --service-user-id <protector-user-id-on-site-b>

Why --service-user-id? Trust creation resolves the service user by ID. Without it, the first tenant mapping fails with Cannot resolve service user ID. Each site has its own Keystone, so read the ID from the site being registered:

Copy

openstack user show protector -f value -c id

The --auth-url value must match keystone_authtoken.auth_url in that site's protector.conf exactly (only trailing slash and /v3 suffix are normalized). The engine identifies which site it is running on by comparing these as strings.

List registered sites:

openstack dr site list

Create tenant mappings

Tenant mappings translate project UUIDs between sites (each site's Keystone assigns different UUIDs to the same tenant):

openstack dr tenant mapping create \
  --local-tenant <project-id-on-this-site> \
  --remote-site <site-b-id> \
  --remote-tenant <project-id-on-site-b>

Validate an existing mapping:

openstack dr tenant mapping validate <mapping-id>

Create a Protection Group

openstack dr protection group create my-pg \
  --primary-site <site-a-id> \
  --secondary-site <site-b-id> \
  --replication-type async \
  --volume-type <replicated-volume-type-id>

For near-zero RPO with Pure Storage ActiveDR (requires Purity//FA 6.0+):

openstack dr protection group create my-pg \
  --primary-site <site-a-id> \
  --secondary-site <site-b-id> \
  --replication-type active_dr

--volume-type is required for async replication. The volume type must have replication_enabled='<is> True' set as an extra-spec, or the Protection Group cannot accept members.

Add VMs to a Protection Group

Pre-check a VM's eligibility without side effects:

openstack dr protection group member validate my-pg \
  --instance <nova-instance-id>

Add a VM:

openstack dr protection group member add my-pg \
  --instance <nova-instance-id> \
  --boot-order 1

All VMs must boot from a Cinder volume (not ephemeral storage). Instances without Cinder-backed boot volumes are rejected with InstanceNotCinderBoot (HTTP 422). All attached volumes must use a volume type with replication_enabled='<is> True'; non-replicated volumes are rejected with VolumeTypeNotReplicated (HTTP 422).

Set boot order for sequenced startup/shutdown during failover:

openstack dr protection group member set my-pg \
  --member <member-id> \
  --boot-order 2

VMs sharing the same boot order value are processed in parallel; lower-numbered tiers complete before higher-numbered tiers begin.

Monitor replication health

openstack dr replication health <pg-id>

The response includes sync state, lag, RPO compliance, failover readiness (can_failover), and available recovery points. Use can_failover as the programmatic readiness check before triggering a failover.

Run a test failover (non-disruptive DR validation)

A test failover clones replicated volumes from the secondary site and boots isolated test VMs without affecting production:

# Using automatic network assignment
openstack dr test failover <pg-id> --auto-network

# Or with explicit network mapping
openstack dr test failover <pg-id> \
  --network-mapping <primary-net-id>:<secondary-test-net-id>

Either --auto-network or --network-mapping is required.

Clean up test resources when done:

openstack dr test failover cleanup <pg-id>

To clean up a specific test failover operation:

openstack dr test failover cleanup <pg-id> --operation <operation-id>

Execute planned failover

openstack dr failover <pg-id> --failover-type planned

Planned failover gracefully shuts down VMs and takes a final snapshot for zero data loss.

For unplanned failover (skips shutdown, accepts data loss up to RPO):

openstack dr failover <pg-id> --failover-type unplanned

To block until the operation completes:

openstack dr failover <pg-id> --failover-type planned --wait

To restore from a specific recovery point:

openstack dr failover <pg-id> \
  --failover-type planned \
  --recovery-point <recovery-point-id>

Execute failback

After restoring operations on the secondary site, fail back to the original primary:

openstack dr failback <pg-id>

Track operations

Every DR action returns an operation ID. Track progress:

openstack dr operation show <operation-id>

List all operations for a Protection Group:

openstack dr operation list --protection-group <pg-id>

Cancel an in-progress operation:

openstack dr operation cancel <operation-id>

Service management (Juju charm)

Restart services via Juju actions:

# Juju 3.x
juju run trilio-site-recovery/leader restart-services

# Juju 2.9
juju run-action trilio-site-recovery/leader restart-services --wait

View logs on the unit:

juju ssh trilio-site-recovery/0 -- tail -f /var/log/protector/protector-api.log
juju ssh trilio-site-recovery/0 -- tail -f /var/log/protector/protector-engine.log

Run a database schema sync manually (usually not required — the charm handles this):

# Juju 3.x
juju run trilio-site-recovery/leader db-sync

# Juju 2.9
juju run-action trilio-site-recovery/leader db-sync --wait

Example 1 — Verify API version discovery

Check the API is reachable and confirm the supported microversion range (no authentication required):

curl -s http://10.21.67.100:8788/ | python3 -m json.tool

Expected output:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://10.21.67.100:8788/v1/"
        }
      ]
    }
  ]
}

Example 2 — Check service health

curl -s http://10.21.67.100:8788/healthz

Expected output when healthy:

{"status": "ok", "checks": {"database": "ok"}}

Expected output when the database is unavailable (HTTP 503):

{"status": "unhealthy", "checks": {"database": "unavailable (OperationalError)"}}

Example 3 — Register both sites and verify

# Source credentials for site A
source site-a-openrc.sh

# Get the protector service user ID on site A
PROTECTOR_USER_ID=$(openstack user show protector -f value -c id)

# Register site A as the primary
openstack dr site create site-a \
  --site-type PRIMARY \
  --auth-url https://keystone.site-a.example.com:5000 \
  --region RegionOne \
  --service-username protector \
  --service-password MyServicePass123 \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default \
  --service-user-id "${PROTECTOR_USER_ID}"

# Register site B as the secondary
openstack dr site create site-b \
  --site-type SECONDARY \
  --auth-url https://keystone.site-b.example.com:5000 \
  --region RegionOne \
  --service-username protector \
  --service-password MyServicePassSiteB123 \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default \
  --service-user-id "<protector-user-id-on-site-b>"

# List registered sites
openstack dr site list

Expected output:

+--------------------------------------+--------+-----------+--------+---------------------------------------------+------------+
| ID                                   | Name   | Type      | Status | Auth URL                                    | Region     |
+--------------------------------------+--------+-----------+--------+---------------------------------------------+------------+
| a1b2c3d4-...                         | site-a | PRIMARY   | ACTIVE | https://keystone.site-a.example.com:5000    | RegionOne  |
| e5f6a7b8-...                         | site-b | SECONDARY | ACTIVE | https://keystone.site-b.example.com:5000    | RegionOne  |
+--------------------------------------+--------+-----------+--------+---------------------------------------------+------------+

Example 4 — Create a replicated volume type and Protection Group

# Create a Cinder volume type with replication enabled
openstack volume type create replicated-vol \
  --property replication_enabled='<is> True' \
  --property replication_type='<in> async'

VOLTYPE_ID=$(openstack volume type show replicated-vol -f value -c id)

# Create a Protection Group
SITE_A_ID=$(openstack dr site show site-a -f value -c id)
SITE_B_ID=$(openstack dr site show site-b -f value -c id)

openstack dr protection group create prod-pg \
  --primary-site "${SITE_A_ID}" \
  --secondary-site "${SITE_B_ID}" \
  --replication-type async \
  --volume-type "${VOLTTYPE_ID}"

Expected output:

+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| id                 | 9f8e7d6c-...                         |
| name               | prod-pg                              |
| status             | active                               |
| primary_site_id    | a1b2c3d4-...                         |
| secondary_site_id  | e5f6a7b8-...                         |
| replication_type   | async                                |
| created_at         | 2025-01-15T10:30:00Z                 |
+--------------------+--------------------------------------+

Example 5 — Validate and add a VM

PG_ID="9f8e7d6c-..."
INSTANCE_ID="vm-uuid-here"

# Pre-check eligibility without side effects
openstack dr protection group member validate "${PG_ID}" \
  --instance "${INSTANCE_ID}"

# If valid, add the instance
openstack dr protection group member add "${PG_ID}" \
  --instance "${INSTANCE_ID}" \
  --boot-order 1

If the instance uses ephemeral storage, member validate returns:

+------------------+-------------------------------------------------------+
| Field            | Value                                                 |
+------------------+-------------------------------------------------------+
| valid            | No                                                    |
| errors           | [InstanceNotCinderBoot] Instance is not a Cinder boot |
+------------------+-------------------------------------------------------+

Example 6 — Monitor replication health

openstack dr replication health "${PG_ID}"

Expected output (abbreviated):

+----------------------+------------------------------------------+
| Field                | Value                                    |
+----------------------+------------------------------------------+
| protection_group_id  | 9f8e7d6c-...                             |
| protection_group_name| prod-pg                                  |
| failover_validation  | {can_failover: true, blockers: []}       |
+----------------------+------------------------------------------+

For programmatic checks, use the REST API:

TOKEN=$(openstack token issue -f value -c id)
TENANT_ID=$(openstack project show myproject -f value -c id)

curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  http://10.21.67.100:8788/v1/${TENANT_ID}/protection-groups/${PG_ID}/replication-health \
  | python3 -m json.tool

Example 7 — Trigger a planned failover and poll until complete

# Initiate the failover
RESPONSE=$(curl -s \
  -X POST \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"failover": {"failover_type": "planned"}}' \
  http://10.21.67.100:8788/v1/${TENANT_ID}/protection-groups/${PG_ID}/action)

OPERATION_ID=$(echo "${RESPONSE}" | python3 -c "import json,sys; print(json.load(sys.stdin)['operation_id'])")
echo "Operation ID: ${OPERATION_ID}"

# Poll until complete
while true; do
  STATUS=$(curl -s \
    -H "X-Auth-Token: ${TOKEN}" \
    http://10.21.67.100:8788/v1/${TENANT_ID}/operations/${OPERATION_ID} \
    | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('status'))")
  echo "Status: ${STATUS}"
  [[ "${STATUS}" == "completed" || "${STATUS}" == "failed" ]] && break
  sleep 10
done

Or using the CLI with --wait:

openstack dr failover "${PG_ID}" --failover-type planned --wait

Example 8 — Non-disruptive test failover and cleanup

# Run a test failover using automatic network assignment
openstack dr test failover "${PG_ID}" --auto-network --wait

# Inspect the test VMs on the secondary site (they have a 'test-' prefix)
source site-b-openrc.sh
openstack server list

# Clean up — removes all test VMs and cloned volumes
openstack dr test failover cleanup "${PG_ID}"

To force cleanup even if some resources fail to delete:

openstack dr test failover cleanup "${PG_ID}" --force

Example 9 — List and inspect operations

# List all operations for a Protection Group
openstack dr operation list --protection-group "${PG_ID}"

# Show details for a specific operation
openstack dr operation show "${OPERATION_ID}"

Issue: Unit stuck in waiting — charm never becomes active

Symptom: juju status shows the unit in waiting/idle indefinitely.

Likely cause: One or more required relations (shared-db, amqp, identity-credentials) are not yet established, or the related charm is not yet ready.

Fix:

juju status trilio-site-recovery --relations

Confirm all three relations show joined. If identity-credentials is missing, verify you added the relation to keystone, not just identity-service.

Issue: mysql-router or mysql-innodb-cluster fails to install on Jammy

Symptom: The mysql-router or mysql-innodb-cluster unit enters an error state with a MarkupSafe or wheel-related Python error.

Likely cause: mysql-router revision 1178 or mysql-innodb-cluster revision 179 ships a broken wheelhouse for Ubuntu Jammy.

Fix: Pin to the tested revisions:

# In deploy.sh, use:
--router-revision 1154
--dedicated-db-revision 164
--dedicated-amqp-revision 295

If already deployed:

juju refresh mysql-router --revision 1154
juju refresh mysql-innodb-cluster --revision 164

Issue: apt.fury.io returns 401 Unauthorized

Symptom: The unit enters blocked state with a message about the Trilio apt repository rejecting the request.

Likely cause: The trilio-pkg-source value is missing a read token for a private Gemfury repository.

Fix:

juju config trilio-site-recovery \
  trilio-pkg-source="deb [trusted=yes] https://<read-token>@apt.fury.io/<repo>/ /"

Issue: openstack dr commands not found after installation

Symptom: openstack dr --help returns No such command or the dr namespace does not appear.

Likely cause: The CLI plugin (python3-triliositerecovery) is not installed, or you installed it via the openstackclient snap which is incompatible with the plugin.

Fix: Ensure the package is installed via apt or pip (not snap):

# Via apt
apt-get install python3-triliositerecovery

# Or from the repository (development)
pip install -e ./client

Do not use pip install -e . from the repository root — that does not install the CLI.

Issue: Role dr_operator does not exist on site <site>

Symptom: Trust creation fails with this message when a tenant first attempts a DR operation.

Likely cause: The dr_operator or dr_site_admin Keystone role does not exist on one or both sites.

Fix: Create both roles on both sites:

openstack role create dr_operator
openstack role create dr_site_admin

Then grant them to DR users:

openstack role add --project <project> --user <dr-user> member
openstack role add --project <project> --user <dr-user> dr_operator

Issue: InstanceNotCinderBoot when adding a VM

Symptom: openstack dr protection group member add returns HTTP 422 with InstanceNotCinderBoot.

Likely cause: The Nova instance uses ephemeral storage for its boot disk rather than a Cinder volume.

Fix: The instance cannot be protected until it is rebuilt with a Cinder-backed boot volume. Use openstack dr protection group member validate to pre-check eligibility:

openstack dr protection group member validate <pg-id> --instance <instance-id>

Issue: VolumeTypeNotReplicated when adding a VM

Symptom: openstack dr protection group member add returns HTTP 422 with VolumeTypeNotReplicated.

Likely cause: One or more Cinder volumes attached to the instance use a volume type that does not have replication_enabled='<is> True' as an extra-spec.

Fix: Retype the volume to a replicated type, or create a new volume with the correct type:

# Check current extra-specs on the volume's type
openstack volume type show <volume-type-id>

# Create a replicated type if one does not exist
openstack volume type create replicated-vol \
  --property replication_enabled='<is> True' \
  --property replication_type='<in> async'

Issue: Tenant mapping creates locally but does not sync to the remote site

Symptom: openstack dr tenant mapping list shows the mapping as pending_validation or the remote site reports it does not exist.

Likely cause: Port 8788 is not reachable from this site to the remote site (or vice versa). Cross-site metadata sync requires bidirectional connectivity on port 8788.

Fix: Verify bidirectional reachability:

# From site A to site B
curl -s http://<site-b-protector-api>:8788/healthz

# From site B to site A
curl -s http://<site-a-protector-api>:8788/healthz

Both should return {"status": "ok", ...}. Open firewall rules in both directions on port 8788.

Issue: Cannot resolve service user ID on first tenant mapping

Symptom: The first tenant mapping fails with this error.

Likely cause: The site was registered without --service-user-id.

Fix: Update the site registration with the correct user ID:

# Get the service user ID on the target site
openstack user show protector -f value -c id

# Update the site
openstack dr site set <site-id> \
  --service-user-id <protector-user-id-on-that-site>

Issue: FailoverNotReady when triggering failover

Symptom: openstack dr failover returns HTTP 422 with FailoverNotReady.

Likely cause: The Protection Group is not ready for failover — replication may be lagging or the health check has blockers.

Fix: Check replication health and review the failover_validation.blockers field:

openstack dr replication health <pg-id>

Address any blockers listed (e.g., replication lag exceeding RPO, missing resource mappings). Use openstack dr protection group mapping list <pg-id> --unmapped to identify unmapped resources.

Issue: config-flags changes have no effect

Symptom: A config-flags value is set but the resulting protector.conf does not contain the expected option.

Likely cause: The config-flags value uses dot separators (e.g., DEFAULT.rpc_response_timeout=120) instead of colon separators.

Fix: Use colon separators in section:key=value format:

# Correct
juju config trilio-site-recovery config-flags="DEFAULT:rpc_conn_pool_size=50,engine:cross_site_timeout=60"

# Wrong — dot separators fail silently
juju config trilio-site-recovery config-flags="DEFAULT.rpc_conn_pool_size=50"

Issue: /healthz returns 503 after deployment

Symptom: The health endpoint returns {"status": "unhealthy", "checks": {"database": "unavailable (...)"}} after the charm becomes active.

Likely cause: The protector-api service cannot reach the database. The [database] connection option may be missing, misconfigured, or point to an unreachable host.

Fix: Verify the database relation is active and inspect the rendered config:

# Juju 3.x
juju run trilio-site-recovery/leader show-config

# Juju 2.9
juju run-action trilio-site-recovery/leader show-config --wait

Check that the [database] connection string in the rendered protector.conf points to a reachable MariaDB/MySQL instance. In Juju charm deployments, the database is provisioned automatically via the shared-db relation — confirm this relation is joined.