Guide

Juju Charms

Installation guide for Canonical Juju Charms

Overview

This page walks you through deploying Trilio Site Recovery for OpenStack using its Juju charm on a Charmed OpenStack cluster. The charm installs and manages two systemd services — protector-api and protector-engine — via the trilio-site-recovery Debian package, renders protector.conf from relation data, migrates the database schema, and registers with Keystone automatically once all required relations are satisfied. This is the only validated, customer-facing deployment method for the tech preview; no other deployment path is documented for end users.

Prerequisites

Before deploying the charm, confirm that each of the following is in place on both the primary and secondary OpenStack sites.

OpenStack and infrastructure

Two independent OpenStack deployments at Victoria or later
Nova, Cinder (with a volume type carrying replication_enabled='<is> True'), Keystone, and Neutron running on each site
Pure Storage FlashArray arrays with an async replication connection between sites
MariaDB 10.3 or later, or MySQL 8.0 (a Charmed OpenStack cloud running mysql-innodb-cluster provides MySQL 8.0)
RabbitMQ (oslo.messaging)
Network connectivity between the two sites on the following ports: Keystone 5000, Nova 8774, Cinder 8776, and protector-api 8788 (bidirectional) — one-way connectivity on port 8788 leaves tenant mappings inconsistent

Juju and charm requirements

Juju 2.9 or later (Juju 3.x is also supported; note that action syntax differs — see Usage)
A MAAS-backed Juju model with machines pre-provisioned on Ubuntu Jammy (22.04) or Noble (24.04); --to placement is required at deploy time to avoid Juju provisioning additional machines
Access to the Trilio apt repository at https://apt.fury.io/trilio-os-site-recovery-master/ (or a private Gemfury mirror with a read token in the URL)
python-openstackclient installed via pip on any machine from which you will run openstack dr commands — do not install via the openstackclient snap, as the snap is incompatible with the triliositerecovery CLI plugin

Keystone roles

The service never creates Keystone roles — it only looks them up. Both of the following roles must exist on both sites before any cross-site operation:

dr_operator — delegated in per-tenant Keystone trusts; must be granted to every DR-enabled project and to the protector service account on each site
dr_site_admin — gates site-level failover API calls; must be granted to site-level operator accounts

Trust creation fails and names the missing role if either is absent.

If you set default_trust_roles through config-flags, check it before upgrading. The charm renders default_trust_roles = member,dr_operator, but a config-flags entry for the same key overrides it. A deployment carrying config-flags="service_credentials:default_trust_roles=member" will keep delegating member alone after an upgrade, dr_operator will never reach the trust, and cross-site sync calls will be refused by policy. Remove the override unless you have a specific reason to keep it.

Infrastructure charm revisions to avoid

The following charm revisions ship a Python wheelhouse that fails to install on Jammy (it omits tomli for Python versions below 3.11), and must not be used:

mysql-innodb-cluster revision 179
mysql-router revision 1178

Pin to mysql-innodb-cluster revision 164 and mysql-router revision 1154. Pin rabbitmq-server to revision 295 on the 3.9/stable channel.

CLI plugin

The openstack dr CLI ships as the Debian package python3-triliositerecovery from the Trilio apt repository. It is not published to PyPI — pip install triliositerecovery cannot work, because triliositerecovery is only the distribution name inside the source tree, not a package on any index.

On a unit deployed by this charm the CLI is already present: the install-client option defaults to true and installs that package for you. You only need to install it by hand on other machines — a workstation or bastion host from which you intend to run openstack dr:

apt-get install python3-triliositerecovery

If you are working from a source checkout, the editable install is the developer path:

pip install -e ./client

Installing the server package (pip install -e . from the repository root) does not install the CLI.

Installation

Prerequisites

Before you deploy, confirm the following on both sites:

Two independent OpenStack deployments ≥ Victoria, each with Nova, Cinder, Keystone, and Neutron.
A Cinder volume type with replication_enabled='<is> True' as an extra-spec on both sites.
Pure Storage FlashArray arrays with an async replication connection between the sites (or the mock storage driver for lab use).
MariaDB/MySQL ≥ 10.3 and RabbitMQ available in the Juju model.
Network connectivity in both directions on port 8788 between the two sites. One-way connectivity leaves metadata sync in an inconsistent state.
The Keystone roles dr_operator and dr_site_admin must exist on both sites before any cross-site operation. The service looks them up; it does not create them. Trust creation fails immediately if either role is absent.
python-openstackclient installed via pip (not the snap) on every machine from which you will run openstack dr commands. The openstackclient snap is incompatible with the CLI plugin.

Create the required Keystone roles on each site:

openstack role create dr_operator
openstack role create dr_site_admin

Grant roles to each DR user on their project:

# member + dr_operator on the tenant project (repeat for every DR-enabled project)
openstack role add --user <dr-user> --project <project> member
openstack role add --user <dr-user> --project <project> dr_operator

Step 1 — Pin infrastructure charm revisions

Before deploying, pin the MySQL and RabbitMQ charm revisions. Revisions 179 (mysql-innodb-cluster) and 1178 (mysql-router) ship wheelhouses that fail to install on Jammy.

# Deploy mysql-innodb-cluster at the known-good revision
juju deploy mysql-innodb-cluster --revision 164 --to <machine-id> \
  -n 3 trilio-mysql

# Deploy rabbitmq-server at the known-good revision
juju deploy rabbitmq-server --revision 295 --to <machine-id> trilio-rabbitmq

If you are sharing an existing mysql-innodb-cluster and rabbitmq-server with the rest of the cloud (the "shared topology"), skip this step and use the existing application names in the relation commands below.

Step 2 — Deploy the service charm

Deploy the trilio-site-recovery-for-openstack charm. The --base ubuntu@22.04 flag is required; without it, a Noble host creates a Noble container where the mysql-router subordinate fails with a MarkupSafe wheel mismatch. The --to placement flag is required to target a specific machine and avoid Juju provisioning a new one.

juju deploy trilio-site-recovery-for-openstack trilio-site-recovery \
  --base ubuntu@22.04 \
  --to <machine-id> \
  --config storage-driver=pure \
  --config pure-primary-management-ip=<primary-fa-ip> \
  --config pure-primary-api-token=<primary-fa-token> \
  --config pure-secondary-management-ip=<secondary-fa-ip> \
  --config pure-secondary-api-token=<secondary-fa-token>

For a lab deployment using the mock storage driver (no FlashArray required), omit the pure-* options and leave storage-driver=mock.

Step 3 — Deploy the mysql-router subordinate

Pin mysql-router to revision 1154 (the tested, known-good revision):

juju deploy mysql-router --revision 1154 trilio-site-recovery-mysql-router

Step 4 — Add relations

Relate the charm to its infrastructure. The database and database user are created automatically through the shared-db relation — you do not need to create them manually or supply admin credentials.

# Database
juju integrate trilio-site-recovery:shared-db \
  trilio-site-recovery-mysql-router:shared-db
juju integrate trilio-site-recovery-mysql-router:db-router \
  trilio-mysql:db-router

# Message broker
juju integrate trilio-site-recovery:amqp trilio-rabbitmq:amqp

# Keystone (preferred relation — works even when Keystone does not know the
# protector service type; the charm registers its own service catalogue entries)
juju integrate trilio-site-recovery:identity-credentials \
  keystone:identity-credentials

Once all three relations are satisfied the charm:

Renders /etc/protector/protector.conf.
Runs the database schema migration (protector-manage db sync).
Registers the service and endpoints with Keystone.
Starts protector-api and protector-engine.

No separate migration or Keystone registration step is needed.

Step 5 — Verify the deployment

Check that both units reached active:

juju status trilio-site-recovery

Expected output (abbreviated):

App                     Version  Status  Scale  Charm                                    ...
trilio-site-recovery             active      1  trilio-site-recovery-for-openstack       ...

Unit                       Workload  Agent  Machine  ...
trilio-site-recovery/0*    active    idle   <id>     ...

Run the built-in health check action:

# Juju 3.x
juju run trilio-site-recovery/leader health-check

# Juju 2.9
juju run-action trilio-site-recovery/leader health-check --wait

Confirm the API is responding with version discovery (no authentication required):

curl http://<unit-ip>:8788/

Expected response:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://<unit-ip>:8788/v1/"}]
    }
  ]
}

Step 6 — Install the CLI on operator workstations

On any unit deployed by this charm, openstack dr is already installed because install-client defaults to true, which installs the python3-triliositerecovery Debian package automatically.

For other machines from which you want to run openstack dr commands (for example, your local workstation or a bastion host), install the CLI manually:

# Add the Trilio apt repository
echo 'deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /' \
  | sudo tee /etc/apt/sources.list.d/trilio-site-recovery.list
sudo apt-get update

# Install the CLI plugin
sudo apt-get install python3-triliositerecovery

# python-openstackclient must come from pip, NOT the snap
pip install python-openstackclient

Important: Do not use the openstackclient snap. It is incompatible with the triliositerecovery CLI plugin. Install python-openstackclient via pip.

For a source checkout (developer path only):

git clone <repo>
pip install python-openstackclient
pip install -e ./client

Verify the plugin is registered:

openstack dr --help

Step 7 — Deploy the optional Horizon dashboard plugin

The dashboard plugin is a subordinate charm that co-locates on the openstack-dashboard unit. It is independent of the service charm — skip this step if you prefer the CLI.

juju deploy trilio-site-recovery-for-openstack-dashboard \
  trilio-site-recovery-dashboard

juju integrate trilio-site-recovery-dashboard:dashboard-plugin \
  openstack-dashboard:dashboard-plugin

The charm installs python3-triliositerecovery-dashboards from the Trilio apt repository and places the panel files in Horizon's local/enabled directory. The dashboard node must be able to reach the Trilio apt repository.

Repeat on the secondary site

Repeat Steps 1–6 on the secondary (DR) OpenStack site. Each site runs its own independent protector-api and protector-engine; cross-site coordination is performed by the engine using Keystone trusts, not by sharing a database.

Configuration

The charm renders /etc/protector/protector.conf automatically from relation data and config values. Do not edit this file directly — the charm overwrites manual changes on every config-changed event. Set all options via juju config:

juju config trilio-site-recovery <option>=<value>

Package source

Option	Default	Description
`trilio-pkg-source`	`deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /`	Apt source line written to `/etc/apt/sources.list.d/trilio-site-recovery.list`. For private Gemfury repos, include a read token in the URL.
`openstack-origin`	`distro`	OpenStack library source. Use `cloud:jammy-caracal` (or the appropriate pocket) to match the OpenStack release of the cloud being protected. `distro` means the Yoga-era oslo libraries on Jammy.

API

Option	Default	Description
`bind-port`	`8788`	TCP port `protector-api` listens on. Maps to `bind_port` in `[api]`.
`api-workers`	`2`	Gunicorn sync worker count. The charm default is 2 (the compiled-in oslo.config default is 4; charmed deployments render 2 unless overridden).
`region`	`RegionOne`	OpenStack region for Keystone endpoint registration.

Storage driver

Option	Default	Description
`storage-driver`	`mock`	`mock` for development (no hardware required) or `pure` for Pure Storage FlashArray. Production deployments must set `pure`. Maps to `storage_driver` in `[storage]`.
`pure-primary-management-ip`	(empty)	Primary FlashArray management IP or hostname. Required when `storage-driver=pure`.
`pure-primary-api-token`	(empty)	Primary FlashArray REST API token.
`pure-secondary-management-ip`	(empty)	Secondary FlashArray management IP or hostname.
`pure-secondary-api-token`	(empty)	Secondary FlashArray REST API token.
`pure-default-rpo-seconds`	`900`	Default RPO target in seconds for async replication (15 minutes).
`pure-snapshot-interval-seconds`	`300`	Replication snapshot interval in seconds (5 minutes).
`pure-verify-ssl`	`false`	Verify TLS certificates presented by the FlashArrays.

Engine

Option	Default	Description
`engine-topic`	`protector-engine`	oslo.messaging RPC topic shared by the API and engine.
`recovery-point-interval-minutes`	`0`	Interval between scheduled recovery point snapshots for ActiveDR groups. `0` disables scheduled recovery points.
`recovery-point-max-count`	`20`	On-demand recovery point snapshots retained per group.
`recovery-point-ttl-hours`	`24`	Age in hours at which recovery point snapshots are deleted.
`auto-create-security-groups`	`false`	Create missing security groups on the DR site during sync. Off by default because absent groups may be absent intentionally.

TLS and connectivity

Option	Default	Description
`verify-ssl`	`true`	Verify TLS certificates when calling OpenStack APIs. Set `false` only in lab environments. Maps to `insecure = true` in `[service_credentials]` and `[keystone_authtoken]`.
`ssl-ca`	(empty)	Base64-encoded PEM CA certificate for verifying OpenStack endpoints. Overridden by a `certificates` relation to Vault when present. Maps to `cafile` in `[service_credentials]`.

Keystone registration

Option	Default	Description
`keystone-service-type`	`disaster-recovery`	Primary Keystone service type registered for this deployment.
`keystone-extra-service-types`	`protector`	Additional service types registered with the same endpoints. The `openstack dr` CLI resolves its endpoint by type `protector`.
`keystone-requested-roles`	`dr_site_admin`	Roles Keystone should ensure exist.
`deregister-on-remove`	`true`	Remove Keystone service and endpoint entries when the application is removed. Set `false` if the service catalogue is managed externally.

Advanced: arbitrary config overrides

The config-flags option lets you inject arbitrary protector.conf settings that the charm does not expose as first-class options. Use section:key=value format with colon separators (dot separators fail silently):

juju config trilio-site-recovery \
  config-flags="DEFAULT:rpc_response_timeout=120,engine:topic=protector-engine"

Cross-site name resolution

Each site's resolver typically serves only its own DNS zone. If the primary site unit cannot resolve the secondary site's Keystone or protector-api hostname, use extra-hosts:

juju config trilio-site-recovery extra-hosts="
10.80.65.60  keystone.site-a.example.com
10.32.65.166 keystone.site-b.example.com"

This is a workaround for environments where DNS cannot be fixed at the network layer. Prefer adding an upstream forwarder in MAAS for a permanent solution.

Key `protector.conf` sections (reference)

The following table covers options you may need to set via config-flags or review in the rendered config. All values are managed by the charm — do not edit the file directly.

[api]

Key	Default	Notes
`bind_host`	`0.0.0.0`
`bind_port`	`8788`
`workers`	`2`	Charm default; oslo.config compiled-in default is 4.

[database]

Key	Default	Notes
`connection`	(set by shared-db relation)	`mysql+pymysql://protector:<pass>@<host>/protector`. Never set manually in charmed deployments.

[service_credentials]

Key	Default	Notes
`default_trust_roles`	`member,dr_operator`	Roles delegated to the service in each tenant's Keystone trust. Both roles must exist on both sites.
`cafile`	(set from ssl-ca or certificates relation)	PEM CA bundle path for cross-site TLS verification.
`insecure`	`false`	Set `true` only in development.

[storage]

Key	Default	Notes
`storage_driver`	`mock`	Set to `pure` for production.

[pure_storage] (rendered only when storage-driver=pure)

Key	Notes
`primary_management_ip`	Primary FlashArray IP
`primary_api_token`	Primary FlashArray token
`secondary_management_ip`	Secondary FlashArray IP
`secondary_api_token`	Secondary FlashArray token
`default_rpo_seconds`	Default 900 (15 min)
`snapshot_interval_seconds`	Default 300 (5 min)

Usage

Once both sites are deployed and the charm reaches active, you interact with Trilio Site Recovery through the openstack dr CLI. All commands follow the pattern openstack dr <resource> <action>.

Register sites

Sites are registered from either site's openstack dr CLI. Run openstack dr site create --help for the full list of required arguments — additional mandatory flags beyond --auth-url and --site-type are required.

# Register the primary site (run on either site)
openstack dr site create primary-site \
  --site-type PRIMARY \
  --auth-url https://keystone.site-a.example.com:5000/v3

# Register the secondary site
openstack dr site create secondary-site \
  --site-type SECONDARY \
  --auth-url https://keystone.site-b.example.com:5000/v3

List registered sites:

openstack dr site list

Create tenant mappings

Because each site has an independent Keystone that assigns different UUIDs to the same project, you must create a tenant mapping before cross-site operations can proceed.

openstack dr tenant mapping create \
  --local-tenant <local-project-id> \
  --remote-site <secondary-site-id> \
  --remote-tenant <remote-project-id>

Validate the mapping resolves correctly on both sides:

openstack dr tenant mapping validate <mapping-id>

Port 8788 must be reachable bidirectionally between sites for tenant mappings to sync. One-way connectivity causes the mapping to create locally but fail to propagate to the remote site.

Configure a replicated Cinder volume type

All volumes attached to VMs in a Protection Group must use a Cinder volume type with replication_enabled='<is> True'. Create one if it does not already exist:

openstack volume type create replicated-pure \
  --property replication_enabled='<is> True'

For async replication, also set:

openstack volume type set replicated-pure \
  --property replication_type='<in> async'

Create a Protection Group

openstack dr protection group create my-pg \
  --primary-site <primary-site-id> \
  --secondary-site <secondary-site-id> \
  --replication-type async \
  --volume-type replicated-pure

List Protection Groups:

openstack dr protection group list

Add VMs to a Protection Group

All VMs must boot from a Cinder volume (not ephemeral storage). Instances using ephemeral boot disks are rejected with InstanceNotCinderBoot. Volumes using non-replicated types are rejected with VolumeTypeNotReplicated.

Pre-check a VM's eligibility without making changes:

openstack dr protection group member validate <pg-id> \
  --instance <nova-instance-id>

If validation passes, add the VM:

openstack dr protection group member add <pg-id> \
  --instance <nova-instance-id> \
  --boot-order 1

Boot order controls startup and shutdown sequence during failover. VMs with the same boot order are processed in parallel; lower-numbered tiers complete before higher ones begin.

Set boot order on an existing member:

openstack dr protection group member set <pg-id> \
  --member <member-id> \
  --boot-order 2

Monitor replication health

openstack dr replication health <pg-id>

The response includes failover_validation.can_failover (boolean), sync lag, RPO compliance, and the number of available recovery points. Use can_failover as your programmatic readiness check before executing a planned failover.

Run a test failover (non-disruptive DR validation)

A test failover clones replicated volumes on the secondary site and boots isolated VMs prefixed with test- on a separate network, leaving production untouched. Either --auto-network or --network-mapping is required:

openstack dr test failover <pg-id> --auto-network

When the test is complete, clean up the test VMs and volumes:

# Clean up the most recent test failover for this PG
openstack dr test failover cleanup <pg-id>

# Target a specific test failover operation
openstack dr test failover cleanup <pg-id> --operation <operation-id>

# Force cleanup even if some resources fail to delete
openstack dr test failover cleanup <pg-id> --force

Execute a planned failover

A planned failover gracefully shuts down VMs, takes a final snapshot for zero data loss, then starts VMs on the secondary site:

openstack dr failover <pg-id> --failover-type planned

For an unplanned failover (disaster scenario — skips shutdown, accepts data loss up to RPO):

openstack dr failover <pg-id> --failover-type unplanned

To fail over to a specific recovery point:

openstack dr failover <pg-id> \
  --failover-type planned \
  --recovery-point <recovery-point-id>

Use --wait to block until the operation completes:

openstack dr failover <pg-id> --failover-type planned --wait

Execute failback

Once the original site is restored, fail workloads back:

openstack dr failback <pg-id>

For a graceful planned failback:

openstack dr failback <pg-id> --failback-type planned

Track operations

All DR actions (failover, failback, test failover, sync) create a tracked operation with a unique ID and progress from 0–100:

# List operations, optionally filtered
openstack dr operation list --protection-group <pg-id>

# Show a specific operation
openstack dr operation show <operation-id>

# Cancel an in-progress operation
openstack dr operation cancel <operation-id>

Perform a site-level failover

Site-level failover (distinct from protection-group failover) is triggered with:

openstack dr site failover <site-id> --failover-type planned

Or, when running on the secondary site:

openstack dr failover to this site --failover-type planned

Examples

Example 1: Verify the API is running

Query the version discovery endpoint (no authentication required):

curl http://<unit-ip>:8788/

Expected response:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://<unit-ip>:8788/v1/"
        }
      ]
    }
  ]
}

Example 2: List Protection Groups via API

Obtain a token from Keystone first, then query the protection groups resource:

TOKEN=$(openstack token issue -f value -c id)
TENANT=$(openstack project show <project-name> -f value -c id)

curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://<unit-ip>:8788/v1/${TENANT}/protection-groups | python3 -m json.tool

The OpenStack-API-Version header is optional. When omitted the API defaults to microversion 1.0 and processes the request normally. Versions outside the 1.0–1.1 range return HTTP 400.

Example 3: Trigger a planned failover via API

Failover, failback, and test failover are all triggered via POST /v1/{tenant_id}/protection-groups/{pg_id}/action:

curl -s -X POST \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "Content-Type: application/json" \
  http://<unit-ip>:8788/v1/${TENANT}/protection-groups/<pg-id>/action \
  -d '{"failover": {"failover_type": "planned"}}'

Expected response (HTTP 202):

{
  "message": "Failover initiated",
  "status": "pending",
  "protection_group_id": "<pg-id>",
  "operation_id": "<operation-id>",
  "operation": {
    "id": "<operation-id>",
    "status": "pending",
    "progress": 0
  }
}

Poll the operation until it completes:

curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  http://<unit-ip>:8788/v1/${TENANT}/operations/<operation-id> | python3 -m json.tool

Example 4: Full Protection Group workflow via CLI

This example creates a Protection Group, adds two VMs with boot ordering, checks replication health, and runs a test failover.

# 1. Create the Protection Group
openstack dr protection group create production-pg \
  --primary-site <primary-site-id> \
  --secondary-site <secondary-site-id> \
  --replication-type async \
  --volume-type replicated

# 2. Add a database VM (tier 1 — starts first)
openstack dr protection group member add production-pg \
  --instance <db-vm-id> \
  --boot-order 1

# 3. Add an application VM (tier 2 — starts after DB is up)
openstack dr protection group member add production-pg \
  --instance <app-vm-id> \
  --boot-order 2

# 4. List members to confirm
openstack dr protection group member list production-pg

# 5. Check replication health
openstack dr replication health <pg-id>

# 6. Run a non-disruptive test failover
openstack dr test failover <pg-id> --auto-network --wait

# 7. Inspect test VMs on the secondary site, then clean up
openstack dr test failover cleanup <pg-id>

Example 5: Check replication health via API

curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  http://<unit-ip>:8788/v1/${TENANT}/protection-groups/<pg-id>/replication-health \
  | python3 -m json.tool

The response nests fields under replication_health.health_report.*. The failover_validation object at replication_health.failover_validation contains can_failover (boolean) — use this as your programmatic readiness check before initiating failover:

{
  "replication_health": {
    "protection_group_id": "<pg-id>",
    "protection_group_name": "production-pg",
    "health_report": {
      "sync_state": "in_sync",
      "lag_seconds": 12,
      "rpo_compliant": true,
      "recovery_points_available": 4
    },
    "failover_validation": {
      "can_failover": true,
      "warnings": [],
      "blockers": [],
      "message": "Protection group is ready for failover"
    }
  }
}

Example 6: Run Juju charm actions

# Juju 2.9 syntax
juju run-action trilio-site-recovery/leader health-check --wait
juju run-action trilio-site-recovery/leader db-sync --wait
juju run-action trilio-site-recovery/leader register-service --wait

# Juju 3.x syntax
juju run trilio-site-recovery/leader health-check
juju run trilio-site-recovery/leader db-sync
juju run trilio-site-recovery/leader register-service

Upgrade packages on a specific unit (run on every unit, not just the leader):

# Juju 2.9
juju run-action trilio-site-recovery/0 upgrade-packages --wait

# Pin to a specific version (also the rollback path)
juju run-action trilio-site-recovery/0 upgrade-packages version=1.0.1 --wait

Example 7: Automate failover readiness polling

This shell snippet polls replication health and triggers failover when the link is ready:

#!/bin/bash
TOKEN=$(openstack token issue -f value -c id)
TENANT=$(openstack project show myproject -f value -c id)
PG_ID="<pg-id>"

while true; do
  CAN_FAILOVER=$(curl -s \
    -H "X-Auth-Token: ${TOKEN}" \
    "http://<unit-ip>:8788/v1/${TENANT}/protection-groups/${PG_ID}/replication-health" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['replication_health']['failover_validation']['can_failover'])")

  if [ "${CAN_FAILOVER}" = "True" ]; then
    echo "Replication healthy — initiating planned failover"
    openstack dr failover "${PG_ID}" --failover-type planned --wait
    break
  else
    echo "Waiting for replication to be ready..."
    sleep 30
  fi
done

Troubleshooting

Unit is blocked: `apt could not install trilio-site-recovery`

Symptom: juju status shows the unit in blocked state with a message about apt.

Likely cause: The trilio-pkg-source URL is unreachable, or a private Gemfury repository requires a read token that was not supplied.

Fix:

# Check the URL is reachable from the unit
juju ssh trilio-site-recovery/0 curl -I 'https://apt.fury.io/trilio-os-site-recovery-master/'

# For a private repository, add the read token
juju config trilio-site-recovery \
  trilio-pkg-source="deb [trusted=yes] https://<read-token>@apt.fury.io/<repo>/ /"

Unit is blocked: `the Trilio apt repository rejected us (401/403)`

Symptom: juju status shows a 401 Unauthorized or 403 Forbidden message.

Likely cause: Missing or invalid read token in trilio-pkg-source.

Fix: Supply a valid token in the apt source URL (see above).

`mysql-router` fails with MarkupSafe wheel mismatch on Jammy

Symptom: The mysql-router subordinate unit errors during install with a Python wheel build failure.

Likely cause: mysql-router revision 1178 or mysql-innodb-cluster revision 179 ship broken wheelhouses for Jammy.

Fix: Deploy with the pinned known-good revisions:

juju deploy mysql-router --revision 1154 trilio-site-recovery-mysql-router
juju deploy mysql-innodb-cluster --revision 164 trilio-mysql

`juju deploy` without `--base ubuntu@22.04` places unit on wrong series

Symptom: The unit deploys on a Noble container; the mysql-router subordinate fails to install.

Likely cause: The --base ubuntu@22.04 flag was omitted.

Fix: Always include --base ubuntu@22.04 when deploying the charm:

juju deploy trilio-site-recovery-for-openstack trilio-site-recovery \
  --base ubuntu@22.04 \
  --to <machine-id>

`openstack dr` commands fail: `No endpoint found`

Symptom: openstack dr site list returns EndpointNotFound or similar.

Likely cause: The service is not registered in the Keystone catalogue, or the identity-credentials relation is not yet satisfied.

Fix:

# Check the relation is up
juju status trilio-site-recovery

# Confirm endpoints are registered
openstack endpoint list --service protector

# Re-register if needed
juju run trilio-site-recovery/leader register-service
# (Juju 2.9: juju run-action trilio-site-recovery/leader register-service --wait)

`openstack dr` commands fail: plugin not found

Symptom: Running openstack dr returns Invalid command: 'dr'.

Likely cause: python3-triliositerecovery is not installed on this machine, or python-openstackclient was installed via the snap (which is incompatible with the plugin).

Fix:

# Remove the snap if present
sudo snap remove openstackclient

# Install python-openstackclient via pip
pip install python-openstackclient

# Install the CLI plugin from the Trilio apt repository
echo 'deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /' \
  | sudo tee /etc/apt/sources.list.d/trilio-site-recovery.list
sudo apt-get update
sudo apt-get install python3-triliositerecovery

Trust creation fails: `Role dr_operator does not exist on site <site>`

Symptom: The first cross-site operation fails with a message about a missing role.

Likely cause: The dr_operator or dr_site_admin Keystone role does not exist on one or both sites. The service looks up roles — it does not create them.

Fix: Create the missing roles on both sites before any cross-site operation:

openstack role create dr_operator
openstack role create dr_site_admin

Tenant mapping creates locally but fails to sync to the remote site

Symptom: openstack dr tenant mapping list shows the mapping, but the remote site does not see it. Validation fails.

Likely cause: Port 8788 is not open in both directions between the two sites. Cross-site metadata sync requires bidirectional connectivity on port 8788.

Fix: Open port 8788 bidirectionally in the firewall or security group rules between the two sites and retry:

openstack dr tenant mapping validate <mapping-id>

`InstanceNotCinderBoot` error when adding a VM

Symptom: openstack dr protection group member add returns HTTP 422 with InstanceNotCinderBoot.

Likely cause: The instance uses an ephemeral boot disk rather than a Cinder volume.

Fix: Rebuild the instance with a Cinder-backed boot volume. Validate eligibility first:

openstack dr protection group member validate <pg> --instance <instance-id>

`VolumeTypeNotReplicated` error when adding a VM

Symptom: openstack dr protection group member add returns HTTP 422 with VolumeTypeNotReplicated.

Likely cause: One or more Cinder volumes attached to the instance use a volume type that does not have replication_enabled='<is> True' as an extra-spec.

Fix: Set the correct extra-spec on the volume type, or retype the volume:

openstack volume type set <volume-type> \
  --property replication_enabled='<is> True'

`config-flags` changes have no effect

Symptom: Settings passed via config-flags are not reflected in protector.conf.

Likely cause: Dot separators were used instead of colon separators. Dot separators fail silently.

Fix: Use section:key=value with colon separators:

# Correct
juju config trilio-site-recovery \
  config-flags="DEFAULT:rpc_response_timeout=120"

# Incorrect (fails silently)
juju config trilio-site-recovery \
  config-flags="DEFAULT.rpc_response_timeout=120"

Viewing logs

Both services write to /var/log/protector and to stderr (captured by systemd):

# Live log from the API service
juju ssh trilio-site-recovery/0 journalctl -u protector-api -f

# Live log from the engine
juju ssh trilio-site-recovery/0 journalctl -u protector-engine -f

# View the rendered config (secrets redacted)
juju run trilio-site-recovery/leader show-config

Restarting services

juju run trilio-site-recovery/leader restart-services
# (Juju 2.9: juju run-action trilio-site-recovery/leader restart-services --wait)

Juju Charms

Prerequisites

Prerequisites

Step 1 — Pin infrastructure charm revisions

Step 2 — Deploy the service charm

Step 3 — Deploy the mysql-router subordinate

Step 4 — Add relations

Step 5 — Verify the deployment

Step 6 — Install the CLI on operator workstations

Step 7 — Deploy the optional Horizon dashboard plugin

Repeat on the secondary site

Package source

API

Storage driver

Engine

TLS and connectivity

Keystone registration

Advanced: arbitrary config overrides

Cross-site name resolution

Key protector.conf sections (reference)

Register sites

Create tenant mappings

Configure a replicated Cinder volume type

Create a Protection Group

Add VMs to a Protection Group

Monitor replication health

Run a test failover (non-disruptive DR validation)

Execute a planned failover

Execute failback

Track operations

Perform a site-level failover

Example 1: Verify the API is running

Example 2: List Protection Groups via API

Example 3: Trigger a planned failover via API

Example 4: Full Protection Group workflow via CLI

Example 5: Check replication health via API

Example 6: Run Juju charm actions

Example 7: Automate failover readiness polling

Unit is blocked: apt could not install trilio-site-recovery

Unit is blocked: the Trilio apt repository rejected us (401/403)

mysql-router fails with MarkupSafe wheel mismatch on Jammy

juju deploy without --base ubuntu@22.04 places unit on wrong series

openstack dr commands fail: No endpoint found

openstack dr commands fail: plugin not found

Trust creation fails: Role dr_operator does not exist on site <site>

Tenant mapping creates locally but fails to sync to the remote site

InstanceNotCinderBoot error when adding a VM

VolumeTypeNotReplicated error when adding a VM

config-flags changes have no effect

Viewing logs

Restarting services

Key `protector.conf` sections (reference)

Unit is blocked: `apt could not install trilio-site-recovery`

Unit is blocked: `the Trilio apt repository rejected us (401/403)`

`mysql-router` fails with MarkupSafe wheel mismatch on Jammy

`juju deploy` without `--base ubuntu@22.04` places unit on wrong series

`openstack dr` commands fail: `No endpoint found`

`openstack dr` commands fail: plugin not found

Trust creation fails: `Role dr_operator does not exist on site <site>`

`InstanceNotCinderBoot` error when adding a VM

`VolumeTypeNotReplicated` error when adding a VM

`config-flags` changes have no effect