Kolla

This page explains how to install Trilio Site Recovery for OpenStack on a Kolla-Ansible containerized OpenStack cluster using the provided shell-based installer. The installer builds a Docker image containing the protector-api and protector-engine services, creates the MariaDB schema, registers Keystone resources, deploys the Horizon dashboard plugin, and applies Cinder policy overrides — all in a single script invocation. This deployment path is intended for internal development and evaluation purposes only; the only validated, customer-facing deployment method is the Juju charm described in the Charmed OpenStack installation guide.

Before running the installer, make sure the following are in place on both clusters:

• Two Kolla-Ansible deployed OpenStack clusters (Victoria or later) each with:
  • MariaDB, RabbitMQ, Keystone, Nova, Cinder, Neutron, and Horizon deployed by Kolla
  • Cinder configured with a Pure Storage FlashArray iSCSI backend
  • A replicated Cinder volume type (e.g. pure-replicated) with extra-spec replication_enabled='<is> True'
  • Pure Storage FlashArray arrays connected with async replication between sites (Purity//FA 6.0+ required for ActiveDR; any supported version for async)
• SSH access to both cluster nodes as pureuser with passwordless sudo (the installer uses sudo -S internally)
• Docker installed and running on both cluster nodes
• rsync and scp available on the workstation running the installer
• Bidirectional network connectivity between clusters on the following ports:
  • 8788/TCP — protector-api (required for cross-site metadata sync)
  • 5672/TCP — RabbitMQ AMQP
  • 5000/TCP — Keystone
  • 8774/TCP — Nova
  • 8776/TCP — Cinder
• The Keystone roles dr_operator and dr_site_admin must exist on both sites before any cross-site operation; the installer creates them if they are absent, but if you run with --skip-keystone you must create them manually:
  Copy

  openstack role create dr_operator
  openstack role create dr_site_admin
  

• python-openstackclient installed via pip on the workstation (the openstackclient snap is incompatible with the triliositerecovery CLI plugin):
  Copy

  pip install python-openstackclient
  pip install -e ./client   # installs the triliositerecovery CLI plugin
  

1. Set cluster IP environment variables

Export the management IPs of your two clusters so the installer can reach them over SSH:

export CLUSTER1_IP=10.21.67.100
export CLUSTER2_IP=10.21.67.101

2. Prepare cluster-specific configuration files

The installer looks for a cluster-specific protector.conf at etc/protector-<cluster_name>/protector.conf. Create one for each cluster before running the script. A minimal example for cluster1 is shown in the Configuration section below. At minimum you must set:

• [database] connection — pointing at the cluster's MariaDB via ProxySQL
• [DEFAULT] transport_url — pointing at the cluster's RabbitMQ
• [keystone_authtoken] — all fields pointing at the cluster's Keystone
• [storage] storage_driver = pure — activates the Pure Storage FlashArray driver (the compiled-in default is mock, which is not suitable for production)
• [pure_storage] — FlashArray IPs, API tokens, and site names

3. Run the installer

Install on both clusters at once (recommended for initial setup):

bash deploy/kolla/install.sh all

Install on a single cluster (supply the IP and a short name):

bash deploy/kolla/install.sh 10.21.67.100 cluster1

The script performs the following steps on each cluster, in order:

1. Builds the protector:latest Docker image on the target host (requires the full repository source to be rsync'd first).
2. Creates /etc/protector, /var/log/protector, and /var/lib/protector on the target host.
3. Copies the cluster-specific protector.conf, policy.yaml, and api-paste.ini to /etc/protector/.
4. Creates the protector database and user in MariaDB, and registers the user in ProxySQL.
5. Runs database table creation and schema migration (protector-manage db sync).
6. Creates the Keystone user protector, the roles dr_operator and dr_site_admin, the service catalog entry, and public/internal/admin endpoints.
7. Starts the protector_api and protector_engine Docker containers with --network=host and --restart=unless-stopped.
8. Deploys the Horizon dashboard plugin into the running horizon container.
9. Applies Cinder policy overrides for volume_extension:volume_manage and related rules across cinder_api, cinder_volume, and cinder_scheduler.
10. Verifies that the API is reachable by polling http://<CLUSTER_IP>:8788/ repeatedly (up to 30 seconds).

4. Selective steps with flags

You can skip individual steps if they have already been completed or are managed separately:

Example — skip Horizon and Cinder steps:

bash deploy/kolla/install.sh 10.21.67.100 cluster1 --skip-horizon --skip-cinder

5. Register sites and configure tenant mappings

Once containers are running on both clusters, register each cluster as a site. Run the following on cluster1:

source cluster1-openrc.sh

# Register both sites from cluster1's perspective
openstack dr site create cluster1 \
  --site-type PRIMARY \
  --auth-url http://10.21.67.105:5000 \
  --region USEAST

openstack dr site create cluster2 \
  --site-type SECONDARY \
  --auth-url http://10.21.67.106:5000 \
  --region USEAST

# Set the cross-site service credentials on each site record
TOKEN=$(openstack token issue -f value -c id)
PROJECT_ID=$(openstack token issue -f value -c project_id)

for SITE_ID in $(openstack dr site list -f value -c ID); do
  curl -s -X PUT "http://${CLUSTER1_IP}:8788/v1/${PROJECT_ID}/sites/${SITE_ID}" \
    -H "X-Auth-Token: ${TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{"site": {
      "service_username": "protector",
      "service_password": "ProtectorServicePass123",
      "service_project_name": "service",
      "service_user_domain_name": "Default",
      "service_project_domain_name": "Default"
    }}'
done

Repeat on cluster2 with site types swapped (cluster2 = PRIMARY, cluster1 = SECONDARY).

Then create tenant mappings so each cluster knows which remote project corresponds to the local one:

# On cluster1:
openstack dr tenant mapping create \
  --local-tenant <cluster1-project-id> \
  --remote-site <cluster2-site-id> \
  --remote-tenant <cluster2-project-id>

# On cluster2:
openstack dr tenant mapping create \
  --local-tenant <cluster2-project-id> \
  --remote-site <cluster1-site-id> \
  --remote-tenant <cluster1-project-id>

6. Grant required Keystone roles to DR users

Each tenant user who will perform DR operations needs member and dr_operator on their project on both sites. The protector service account also needs these roles on every DR-enabled project:

# Grant roles to the protector service account on the tenant project
openstack role add --project <tenant-project> --user protector member
openstack role add --project <tenant-project> --user protector dr_operator

# Grant roles to the end user
openstack role add --project <tenant-project> --user <dr-user> member
openstack role add --project <tenant-project> --user <dr-user> dr_operator

Site-level operators who need to trigger site failovers require dr_site_admin on a dedicated account (this role is not delegated in Keystone trusts and does not affect per-PG operations).

The service is configured via /etc/protector/protector.conf. In the Kolla deployment model you manage this file directly — unlike the Juju charm model where the charm renders it automatically. The table below lists the key options you are most likely to need.

Warning: The compiled-in default for [storage] storage_driver is mock. You must set storage_driver = pure in every production protector.conf; leaving it at mock means no real storage operations will be performed.

[DEFAULT]

The RabbitMQ broker URL is specified via the oslo.messaging transport_url option. In a Kolla environment, set it directly in [DEFAULT] as oslo.messaging registers it there:

[DEFAULT]
transport_url = rabbit://openstack:<password>@<cluster-ip>:5672//

Note: The [drbd_storage] transport_url option is an unrelated override used only by the DRBD driver agent path and has no effect on the main protector-api/protector-engine RPC communication.

[api]

[database]

[engine]

[keystone_authtoken]

Standard keystonemiddleware options pointing at the local cluster's Keystone. All fields are required:

[keystone_authtoken]
www_authenticate_uri = http://<keystone-ip>:5000
auth_url = http://<keystone-ip>:5000
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = protector
password = <protector-service-password>
memcached_servers = 127.0.0.1:11211

[service_credentials]

[storage]

[pure_storage]

Snapshot retention (all under [pure_storage]):

Minimal protector.conf for cluster1

[DEFAULT]
debug = true
log_dir = /var/log/protector
use_stderr = true
state_path = /var/lib/protector
host = cluster1
transport_url = rabbit://openstack:<rabbit-password>@10.21.67.100:5672//

[api]
bind_host = 0.0.0.0
bind_port = 8788
workers = 2

[database]
connection = mysql+pymysql://protector:ProtectorPass123@10.21.67.105:3306/protector

[engine]
topic = protector-engine

[keystone_authtoken]
www_authenticate_uri = http://10.21.67.105:5000
auth_url = http://10.21.67.105:5000
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = protector
password = ProtectorServicePass123
memcached_servers = 127.0.0.1:11211

[oslo_policy]
policy_file = /etc/protector/policy.yaml

[service_credentials]
default_service_project = service
default_service_user_domain = Default
default_service_project_domain = Default
default_trust_roles = member,dr_operator

[storage]
storage_driver = pure

[pure_storage]
primary_management_ip = 10.21.67.50
primary_api_token = <primary-fa-api-token>
secondary_management_ip = 10.21.67.60
secondary_api_token = <secondary-fa-api-token>
primary_site_name = cluster1
secondary_site_name = cluster2
verify_ssl = false
default_rpo_seconds = 900
snapshot_interval_seconds = 300
retention_all_for_hours = 24
retention_per_day = 24
retention_days = 7

Once the service is running and sites are registered, you interact with it via the openstack dr CLI (package triliositerecovery, installed separately with pip install -e ./client).

Check service status

# Verify containers are running
sudo docker ps --filter name=protector_api
sudo docker ps --filter name=protector_engine

List registered sites

openstack dr site list

Create a Protection Group

All VMs in a Protection Group must boot from Cinder volumes, and all attached Cinder volumes must use a volume type with replication_enabled='<is> True'. Attempts to add non-compliant VMs return InstanceNotCinderBoot (HTTP 422) or VolumeTypeNotReplicated (HTTP 422).

source cluster1-openrc.sh

openstack dr protection group create my-pg \
  --primary-site cluster1 \
  --secondary-site cluster2 \
  --replication-type async \
  --volume-type pure-replicated

Add a VM to a Protection Group

The --instance flag takes a Nova instance ID (not a name). Use --boot-order to control startup sequencing during failover: lower numbers start first; VMs with the same value start in parallel.

openstack dr protection group member add <pg-id> \
  --instance <nova-instance-id> \
  --boot-order 1

Pre-validate a VM's eligibility without adding it:

openstack dr protection group member validate <pg-id> \
  --instance <nova-instance-id>

This calls POST /v1/{tenant_id}/protection-groups/{pg_id}/members/validate and returns HTTP 200 with a structured errors array — no side effects.

Sync metadata to the secondary site

After adding members, push the Protection Group metadata to cluster2 so the secondary engine is aware of the group:

openstack dr metadata sync <pg-id>

Check replication health

openstack dr replication health <pg-id>

The response includes failover_validation.can_failover (boolean) — use this as a programmatic readiness check before triggering failover.

Trigger a test failover (non-disruptive)

Run this from the secondary cluster. Test failover clones replicated volumes and boots isolated VMs with a test- prefix on a separate network, leaving production untouched:

source cluster2-openrc.sh

openstack dr test failover <pg-id> --auto-network

# Monitor progress
openstack dr operation show <operation-id>

Clean up when done:

openstack dr test failover cleanup <pg-id>

Trigger a planned failover

Run this from the secondary cluster. Planned failover gracefully shuts down VMs on the primary and takes a final snapshot for zero data loss:

source cluster2-openrc.sh

openstack dr failover <pg-id> --failover-type planned

Add --wait to block until the operation completes.

Trigger an unplanned failover

Use --failover-type unplanned when the primary site is unreachable. Data loss is bounded by the RPO:

openstack dr failover <pg-id> --failover-type unplanned

Failback to the original primary

Once the primary site is restored, run failback from the primary cluster:

source cluster1-openrc.sh

openstack dr failback <pg-id>

Restart services

sudo docker restart protector_api protector_engine

View logs

# Container stdout/stderr
sudo docker logs protector_api
sudo docker logs protector_engine

# File-based logs (more detail)
sudo tail -f /var/log/protector/protector-api-wrapper.py.log
sudo tail -f /var/log/protector/protector-engine-wrapper.py.log

Example 1 — Install on both clusters and verify

export CLUSTER1_IP=10.21.67.100
export CLUSTER2_IP=10.21.67.101

bash deploy/kolla/install.sh all

Expected tail output (per cluster):

[INFO] [cluster1] Protector API is healthy
[INFO] [cluster1] Protector Engine is healthy
[INFO] [cluster1] Installation complete

Example 2 — Version discovery (no authentication required)

curl -s http://10.21.67.100:8788/

Expected response:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://10.21.67.100:8788/v1/"
        }
      ]
    }
  ]
}

Example 3 — Create a Protection Group and add a VM

source cluster1-openrc.sh

# Create the Protection Group
PG_ID=$(openstack dr protection group create web-tier \
  --primary-site cluster1 \
  --secondary-site cluster2 \
  --replication-type async \
  --volume-type pure-replicated \
  -f value -c id)

echo "Protection Group: ${PG_ID}"

# Validate VM eligibility before adding
openstack dr protection group member validate ${PG_ID} \
  --instance <nova-instance-id>

# Add the VM
openstack dr protection group member add ${PG_ID} \
  --instance <nova-instance-id> \
  --boot-order 1

# Sync metadata to the secondary site
openstack dr metadata sync ${PG_ID}

Example 4 — Check replication health via CLI and API

CLI:

openstack dr replication health ${PG_ID}

Direct API call:

TOKEN=$(openstack token issue -f value -c id)
PROJECT_ID=$(openstack token issue -f value -c project_id)

curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "OpenStack-API-Version: protector 1.1" \
  "http://10.21.67.100:8788/v1/${PROJECT_ID}/protection-groups/${PG_ID}/replication-health"

Expected response structure (fields nested under replication_health):

{
  "replication_health": {
    "protection_group_id": "<pg-id>",
    "protection_group_name": "web-tier",
    "health_report": {
      "status": "healthy",
      "lag_seconds": 42,
      "rpo_compliant": true,
      "recovery_points_available": 12
    },
    "failover_validation": {
      "can_failover": true,
      "warnings": [],
      "blockers": [],
      "message": "Protection group is ready for failover"
    },
    "storage_config": {}
  }
}

Use replication_health.failover_validation.can_failover as the programmatic readiness gate.

Example 5 — Trigger a planned failover and poll for completion

source cluster2-openrc.sh

TOKEN=$(openstack token issue -f value -c id)
PROJECT_ID=$(openstack token issue -f value -c project_id)

# Trigger planned failover
RESPONSE=$(curl -s -X POST \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "Content-Type: application/json" \
  -H "OpenStack-API-Version: protector 1.1" \
  "http://10.21.67.101:8788/v1/${PROJECT_ID}/protection-groups/${PG_ID}/action" \
  -d '{"failover": {"failover_type": "planned"}}')

OP_ID=$(echo ${RESPONSE} | python3 -c "import sys,json; print(json.load(sys.stdin)['operation_id'])")
echo "Operation: ${OP_ID}  (initial status: pending)"

# Poll until complete
while true; do
  STATUS=$(curl -s \
    -H "X-Auth-Token: ${TOKEN}" \
    "http://10.21.67.101:8788/v1/${PROJECT_ID}/operations/${OP_ID}" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['operation']['status'], d['operation']['progress'])")
  echo "${STATUS}"
  [[ "${STATUS}" == completed* ]] || [[ "${STATUS}" == failed* ]] && break
  sleep 10
done

Example 6 — Test failover with a specific recovery point

source cluster2-openrc.sh

# List available recovery points
openstack dr recovery point list ${PG_ID}

# Run a test failover against a specific recovery point, using auto network assignment
openstack dr test failover ${PG_ID} \
  --auto-network \
  --recovery-point <recovery-point-id>

# Monitor
openstack dr operation show <operation-id>

# Clean up test resources when finished
openstack dr test failover cleanup ${PG_ID}

Example 7 — Uninstall from both clusters

export CLUSTER1_IP=10.21.67.100
export CLUSTER2_IP=10.21.67.101

bash deploy/kolla/uninstall.sh all

This removes: protector_api and protector_engine containers and their Docker images, Keystone user/service/endpoints, the protector database and database user, the Horizon plugin from the horizon container, Cinder policy overrides, and all files under /etc/protector, /var/log/protector, /var/lib/protector, and /opt/openstack-protector.

Use a consistent structure for each issue: Symptom → Likely cause → Fix.

Symptom: docker logs protector_api shows connection refused or the API returns HTTP 000.

Likely cause: The protector_api container failed to start, usually due to a misconfigured protector.conf (bad database URL, missing [keystone_authtoken] fields) or a port conflict.

Fix:

sudo docker ps -a --filter name=protector_api
sudo docker logs protector_api --tail 50

Correct the offending config key and restart:

sudo docker restart protector_api

Symptom: openstack dr protection group member add returns HTTP 422 with error type InstanceNotCinderBoot.

Likely cause: The Nova instance boots from an ephemeral (local) disk rather than a Cinder volume. All VMs in a Protection Group must use Cinder-backed boot volumes.

Fix: Recreate the instance using --block-device source_type=image,...,destination_type=volume,volume_type=pure-replicated,... to ensure the boot disk is a Cinder volume.

Symptom: openstack dr protection group member add returns HTTP 422 with error type VolumeTypeNotReplicated.

Likely cause: One or more Cinder volumes attached to the instance use a volume type that does not have replication_enabled='<is> True' set as an extra-spec.

Fix:

openstack volume type set --property replication_enabled='<is> True' pure-replicated

Confirm with:

openstack volume type show pure-replicated -c extra_specs

Symptom: Engine logs show py-pure-client SDK not installed or ImportError: No module named 'pypureclient'.

Likely cause: The Docker image was built without the py-pure-client dependency, or the [storage] storage_driver is set to pure but the Pure SDK is missing inside the container.

Fix: Verify the SDK is installed in the image:

sudo docker exec protector_engine pip3 list | grep pure

If missing, rebuild the image (the Dockerfile installs py-pure-client via pip3 in the builder stage). If the problem persists after a rebuild, check that the build completed without errors.

Symptom: Failover fails with InvalidExecutionSite (HTTP 400).

Likely cause: You are attempting to run failover from the primary site. Failover must be triggered from the secondary (DR) site.

Fix: Source the openrc for cluster2 and re-run the command:

source cluster2-openrc.sh
openstack dr failover <pg-id> --failover-type planned

Symptom: openstack dr metadata sync succeeds locally but the remote site has no record of the Protection Group.

Likely cause: Port 8788/TCP is blocked in one direction between the clusters. Cross-site metadata sync requires bidirectional connectivity on port 8788.

Fix: Confirm connectivity in both directions:

# From cluster1 to cluster2
curl -s http://10.21.67.101:8788/
# From cluster2 to cluster1
curl -s http://10.21.67.100:8788/

Both should return the version discovery JSON. Open firewall rules as needed for port 8788/TCP in both directions.

Symptom: Trust creation fails with Role dr_operator does not exist on site <site>.

Likely cause: The dr_operator or dr_site_admin Keystone role was not created on one of the sites before the first cross-site operation. The service looks up roles by name and does not create them.

Fix: Create both roles on both sites:

openstack role create dr_operator
openstack role create dr_site_admin

Symptom: VM remains in BUILD state for several minutes after failover or test failover.

Likely cause: iSCSI multipath attachment for the promoted Pure Storage volume is slow on the DR site's compute node.

Fix: Wait 2–3 minutes. If the VM eventually reaches ACTIVE, the attach succeeded. If it transitions to ERROR, check nova-compute logs on the DR site compute node for iSCSI or multipath errors. Ensure the compute node has the Pure Storage iSCSI initiator configured.

Symptom: openstack dr site create returns an error about missing arguments.

Likely cause: dr site create requires --site-type and --auth-url at minimum; other arguments may also be required depending on your environment.

Fix: Check the full required argument list:

openstack dr site create --help

Symptom: The Horizon Site Recovery panel is not visible after deployment.

Likely cause: The Horizon plugin files were not copied correctly, or collectstatic was not run, or the horizon container was not restarted.

Fix:

# Verify plugin files are present inside the horizon container
sudo docker exec horizon ls /var/lib/kolla/venv/lib/python3.12/site-packages/protector_dashboards

# Force collectstatic and restart if files are present
sudo docker exec horizon python3 /var/lib/kolla/venv/bin/manage.py collectstatic --noinput
sudo docker restart horizon

If files are missing, re-run the installer with only the Horizon step:

bash deploy/kolla/install.sh ${CLUSTER1_IP} cluster1 --skip-db --skip-keystone --skip-cinder

Symptom: openstack dr commands are not recognized (openstack: 'dr' is not an openstack command).

Likely cause: The triliositerecovery CLI plugin is not installed. Running pip install -e . from the repository root installs only the server binaries and does not install the CLI.

Fix:

pip install -e ./client

Also ensure you are using pip-installed python-openstackclient, not the openstackclient snap, which is incompatible with third-party CLI plugins.