--- title: Kolla product: trilio-site-recovery-for-openstack doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-openstack/trilio-site-recovery-for-openstack/kolla --- # Kolla _Trilio Site Recovery installation on Koll a OpenStack_ ## Overview This page explains how to install Trilio Site Recovery for OpenStack on a Kolla-Ansible containerized OpenStack cluster using the provided shell-based installer. The installer builds a Docker image containing the `protector-api` and `protector-engine` services, creates the MariaDB schema, registers Keystone resources, deploys the Horizon dashboard plugin, and applies Cinder policy overrides — all in a single script invocation. This deployment path is intended for **internal development and evaluation purposes only**; the only validated, customer-facing deployment method is the Juju charm described in the Charmed OpenStack installation guide. ## Prerequisites Before running the installer, make sure the following are in place on **both** clusters: - Two Kolla-Ansible deployed OpenStack clusters (Victoria or later) each with: - MariaDB, RabbitMQ, Keystone, Nova, Cinder, Neutron, and Horizon deployed by Kolla - Cinder configured with a Pure Storage FlashArray iSCSI backend - A replicated Cinder volume type (e.g. `pure-replicated`) with extra-spec `replication_enabled=' True'` - Pure Storage FlashArray arrays connected with async replication between sites (Purity//FA 6.0+ required for ActiveDR; any supported version for async) - SSH access to both cluster nodes as `pureuser` with passwordless `sudo` (the installer uses `sudo -S` internally) - Docker installed and running on both cluster nodes - `rsync` and `scp` available on the workstation running the installer - Bidirectional network connectivity between clusters on the following ports: - `8788/TCP` — protector-api (required for cross-site metadata sync) - `5672/TCP` — RabbitMQ AMQP - `5000/TCP` — Keystone - `8774/TCP` — Nova - `8776/TCP` — Cinder - The Keystone roles `dr_operator` and `dr_site_admin` must exist on both sites before any cross-site operation; the installer creates them if they are absent, but if you run with `--skip-keystone` you must create them manually: ```bash openstack role create dr_operator openstack role create dr_site_admin ``` - `python-openstackclient` installed via `pip` on the workstation (the `openstackclient` snap is incompatible with the `triliositerecovery` CLI plugin): ```bash pip install python-openstackclient pip install -e ./client # installs the triliositerecovery CLI plugin ``` ## Installation ### 1. Set cluster IP environment variables Export the management IPs of your two clusters so the installer can reach them over SSH: ```bash export CLUSTER1_IP=10.21.67.100 export CLUSTER2_IP=10.21.67.101 ``` ### 2. Prepare cluster-specific configuration files The installer looks for a cluster-specific `protector.conf` at `etc/protector-/protector.conf`. Create one for each cluster before running the script. A minimal example for cluster1 is shown in the [Configuration](#configuration) section below. At minimum you must set: - `[database] connection` — pointing at the cluster's MariaDB via ProxySQL - `[DEFAULT] transport_url` — pointing at the cluster's RabbitMQ - `[keystone_authtoken]` — all fields pointing at the cluster's Keystone - `[storage] storage_driver = pure` — activates the Pure Storage FlashArray driver (the compiled-in default is `mock`, which is not suitable for production) - `[pure_storage]` — FlashArray IPs, API tokens, and site names ### 3. Run the installer **Install on both clusters at once** (recommended for initial setup): ```bash bash deploy/kolla/install.sh all ``` **Install on a single cluster** (supply the IP and a short name): ```bash bash deploy/kolla/install.sh 10.21.67.100 cluster1 ``` The script performs the following steps on each cluster, in order: 1. Builds the `protector:latest` Docker image on the target host (requires the full repository source to be rsync'd first). 2. Creates `/etc/protector`, `/var/log/protector`, and `/var/lib/protector` on the target host. 3. Copies the cluster-specific `protector.conf`, `policy.yaml`, and `api-paste.ini` to `/etc/protector/`. 4. Creates the `protector` database and user in MariaDB, and registers the user in ProxySQL. 5. Runs database table creation and schema migration (`protector-manage db sync`). 6. Creates the Keystone user `protector`, the roles `dr_operator` and `dr_site_admin`, the service catalog entry, and public/internal/admin endpoints. 7. Starts the `protector_api` and `protector_engine` Docker containers with `--network=host` and `--restart=unless-stopped`. 8. Deploys the Horizon dashboard plugin into the running `horizon` container. 9. Applies Cinder policy overrides for `volume_extension:volume_manage` and related rules across `cinder_api`, `cinder_volume`, and `cinder_scheduler`. 10. Verifies that the API is reachable by polling `http://:8788/` repeatedly (up to 30 seconds). ### 4. Selective steps with flags You can skip individual steps if they have already been completed or are managed separately: | Flag | Skips | |------|-------| | `--skip-keystone` | Keystone user, service, and endpoint creation | | `--skip-horizon` | Horizon plugin deployment | | `--skip-cinder` | Cinder policy overrides | | `--skip-db` | Database and user creation | | `--image ` | Uses a custom Docker image instead of `protector:latest` | | `--dev` | Mounts host code directories into containers for live code changes | Example — skip Horizon and Cinder steps: ```bash bash deploy/kolla/install.sh 10.21.67.100 cluster1 --skip-horizon --skip-cinder ``` ### 5. Register sites and configure tenant mappings Once containers are running on both clusters, register each cluster as a site. Run the following on **cluster1**: ```bash source cluster1-openrc.sh # Register both sites from cluster1's perspective openstack dr site create cluster1 \ --site-type PRIMARY \ --auth-url http://10.21.67.105:5000 \ --region USEAST openstack dr site create cluster2 \ --site-type SECONDARY \ --auth-url http://10.21.67.106:5000 \ --region USEAST # Set the cross-site service credentials on each site record TOKEN=$(openstack token issue -f value -c id) PROJECT_ID=$(openstack token issue -f value -c project_id) for SITE_ID in $(openstack dr site list -f value -c ID); do curl -s -X PUT "http://${CLUSTER1_IP}:8788/v1/${PROJECT_ID}/sites/${SITE_ID}" \ -H "X-Auth-Token: ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"site": { "service_username": "protector", "service_password": "ProtectorServicePass123", "service_project_name": "service", "service_user_domain_name": "Default", "service_project_domain_name": "Default" }}' done ``` Repeat on **cluster2** with site types swapped (cluster2 = PRIMARY, cluster1 = SECONDARY). Then create tenant mappings so each cluster knows which remote project corresponds to the local one: ```bash # On cluster1: openstack dr tenant mapping create \ --local-tenant \ --remote-site \ --remote-tenant # On cluster2: openstack dr tenant mapping create \ --local-tenant \ --remote-site \ --remote-tenant ``` ### 6. Grant required Keystone roles to DR users Each tenant user who will perform DR operations needs `member` and `dr_operator` on their project on **both** sites. The `protector` service account also needs these roles on every DR-enabled project: ```bash # Grant roles to the protector service account on the tenant project openstack role add --project --user protector member openstack role add --project --user protector dr_operator # Grant roles to the end user openstack role add --project --user member openstack role add --project --user dr_operator ``` Site-level operators who need to trigger site failovers require `dr_site_admin` on a dedicated account (this role is not delegated in Keystone trusts and does not affect per-PG operations). ## Configuration The service is configured via `/etc/protector/protector.conf`. In the Kolla deployment model you manage this file directly — unlike the Juju charm model where the charm renders it automatically. The table below lists the key options you are most likely to need. > **Warning:** The compiled-in default for `[storage] storage_driver` is `mock`. You **must** set `storage_driver = pure` in every production `protector.conf`; leaving it at `mock` means no real storage operations will be performed. ### `[DEFAULT]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `host` | string | `localhost` | Logical name for this site's engine host. Set to a unique, meaningful name per cluster. | | `rpc_conn_pool_size` | integer | `30` | Oslo.messaging RPC connection pool size. | | `debug` | boolean | `false` | Enable debug-level logging. | | `log_dir` | string | — | Directory for log files (e.g. `/var/log/protector`). | | `state_path` | string | — | Writable state directory (e.g. `/var/lib/protector`). | The RabbitMQ broker URL is specified via the oslo.messaging `transport_url` option. In a Kolla environment, set it directly in `[DEFAULT]` as oslo.messaging registers it there: ```ini [DEFAULT] transport_url = rabbit://openstack:@:5672// ``` > **Note:** The `[drbd_storage] transport_url` option is an unrelated override used only by the DRBD driver agent path and has no effect on the main `protector-api`/`protector-engine` RPC communication. ### `[api]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `bind_host` | string | `0.0.0.0` | IP address the API process listens on. | | `bind_port` | integer | `8788` | TCP port. | | `workers` | integer | `4` | Number of API worker processes. | ### `[database]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `connection` | string | `mysql+pymysql://protector:password@localhost/protector` | SQLAlchemy connection URL. **Always override this.** In Kolla environments, ProxySQL fronts MariaDB on port `3306`. | ### `[engine]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `topic` | string | `protector-engine` | Oslo.messaging topic the engine subscribes to. Must match between API and engine. | | `cross_site_timeout` | integer | `30` | Seconds to wait for responses from the remote site's API. | | `auto_create_security_groups` | boolean | `false` | When `true`, security groups missing on the DR site are created automatically during failover. | | `recovery_point_interval_minutes` | integer | `0` | Interval between automatic recovery point creation (0 = disabled). | | `recovery_point_max_count` | integer | `20` | Maximum number of recovery points to retain per Protection Group. | ### `[keystone_authtoken]` Standard keystonemiddleware options pointing at the **local** cluster's Keystone. All fields are required: ```ini [keystone_authtoken] www_authenticate_uri = http://:5000 auth_url = http://:5000 auth_type = password project_domain_name = Default user_domain_name = Default project_name = service username = protector password = memcached_servers = 127.0.0.1:11211 ``` ### `[service_credentials]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `default_service_project` | string | `service` | The service project name used when creating Keystone trusts. | | `default_service_user_domain` | string | `Default` | Domain for the service user. | | `default_service_project_domain` | string | `Default` | Domain for the service project. | | `default_trust_roles` | list | `member,dr_operator` | Roles delegated in Keystone trusts. Both roles must exist on both sites before any cross-site operation. | | `cafile` | string | — | Path to a PEM-encoded CA bundle for TLS verification on cross-site OpenStack API calls. | | `insecure` | boolean | `false` | Disable TLS certificate verification. Set `true` only in development or lab environments. | ### `[storage]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `storage_driver` | string | `mock` | Storage backend driver. **Must be set to `pure` for production.** The `mock` default performs no real storage operations. | ### `[pure_storage]` | Option | Type | Default | Description | |--------|------|---------|-------------| | `primary_management_ip` | string | — | Management IP of the primary FlashArray. | | `primary_api_token` | string | — | API token for the primary FlashArray. | | `secondary_management_ip` | string | — | Management IP of the secondary FlashArray. | | `secondary_api_token` | string | — | API token for the secondary FlashArray. | | `primary_site_name` | string | `""` | Logical name of the primary site (must match how the Protection Group is registered). | | `secondary_site_name` | string | `""` | Logical name of the secondary site. | | `verify_ssl` | boolean | `false` | Verify TLS certificates when calling FlashArray management APIs. | | `default_rpo_seconds` | integer | `900` | RPO target in seconds used to generate replication health warnings. | | `snapshot_interval_seconds` | integer | `300` | Desired replication snapshot interval. The array may enforce a higher minimum; a warning is logged when it does. | **Snapshot retention** (all under `[pure_storage]`): | Option | Default | Description | |--------|---------|-------------| | `retention_all_for_hours` | `24` | Retain every snapshot for this many hours. | | `retention_per_day` | `24` | After the `all_for` window, retain this many snapshots per day. | | `retention_days` | `7` | Total number of days to retain snapshots (recovery points). | ### Minimal `protector.conf` for cluster1 ```ini [DEFAULT] debug = true log_dir = /var/log/protector use_stderr = true state_path = /var/lib/protector host = cluster1 transport_url = rabbit://openstack:@10.21.67.100:5672// [api] bind_host = 0.0.0.0 bind_port = 8788 workers = 2 [database] connection = mysql+pymysql://protector:ProtectorPass123@10.21.67.105:3306/protector [engine] topic = protector-engine [keystone_authtoken] www_authenticate_uri = http://10.21.67.105:5000 auth_url = http://10.21.67.105:5000 auth_type = password project_domain_name = Default user_domain_name = Default project_name = service username = protector password = ProtectorServicePass123 memcached_servers = 127.0.0.1:11211 [oslo_policy] policy_file = /etc/protector/policy.yaml [service_credentials] default_service_project = service default_service_user_domain = Default default_service_project_domain = Default default_trust_roles = member,dr_operator [storage] storage_driver = pure [pure_storage] primary_management_ip = 10.21.67.50 primary_api_token = secondary_management_ip = 10.21.67.60 secondary_api_token = primary_site_name = cluster1 secondary_site_name = cluster2 verify_ssl = false default_rpo_seconds = 900 snapshot_interval_seconds = 300 retention_all_for_hours = 24 retention_per_day = 24 retention_days = 7 ``` ## Usage Once the service is running and sites are registered, you interact with it via the `openstack dr` CLI (package `triliositerecovery`, installed separately with `pip install -e ./client`). ### Check service status ```bash # Verify containers are running sudo docker ps --filter name=protector_api sudo docker ps --filter name=protector_engine ``` ### List registered sites ```bash openstack dr site list ``` ### Create a Protection Group All VMs in a Protection Group must boot from Cinder volumes, and all attached Cinder volumes must use a volume type with `replication_enabled=' True'`. Attempts to add non-compliant VMs return `InstanceNotCinderBoot` (HTTP 422) or `VolumeTypeNotReplicated` (HTTP 422). ```bash source cluster1-openrc.sh openstack dr protection group create my-pg \ --primary-site cluster1 \ --secondary-site cluster2 \ --replication-type async \ --volume-type pure-replicated ``` ### Add a VM to a Protection Group The `--instance` flag takes a Nova instance ID (not a name). Use `--boot-order` to control startup sequencing during failover: lower numbers start first; VMs with the same value start in parallel. ```bash openstack dr protection group member add \ --instance \ --boot-order 1 ``` Pre-validate a VM's eligibility without adding it: ```bash openstack dr protection group member validate \ --instance ``` This calls `POST /v1/{tenant_id}/protection-groups/{pg_id}/members/validate` and returns HTTP 200 with a structured `errors` array — no side effects. ### Sync metadata to the secondary site After adding members, push the Protection Group metadata to cluster2 so the secondary engine is aware of the group: ```bash openstack dr metadata sync ``` ### Check replication health ```bash openstack dr replication health ``` The response includes `failover_validation.can_failover` (boolean) — use this as a programmatic readiness check before triggering failover. ### Trigger a test failover (non-disruptive) Run this from the **secondary** cluster. Test failover clones replicated volumes and boots isolated VMs with a `test-` prefix on a separate network, leaving production untouched: ```bash source cluster2-openrc.sh openstack dr test failover --auto-network # Monitor progress openstack dr operation show ``` Clean up when done: ```bash openstack dr test failover cleanup ``` ### Trigger a planned failover Run this from the **secondary** cluster. Planned failover gracefully shuts down VMs on the primary and takes a final snapshot for zero data loss: ```bash source cluster2-openrc.sh openstack dr failover --failover-type planned ``` Add `--wait` to block until the operation completes. ### Trigger an unplanned failover Use `--failover-type unplanned` when the primary site is unreachable. Data loss is bounded by the RPO: ```bash openstack dr failover --failover-type unplanned ``` ### Failback to the original primary Once the primary site is restored, run failback from the **primary** cluster: ```bash source cluster1-openrc.sh openstack dr failback ``` ### Restart services ```bash sudo docker restart protector_api protector_engine ``` ### View logs ```bash # Container stdout/stderr sudo docker logs protector_api sudo docker logs protector_engine # File-based logs (more detail) sudo tail -f /var/log/protector/protector-api-wrapper.py.log sudo tail -f /var/log/protector/protector-engine-wrapper.py.log ``` ## Examples ### Example 1 — Install on both clusters and verify ```bash export CLUSTER1_IP=10.21.67.100 export CLUSTER2_IP=10.21.67.101 bash deploy/kolla/install.sh all ``` Expected tail output (per cluster): ``` [INFO] [cluster1] Protector API is healthy [INFO] [cluster1] Protector Engine is healthy [INFO] [cluster1] Installation complete ``` --- ### Example 2 — Version discovery (no authentication required) ```bash curl -s http://10.21.67.100:8788/ ``` Expected response: ```json { "versions": [ { "id": "v1.0", "status": "CURRENT", "version": "1.1", "min_version": "1.0", "links": [ { "rel": "self", "href": "http://10.21.67.100:8788/v1/" } ] } ] } ``` --- ### Example 3 — Create a Protection Group and add a VM ```bash source cluster1-openrc.sh # Create the Protection Group PG_ID=$(openstack dr protection group create web-tier \ --primary-site cluster1 \ --secondary-site cluster2 \ --replication-type async \ --volume-type pure-replicated \ -f value -c id) echo "Protection Group: ${PG_ID}" # Validate VM eligibility before adding openstack dr protection group member validate ${PG_ID} \ --instance # Add the VM openstack dr protection group member add ${PG_ID} \ --instance \ --boot-order 1 # Sync metadata to the secondary site openstack dr metadata sync ${PG_ID} ``` --- ### Example 4 — Check replication health via CLI and API **CLI:** ```bash openstack dr replication health ${PG_ID} ``` **Direct API call:** ```bash TOKEN=$(openstack token issue -f value -c id) PROJECT_ID=$(openstack token issue -f value -c project_id) curl -s \ -H "X-Auth-Token: ${TOKEN}" \ -H "OpenStack-API-Version: protector 1.1" \ "http://10.21.67.100:8788/v1/${PROJECT_ID}/protection-groups/${PG_ID}/replication-health" ``` Expected response structure (fields nested under `replication_health`): ```json { "replication_health": { "protection_group_id": "", "protection_group_name": "web-tier", "health_report": { "status": "healthy", "lag_seconds": 42, "rpo_compliant": true, "recovery_points_available": 12 }, "failover_validation": { "can_failover": true, "warnings": [], "blockers": [], "message": "Protection group is ready for failover" }, "storage_config": {} } } ``` Use `replication_health.failover_validation.can_failover` as the programmatic readiness gate. --- ### Example 5 — Trigger a planned failover and poll for completion ```bash source cluster2-openrc.sh TOKEN=$(openstack token issue -f value -c id) PROJECT_ID=$(openstack token issue -f value -c project_id) # Trigger planned failover RESPONSE=$(curl -s -X POST \ -H "X-Auth-Token: ${TOKEN}" \ -H "Content-Type: application/json" \ -H "OpenStack-API-Version: protector 1.1" \ "http://10.21.67.101:8788/v1/${PROJECT_ID}/protection-groups/${PG_ID}/action" \ -d '{"failover": {"failover_type": "planned"}}') OP_ID=$(echo ${RESPONSE} | python3 -c "import sys,json; print(json.load(sys.stdin)['operation_id'])") echo "Operation: ${OP_ID} (initial status: pending)" # Poll until complete while true; do STATUS=$(curl -s \ -H "X-Auth-Token: ${TOKEN}" \ "http://10.21.67.101:8788/v1/${PROJECT_ID}/operations/${OP_ID}" \ | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['operation']['status'], d['operation']['progress'])") echo "${STATUS}" [[ "${STATUS}" == completed* ]] || [[ "${STATUS}" == failed* ]] && break sleep 10 done ``` --- ### Example 6 — Test failover with a specific recovery point ```bash source cluster2-openrc.sh # List available recovery points openstack dr recovery point list ${PG_ID} # Run a test failover against a specific recovery point, using auto network assignment openstack dr test failover ${PG_ID} \ --auto-network \ --recovery-point # Monitor openstack dr operation show # Clean up test resources when finished openstack dr test failover cleanup ${PG_ID} ``` --- ### Example 7 — Uninstall from both clusters ```bash export CLUSTER1_IP=10.21.67.100 export CLUSTER2_IP=10.21.67.101 bash deploy/kolla/uninstall.sh all ``` This removes: `protector_api` and `protector_engine` containers and their Docker images, Keystone user/service/endpoints, the `protector` database and database user, the Horizon plugin from the `horizon` container, Cinder policy overrides, and all files under `/etc/protector`, `/var/log/protector`, `/var/lib/protector`, and `/opt/openstack-protector`. ## Troubleshooting Use a consistent structure for each issue: **Symptom → Likely cause → Fix**. --- **Symptom:** `docker logs protector_api` shows `connection refused` or the API returns HTTP 000. **Likely cause:** The `protector_api` container failed to start, usually due to a misconfigured `protector.conf` (bad database URL, missing `[keystone_authtoken]` fields) or a port conflict. **Fix:** ```bash sudo docker ps -a --filter name=protector_api sudo docker logs protector_api --tail 50 ``` Correct the offending config key and restart: ```bash sudo docker restart protector_api ``` --- **Symptom:** `openstack dr protection group member add` returns HTTP 422 with error type `InstanceNotCinderBoot`. **Likely cause:** The Nova instance boots from an ephemeral (local) disk rather than a Cinder volume. All VMs in a Protection Group must use Cinder-backed boot volumes. **Fix:** Recreate the instance using `--block-device source_type=image,...,destination_type=volume,volume_type=pure-replicated,...` to ensure the boot disk is a Cinder volume. --- **Symptom:** `openstack dr protection group member add` returns HTTP 422 with error type `VolumeTypeNotReplicated`. **Likely cause:** One or more Cinder volumes attached to the instance use a volume type that does not have `replication_enabled=' True'` set as an extra-spec. **Fix:** ```bash openstack volume type set --property replication_enabled=' True' pure-replicated ``` Confirm with: ```bash openstack volume type show pure-replicated -c extra_specs ``` --- **Symptom:** Engine logs show `py-pure-client SDK not installed` or `ImportError: No module named 'pypureclient'`. **Likely cause:** The Docker image was built without the `py-pure-client` dependency, or the `[storage] storage_driver` is set to `pure` but the Pure SDK is missing inside the container. **Fix:** Verify the SDK is installed in the image: ```bash sudo docker exec protector_engine pip3 list | grep pure ``` If missing, rebuild the image (the `Dockerfile` installs `py-pure-client` via `pip3` in the builder stage). If the problem persists after a rebuild, check that the build completed without errors. --- **Symptom:** Failover fails with `InvalidExecutionSite` (HTTP 400). **Likely cause:** You are attempting to run failover from the **primary** site. Failover must be triggered from the secondary (DR) site. **Fix:** Source the openrc for cluster2 and re-run the command: ```bash source cluster2-openrc.sh openstack dr failover --failover-type planned ``` --- **Symptom:** `openstack dr metadata sync` succeeds locally but the remote site has no record of the Protection Group. **Likely cause:** Port `8788/TCP` is blocked in one direction between the clusters. Cross-site metadata sync requires **bidirectional** connectivity on port 8788. **Fix:** Confirm connectivity in both directions: ```bash # From cluster1 to cluster2 curl -s http://10.21.67.101:8788/ # From cluster2 to cluster1 curl -s http://10.21.67.100:8788/ ``` Both should return the version discovery JSON. Open firewall rules as needed for port 8788/TCP in both directions. --- **Symptom:** Trust creation fails with `Role dr_operator does not exist on site `. **Likely cause:** The `dr_operator` or `dr_site_admin` Keystone role was not created on one of the sites before the first cross-site operation. The service looks up roles by name and does not create them. **Fix:** Create both roles on **both** sites: ```bash openstack role create dr_operator openstack role create dr_site_admin ``` --- **Symptom:** VM remains in `BUILD` state for several minutes after failover or test failover. **Likely cause:** iSCSI multipath attachment for the promoted Pure Storage volume is slow on the DR site's compute node. **Fix:** Wait 2–3 minutes. If the VM eventually reaches `ACTIVE`, the attach succeeded. If it transitions to `ERROR`, check `nova-compute` logs on the DR site compute node for iSCSI or multipath errors. Ensure the compute node has the Pure Storage iSCSI initiator configured. --- **Symptom:** `openstack dr site create` returns an error about missing arguments. **Likely cause:** `dr site create` requires `--site-type` and `--auth-url` at minimum; other arguments may also be required depending on your environment. **Fix:** Check the full required argument list: ```bash openstack dr site create --help ``` --- **Symptom:** The Horizon Site Recovery panel is not visible after deployment. **Likely cause:** The Horizon plugin files were not copied correctly, or `collectstatic` was not run, or the `horizon` container was not restarted. **Fix:** ```bash # Verify plugin files are present inside the horizon container sudo docker exec horizon ls /var/lib/kolla/venv/lib/python3.12/site-packages/protector_dashboards # Force collectstatic and restart if files are present sudo docker exec horizon python3 /var/lib/kolla/venv/bin/manage.py collectstatic --noinput sudo docker restart horizon ``` If files are missing, re-run the installer with only the Horizon step: ```bash bash deploy/kolla/install.sh ${CLUSTER1_IP} cluster1 --skip-db --skip-keystone --skip-cinder ``` --- **Symptom:** `openstack dr` commands are not recognized (`openstack: 'dr' is not an openstack command`). **Likely cause:** The `triliositerecovery` CLI plugin is not installed. Running `pip install -e .` from the repository root installs only the server binaries and does **not** install the CLI. **Fix:** ```bash pip install -e ./client ``` Also ensure you are using `pip`-installed `python-openstackclient`, not the `openstackclient` snap, which is incompatible with third-party CLI plugins.