---
title: RHOSO18
product: trilio-site-recovery-for-openstack
doc_type: guide
version: master
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-openstack/trilio-site-recovery-for-openstack/rhoso18
---

# RHOSO18

_Coming soon (preview): install Trilio Site Recovery on Red Hat OpenStack Services on OpenShift (RHOSO18) — not yet generally available._

## Overview

This page explains how to deploy Trilio Site Recovery for OpenStack on Red Hat OpenStack Services on OpenShift 18 (RHOSO18). The service runs as two Kubernetes Deployments — `trilio-site-recovery-api` and `trilio-site-recovery-engine` — inside a dedicated `trilio-site-recovery` namespace, sharing the Galera cluster and RabbitMQ cluster already present in the `trilio-openstack` namespace. Because this deployment path targets an internal development environment and is not the validated customer-facing method, you should treat these instructions as a technical reference for contributors and evaluators rather than a production runbook. The Juju charm deployment remains the only validated, customer-facing installation method for the tech preview.

## Prerequisites

Before you begin, confirm that the following are in place on every site where you intend to run the service.

**OpenShift and OpenStack**
- Red Hat OpenShift Container Platform with RHOSO18 installed and healthy
- Two independent OpenStack deployments ≥ Victoria (one primary, one secondary), each with Nova, Cinder, Keystone, and Neutron
- Cinder configured with a volume type that has `replication_enabled='<is> True'` as an extra-spec, backed by Pure Storage FlashArray
- Pure Storage FlashArray arrays with an async replication connection between sites (Purity//FA 6.0+ required for ActiveDR; any supported Purity//FA version for async snapshot replication)

**Shared infrastructure (already deployed in `trilio-openstack` namespace)**
- `trilio-galera-cluster` — MariaDB ≥ 10.3 Galera cluster; the install script creates the `protector` database and user automatically
- `trilio-rabbitmq-cluster` — RabbitMQ cluster with management API enabled on port 15671 (TLS)
- `combined-ca-bundle` Secret in the `trilio-openstack` namespace containing `tls-ca-bundle.pem`
- `trilio-openstack-secret` Secret in the `trilio-openstack` namespace (shared with other Trilio products; the install script adds `SrDatabasePassword`, `SrKeystonePassword`, and `SrRabbitPassword` keys)
- `osp-secret` in the `openstack` namespace containing `AdminPassword`

**Tooling**
- `oc` CLI installed and logged in (`oc whoami` must succeed)
- `podman` installed if you need to build the container image locally
- Network access from the `trilio-site-recovery` namespace to:
  - `keystone-public.openstack.svc:5000` (Keystone)
  - `trilio-galera-cluster.trilio-openstack.svc:3306` (MariaDB)
  - `trilio-rabbitmq-cluster.trilio-openstack.svc:5671` (AMQP) and `:15671` (management API)
  - Port 8788 on the remote site's `protector-api` (bidirectional — one-way connectivity causes metadata sync failures)

**Keystone roles**
- The Keystone roles `dr_operator` and `dr_site_admin` must exist on both sites before you run any DR operation. The service looks them up and grants them; it does not create them. Missing roles cause trust creation to fail with a message naming the absent role.

**CLI (optional, on any workstation)**
- `python-openstackclient` installed via `pip` (not the `snap` — the snap is incompatible with the CLI plugin)
- `pip install triliositerecovery` to install the `openstack dr` plugin separately from the server packages

## Installation

The install script at `deploy/openshift/scripts/install.sh` is idempotent — you can re-run it safely. It performs every bootstrap step in order: namespace, RBAC, CA bundle, credential provisioning, init jobs (database, RabbitMQ, Keystone), schema migration, service/route, and deployment rollout.

### Step 1 — Clone the repository and navigate to the scripts directory

```bash
git clone <repository-url>
cd deploy/openshift/scripts
```

### Step 2 — Log in to OpenShift

```bash
oc login --token=<token> --server=https://<api-server>:6443
oc whoami   # must succeed before continuing
```

### Step 3 — Run the install script with default settings

The defaults assume the Trilio shared infrastructure lives in `trilio-openstack` and the OpenStack control plane lives in `openstack`. Adjust with flags if your environment differs.

```bash
bash install.sh
```

To override any of the defaults:

```bash
bash install.sh \
  --namespace trilio-site-recovery \
  --trilio-ns trilio-openstack \
  --openstack-ns openstack
```

To use a custom image (for example, a build from your own registry):

```bash
bash install.sh --image registry.example.com/trilio/trilio-site-recovery-openstack:1.0.0
```

The script performs the following steps automatically; you do not need to run any of them manually:

1. **Creates the `trilio-site-recovery` namespace** from `manifests/namespace.yaml`.
2. **Creates the ServiceAccount and RBAC** from `manifests/serviceaccount.yaml`.
3. **Copies `tls-ca-bundle.pem`** from `combined-ca-bundle` in `trilio-openstack` into the `trilio-site-recovery-ca-bundle` Secret in the service namespace.
4. **Reads admin credentials** from `trilio-galera-cluster-mariadb-root-db-secret`, `trilio-rabbitmq-cluster-default-user`, and `osp-secret`.
5. **Generates or reuses `Sr*` passwords** (`SrDatabasePassword`, `SrKeystonePassword`, `SrRabbitPassword`) in `trilio-openstack-secret`. Use `--rotate-passwords` to force regeneration.
6. **Creates the transient `trilio-site-recovery-init-creds` Secret** used by the init Jobs, then deletes it at the end.
7. **Applies the ConfigMap** (`manifests/configmap.yaml`) with the base `protector.conf` and `api-paste.ini`.
8. **Renders and applies `trilio-site-recovery-secrets`** with live `Sr*` passwords substituted into `protector-secrets.conf`.
9. **Runs the `trilio-site-recovery-db-init` Job** — creates the `protector` database and user on the Galera cluster using the Galera root credentials.
10. **Runs the `trilio-site-recovery-rabbitmq-init` Job** — creates the `protector` RabbitMQ user and grants permissions on the default vhost.
11. **Runs the `trilio-site-recovery-keystone-init` Job** — creates or updates the `protector` Keystone user, ensures `dr_operator` and `dr_site_admin` roles exist, registers the `disaster-recovery` service, and upserts the public/internal/admin endpoints.
12. **Runs the `trilio-site-recovery-db-sync` Job** — migrates the database schema using `protector-manage db sync` (the canonical form of the schema migration command).
13. **Applies the Service and Route** from `manifests/service.yaml` — exposes port 8788 via a ClusterIP Service and an edge-terminated OpenShift Route.
14. **Deploys the API** (`manifests/deployment-api.yaml`) — 2 replicas of the `api` container listening on port 8788.
15. **Deploys the Engine** (`manifests/deployment-engine.yaml`) — 1 replica (singleton; multiple engine replicas cause duplicate RPC handling).
16. **Waits for rollouts** to complete (`oc rollout status`).
17. **Deletes the transient `trilio-site-recovery-init-creds` Secret**.

### Step 4 — Verify the installation

```bash
# Check that both pods are Running
oc get pods -n trilio-site-recovery -l app.kubernetes.io/name=trilio-site-recovery

# Get the public Route URL
ROUTE=$(oc get route trilio-site-recovery-api \
  -n trilio-site-recovery \
  -o jsonpath='{.spec.host}')
echo "API URL: https://${ROUTE}"

# Version discovery — no authentication required
curl -kSs "https://${ROUTE}/" | python3 -m json.tool
```

A healthy response looks like:

```json
{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "https://<route-host>/v1/"
        }
      ]
    }
  ]
}
```

### Building the container image locally

If you need to build and push your own image:

```bash
cd deploy/openshift/scripts
bash build.sh --registry registry.example.com/trilio --tag 1.0.0 --push
```

Then pass `--image registry.example.com/trilio/trilio-site-recovery-openstack:1.0.0` to `install.sh`.

### Uninstalling

To remove in-namespace resources while leaving the database, RabbitMQ user, Keystone artefacts, and `Sr*` secret keys intact:

```bash
bash uninstall.sh
```

To also drop the `protector` database and user on Galera, delete the RabbitMQ user, remove the Keystone service and endpoints, and strip the `Sr*` keys from `trilio-openstack-secret`:

```bash
bash uninstall.sh --purge
```

To additionally delete the namespace itself:

```bash
bash uninstall.sh --purge --delete-namespace
```

## Configuration

Configuration is split into two Kubernetes objects so that secrets never appear in the ConfigMap:

- **`trilio-site-recovery-config` ConfigMap** — base `protector.conf` containing non-sensitive options, plus `api-paste.ini` and `policy.yaml`.
- **`trilio-site-recovery-secrets` Secret** — `protector-secrets.conf` containing the database connection string, RabbitMQ transport URL, Keystone credentials, and Pure Storage tokens. This file is applied on top of the base config at startup via a second `--config-file` argument, so its values override any matching key in the ConfigMap.

Both are projected into the container at `/etc/trilio-site-recovery`.

> **Important:** Do not edit these objects directly in a running cluster and then re-run the install script — the script will overwrite your changes. Make all persistent configuration changes in the manifests before running `install.sh`, or patch the objects with `oc patch` / `oc edit` if you are making a one-off adjustment to a running deployment.

---

### Base configuration (ConfigMap — `protector.conf`)

#### `[DEFAULT]`

| Option | Default | Description |
|---|---|---|
| `debug` | `false` | Enable verbose debug logging. Set to `true` only when diagnosing a problem — it produces very large log volumes. |
| `use_stderr` | `true` | Log to stderr (captured by the container runtime). |
| `use_journal` | `false` | Log to systemd journal — leave `false` in OpenShift. |
| `state_path` | `/var/lib/trilio-site-recovery` | Directory for runtime state files (engine emptyDir volume). |
| `rpc_conn_pool_size` | `30` | oslo.messaging connection pool size between API and engine. |
| `use_mock_storage` | `false` | Skip real Pure Storage array calls. Set to `true` in development; **must be `false` in production**. |
| `use_mock_cinder` | `false` | Skip real Cinder calls. Set to `true` in development; **must be `false` in production**. |

#### `[oslo_messaging_rabbit]`

| Option | Default in this deployment | Description |
|---|---|---|
| `rabbit_durable_queues` | `true` | Durable queues survive broker restarts. |
| `amqp_durable_queues` | `true` | Durable AMQP queues. |
| `ssl` | `true` | Enable TLS for RabbitMQ connections (required when `trilio-rabbitmq-cluster` uses TLS). |
| `ssl_ca_file` | `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` | Path to the CA bundle for verifying the broker's certificate. |

#### `[api]`

| Option | Default | Description |
|---|---|---|
| `bind_host` | `0.0.0.0` | Address the API worker binds to. Leave as `0.0.0.0` in containers. |
| `bind_port` | `8788` | TCP port. Must match the Service `targetPort`. |
| `workers` | `2` | Number of API worker processes per pod. The compiled-in oslo.config default is 4, but the OpenShift ConfigMap ships `workers = 2`. Increase for higher throughput, keeping in mind pod memory limits. |

#### `[engine]`

| Option | Default | Description |
|---|---|---|
| `topic` | `protector-engine` | RabbitMQ topic the engine listens on. Must match the API's `engine.topic`. |

#### `[keystone_authtoken]`

| Option | Value in this deployment | Description |
|---|---|---|
| `auth_type` | `password` | Keystone auth method. |
| `project_domain_name` | `Default` | Domain of the service project. |
| `user_domain_name` | `Default` | Domain of the `protector` service user. |
| `project_name` | `service` | The service project name. |
| `memcached_servers` | `memcached.openstack.svc:11211` | Token cache; reduces Keystone load. |

#### `[service_credentials]`

| Option | Default | Description |
|---|---|---|
| `default_service_project` | `service` | Project used for cross-site service API calls. |
| `default_service_user_domain` | `Default` | Domain of the service user. |
| `default_service_project_domain` | `Default` | Domain of the service project. |
| `default_trust_roles` | `member,dr_operator` | Roles delegated in Keystone trusts for cross-site operations. Both roles must exist on both sites. |
| `cafile` | _(none)_ | PEM CA bundle for verifying TLS on cross-site OpenStack API calls. Set this to the bundle path when your endpoints use private CAs. |
| `insecure` | `false` | Disable TLS verification for cross-site calls. Set to `true` only in development. |

#### `[pure_storage]`

| Option | Default | Description |
|---|---|---|
| `verify_ssl` | `false` | Verify TLS certificates when connecting to the FlashArray management API. |
| `default_rpo_seconds` | `900` | RPO target in seconds used for replication health warnings. |
| `snapshot_interval_seconds` | `300` | Desired replication snapshot interval. The array may enforce a higher minimum; a warning is logged when it clamps this value. |

#### `[mock_storage]`

| Option | Default in this deployment | Description |
|---|---|---|
| `storage_base_path` | `/var/lib/trilio-site-recovery/mock_storage` | Base path for the mock storage driver's state files. The path must be writable by the engine pod's UID. |

---

### Secrets overlay (`protector-secrets.conf`)

The install script renders `manifests/secret.yaml` with live `Sr*` passwords and applies it. The resulting Secret contains a single key `protector-secrets.conf` that overrides the ConfigMap at startup. The sensitive sections are:

#### `[DEFAULT]`

```ini
transport_url = rabbit://protector:<SrRabbitPassword>@trilio-rabbitmq-cluster.trilio-openstack.svc:5671//
```

#### `[database]`

```ini
connection = mysql+pymysql://protector:<SrDatabasePassword>@trilio-galera-cluster.trilio-openstack.svc:3306/protector?ssl_ca=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
```

#### `[keystone_authtoken]`

```ini
auth_url = https://keystone-public.openstack.svc:5000
www_authenticate_uri = https://keystone-public-openstack.apps.<cluster-domain>
cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
username = protector
password = <SrKeystonePassword>
```

#### `[service_credentials]`

```ini
auth_url = https://keystone-public.openstack.svc:5000
cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
username = protector
password = <SrKeystonePassword>
project_name = service
user_domain_name = Default
project_domain_name = Default
```

#### `[pure_storage]` — production values (uncomment when ready)

```ini
# primary_management_ip = 10.21.67.50
# primary_api_token = <token>
# primary_site_name = cluster1
# secondary_management_ip = 10.21.67.60
# secondary_api_token = <token>
# secondary_site_name = cluster2
```

When you are ready to connect real Pure Storage arrays, edit the Secret and also flip `use_mock_storage = false` and `use_mock_cinder = false` in the ConfigMap, then update `[storage]` with `storage_driver = pure`:

```bash
oc edit configmap trilio-site-recovery-config -n trilio-site-recovery
# Change use_mock_storage = false, use_mock_cinder = false
# Add:
# [storage]
# storage_driver = pure

oc edit secret trilio-site-recovery-secrets -n trilio-site-recovery
# Populate [pure_storage] with real IPs and API tokens (base64-encode the file)

# Restart both workloads to pick up the new config
oc rollout restart deployment/trilio-site-recovery-api -n trilio-site-recovery
oc rollout restart deployment/trilio-site-recovery-engine -n trilio-site-recovery
```

---

### Policy

The `policy.yaml` shipped in the ConfigMap grants all protection-group and member operations to project members (`rule:default` = `is_admin:True or project_id:%(project_id)s`). Site-level operations (`sites:action`, `sites:failover`) and site operation listing require `role:dr_site_admin or role:admin`. Adjust this file to tighten or loosen access control for your environment.

## Usage

Once the service is running and endpoints are registered in Keystone, you interact with it via the `openstack dr` CLI plugin or directly through the REST API.

### Installing the CLI plugin

The CLI plugin is a separate Python package. Install it on any workstation that has `python-openstackclient` available (install via `pip`, not the `snap`):

```bash
pip install triliositerecovery
```

Verify the plugin is loaded:

```bash
openstack dr site list
```

### Authenticating

All API calls require an `X-Auth-Token` obtained from Keystone on port 5000. The CLI handles this automatically when your `OS_*` environment variables are set. For direct API calls:

```bash
export OS_AUTH_URL=https://keystone-public-openstack.apps.<cluster-domain>
export OS_USERNAME=<your-username>
export OS_PASSWORD=<your-password>
export OS_PROJECT_NAME=<your-project>
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3

TOKEN=$(openstack token issue -f value -c id)
```

### Registering sites

Register both your primary and secondary OpenStack sites. The `dr site create` command requires `--site-type`, `--auth-url`, and additional mandatory arguments — run `openstack dr site create --help` for the full list before proceeding.

```bash
openstack dr site create primary-site \
  --site-type PRIMARY \
  --auth-url https://keystone-primary.example.com:5000 \
  --service-username protector \
  --service-password <password> \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default

openstack dr site create secondary-site \
  --site-type SECONDARY \
  --auth-url https://keystone-secondary.example.com:5000 \
  --service-username protector \
  --service-password <password> \
  --service-project-name service \
  --service-user-domain-name Default \
  --service-project-domain-name Default
```

### Creating a tenant mapping

Because each site has an independent Keystone, the same tenant has different UUIDs on each site. Register that mapping:

```bash
openstack dr tenant mapping create \
  --local-tenant <primary-project-id> \
  --remote-site <secondary-site-id> \
  --remote-tenant <secondary-project-id>
```

### Creating a Protection Group

A Protection Group names the collection of VMs you want to replicate and fail over together:

```bash
openstack dr protection group create my-app-pg \
  --primary-site <primary-site-id> \
  --secondary-site <secondary-site-id> \
  --replication-type async
```

### Adding VMs to a Protection Group

Every VM must boot from a Cinder volume (not ephemeral storage), and all attached volumes must use a volume type with `replication_enabled='<is> True'`. Validate eligibility first to avoid side effects:

```bash
# Pre-check — returns HTTP 200 with an 'errors' array; no side effects
openstack dr protection group member validate my-app-pg \
  --instance <nova-instance-id>

# Add if validation passes
openstack dr protection group member add my-app-pg \
  --instance <nova-instance-id> \
  --boot-order 1
```

VMs with the same `--boot-order` value are started and stopped in parallel during failover; lower-numbered tiers complete before higher-numbered tiers begin.

### Running a test failover (non-disruptive DR validation)

A test failover clones replicated volumes and boots isolated test VMs on a separate network without touching production:

```bash
openstack dr test failover <pg-id> --auto-network
```

You must supply either `--auto-network` or `--network-mapping`; the command is rejected without one of them.

Clean up test resources when done:

```bash
openstack dr test failover cleanup <pg-id>
```

To target a specific test failover operation rather than the most recent one:

```bash
openstack dr test failover cleanup <pg-id> --operation <operation-id>
```

### Executing a planned failover

```bash
openstack dr failover <pg-id> --failover-type planned --wait
```

`--failover-type planned` gracefully shuts down VMs and takes a final snapshot before promoting the replica — zero data loss. Use `--failover-type unplanned` when the primary site is unavailable and you accept data loss up to the RPO.

### Monitoring replication health

```bash
openstack dr replication health <pg-id>
```

Check `replication_health.failover_validation.can_failover` programmatically to gate automated failover workflows:

```bash
curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  "https://${ROUTE}/v1/${OS_PROJECT_ID}/protection-groups/<pg-id>/replication-health" \
  | python3 -m json.tool
```

### Tracking operations

All long-running DR operations (failover, failback, test failover, sync) return an `operation_id` immediately with HTTP 202. Poll the operation until it reaches 100% progress:

```bash
openstack dr operation show <operation-id>

# Or via the API
curl -s \
  -H "X-Auth-Token: ${TOKEN}" \
  "https://${ROUTE}/v1/${OS_PROJECT_ID}/operations/<operation-id>" \
  | python3 -m json.tool
```

To cancel an in-progress operation:

```bash
openstack dr operation cancel <operation-id>
```

## Examples

### Example 1 — Verify service health (no authentication required)

The `/healthz` endpoint returns `{"status": "ok"}` when the service is up and the database is reachable.

```bash
ROUTE=$(oc get route trilio-site-recovery-api \
  -n trilio-site-recovery \
  -o jsonpath='{.spec.host}')

curl -kSs "https://${ROUTE}/healthz"
```

**Expected output:**
```json
{"status": "ok", "checks": {"database": "ok"}}
```

If the database is unreachable the status code is 503:
```json
{"status": "unhealthy", "checks": {"database": "OperationalError"}}
```

---

### Example 2 — Version discovery (no authentication required)

```bash
curl -kSs "https://${ROUTE}/" | python3 -m json.tool
```

**Expected output:**
```json
{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "https://<route-host>/v1/"
        }
      ]
    }
  ]
}
```

---

### Example 3 — List Protection Groups via the REST API

```bash
TOKEN=$(openstack token issue -f value -c id)
PROJECT_ID=$(openstack token issue -f value -c project_id)

curl -kSs \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "OpenStack-API-Version: protector 1.1" \
  "https://${ROUTE}/v1/${PROJECT_ID}/protection-groups" \
  | python3 -m json.tool
```

The `OpenStack-API-Version` header is optional. When omitted the service defaults to microversion 1.0 and processes the request normally.

**Expected output (empty project):**
```json
{"protection_groups": []}
```

---

### Example 4 — Trigger a planned failover and poll to completion

```bash
PG_ID="<protection-group-id>"

# Trigger the failover
RESPONSE=$(curl -ksSf \
  -X POST \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "Content-Type: application/json" \
  "https://${ROUTE}/v1/${PROJECT_ID}/protection-groups/${PG_ID}/action" \
  -d '{"failover": {"failover_type": "planned"}}')

echo "$RESPONSE" | python3 -m json.tool
OP_ID=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['operation_id'])")
```

**Initial response (HTTP 202):**
```json
{
  "message": "Failover initiated",
  "status": "pending",
  "protection_group_id": "<pg-id>",
  "operation_id": "<op-id>",
  "operation": { ... }
}
```

The initial `status` is `pending`, not `running`. Poll until `progress` reaches 100:

```bash
while true; do
  STATUS=$(curl -ksSf \
    -H "X-Auth-Token: ${TOKEN}" \
    "https://${ROUTE}/v1/${PROJECT_ID}/operations/${OP_ID}" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['status'], d.get('progress',0))")
  echo "$(date): ${STATUS}"
  [[ "$STATUS" == completed* || "$STATUS" == failed* ]] && break
  sleep 10
done
```

Alternatively, pass `--wait` to the CLI and it blocks automatically:

```bash
openstack dr failover "${PG_ID}" --failover-type planned --wait
```

---

### Example 5 — Pre-validate a member before adding it

Use the validate endpoint to check eligibility without creating anything:

```bash
curl -ksSf \
  -X POST \
  -H "X-Auth-Token: ${TOKEN}" \
  -H "Content-Type: application/json" \
  "https://${ROUTE}/v1/${PROJECT_ID}/protection-groups/${PG_ID}/members/validate" \
  -d '{"instance_id": "<nova-instance-id>"}' \
  | python3 -m json.tool
```

**Expected output when eligible (HTTP 200):**
```json
{"errors": []}
```

**Output when ineligible (HTTP 200, errors populated):**
```json
{
  "errors": [
    {
      "type": "InstanceNotCinderBoot",
      "message": "Instance <id> is not a Cinder boot instance. All protected VMs must boot from a Cinder volume."
    }
  ]
}
```

---

### Example 6 — Check replication health programmatically

```bash
HEALTH=$(curl -ksSf \
  -H "X-Auth-Token: ${TOKEN}" \
  "https://${ROUTE}/v1/${PROJECT_ID}/protection-groups/${PG_ID}/replication-health")

echo "$HEALTH" | python3 -m json.tool

# Extract the failover readiness flag
CAN_FAILOVER=$(echo "$HEALTH" | python3 -c \
  "import sys,json; d=json.load(sys.stdin); \
   print(d['replication_health']['failover_validation']['can_failover'])")

echo "Can failover: ${CAN_FAILOVER}"
```

---

### Example 7 — Rotate `Sr*` passwords

To regenerate all `Sr*` passwords (database, Keystone, RabbitMQ) without a full reinstall:

```bash
bash install.sh --rotate-passwords
```

The script generates new random passwords, patches `trilio-openstack-secret`, re-renders `trilio-site-recovery-secrets`, and re-runs the init jobs. Both deployments are then restarted automatically as part of the rollout steps.

## Troubleshooting

Use a consistent diagnostic workflow: check pod status → inspect logs → verify the specific component or secret involved.

```bash
# Pod overview
oc get pods -n trilio-site-recovery

# API logs (last 100 lines)
oc logs -n trilio-site-recovery \
  -l app.kubernetes.io/component=api --tail=100

# Engine logs
oc logs -n trilio-site-recovery \
  -l app.kubernetes.io/component=engine --tail=100

# Init job logs (replace <component> with db-init, keystone-init, etc.)
oc logs -n trilio-site-recovery \
  -l app.kubernetes.io/component=<component> --tail=200
```

---

### Issue: Install script fails with "combined-ca-bundle not found"

**Symptom:** `install.sh` exits with `combined-ca-bundle not found in trilio-openstack`.

**Cause:** The shared `trilio-openstack` namespace does not contain the expected CA bundle Secret, or the `--trilio-ns` flag points to the wrong namespace.

**Fix:**
1. Confirm the Secret exists: `oc get secret combined-ca-bundle -n trilio-openstack`
2. If you used a non-default namespace for shared infra, pass the correct value: `bash install.sh --trilio-ns <your-trilio-ns>`

---

### Issue: `trilio-site-recovery-db-init` Job fails

**Symptom:** Job enters `Failed` state; logs show `Access denied for user 'root'` or a connection refused error.

**Cause:** Either the Galera root password Secret key is wrong, or the Galera service is not reachable from the `trilio-site-recovery` namespace.

**Fix:**
1. Verify the Secret key: `oc get secret trilio-galera-cluster-mariadb-root-db-secret -n trilio-openstack -o jsonpath='{.data.DatabasePassword}' | base64 -d`
2. Check network policy: from a debug pod in `trilio-site-recovery`, test `nc -zv trilio-galera-cluster.trilio-openstack.svc 3306`
3. Re-run the install script after fixing connectivity — it is idempotent.

---

### Issue: `trilio-site-recovery-keystone-init` Job fails with "Role dr_operator does not exist"

**Symptom:** The keystone-init Job log shows `Role dr_operator does not exist on site <site>` and exits non-zero.

**Cause:** The Keystone init job creates the roles if they are absent on the **local** site (see the `for R in dr_operator dr_site_admin` loop in `job-keystone-init.yaml`). However, if the remote site does not have the roles, trust creation fails during the first cross-site DR operation.

**Fix:** Create both roles on the remote site using an admin account on that site:
```bash
openstack role create dr_operator
openstack role create dr_site_admin
```

---

### Issue: API pod starts but returns HTTP 500 on all requests

**Symptom:** `curl https://<route>/` returns a version document, but authenticated calls to `/v1/<project>/protection-groups` return 500.

**Cause:** Most commonly a database connectivity failure — the `[database] connection` string in the secrets overlay is incorrect, or the schema has not been migrated.

**Fix:**
1. Check API logs for `OperationalError` or `Can't connect to MySQL`.
2. Verify the database connection string in the Secret: `oc get secret trilio-site-recovery-secrets -n trilio-site-recovery -o jsonpath='{.data.protector-secrets\.conf}' | base64 -d`
3. If the `db-sync` Job failed, check its logs and re-run the install script to retry it.

---

### Issue: Engine pod is in `CrashLoopBackOff`

**Symptom:** `oc get pods` shows `trilio-site-recovery-engine-<hash>` with status `CrashLoopBackOff`.

**Cause:** Usually a misconfigured `transport_url` — the engine cannot connect to RabbitMQ — or a missing `[storage]` section when `use_mock_storage = false`.

**Fix:**
1. `oc logs -n trilio-site-recovery -l app.kubernetes.io/component=engine --previous --tail=50`
2. Look for `AMQP server ... is unreachable` → verify the `transport_url` in `trilio-site-recovery-secrets`.
3. Look for `StorageDriverConfigError` → confirm `storage_driver = pure` and all `[pure_storage]` options are populated in the Secret.
4. After editing the Secret, restart the engine: `oc rollout restart deployment/trilio-site-recovery-engine -n trilio-site-recovery`

---

### Issue: `openstack dr site create` fails with "Authentication failed"

**Symptom:** `SiteAuthenticationFailed` (HTTP 401) when registering a remote site.

**Cause:** The `--service-username` / `--service-password` credentials are wrong, or the `protector` user does not exist on the remote site's Keystone.

**Fix:**
1. Confirm the user exists on the remote site: `openstack user show protector` (authenticated against the remote Keystone).
2. Re-run the keystone-init Job on the remote site, or create the user manually and set the password.
3. Retry `openstack dr site set <site> --service-password <correct-password>`.

---

### Issue: Member add fails with HTTP 422 `InstanceNotCinderBoot`

**Symptom:** `openstack dr protection group member add` returns `InstanceNotCinderBoot`.

**Cause:** The Nova instance uses ephemeral (local) storage for its boot disk rather than a Cinder volume.

**Fix:** You cannot protect this instance without migrating it to a Cinder-backed boot volume. Use `openstack server rebuild` or create a new instance from a Cinder volume, then add it to the Protection Group.

---

### Issue: Member add fails with HTTP 422 `VolumeTypeNotReplicated`

**Symptom:** `openstack dr protection group member add` returns `VolumeTypeNotReplicated`.

**Cause:** One or more Cinder volumes attached to the instance use a volume type that lacks `replication_enabled='<is> True'` as an extra-spec.

**Fix:**
1. Identify the volume type: `openstack volume show <volume-id> -f value -c volume_type`
2. Check its extra-specs: `openstack volume type show <type> -f json`
3. Either set the extra-spec on the existing type (admin operation): `openstack volume type set --property replication_enabled='<is> True' <type>` — or retype the volume to a replicated type before adding the instance.

---

### Issue: Cross-site metadata sync fails / tenant mapping does not propagate

**Symptom:** `openstack dr tenant mapping create` succeeds locally but operations on the remote site report `TenantMappingNotFound`.

**Cause:** Port 8788 connectivity between sites is one-directional or blocked. Cross-site metadata sync requires **bidirectional** connectivity on port 8788.

**Fix:**
1. From the engine pod on site A, test reachability to site B: `curl -ksSf https://<site-b-protector-api>:8788/` should return the version document.
2. Repeat the test in the opposite direction from site B to site A.
3. Update network policies, firewall rules, or OpenShift Routes to allow traffic in both directions.
4. Force a re-sync after restoring connectivity: `openstack dr metadata sync <pg-id>`

---

### Issue: `openstack dr` commands not found after `pip install triliositerecovery`

**Symptom:** `openstack dr site list` returns `openstack: 'dr' is not an openstack command`.

**Cause:** Either the `openstackclient` snap is in use (incompatible with the plugin), or the plugin was installed into a different Python environment.

**Fix:**
1. Confirm you are using pip-installed `python-openstackclient`, not the snap: `which openstack` — if the path contains `snap`, remove it and reinstall via pip.
2. Ensure the plugin is installed in the same environment: `pip show triliositerecovery`
3. Verify the entry point is registered: `openstack --help | grep dr`
