---
title: Observability
product: trilio-site-recovery-for-openstack
doc_type: guide
version: master
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-openstack/trilio-site-recovery-for-openstack/observability
---

# Observability

_Logging, metrics, tracing, health checks_

## Overview

This page covers the observability surface of Trilio Site Recovery for OpenStack: structured logging, replication health metrics, DR operation tracking, and service health monitoring. Understanding these signals lets you detect replication degradation before it affects your recovery point objective, confirm that failover is safe to execute, and diagnose failures when a DR operation does not complete successfully. The storage driver exposes a rich set of replication metrics and health checks — including link status, lag, sync state, recovery point availability, and transfer throughput — that flow through the API and CLI so you can monitor and automate DR readiness without vendor-specific tooling.

## Prerequisites

Before using the observability features described here:

- Trilio Site Recovery deployed and configured on both sites (Juju charm deployment — the only validated customer-facing method).
- `python-openstackclient` installed via `pip` (not snap — the snap is incompatible with the `triliositerecovery` CLI plugin).
- The `triliositerecovery` CLI plugin installed separately: `pip install -e ./client` from the repository root, or the `python3-triliositerecovery` Debian package.
- A valid Keystone token with `member` and `dr_operator` roles on your project.
- At least one Protection Group created and at least one VM added to it.
- Network connectivity from your workstation to `protector-api` on port 8788 on each site.
- For service-level health, access to the `protector-api` endpoint (port 8788).

## Installation

The observability tools are part of the standard Trilio Site Recovery installation. No additional components are required.

**1. Verify the CLI plugin is installed**

```bash
pip show triliositerecovery
```

If not installed, install it separately from the repository root:

```bash
pip install -e ./client
```

Or, on a Juju-managed node, install the Debian package:

```bash
sudo apt-get install python3-triliositerecovery
```

**2. Confirm the API is reachable**

Retrieve the version discovery document (no authentication required):

```bash
curl -s http://<protector-api-host>:8788/
```

Expected response:

```json
{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://<protector-api-host>:8788/v1/"}]
    }
  ]
}
```

A 200 response confirms the API process is running.

**3. Obtain a Keystone token**

All authenticated API calls require an `X-Auth-Token` obtained from Keystone on port 5000:

```bash
export OS_TOKEN=$(openstack token issue -f value -c id)
```

**4. Confirm replication health access**

```bash
openstack dr replication health <protection-group-id>
```

A successful response confirms that observability is working end-to-end.

## Configuration

Observability-related configuration lives in `protector.conf`. In Juju deployments, **do not edit this file directly** — the charm overwrites it on every `config-changed` event. Use `juju config trilio-site-recovery <key>=<value>` with hyphenated option names instead.

### Logging

Trilio Site Recovery uses the standard `oslo.log` framework. Log output goes to `/var/log/protector/` on each site. Log verbosity is controlled by the standard `oslo.log` options (`debug`, `verbose`) set via Juju config or `config-flags`.

| Juju option (hyphenated) | protector.conf equivalent | Default | Effect |
|---|---|---|---|
| `config-flags` with `DEFAULT:debug=true` | `[DEFAULT] debug = true` | `false` | Enable DEBUG-level log output |

### Engine: Recovery Point Scheduling

The engine can automatically create recovery points (snapshots) on a configurable interval. This affects how much history appears in the replication health report.

| Section | Option | Default | Description |
|---|---|---|---|
| `[engine]` | `recovery_point_interval_minutes` | `0` | How often to create scheduled recovery points. Set to `0` to disable. |
| `[engine]` | `recovery_point_max_count` | `20` | Maximum number of recovery points to retain per Protection Group. |
| `[engine]` | `recovery_point_ttl_hours` | `24` | How long recovery points are retained (hours). |

In Juju deployments, set these via `config-flags`:

```bash
juju config trilio-site-recovery \
  config-flags='engine:recovery_point_interval_minutes=15,engine:recovery_point_max_count=48'
```

### Storage Driver: RPO and Snapshot Interval

For async replication, the following `[pure_storage]` options control the replication cadence that underpins lag and RPO metrics:

| Section | Option | Default | Description |
|---|---|---|---|
| `[pure_storage]` | `default_rpo_seconds` | `900` | Target RPO in seconds (15 minutes). Used to evaluate `rpo_compliant` in health reports. |
| `[pure_storage]` | `snapshot_interval_seconds` | `300` | How often the driver takes a Protection Group snapshot. |
| `[pure_storage]` | `verify_ssl` | `false` | Whether to verify SSL certificates when connecting to FlashArray management IPs. Set `true` in production. |

In Juju deployments, set these via charm config options (hyphenated):

```bash
juju config trilio-site-recovery \
  pure-primary-management-ip=<ip> \
  pure-secondary-management-ip=<ip>
```

### TLS for Cross-Site Calls

| Section | Option | Default | Description |
|---|---|---|---|
| `[service_credentials]` | `cafile` | _(none)_ | Path to a PEM-encoded CA bundle for verifying TLS on cross-site OpenStack endpoint calls. |
| `[service_credentials]` | `insecure` | `false` | Disable TLS certificate verification for cross-site calls. Use only in development. |

## Usage

### Checking Replication Health

The primary observability command is `openstack dr replication health`. It queries the storage driver and returns a structured report covering link status, lag, RPO compliance, sync state, transfer throughput, and available recovery points.

```bash
openstack dr replication health <protection-group-id-or-name>
```

The API equivalent is:

```bash
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/protection-groups/<pg-id>/replication-health
```

The response nests all fields under `replication_health`. Key fields to watch:

| Field path | Type | What it tells you |
|---|---|---|
| `replication_health.health_report.link.status` | string | `healthy`, `degraded`, `broken`, or `unknown` |
| `replication_health.health_report.link.connected` | boolean | Whether the two arrays can see each other |
| `replication_health.health_report.lag.lag_seconds` | integer | Current replication lag in seconds |
| `replication_health.health_report.lag.rpo_compliant` | boolean | Whether lag is within the configured RPO target |
| `replication_health.health_report.lag.rpo_violation_duration` | integer | How long (seconds) the RPO has been violated |
| `replication_health.health_report.sync_status.in_sync` | boolean | Whether primary and secondary are fully synchronized |
| `replication_health.health_report.sync_status.sync_progress_pct` | float | Percentage complete if a resync is in progress |
| `replication_health.health_report.recovery_points.total_count` | integer | Number of available recovery points on the secondary |
| `replication_health.health_report.recovery_points.newest_replicated` | object | Most recent recovery point confirmed on the target |
| `replication_health.health_report.recovery_points.missed_snapshots_24h` | integer | Snapshots missed in the last 24 hours |
| `replication_health.health_report.transfer.throughput_bytes_per_sec` | float | Current replication throughput |
| `replication_health.health_report.transfer.replication_keeping_up` | boolean | Whether replication rate matches the source write rate |
| `replication_health.failover_validation.can_failover` | boolean | **Primary readiness indicator** — use this for automation |
| `replication_health.failover_validation.blockers` | array | Reasons failover is blocked (if `can_failover` is false) |
| `replication_health.failover_validation.warnings` | array | Non-blocking warnings |
| `replication_health.data_at_risk_seconds` | integer | Estimated data loss (seconds) if the primary fails right now |

Use `replication_health.failover_validation.can_failover` as the single programmatic readiness check in automation scripts.

### Tracking DR Operations

Every failover, failback, test failover, and sync creates a tracked DR Operation with a unique ID. Operations progress from `pending` → `running` → `completed` (or `failed`).

**List all operations:**

```bash
openstack dr operation list
```

Filter by Protection Group or status:

```bash
openstack dr operation list --protection-group <pg-id> --status running
```

**Inspect a specific operation:**

```bash
openstack dr operation show <operation-id>
```

The API equivalent:

```bash
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/operations/<operation-id>
```

The operation response includes:
- `status` — current status (`pending`, `running`, `completed`, `failed`)
- `progress` — integer 0–100
- `current_phase` — which workflow step is executing
- `steps_completed` — list of completed steps
- `steps_failed` — list of failed steps with details
- `instances_created`, `volumes_managed`, `networks_configured` — resources touched
- `error_message` — failure reason if `status` is `failed`
- `warnings` — non-fatal issues encountered

**Cancel an in-progress operation:**

```bash
openstack dr operation cancel <operation-id>
```

### Listing Recovery Points

Recovery points are point-in-time snapshots available on the secondary site:

```bash
openstack dr recovery point list <protection-group-id>
```

The API equivalent:

```bash
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/protection-groups/<pg-id>/recovery-points
```

Each recovery point reports its name, creation time, age in seconds, whether it has been replicated to the target, and whether it is a consistency-group-level snapshot.

### Checking Metadata Sync Status

Metadata synchronization state (whether both sites agree on the Protection Group's configuration) is at:

```bash
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/protection-groups/<pg-id>/metadata/status
```

If metadata is out of sync, trigger a sync:

```bash
openstack dr metadata sync <protection-group-id>
```

or

```bash
openstack dr protection group sync <protection-group-id>
```

### Viewing Logs

Logs for both services are written to `/var/log/protector/` on each site. In a Juju deployment, tail logs with:

```bash
# On the unit running protector-api/protector-engine
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-api.log
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-engine.log
```

Enable debug logging temporarily:

```bash
juju config trilio-site-recovery config-flags='DEFAULT:debug=true'
# Review logs, then disable:
juju config trilio-site-recovery config-flags='DEFAULT:debug=false'
```

## Examples

### Example 1: Check whether a Protection Group is ready to fail over

```bash
openstack dr replication health my-pg
```

Expected output (async replication, healthy):

```
+-----------------------------------+----------------------------------------------------------+
| Field                             | Value                                                    |
+-----------------------------------+----------------------------------------------------------+
| protection_group_id               | a1b2c3d4-...                                             |
| protection_group_name             | my-pg                                                    |
| failover_validation.can_failover  | True                                                     |
| failover_validation.message       | Ready with 245s data at risk                             |
| health_report.link.status         | healthy                                                  |
| health_report.link.connected      | True                                                     |
| health_report.lag.lag_seconds     | 245                                                      |
| health_report.lag.rpo_compliant   | True                                                     |
| health_report.sync_status.in_sync | True                                                     |
| health_report.recovery_points.total_count | 12                                               |
| data_at_risk_seconds              | 245                                                      |
+-----------------------------------+----------------------------------------------------------+
```

---

### Example 2: Poll replication health via the REST API

```bash
export OS_TOKEN=$(openstack token issue -f value -c id)
export TENANT_ID=$(openstack project show -f value -c id $OS_PROJECT_NAME)
export PG_ID=a1b2c3d4-0000-0000-0000-000000000001

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  "http://protector-primary.example.com:8788/v1/${TENANT_ID}/protection-groups/${PG_ID}/replication-health" \
  | python3 -m json.tool
```

Expected response structure:

```json
{
  "replication_health": {
    "protection_group_id": "a1b2c3d4-0000-0000-0000-000000000001",
    "protection_group_name": "my-pg",
    "failover_validation": {
      "can_failover": true,
      "message": "Ready with 245s data at risk",
      "warnings": [],
      "blockers": []
    },
    "health_report": {
      "storage_group_name": "protector-my-pg",
      "replication_type": "async",
      "collected_at": "2025-01-15T10:30:00",
      "link": {
        "status": "healthy",
        "direction": "source_to_target",
        "connected": true,
        "last_healthy_at": "2025-01-15T10:29:55",
        "message": "Replication active"
      },
      "sync_status": {
        "in_sync": true,
        "sync_progress_pct": null,
        "last_consistent_point": "2025-01-15T10:25:45",
        "seconds_since_consistent": 255
      },
      "lag": {
        "lag_seconds": 245,
        "lag_bytes": 10485760,
        "rpo_compliant": true,
        "rpo_target_seconds": 900,
        "rpo_violation_duration": null,
        "lag_trend": "stable",
        "avg_lag_seconds_1h": 230.5
      },
      "recovery_points": {
        "total_count": 12,
        "newest_replicated": {
          "name": "protector-my-pg.snap-20250115T102545",
          "created_at": "2025-01-15T10:25:45",
          "age_seconds": 255,
          "is_replicated": true,
          "is_consistent": true
        },
        "missed_snapshots_24h": 0
      },
      "transfer": {
        "throughput_bytes_per_sec": 52428800,
        "replication_keeping_up": true
      },
      "failover_ready": true,
      "failover_ready_reason": "Ready with 245s data at risk",
      "data_at_risk_seconds": 245,
      "data_at_risk_bytes": 10485760
    }
  }
}
```

---

### Example 3: Automate a readiness check before failover

```bash
#!/usr/bin/env bash
# Fails with exit code 1 if the Protection Group is not ready to fail over.

PG_ID="$1"
OS_TOKEN=$(openstack token issue -f value -c id)
TENANT_ID=$(openstack project show -f value -c id "$OS_PROJECT_NAME")

HEALTH=$(curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  "http://protector-primary.example.com:8788/v1/${TENANT_ID}/protection-groups/${PG_ID}/replication-health")

CAN_FAILOVER=$(echo "$HEALTH" | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['replication_health']['failover_validation']['can_failover'])")

if [ "$CAN_FAILOVER" = "True" ]; then
  echo "Protection group $PG_ID is ready for failover."
  exit 0
else
  BLOCKERS=$(echo "$HEALTH" | python3 -c \
    "import sys,json; d=json.load(sys.stdin); print(d['replication_health']['failover_validation']['blockers'])")
  echo "ERROR: Failover not ready. Blockers: $BLOCKERS"
  exit 1
fi
```

---

### Example 4: Track an in-progress failover operation

```bash
# Initiate a planned failover
openstack dr failover my-pg --failover-type planned --wait

# Or, initiate and poll manually:
OPERATION_ID=$(openstack dr failover my-pg --failover-type planned \
  -f value -c operation_id)

echo "Tracking operation: $OPERATION_ID"
while true; do
  STATUS=$(openstack dr operation show "$OPERATION_ID" -f value -c status)
  PROGRESS=$(openstack dr operation show "$OPERATION_ID" -f value -c progress)
  echo "  Status: $STATUS  Progress: ${PROGRESS}%"
  if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
    break
  fi
  sleep 10
done

openstack dr operation show "$OPERATION_ID"
```

---

### Example 5: List recovery points available on the secondary

```bash
openstack dr recovery point list my-pg --limit 10
```

Expected output:

```
+------------------------------------------+---------------------+-------------+--------------+--------------+
| Name                                     | Created At          | Age (s)     | Replicated   | Consistent   |
+------------------------------------------+---------------------+-------------+--------------+--------------+
| protector-my-pg.snap-20250115T102545     | 2025-01-15T10:25:45 | 255         | True         | True         |
| protector-my-pg.snap-20250115T102045     | 2025-01-15T10:20:45 | 555         | True         | True         |
| protector-my-pg.snap-20250115T101545     | 2025-01-15T10:15:45 | 855         | True         | True         |
+------------------------------------------+---------------------+-------------+--------------+--------------+
```

---

### Example 6: Check the API version discovery endpoint

This endpoint requires no authentication and is a quick smoke test that the API process is up:

```bash
curl -s http://protector-primary.example.com:8788/ | python3 -m json.tool
```

Expected output:

```json
{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {"rel": "self", "href": "http://protector-primary.example.com:8788/v1/"}
      ]
    }
  ]
}
```

## Troubleshooting

### Issue: `openstack dr replication health` returns RPO violation

**Symptom:** The command returns `rpo_compliant: false` and `rpo_violation_duration` is non-zero.

**Likely causes:**
- The replication link is degraded or throttled, causing snapshots to fall behind.
- The snapshot interval (`snapshot_interval_seconds`) is too long relative to your RPO target (`default_rpo_seconds`).
- A transient network issue between the FlashArray management IPs.

**Fix:**
1. Check the link status: `openstack dr replication health <pg>` → `health_report.link.status`.
2. If `status` is `degraded` or `broken`, check network connectivity between the arrays.
3. If the link is healthy but lag is growing, check `transfer.replication_keeping_up`. If `false`, the write rate on the source is exceeding replication throughput.
4. Reduce `snapshot_interval_seconds` or increase `default_rpo_seconds` in `[pure_storage]` to relax the compliance threshold.
5. In Juju: `juju config trilio-site-recovery config-flags='pure_storage:snapshot_interval_seconds=120'`.

---

### Issue: `failover_validation.can_failover` is `false` with `No replicated recovery points available`

**Symptom:** Health check reports the PG is not failover-ready because there are no replicated recovery points.

**Likely causes:**
- The Protection Group was just created and the first snapshot has not yet been taken and replicated.
- Snapshot scheduling is disabled (`recovery_point_interval_minutes = 0`) and no manual recovery point has been created.
- The replication link was broken before any snapshot completed.

**Fix:**
1. Manually create a recovery point: `openstack dr recovery point create <pg>`.
2. Wait for replication to complete (check `recovery_points.newest_replicated` in the health report).
3. If the link is broken, restore connectivity to the secondary FlashArray and allow replication to catch up.
4. Enable scheduled recovery points if they are disabled: set `recovery_point_interval_minutes` to a value greater than 0 via Juju config.

---

### Issue: DR operation stuck at `pending`

**Symptom:** `openstack dr operation show <id>` shows `status: pending` and `progress: 0` for several minutes.

**Likely causes:**
- `protector-engine` is not running or is not consuming from the RabbitMQ queue.
- The RabbitMQ connection (`[DEFAULT] transport_url`) is misconfigured or the broker is unreachable.
- The engine crashed after accepting the operation and has not yet recovered.

**Fix:**
1. In a Juju deployment, check the engine service status:
   ```bash
   juju ssh trilio-site-recovery/0 -- sudo systemctl status protector-engine
   ```
2. Check engine logs for connection errors:
   ```bash
   juju ssh trilio-site-recovery/0 -- sudo tail -50 /var/log/protector/protector-engine.log
   ```
3. If the engine is running but not consuming, verify `[DEFAULT] transport_url` in `/etc/protector/protector.conf` matches the RabbitMQ endpoint. Remember not to edit this file directly in Juju — check with `juju config trilio-site-recovery`.
4. After an engine restart, previously stuck operations are automatically recovered: operations that had not committed storage changes are marked `failed`; the Protection Group status is restored to `active` or `failed_over` as appropriate.

---

### Issue: Operation failed with `FailoverNotReady` (HTTP 422)

**Symptom:** A failover attempt returns HTTP 422 with error type `FailoverNotReady`.

**Likely causes:**
- The replication link is disconnected or broken.
- No replicated recovery points exist on the secondary (async replication).
- The ActiveDR replica link is not in a `healthy` or `replicating` state.
- RPO is violated and a planned failover requires compliance.

**Fix:**
1. Run `openstack dr replication health <pg>` and inspect `failover_validation.blockers`.
2. Address each blocker before retrying. Common blockers and fixes:
   - **"Replication link disconnected"** → Restore network between FlashArrays.
   - **"RPO violation: Xsec lag exceeds Ysec target"** → Wait for replication to catch up, or use unplanned failover (`--failover-type unplanned`) if data loss is acceptable.
   - **"No replicated recovery points available"** → Create a recovery point manually: `openstack dr recovery point create <pg>`.
3. Pre-validate member eligibility without side effects:
   ```bash
   openstack dr protection group member validate <pg> --instance <instance-id>
   ```

---

### Issue: `openstack dr operation list` shows no operations

**Symptom:** The list is empty even though failovers have been run.

**Likely causes:**
- You are querying the wrong site. Each site has an independent database; operations created on the primary are recorded there.
- Your token's project scope does not match the Protection Group's tenant.

**Fix:**
1. Ensure you are sourcing the OpenStack credentials for the correct site.
2. If you have admin rights, try filtering with `--protection-group <pg-id>` and check both sites.
3. Verify the operation ID using the API directly:
   ```bash
   curl -s \
     -H "X-Auth-Token: $OS_TOKEN" \
     "http://protector-primary.example.com:8788/v1/${TENANT_ID}/operations"
   ```

---

### Issue: Replication health shows `link.status: broken` after engine restart

**Symptom:** After restarting the engine, the health report shows the replication link as broken, but the FlashArrays are actually connected.

**Likely causes:**
- The storage driver lost its session with the FlashArray management IP during the restart.
- `[pure_storage] verify_ssl = true` but the FlashArray certificate has expired or changed.

**Fix:**
1. Check engine logs for SSL or connection errors:
   ```bash
   juju ssh trilio-site-recovery/0 -- sudo grep -i 'pure\|ssl\|connect' /var/log/protector/protector-engine.log | tail -30
   ```
2. If using self-signed certificates, set `verify_ssl = false` in `[pure_storage]` for development, or configure `cafile` in `[service_credentials]` for production.
3. In Juju: `juju config trilio-site-recovery config-flags='pure_storage:verify_ssl=false'` (development only).
4. If the link is genuinely broken, restore the replication connection on the FlashArrays before retrying.

---

### Issue: `InvalidState` (HTTP 400) when triggering a DR action

**Symptom:** POST to `/v1/{tenant_id}/protection-groups/{pg_id}/action` returns HTTP 400 with type `InvalidState`.

**Likely causes:**
- The Protection Group is in a transitional state (`creating`, `failing_over`, `failing_back`) that does not allow the requested action.
- A previous operation failed mid-flight and left the PG in an inconsistent state.

**Fix:**
1. Check the PG status: `openstack dr protection group show <pg>`.
2. If the status is stuck in a transitional state, use the reconcile action to recover:
   ```bash
   openstack dr protection group reconcile <pg>
   ```
3. If reconcile does not resolve it, reset the status manually (admin only):
   ```bash
   openstack dr protection group set <pg> --status active --reset-to-primary
   ```
   Use `--reset-to-secondary` and `--status failed_over` if the PG is effectively on the secondary site.
