Observability

This page covers the observability surface of Trilio Site Recovery for OpenStack: structured logging, replication health metrics, DR operation tracking, and service health monitoring. Understanding these signals lets you detect replication degradation before it affects your recovery point objective, confirm that failover is safe to execute, and diagnose failures when a DR operation does not complete successfully. The storage driver exposes a rich set of replication metrics and health checks — including link status, lag, sync state, recovery point availability, and transfer throughput — that flow through the API and CLI so you can monitor and automate DR readiness without vendor-specific tooling.

Before using the observability features described here:

• Trilio Site Recovery deployed and configured on both sites (Juju charm deployment — the only validated customer-facing method).
• python-openstackclient installed via pip (not snap — the snap is incompatible with the triliositerecovery CLI plugin).
• The triliositerecovery CLI plugin installed separately: pip install -e ./client from the repository root, or the python3-triliositerecovery Debian package.
• A valid Keystone token with member and dr_operator roles on your project.
• At least one Protection Group created and at least one VM added to it.
• Network connectivity from your workstation to protector-api on port 8788 on each site.
• For service-level health, access to the protector-api endpoint (port 8788).

The observability tools are part of the standard Trilio Site Recovery installation. No additional components are required.

1. Verify the CLI plugin is installed

pip show triliositerecovery

If not installed, install it separately from the repository root:

pip install -e ./client

Or, on a Juju-managed node, install the Debian package:

sudo apt-get install python3-triliositerecovery

2. Confirm the API is reachable

Retrieve the version discovery document (no authentication required):

curl -s http://<protector-api-host>:8788/

Expected response:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://<protector-api-host>:8788/v1/"}]
    }
  ]
}

A 200 response confirms the API process is running.

3. Obtain a Keystone token

All authenticated API calls require an X-Auth-Token obtained from Keystone on port 5000:

export OS_TOKEN=$(openstack token issue -f value -c id)

4. Confirm replication health access

openstack dr replication health <protection-group-id>

A successful response confirms that observability is working end-to-end.

Observability-related configuration lives in protector.conf. In Juju deployments, do not edit this file directly — the charm overwrites it on every config-changed event. Use juju config trilio-site-recovery <key>=<value> with hyphenated option names instead.

Logging

Trilio Site Recovery uses the standard oslo.log framework. Log output goes to /var/log/protector/ on each site. Log verbosity is controlled by the standard oslo.log options (debug, verbose) set via Juju config or config-flags.

Engine: Recovery Point Scheduling

The engine can automatically create recovery points (snapshots) on a configurable interval. This affects how much history appears in the replication health report.

In Juju deployments, set these via config-flags:

juju config trilio-site-recovery \
  config-flags='engine:recovery_point_interval_minutes=15,engine:recovery_point_max_count=48'

Storage Driver: RPO and Snapshot Interval

For async replication, the following [pure_storage] options control the replication cadence that underpins lag and RPO metrics:

In Juju deployments, set these via charm config options (hyphenated):

juju config trilio-site-recovery \
  pure-primary-management-ip=<ip> \
  pure-secondary-management-ip=<ip>

TLS for Cross-Site Calls

Checking Replication Health

The primary observability command is openstack dr replication health. It queries the storage driver and returns a structured report covering link status, lag, RPO compliance, sync state, transfer throughput, and available recovery points.

openstack dr replication health <protection-group-id-or-name>

The API equivalent is:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/protection-groups/<pg-id>/replication-health

The response nests all fields under replication_health. Key fields to watch:

Use replication_health.failover_validation.can_failover as the single programmatic readiness check in automation scripts.

Tracking DR Operations

Every failover, failback, test failover, and sync creates a tracked DR Operation with a unique ID. Operations progress from pending → running → completed (or failed).

List all operations:

openstack dr operation list

Filter by Protection Group or status:

openstack dr operation list --protection-group <pg-id> --status running

Inspect a specific operation:

openstack dr operation show <operation-id>

The API equivalent:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/operations/<operation-id>

The operation response includes:

• status — current status (pending, running, completed, failed)
• progress — integer 0–100
• current_phase — which workflow step is executing
• steps_completed — list of completed steps
• steps_failed — list of failed steps with details
• instances_created, volumes_managed, networks_configured — resources touched
• error_message — failure reason if status is failed
• warnings — non-fatal issues encountered

Cancel an in-progress operation:

openstack dr operation cancel <operation-id>

Listing Recovery Points

Recovery points are point-in-time snapshots available on the secondary site:

openstack dr recovery point list <protection-group-id>

The API equivalent:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/protection-groups/<pg-id>/recovery-points

Each recovery point reports its name, creation time, age in seconds, whether it has been replicated to the target, and whether it is a consistency-group-level snapshot.

Checking Metadata Sync Status

Metadata synchronization state (whether both sites agree on the Protection Group's configuration) is at:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/<tenant-id>/protection-groups/<pg-id>/metadata/status

If metadata is out of sync, trigger a sync:

openstack dr metadata sync <protection-group-id>

or

openstack dr protection group sync <protection-group-id>

Viewing Logs

Logs for both services are written to /var/log/protector/ on each site. In a Juju deployment, tail logs with:

# On the unit running protector-api/protector-engine
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-api.log
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-engine.log

Enable debug logging temporarily:

juju config trilio-site-recovery config-flags='DEFAULT:debug=true'
# Review logs, then disable:
juju config trilio-site-recovery config-flags='DEFAULT:debug=false'

Example 1: Check whether a Protection Group is ready to fail over

openstack dr replication health my-pg

Expected output (async replication, healthy):

+-----------------------------------+----------------------------------------------------------+
| Field                             | Value                                                    |
+-----------------------------------+----------------------------------------------------------+
| protection_group_id               | a1b2c3d4-...                                             |
| protection_group_name             | my-pg                                                    |
| failover_validation.can_failover  | True                                                     |
| failover_validation.message       | Ready with 245s data at risk                             |
| health_report.link.status         | healthy                                                  |
| health_report.link.connected      | True                                                     |
| health_report.lag.lag_seconds     | 245                                                      |
| health_report.lag.rpo_compliant   | True                                                     |
| health_report.sync_status.in_sync | True                                                     |
| health_report.recovery_points.total_count | 12                                               |
| data_at_risk_seconds              | 245                                                      |
+-----------------------------------+----------------------------------------------------------+

Example 2: Poll replication health via the REST API

export OS_TOKEN=$(openstack token issue -f value -c id)
export TENANT_ID=$(openstack project show -f value -c id $OS_PROJECT_NAME)
export PG_ID=a1b2c3d4-0000-0000-0000-000000000001

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  "http://protector-primary.example.com:8788/v1/${TENANT_ID}/protection-groups/${PG_ID}/replication-health" \
  | python3 -m json.tool

Expected response structure:

{
  "replication_health": {
    "protection_group_id": "a1b2c3d4-0000-0000-0000-000000000001",
    "protection_group_name": "my-pg",
    "failover_validation": {
      "can_failover": true,
      "message": "Ready with 245s data at risk",
      "warnings": [],
      "blockers": []
    },
    "health_report": {
      "storage_group_name": "protector-my-pg",
      "replication_type": "async",
      "collected_at": "2025-01-15T10:30:00",
      "link": {
        "status": "healthy",
        "direction": "source_to_target",
        "connected": true,
        "last_healthy_at": "2025-01-15T10:29:55",
        "message": "Replication active"
      },
      "sync_status": {
        "in_sync": true,
        "sync_progress_pct": null,
        "last_consistent_point": "2025-01-15T10:25:45",
        "seconds_since_consistent": 255
      },
      "lag": {
        "lag_seconds": 245,
        "lag_bytes": 10485760,
        "rpo_compliant": true,
        "rpo_target_seconds": 900,
        "rpo_violation_duration": null,
        "lag_trend": "stable",
        "avg_lag_seconds_1h": 230.5
      },
      "recovery_points": {
        "total_count": 12,
        "newest_replicated": {
          "name": "protector-my-pg.snap-20250115T102545",
          "created_at": "2025-01-15T10:25:45",
          "age_seconds": 255,
          "is_replicated": true,
          "is_consistent": true
        },
        "missed_snapshots_24h": 0
      },
      "transfer": {
        "throughput_bytes_per_sec": 52428800,
        "replication_keeping_up": true
      },
      "failover_ready": true,
      "failover_ready_reason": "Ready with 245s data at risk",
      "data_at_risk_seconds": 245,
      "data_at_risk_bytes": 10485760
    }
  }
}

Example 3: Automate a readiness check before failover

#!/usr/bin/env bash
# Fails with exit code 1 if the Protection Group is not ready to fail over.

PG_ID="$1"
OS_TOKEN=$(openstack token issue -f value -c id)
TENANT_ID=$(openstack project show -f value -c id "$OS_PROJECT_NAME")

HEALTH=$(curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  "http://protector-primary.example.com:8788/v1/${TENANT_ID}/protection-groups/${PG_ID}/replication-health")

CAN_FAILOVER=$(echo "$HEALTH" | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['replication_health']['failover_validation']['can_failover'])")

if [ "$CAN_FAILOVER" = "True" ]; then
  echo "Protection group $PG_ID is ready for failover."
  exit 0
else
  BLOCKERS=$(echo "$HEALTH" | python3 -c \
    "import sys,json; d=json.load(sys.stdin); print(d['replication_health']['failover_validation']['blockers'])")
  echo "ERROR: Failover not ready. Blockers: $BLOCKERS"
  exit 1
fi

Example 4: Track an in-progress failover operation

# Initiate a planned failover
openstack dr failover my-pg --failover-type planned --wait

# Or, initiate and poll manually:
OPERATION_ID=$(openstack dr failover my-pg --failover-type planned \
  -f value -c operation_id)

echo "Tracking operation: $OPERATION_ID"
while true; do
  STATUS=$(openstack dr operation show "$OPERATION_ID" -f value -c status)
  PROGRESS=$(openstack dr operation show "$OPERATION_ID" -f value -c progress)
  echo "  Status: $STATUS  Progress: ${PROGRESS}%"
  if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
    break
  fi
  sleep 10
done

openstack dr operation show "$OPERATION_ID"

Example 5: List recovery points available on the secondary

openstack dr recovery point list my-pg --limit 10

Expected output:

+------------------------------------------+---------------------+-------------+--------------+--------------+
| Name                                     | Created At          | Age (s)     | Replicated   | Consistent   |
+------------------------------------------+---------------------+-------------+--------------+--------------+
| protector-my-pg.snap-20250115T102545     | 2025-01-15T10:25:45 | 255         | True         | True         |
| protector-my-pg.snap-20250115T102045     | 2025-01-15T10:20:45 | 555         | True         | True         |
| protector-my-pg.snap-20250115T101545     | 2025-01-15T10:15:45 | 855         | True         | True         |
+------------------------------------------+---------------------+-------------+--------------+--------------+

Example 6: Check the API version discovery endpoint

This endpoint requires no authentication and is a quick smoke test that the API process is up:

curl -s http://protector-primary.example.com:8788/ | python3 -m json.tool

Expected output:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {"rel": "self", "href": "http://protector-primary.example.com:8788/v1/"}
      ]
    }
  ]
}

Issue: openstack dr replication health returns RPO violation

Symptom: The command returns rpo_compliant: false and rpo_violation_duration is non-zero.

Likely causes:

• The replication link is degraded or throttled, causing snapshots to fall behind.
• The snapshot interval (snapshot_interval_seconds) is too long relative to your RPO target (default_rpo_seconds).
• A transient network issue between the FlashArray management IPs.

Fix:

1. Check the link status: openstack dr replication health <pg> → health_report.link.status.
2. If status is degraded or broken, check network connectivity between the arrays.
3. If the link is healthy but lag is growing, check transfer.replication_keeping_up. If false, the write rate on the source is exceeding replication throughput.
4. Reduce snapshot_interval_seconds or increase default_rpo_seconds in [pure_storage] to relax the compliance threshold.
5. In Juju: juju config trilio-site-recovery config-flags='pure_storage:snapshot_interval_seconds=120'.

Issue: failover_validation.can_failover is false with No replicated recovery points available

Symptom: Health check reports the PG is not failover-ready because there are no replicated recovery points.

Likely causes:

• The Protection Group was just created and the first snapshot has not yet been taken and replicated.
• Snapshot scheduling is disabled (recovery_point_interval_minutes = 0) and no manual recovery point has been created.
• The replication link was broken before any snapshot completed.

Fix:

1. Manually create a recovery point: openstack dr recovery point create <pg>.
2. Wait for replication to complete (check recovery_points.newest_replicated in the health report).
3. If the link is broken, restore connectivity to the secondary FlashArray and allow replication to catch up.
4. Enable scheduled recovery points if they are disabled: set recovery_point_interval_minutes to a value greater than 0 via Juju config.

Issue: DR operation stuck at pending

Symptom: openstack dr operation show <id> shows status: pending and progress: 0 for several minutes.

Likely causes:

• protector-engine is not running or is not consuming from the RabbitMQ queue.
• The RabbitMQ connection ([DEFAULT] transport_url) is misconfigured or the broker is unreachable.
• The engine crashed after accepting the operation and has not yet recovered.

Fix:

1. In a Juju deployment, check the engine service status:
   Copy

   juju ssh trilio-site-recovery/0 -- sudo systemctl status protector-engine
   

2. Check engine logs for connection errors:
   Copy

   juju ssh trilio-site-recovery/0 -- sudo tail -50 /var/log/protector/protector-engine.log
   

3. If the engine is running but not consuming, verify [DEFAULT] transport_url in /etc/protector/protector.conf matches the RabbitMQ endpoint. Remember not to edit this file directly in Juju — check with juju config trilio-site-recovery.
4. After an engine restart, previously stuck operations are automatically recovered: operations that had not committed storage changes are marked failed; the Protection Group status is restored to active or failed_over as appropriate.

Issue: Operation failed with FailoverNotReady (HTTP 422)

Symptom: A failover attempt returns HTTP 422 with error type FailoverNotReady.

Likely causes:

• The replication link is disconnected or broken.
• No replicated recovery points exist on the secondary (async replication).
• The ActiveDR replica link is not in a healthy or replicating state.
• RPO is violated and a planned failover requires compliance.

Fix:

1. Run openstack dr replication health <pg> and inspect failover_validation.blockers.
2. Address each blocker before retrying. Common blockers and fixes:
   • "Replication link disconnected" → Restore network between FlashArrays.
   • "RPO violation: Xsec lag exceeds Ysec target" → Wait for replication to catch up, or use unplanned failover (--failover-type unplanned) if data loss is acceptable.
   • "No replicated recovery points available" → Create a recovery point manually: openstack dr recovery point create <pg>.
3. Pre-validate member eligibility without side effects:
   Copy

   openstack dr protection group member validate <pg> --instance <instance-id>
   

Issue: openstack dr operation list shows no operations

Symptom: The list is empty even though failovers have been run.

Likely causes:

• You are querying the wrong site. Each site has an independent database; operations created on the primary are recorded there.
• Your token's project scope does not match the Protection Group's tenant.

Fix:

1. Ensure you are sourcing the OpenStack credentials for the correct site.
2. If you have admin rights, try filtering with --protection-group <pg-id> and check both sites.
3. Verify the operation ID using the API directly:
   Copy

   curl -s \
     -H "X-Auth-Token: $OS_TOKEN" \
     "http://protector-primary.example.com:8788/v1/${TENANT_ID}/operations"
   

Issue: Replication health shows link.status: broken after engine restart

Symptom: After restarting the engine, the health report shows the replication link as broken, but the FlashArrays are actually connected.

Likely causes:

• The storage driver lost its session with the FlashArray management IP during the restart.
• [pure_storage] verify_ssl = true but the FlashArray certificate has expired or changed.

Fix:

1. Check engine logs for SSL or connection errors:
   Copy

   juju ssh trilio-site-recovery/0 -- sudo grep -i 'pure\|ssl\|connect' /var/log/protector/protector-engine.log | tail -30
   

2. If using self-signed certificates, set verify_ssl = false in [pure_storage] for development, or configure cafile in [service_credentials] for production.
3. In Juju: juju config trilio-site-recovery config-flags='pure_storage:verify_ssl=false' (development only).
4. If the link is genuinely broken, restore the replication connection on the FlashArrays before retrying.

Issue: InvalidState (HTTP 400) when triggering a DR action

Symptom: POST to /v1/{tenant_id}/protection-groups/{pg_id}/action returns HTTP 400 with type InvalidState.

Likely causes:

• The Protection Group is in a transitional state (creating, failing_over, failing_back) that does not allow the requested action.
• A previous operation failed mid-flight and left the PG in an inconsistent state.

Fix:

1. Check the PG status: openstack dr protection group show <pg>.
2. If the status is stuck in a transitional state, use the reconcile action to recover:
   Copy

   openstack dr protection group reconcile <pg>
   

3. If reconcile does not resolve it, reset the status manually (admin only):
   Copy

   openstack dr protection group set <pg> --status active --reset-to-primary
   

   Use --reset-to-secondary and --status failed_over if the PG is effectively on the secondary site.