Metrics

Trilio Site Recovery exposes a rich set of internal health metrics through its replication health API and CLI, giving you real-time visibility into replication link status, synchronisation lag, RPO compliance, recovery point availability, and transfer performance for each Protection Group. This page explains what each metric means, how to retrieve it, and how to interpret it to make informed decisions about failover readiness — whether you are monitoring programmatically, building automation, or investigating a replication problem.

Before using the replication health features described on this page:

• Both sites must be registered (openstack dr site list shows both)
• At least one Protection Group must exist with status active
• The triliositerecovery CLI plugin must be installed separately from the server package:
  Copy

  pip install -e ./client
  

• A valid Keystone token scoped to the project that owns the Protection Group is required for all API calls; obtain one from Keystone on port 5000
• Network connectivity on port 8788 must be bidirectional between sites — one-way connectivity causes metadata sync failures that can make health data stale
• The Pure Storage FlashArray driver must be active (storage_driver = pure in [storage]); the default mock driver returns simulated health data suitable only for development

The replication health endpoint and CLI command are part of the core service — no separate installation is needed. Ensure both packages are present:

1. Install the server package (Juju charm — the only supported production deployment method):

Deploy the charm using the validated process. The trilio-site-recovery Debian package is installed automatically by the charm:

bash deploy/charms/deploy.sh

Verify the charm is active:

juju status trilio-site-recovery

2. Install the CLI plugin on the client machine:

The CLI plugin is a separate Python package. Install it with pip (do not use the openstackclient snap — it is incompatible with this plugin):

pip install -e ./client

Verify the plugin is available:

openstack dr replication health --help

3. Confirm the storage driver is configured for production:

In Juju deployments, set the Pure Storage driver via juju config (never edit protector.conf directly — the charm overwrites it):

juju config trilio-site-recovery storage-driver=pure
juju config trilio-site-recovery pure-primary-management-ip=<primary-array-ip>
juju config trilio-site-recovery pure-primary-api-token=<token>
juju config trilio-site-recovery pure-secondary-management-ip=<secondary-array-ip>
juju config trilio-site-recovery pure-secondary-api-token=<token>

Without storage_driver = pure, all health data comes from the mock driver and does not reflect real array state.

The following protector.conf options affect replication health data collection and retention. In Juju deployments, set these via juju config trilio-site-recovery <hyphenated-key>=<value>.

[storage] section

[pure_storage] section

[engine] section

[service_credentials] section

Retrieving replication health via the CLI

The primary way to check replication health is the openstack dr replication health command. The Protection Group ID is a required positional argument:

openstack dr replication health <protection-group-id>

This command queries the engine for live data from the storage arrays and returns a structured report covering link health, sync state, lag, recovery points, and transfer performance.

Retrieving replication health via the API

Send a GET request to the replication health endpoint. All versioned resource endpoints require the /v1/{tenant_id}/ prefix:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://<protector-api-host>:8788/v1/$OS_PROJECT_ID/protection-groups/<pg-id>/replication-health

The OpenStack-API-Version header is optional. When omitted, the service defaults to microversion 1.0 and processes the request normally. Only values outside the 1.0–1.1 range return HTTP 400 Bad Request.

Understanding the response structure

The response nests all fields under replication_health. The top-level keys alongside the nested report are protection_group_id, protection_group_name, failover_validation, and storage_config:

replication_health
  .protection_group_id
  .protection_group_name
  .failover_validation
      .can_failover          ← boolean: use this as your programmatic readiness check
      .warnings
      .blockers
      .message
  .storage_config            ← Pure array configuration details
  .health_report
      .storage_group_name
      .replication_type      ← 'async' or 'active_dr'
      .collected_at
      .link                  ← ReplicationLinkHealth
      .sync_status           ← SyncStatus
      .lag                   ← ReplicationLag
      .recovery_points       ← RecoveryPointSummary
      .transfer              ← TransferPerformance
      .failover_ready        ← boolean rollup
      .failover_ready_reason ← human-readable explanation
      .data_at_risk_seconds  ← estimated data loss if primary fails now
      .data_at_risk_bytes

The failover_validation.can_failover field

For programmatic automation, use replication_health.failover_validation.can_failover as your readiness gate. This boolean reflects the engine's assessment after evaluating link status, RPO compliance, sync state, and recovery point availability together. Individual sub-fields give you the detail needed to diagnose why can_failover is false.

Metric sub-objects

health_report.link — Replication link health:

• status: healthy, degraded, broken, or unknown
• direction: bidirectional (ActiveDR) or source_to_target (async)
• connected: whether the arrays can reach each other
• last_healthy_at: ISO 8601 timestamp of last fully healthy state
• message: human-readable description
• mediator_connected: mediator status (applicable to sync replication only; null otherwise)

health_report.sync_status — Synchronisation state:

• in_sync: whether primary and secondary are fully synchronised
• sync_progress_pct: percentage complete during a resync (0–100), or null if not resyncing
• estimated_sync_eta: seconds until in-sync, or null if already synced
• last_consistent_point: ISO 8601 timestamp of the last confirmed consistent point
• seconds_since_consistent: staleness of the consistency confirmation in seconds

health_report.lag — Replication lag:

• lag_seconds: current replication lag in seconds
• lag_bytes: estimated unreplicated data in bytes
• rpo_compliant: whether current lag is within the configured default_rpo_seconds target
• rpo_target_seconds: the configured RPO target
• rpo_violation_duration: how long the RPO has been violated, in seconds (if applicable)
• lag_trend: stable, increasing, or decreasing
• avg_lag_seconds_1h: average lag over the last hour

health_report.recovery_points — Available recovery points:

• total_count: total number of available recovery points on the secondary
• oldest: the oldest available point (name, created_at, age_seconds, is_replicated, is_consistent)
• newest: the most recent available point
• newest_replicated: the most recent point confirmed replicated to the secondary — use this for RPO calculations
• retention_window_hours: how far back available points extend
• missed_snapshots_24h: snapshots missed in the last 24 hours (compared to snapshot_interval_seconds)
• expected_interval_seconds: configured snapshot interval
• actual_avg_interval_seconds: measured average interval

health_report.transfer — Transfer performance:

• throughput_bytes_per_sec: current replication throughput
• avg_throughput_1h: average throughput over the last hour
• peak_throughput_24h: peak throughput in the last 24 hours
• source_write_rate_bytes_per_sec: write rate on the source array
• replication_keeping_up: whether replication throughput is keeping pace with source writes

Failover readiness logic by replication type

Async replication — The engine reports can_failover: true when:

1. The replication link is connected and not broken
2. rpo_compliant is true (lag is within default_rpo_seconds)
3. At least one replicated recovery point exists on the secondary (newest_replicated is not null)

A missed-snapshot warning is included in failover_validation.warnings but does not block failover.

ActiveDR replication — The engine reports can_failover: true when:

1. The replica link is connected
2. Link status is healthy or replicating

Any non-zero lag is reported but does not block failover — ActiveDR provides near-zero RPO by design.

Monitoring replication health programmatically

Poll the replication health endpoint at your monitoring interval. A typical automation loop:

1. GET /v1/{tenant_id}/protection-groups/{pg_id}/replication-health
2. Check replication_health.failover_validation.can_failover
3. If false, inspect failover_validation.blockers and health_report.lag.rpo_compliant
4. Alert or trigger automated failover based on your RTO/RPO thresholds

Poll interval guidance: for async replication with a 15-minute RPO, polling every 2–5 minutes gives you actionable lag data before an RPO violation becomes critical. For ActiveDR, polling every 60 seconds is sufficient given near-zero RPO.

Example 1 — Check replication health for an async Protection Group

openstack dr replication health pg-prod-east

Expected output (async, healthy):

+-----------------------------+-------------------------------------------+
| Field                       | Value                                     |
+-----------------------------+-------------------------------------------+
| protection_group_id         | a1b2c3d4-e5f6-7890-abcd-ef1234567890      |
| protection_group_name       | pg-prod-east                              |
| replication_type            | async                                     |
| collected_at                | 2024-06-15T14:32:01.123456                |
| link.status                 | healthy                                   |
| link.connected              | True                                      |
| link.direction              | source_to_target                          |
| sync_status.in_sync         | True                                      |
| lag.lag_seconds             | 47                                        |
| lag.rpo_compliant           | True                                      |
| lag.rpo_target_seconds      | 900                                       |
| lag.lag_trend               | stable                                    |
| recovery_points.total_count | 12                                        |
| recovery_points.newest_replicated.age_seconds | 47                      |
| recovery_points.missed_snapshots_24h | 0                              |
| transfer.throughput_bytes_per_sec | 52428800                          |
| transfer.replication_keeping_up | True                               |
| failover_ready              | True                                      |
| failover_ready_reason       | Ready with 47s data at risk               |
| data_at_risk_seconds        | 47                                        |
| can_failover                | True                                      |
+-----------------------------+-------------------------------------------+

Example 2 — Check replication health via the REST API

Obtain a Keystone token, then query the endpoint:

export OS_TOKEN=$(openstack token issue -f value -c id)
export OS_PROJECT_ID=$(openstack project show myproject -f value -c id)
export PG_ID="a1b2c3d4-e5f6-7890-abcd-ef1234567890"

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<protector-api-host>:8788/v1/$OS_PROJECT_ID/protection-groups/$PG_ID/replication-health \
  | python3 -m json.tool

Expected response (async, healthy):

{
  "replication_health": {
    "protection_group_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "protection_group_name": "pg-prod-east",
    "failover_validation": {
      "can_failover": true,
      "warnings": [],
      "blockers": [],
      "message": "Ready with 47s data at risk"
    },
    "storage_config": {
      "pure_pg_name": "protector-pg-prod-east",
      "replication_enabled": true,
      "replication_frequency": 300,
      "snapshot_count": 12,
      "volume_count": 3
    },
    "health_report": {
      "storage_group_name": "protector-pg-prod-east",
      "replication_type": "async",
      "collected_at": "2024-06-15T14:32:01.123456",
      "link": {
        "status": "healthy",
        "direction": "source_to_target",
        "connected": true,
        "last_healthy_at": "2024-06-15T14:32:01.123456",
        "message": "Replication active",
        "mediator_connected": null
      },
      "sync_status": {
        "in_sync": true,
        "sync_progress_pct": null,
        "estimated_sync_eta": null,
        "last_consistent_point": "2024-06-15T14:31:14.000000",
        "seconds_since_consistent": 47
      },
      "lag": {
        "lag_seconds": 47,
        "lag_bytes": 2097152,
        "rpo_compliant": true,
        "rpo_target_seconds": 900,
        "rpo_violation_duration": null,
        "lag_trend": "stable",
        "avg_lag_seconds_1h": 43.2
      },
      "recovery_points": {
        "total_count": 12,
        "oldest": {
          "name": "protector-pg-prod-east.2024-06-15-10:00:00",
          "created_at": "2024-06-15T10:00:00.000000",
          "age_seconds": 16321,
          "is_replicated": true,
          "is_consistent": true,
          "size_bytes": null
        },
        "newest": {
          "name": "protector-pg-prod-east.2024-06-15-14:31:00",
          "created_at": "2024-06-15T14:31:00.000000",
          "age_seconds": 61,
          "is_replicated": true,
          "is_consistent": true,
          "size_bytes": null
        },
        "newest_replicated": {
          "name": "protector-pg-prod-east.2024-06-15-14:31:00",
          "created_at": "2024-06-15T14:31:00.000000",
          "age_seconds": 61,
          "is_replicated": true,
          "is_consistent": true,
          "size_bytes": null
        },
        "retention_window_hours": 4.5,
        "expected_interval_seconds": 300,
        "actual_avg_interval_seconds": 301.4,
        "missed_snapshots_24h": 0
      },
      "transfer": {
        "throughput_bytes_per_sec": 52428800,
        "avg_throughput_1h": 48234496.0,
        "peak_throughput_24h": 104857600.0,
        "source_write_rate_bytes_per_sec": 41943040.0,
        "replication_keeping_up": true,
        "estimated_initial_sync_remaining": null
      },
      "failover_ready": true,
      "failover_ready_reason": "Ready with 47s data at risk",
      "data_at_risk_seconds": 47,
      "data_at_risk_bytes": 2097152
    }
  }
}

Example 3 — Check health for an ActiveDR Protection Group

openstack dr replication health pg-prod-activedr

Expected output (ActiveDR, replicating):

| replication_type            | active_dr                                 |
| link.status                 | healthy                                   |
| link.direction              | bidirectional                             |
| link.connected              | True                                      |
| lag.lag_seconds             | 0                                         |
| lag.rpo_compliant           | True                                      |
| failover_ready              | True                                      |
| failover_ready_reason       | Continuously replicating (near-zero RPO)  |
| data_at_risk_seconds        | 0                                         |
| can_failover                | True                                      |

Example 4 — Programmatic readiness check in a shell script

This pattern is suitable for pre-failover automation:

#!/bin/bash
OS_TOKEN=$(openstack token issue -f value -c id)
PROTECTOR_HOST="protector-api.example.com"
PROJECT_ID=$(openstack project show myproject -f value -c id)
PG_ID="a1b2c3d4-e5f6-7890-abcd-ef1234567890"

RESPONSE=$(curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  "http://${PROTECTOR_HOST}:8788/v1/${PROJECT_ID}/protection-groups/${PG_ID}/replication-health")

CAN_FAILOVER=$(echo "$RESPONSE" | python3 -c "
import json, sys
data = json.load(sys.stdin)
print(data['replication_health']['failover_validation']['can_failover'])
")

if [ "$CAN_FAILOVER" = "True" ]; then
  echo "Protection group is ready for failover."
  DATA_AT_RISK=$(echo "$RESPONSE" | python3 -c "
import json, sys
data = json.load(sys.stdin)
print(data['replication_health']['health_report']['data_at_risk_seconds'])
")
  echo "Data at risk if failover triggered now: ${DATA_AT_RISK}s"
else
  echo "Protection group is NOT ready for failover."
  BLOCKERS=$(echo "$RESPONSE" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for b in data['replication_health']['failover_validation']['blockers']:
    print(' -', b)
")
  echo "Blockers:"
  echo "$BLOCKERS"
  exit 1
fi

Example 5 — List available recovery points

Recovery points are listed separately from the health report:

openstack dr recovery point list pg-prod-east

Or via the API:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  "http://<protector-api-host>:8788/v1/$OS_PROJECT_ID/protection-groups/$PG_ID/recovery-points"

Expected response:

{
  "recovery_points": [
    {
      "name": "protector-pg-prod-east.2024-06-15-14:31:00",
      "created_at": "2024-06-15T14:31:00.000000",
      "age_seconds": 61,
      "is_replicated": true,
      "is_consistent": true
    },
    {
      "name": "protector-pg-prod-east.2024-06-15-14:26:00",
      "created_at": "2024-06-15T14:26:00.000000",
      "age_seconds": 361,
      "is_replicated": true,
      "is_consistent": true
    }
  ]
}

Pass a recovery point name to openstack dr failover or openstack dr test failover using the --recovery-point flag to restore to a specific point in time.

Issue 1: can_failover is false with blocker "RPO violation"

Symptom: The replication health report shows lag.rpo_compliant: false and failover_validation.can_failover: false. The reason string includes the lag in seconds and the configured target.

Likely causes:

• WAN link congestion or instability causing replication to fall behind
• Source write rate exceeding transfer.throughput_bytes_per_sec (replication_keeping_up: false)
• The configured default_rpo_seconds is too tight for your workload

Fix:

1. Check lag.lag_trend — if increasing, the problem is ongoing; if decreasing, replication is recovering.
2. Check transfer.replication_keeping_up. If false, the write rate on the source array exceeds replication capacity — investigate WAN bandwidth or reduce write I/O.
3. If the RPO target is too tight for your workload, increase default_rpo_seconds in [pure_storage] (in Juju: juju config trilio-site-recovery pure-default-rpo-seconds=<seconds>).
4. Check recovery_points.missed_snapshots_24h — missed snapshots indicate recurring scheduling problems. Increase snapshot_interval_seconds if snapshots are consistently being skipped.

Issue 2: link.connected is false

Symptom: The health report shows link.connected: false and link.status: broken or unknown. can_failover is false.

Likely causes:

• Network interruption between sites on the replication network
• FlashArray management credentials wrong or expired (primary_api_token or secondary_api_token)
• verify_ssl: true set in [pure_storage] but the array's certificate is self-signed or expired

Fix:

1. Verify network reachability from the engine host to both FlashArray management IPs.
2. Confirm API tokens are valid — log in to each array's management UI and check the token has not been rotated.
3. If using self-signed certificates, either set verify_ssl = false in [pure_storage] (Juju: juju config trilio-site-recovery pure-verify-ssl=false) or provide a valid CA bundle via [service_credentials] cafile.
4. In Juju, after updating credentials: juju config trilio-site-recovery pure-primary-api-token=<new-token>.

Issue 3: Health report returns stale or missing data

Symptom: collected_at is significantly older than the current time, or the endpoint returns a 503 or an empty health_report.

Likely causes:

• The protector-engine is not running or has lost its RabbitMQ connection
• Cross-site connectivity on port 8788 is one-directional — the engine cannot sync metadata from the secondary site
• storage_driver = mock is still active, returning simulated data

Fix:

1. In Juju deployments, check engine health:
   Copy

   juju status trilio-site-recovery
   juju run trilio-site-recovery/0 health-check
   

2. Check logs in /var/log/protector/ for RabbitMQ connection errors or RPC timeouts.
3. Confirm port 8788 is open and bidirectional between both sites — one-way connectivity causes metadata sync failures and stale health data.
4. Confirm storage_driver = pure is set: juju config trilio-site-recovery | grep storage-driver.

Issue 4: recovery_points.newest_replicated is null

Symptom: Health report shows total_count > 0 but newest_replicated is null. can_failover is false with reason "No replicated recovery points available".

Likely causes:

• Snapshots are being created on the primary but replication to the secondary has not completed yet (initial sync still running)
• The replication link was recently re-established after a break

Fix:

1. Check sync_status.sync_progress_pct — if a value is shown, the initial sync or resync is in progress. Wait for it to complete.
2. Check sync_status.estimated_sync_eta for an estimate of when the first replicated point will be available.
3. If sync_progress_pct is null and link.connected is true, the array may need the Protection Group snapshot to replicate. Create a manual recovery point to trigger an immediate replication:
   Copy

   openstack dr recovery point create <protection-group>
   

   Then re-check health after the snapshot interval.

Issue 5: replication_type shows sync in the report but the Protection Group was created as async

Symptom: The health_report.replication_type field does not match the Protection Group's configured replication_type.

Likely causes:

• The health_metrics.py internal report type reflects how the storage driver classified the group at query time, not the Protection Group database field
• The Protection Group may have been manually modified or recovered to an inconsistent state

Fix:

1. Verify the Protection Group's configured type:
   Copy

   openstack dr protection group show <protection-group>
   

2. If the status is inconsistent, use the reconcile command to recover:
   Copy

   openstack dr protection group reconcile <protection-group>
   

3. If the group needs a status reset after manual recovery:
   Copy

   openstack dr protection group set <protection-group> --status active --reset-to-primary
   

Issue 6: HTTP 400 returned when querying the health endpoint

Symptom: curl returns HTTP 400 with {"error": {"code": 400, "message": "Invalid API version ..."}} when querying the replication health endpoint.

Likely cause: The OpenStack-API-Version header contains a version outside the supported 1.0–1.1 range, or the header value is malformed.

Fix: Omit the header entirely (the service defaults to 1.0 and processes normally) or use a supported value:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  "http://<protector-api-host>:8788/v1/$OS_PROJECT_ID/protection-groups/$PG_ID/replication-health"

The response always includes OpenStack-API-Version and OpenStack-API-Minimum-Version headers showing the supported range.