Metrics

Trilio Site Recovery exposes Prometheus-compatible metrics from both the quorum control-plane manager and the workload-side DRBD node agent. These metrics let you track replication health, failover outcomes, and controller activity without leaving your existing monitoring stack. This page explains what metrics are available, how the metrics server is configured, and how to query and interpret the most operationally important signals — including recovery success rates and failover duration.

Before working with Site Recovery metrics you need:

• A Site Recovery deployment with at least one ProtectionZone in Available phase (quorum and both workload clusters operational).
• Access to the quorum cluster with oc or kubectl and appropriate RBAC permissions.
• Prometheus or a compatible scraper reachable from the quorum and workload cluster networks.
• OpenShift 4.17 or later on all three clusters (quorum, primary, DR) — the validated floor per docs/tech-preview-supported-configurations.md.
• Familiarity with kubectl get and oc commands; oc is required for OpenShift-specific paths such as oc adm must-gather.

Metrics endpoints are enabled by default — no separate installation step is required. The quorum control-plane Deployment and the DRBD node agent DaemonSet both start a metrics server automatically when metrics.enabled is true (the default).

Verify the quorum control-plane metrics server

1. Confirm the quorum control-plane pod is running in the zone namespace (dr-<name>):

oc get pods -n dr-<name> \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery

Expected output (single container named manager, 1/1 Running):

NAME                                              READY   STATUS    RESTARTS   AGE
<release>-site-recovery-quorum-control-plane-...  1/1     Running   0          10m

2. Confirm the metrics port is reachable. The default port is 8443 (HTTPS, authenticated):

oc -n dr-<name> port-forward deploy/<release>-site-recovery-quorum-control-plane 8443:8443

Then in a second terminal:

curl -k https://localhost:8443/metrics

You should see Prometheus exposition-format output beginning with # HELP lines.

Verify the DRBD node agent metrics server

The DRBD node agent (drbd-node-agent) runs on the host network. Its metrics server defaults to disabled (metricsAddr: "0"). When enabled via the chart (agent.metricsPort), each agent pod listens on agent.metricsPort (default 18091) on the host IP.

# Identify an agent pod on a specific worker node
oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent,app.kubernetes.io/part-of=site-recovery \
  -o wide

# Port-forward to that pod (substitute the pod name)
oc -n trilio-site-recovery-system port-forward <agent-pod> 18091:18091

# Scrape in a second terminal
curl http://localhost:18091/metrics

Note: The agent metrics endpoint has no authentication or TLS by default. Enable only in environments where host-network access is controlled.

Metrics behaviour is controlled by values in the embedded Helm charts, which the ProtectionZone controller applies automatically — you do not run helm install directly. To change these settings, update the relevant fields in the ProtectionZone CR or through the site-recovery-logging ConfigMap in the zone namespace.

Quorum control-plane metrics

When metrics.secure is true (the default), the metrics server applies WithAuthenticationAndAuthorization — scraper clients must present a valid bearer token and be authorized by the cluster RBAC. This is the correct setting for production: scraping unauthenticated metrics from a controller that holds kubeconfig Secrets would be a security regression.

DRBD node agent metrics

The agent's metrics server is disabled by default ("0"). The agent runs on the host network where common ports such as 8080 and 8081 may already be bound by other node components. When you enable it, choose a port not already used on your worker nodes.

Runtime log-level and observability

Log verbosity for all controllers is managed through the site-recovery-logging ConfigMap in the zone namespace (dr-<name>). This ConfigMap is watched at runtime — no pod restart is required.

oc -n dr-<name> get configmap site-recovery-logging -o yaml

The level key controls the global level. Per-component keys override it:

Note: There is no warn level. The controller uses logr's two-class model (info/error). Setting trace is equivalent to integer level 2 at startup.

To raise verbosity on the quorum control-plane at runtime:

oc -n dr-<name> patch configmap site-recovery-logging \
  --type merge \
  -p '{"data":{"level.control-plane":"debug"}}'

Site Recovery metrics fall into three categories: controller-runtime framework metrics (work queue depth, reconcile errors, webhook latency), replication health metrics (sync state, out-of-sync bytes, RPO), and failover operation metrics (duration, success/failure counts). You observe replication and failover health primarily through the Custom Resources (ReplicationGroupStatus, RPOEvent, ProtectionGroup, FailoverRequest) and the TSR web console; Prometheus metrics complement this with time-series data for alerting and dashboards.

Scraping the quorum control-plane

Configure your Prometheus scrape job to use token authentication against the quorum API server's RBAC. Example ServiceMonitor (OpenShift / Prometheus Operator):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: site-recovery-quorum
  namespace: dr-<name>
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: site-recovery
      app.kubernetes.io/component: quorum-control-plane
  endpoints:
  - port: metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

Querying replication health via the TSR backend API

The site-recovery-backend Flask API exposes zone-level and per-group replication status endpoints. On OpenShift the API is reachable through the oauth-proxy Route created by the ProtectionZone controller:

# Retrieve the console Route hostname
oc -n dr-<name> get route -o jsonpath='{.items[0].spec.host}'

# Query the replication status for a zone
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://<route-host>/api/protection-zones/<zone_id>/replication-status

# Query per-group replication status
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://<route-host>/api/protection-zones/<zone_id>/protection-groups/<pg_name>/replication-status

# Query recent RPO events
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://<route-host>/api/protection-zones/<zone_id>/rpo-events

# Query RPO compliance summary
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://<route-host>/api/protection-zones/<zone_id>/rpo-compliance

Monitoring failover operations via kubectl

Failover outcome and duration are best read from the FailoverRequest CR status rather than a metric time series, because each failover is a discrete event:

# List all failover requests in a zone namespace
oc -n dr-<name> get failoverrequests

# Describe a specific request to see per-VM status
oc -n dr-<name> describe failoverrequest <name>

# Watch failover progress live
oc -n dr-<name> get failoverrequest <name> -w

The status.state field transitions through Pending → InProgress → Completed | Failed. Per-VM detail is in status.vmStatuses (not a failedVMs field — that field does not exist).

Monitoring replication health via kubectl

# View aggregated replication health for all protection groups in a zone
oc -n dr-<name> get replicationgroupstatuses

# Detailed view of one group (includes per-volume sync progress)
oc -n dr-<name> describe replicationgroupstatus <pg-name>-replication

# View recent RPO violation events
oc -n dr-<name> get rpoevents

# Watch DRBDResource sync state on the primary cluster
oc -n <vm-namespace> get drbdresources -w

Filter RPO events by protection group using the label stamped at creation:

oc -n dr-<name> get rpoevents \
  -l siterecovery.trilio.io/protection-group=<pg-name>

Field selectors on Custom Resources: CRDs in this product do not declare selectableFields, so --field-selector on spec.* or status.* fields is not supported and will return an error. Use the siterecovery.trilio.io/protection-group label selector instead.

Example 1 — Check overall zone health

Use the health API to confirm all components are operational before running a failover:

ROUTE=$(oc -n dr-<name> get route -o jsonpath='{.items[0].spec.host}')
TOKEN=$(oc whoami -t)

curl -sk -H "Authorization: Bearer $TOKEN" \
  https://$ROUTE/api/protection-zones/<zone_id>/health

Expected response (all checks healthy):

{
  "status": "healthy",
  "zone": "<zone_id>",
  "checks": [
    {"name": "quorum-reachable", "status": "healthy"},
    {"name": "cluster1-reachable", "status": "healthy"},
    {"name": "cluster2-reachable", "status": "healthy"}
  ]
}

Example 2 — Query per-group RPO status

curl -sk -H "Authorization: Bearer $TOKEN" \
  https://$ROUTE/api/protection-zones/<zone_id>/protection-groups/<pg_name>/replication-status

Expected fields in the response include rpoStatus (Met | Degraded | Violated | Unknown), currentRPOSeconds, lastFullySyncedAt, and per-volume outOfSyncBytes. A rpoStatus of Met with currentRPOSeconds: 0 confirms Protocol C synchronous replication is working correctly (RPO=0).

Example 3 — Observe ReplicationGroupStatus for a protection group

The ReplicationGroupStatus CR (short name rgs) is auto-created by the replication monitor and named <pg-name>-replication:

oc -n dr-<name> get rgs <pg-name>-replication -o yaml

Key fields in the output:

status:
  rpoStatus: Met                     # Met | Degraded | Violated | Unknown
  currentRPOSeconds: 0               # 0 for synchronous Protocol C
  lastFullySyncedAt: "2026-05-01T10:23:00Z"
  aggregated:
    connectionState: Connected
    outOfSyncBytes: 0
    syncedVolumeCount: 4
    totalVolumeCount: 4
    avgWriteLatencyMilliseconds: 3

Example 4 — Review RPO events for compliance reporting

RPO events are immutable audit records. List all events for a protection group:

oc -n dr-<name> get rpoevents \
  -l siterecovery.trilio.io/protection-group=<pg-name> \
  --sort-by=.spec.timestamp

Example output columns:

PROTECTION-GROUP   EVENT-TYPE        RPO-SECONDS   OUT-OF-SYNC   REASON   TIMESTAMP
pg-prod            SyncCompleted     0             0                      2026-05-01T09:00:00Z
pg-prod            ConnectionLost    45            2097152       NetworkTimeout   2026-05-01T10:00:00Z
pg-prod            ConnectionRestored 0            0                      2026-05-01T10:01:30Z

Example 5 — Measure failover duration from FailoverRequest status

oc -n dr-<name> get failoverrequest <name> \
  -o jsonpath='Start: {.status.startTime}%nEnd: {.status.completionTime}%nState: {.status.state}%n'

Expected output for a completed planned failover:

Start: 2026-05-01T11:00:00Z
End: 2026-05-01T11:03:42Z
State: Completed

For per-VM detail:

oc -n dr-<name> get failoverrequest <name> \
  -o jsonpath='{range .status.vmStatuses[*]}{.name}: {.phase}{"\n"}{end}'

Example 6 — Run a health check and review results

# Trigger an on-demand health check
curl -sk -X POST -H "Authorization: Bearer $TOKEN" \
  https://$ROUTE/api/protection-zones/<zone_id>/health/run

# Retrieve the detailed check results
curl -sk -H "Authorization: Bearer $TOKEN" \
  https://$ROUTE/api/protection-zones/<zone_id>/health/checks

Example 7 — Dashboard statistics for a zone

curl -sk -H "Authorization: Bearer $TOKEN" \
  https://$ROUTE/api/protection-zones/<zone_id>/dashboard/stats

The response includes counts of protected VMs, active failovers, replication health summary, and operation statistics — useful as a top-level health signal for an external dashboard.

Issue: ReplicationGroupStatus shows Degraded or Critical unexpectedly

Symptom: The health field on a ReplicationGroupStatus object transitions to Degraded or Critical without a user-initiated failover.

Likely cause: The observed replication lag has exceeded spec.rpoObjectiveSeconds, or the DRBD connection between primary and DR nodes has dropped. An RPOEvent will be present in the same namespace.

Fix:

1. List recent RPOEvent records for the affected group:
   Copy

   oc get rpoevent -n dr-<name> \
     -o jsonpath='{range .items[*]}{.spec.eventType}\t{.spec.violationReason}\t{.spec.rpoAtEvent}s\n{end}'
   

2. Check DRBD node agent logs on the affected worker nodes:
   Copy

   oc logs -n <workload-namespace> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
     --tail=200
   

3. Verify TCP port 7000–8000 is open between primary and DR worker nodes.
4. If using Protocol C (synchronous), confirm round-trip latency is below 50ms; elevated latency causes write stalls that manifest as lag.

Issue: No metrics appear in Prometheus after configuring the ServiceMonitor

Symptom: The Prometheus target for site-recovery-quorum shows DOWN or never appears.

Likely cause: The Service selector labels do not match the pods, or the metrics port in the ServiceMonitor does not match the port exposed by the Service.

Fix:

1. Confirm the quorum control plane pods are running and 1/1:
   Copy

   oc get pods -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
   

2. Verify the Service endpoint resolves to a pod IP:
   Copy

   oc get endpoints site-recovery-quorum-metrics -n dr-<name>
   

3. Check that the port name in the ServiceMonitor (spec.endpoints[].port) exactly matches the port name defined in the Service (spec.ports[].name).
4. If TLS is enforced on the metrics endpoint, ensure tlsConfig is set correctly in the ServiceMonitor.

Issue: RPOEvent records accumulate but no alerts fire

Symptom: oc get rpoevent -n dr-<name> shows many critical events, but no Prometheus alerts trigger.

Likely cause: PrometheusRule or alerting rules have not been created for RPOEvent-based signals, or the Prometheus instance cannot reach the quorum cluster metrics endpoint.

Fix:

1. Confirm the Prometheus target is UP (see previous issue).
2. Verify that any alerting rules reference the correct metric names and label matchers as exposed by the controller. If the rules were written against invented metric names, they will never match.
3. Use tsr-gather to collect a full diagnostic bundle and share it with support:
   Copy

   oc adm must-gather \
     --image=<tsr-gather-image> \
     --dest-dir=./tsr-bundle
   

Issue: FailoverRequest remains in InProgress indefinitely

Symptom: A FailoverRequest does not transition to Completed or Failed after several minutes.

Likely cause: The failover-controller on the quorum cluster cannot reach the primary or DR API server, DRBD volume promotion is stalled, or a VM failed to start on the target cluster.

Fix:

1. Check quorum control plane logs for failover reconciler output:
   Copy

   oc logs -f deployment/<quorum-release-name> -n dr-<name> -c manager
   

2. Inspect per-VM status for the stuck request:
   Copy

   oc get failoverrequest <name> -n dr-<name> \
     -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool
   

3. Confirm TCP port 6443 is open from the quorum cluster to both workload cluster API servers.
4. Check DRBDResource status on the DR cluster to verify volume promotion succeeded:
   Copy

   oc --kubeconfig <dr-kubeconfig> get drbdresource -n <namespace>
   

Issue: TestFailover is stuck in CleaningUp and never reaches Cleaned

Symptom: A TestFailover object remains in the CleaningUp phase for an extended period.

Likely cause: The test-failover-controller is unable to delete resources in the isolated test namespace, often due to finalizers on test PVCs or VMs that cannot be released.

Fix:

1. Identify the test namespace from the CR status (do not construct it manually):
   Copy

   oc get testfailover <name> -n dr-<name> \
     -o jsonpath='{.status.testNamespace}'
   

2. Check for stuck resources in that namespace:
   Copy

   oc get all -n <test-namespace>
   

3. Review test-failover-controller logs for the specific error:
   Copy

   oc logs deployment/<quorum-release-name> -n dr-<name> -c manager | grep -i "testfailover"
   

4. Do not patch finalizers directly. Contact Trilio support and provide the tsr-gather bundle for guidance on safe remediation.