Metrics

Trilio Site Recovery exposes Prometheus-compatible metrics from two processes: the TSR-Manager control plane (manager-control-plane binary) and the workload-cluster control plane (workload-control-plane binary, which also runs inside the DRBD node agent DaemonSet pods). These metrics cover the health of DR operations, replication state, and controller internals — giving platform engineers and SREs the signal they need to build alerts around recovery time objectives, replication lag, and failover success rates. This page describes the metrics endpoints, how to scrape them, what the most operationally significant metrics measure, and how to interpret them alongside the CRD status fields that feed them.

Before scraping or alerting on Site Recovery metrics, confirm the following:

• Trilio Site Recovery is installed and at least one ProtectionZone has reached status.phase: Available.
• You have oc CLI access to the TSR-Manager cluster with sufficient RBAC to read pods and services in the zone namespace (dr-<name>) and in trilio-site-recovery-system on each workload cluster.
• A Prometheus-compatible scraping stack (OpenShift Monitoring, Prometheus Operator, or equivalent) is running and can reach the metrics endpoints described below.
• The metrics server TLS certificate is signed by a CA your Prometheus instance trusts, or your scrape config sets insecure_skip_verify: true for these targets (not recommended for production).
• Network connectivity: your Prometheus pods must be able to reach port 8443 on the TSR-Manager control-plane pod and on each workload control-plane pod (the default metrics.port).

Metrics serving is enabled by default (metrics.enabled: true, metrics.port: 8443, metrics.secure: true). No extra installation step is required. The ProtectionZone controller deploys the control-plane charts automatically once the zone reaches Available; the metrics server starts as part of that rollout.

To confirm the metrics server is running:

1. Locate the TSR-Manager control-plane pod in the zone namespace.

oc get pods -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane

Expected output shows a 1/1 running pod (the manager container).

2. Locate workload control-plane pods on each workload cluster.

# Run against the primary cluster kubeconfig
oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane

# Run against the DR cluster kubeconfig
oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane

3. Verify the metrics endpoint is reachable (from within the cluster).

# Port-forward the TSR-Manager control-plane pod
oc port-forward -n dr-<zone-name> \
  deployment/<release>-site-recovery-manager-control-plane \
  8443:8443

# In a second terminal, curl with a service-account token
curl -sk \
  -H "Authorization: Bearer $(oc whoami -t)" \
  https://localhost:8443/metrics | head -40

The metrics server uses OpenShift authentication and authorization (filters.WithAuthenticationAndAuthorization) when metrics.secure: true (the default). If you see an HTTP 403, the identity used by your Prometheus service account needs read access to the metrics endpoint.

4. Create a Prometheus ServiceMonitor or equivalent scrape target.

The exact manifest depends on your monitoring stack. A minimal ServiceMonitor for the TSR-Manager control plane (adjust release and zone namespace as appropriate):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tsr-manager-metrics
  namespace: dr-<zone-name>
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: manager-control-plane
      app.kubernetes.io/part-of: site-recovery
  endpoints:
    - port: metrics          # the chart creates a Service exposing port 8443
      scheme: https
      tlsConfig:
        insecureSkipVerify: false
        # Point to the CA bundle that signed webhook-server-cert,
        # or the cluster CA if you rotated to a cluster-signed cert.
        caFile: /etc/prometheus/certs/ca.crt
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  namespaceSelector:
    matchNames:
      - dr-<zone-name>

Repeat the ServiceMonitor pattern for app.kubernetes.io/component: workload-control-plane in trilio-site-recovery-system on each workload cluster.

The metrics server is configured by the Helm chart values embedded in the operator binary (deployed automatically by the ProtectionZone controller — you do not run helm install directly). The values that govern metrics behaviour are:

These chart values are internal — the ProtectionZone controller applies them from the embedded chart at zone reconcile time. To influence logging verbosity (which affects log-based alerting rather than Prometheus metrics directly), use the ProtectionZone spec:

spec:
  logging:
    level: info               # one of: error | info | debug | trace
    controlPlaneLevel: debug  # override for the control-plane component only
    nodeAgentLevel: info      # override for the drbd-node-agent component

You can also change the log level at runtime without a pod restart by editing the site-recovery-logging ConfigMap in the zone namespace or in trilio-site-recovery-system:

# Raise control-plane log level to debug at runtime
oc patch configmap site-recovery-logging \
  -n dr-<zone-name> \
  --type merge \
  -p '{"data":{"level.control-plane":"debug"}}'

Accepted levels: error, info, debug, trace. There is no warn level.

Health probe port: The manager health probe listens on :8081 (not 8080). This is the liveness and readiness port used by Kubernetes, not the Prometheus metrics port.

DRBD node agent metrics: The node agent defaults to metrics disabled (metrics-bind-address: 0) and uses probe port :18090 (controlled by agent.probePort). If you enable agent metrics, they are served without authentication because the agent runs on the host network; plan your network policy accordingly.

The most operationally significant signal in Site Recovery comes from two complementary sources:

1. ReplicationGroupStatus CRs — aggregated replication health per ProtectionGroup, auto-created by the replication monitor as <protection-group-name>-replication in the zone namespace. Query these with:

   Copy

   oc get rgs -n dr-<zone-name>
   

   Key status fields to alert on:

   Field	Meaning
   status.rpoStatus	Met / Degraded / Violated / Unknown — computed against spec.rpoObjectiveSeconds
   status.currentRPOSeconds	Current measured RPO in seconds
   status.aggregated.connectionState	Connected / Degraded / Disconnected
   status.aggregated.outOfSyncBytes	Total bytes not yet replicated across all volumes
   status.lastFullySyncedAt	Timestamp of the last confirmed full sync

2. RPOEvent CRs — per-event audit records stamped with siterecovery.trilio.io/protection-group=<name>. Query events for a specific group:

   Copy

   oc get rpoe -n dr-<zone-name> \
     -l siterecovery.trilio.io/protection-group=<pg-name>
   

   The spec.eventType field records RPOViolated, ConnectionLost, ResyncStarted, and similar lifecycle events. The spec.rpoAtEvent field gives the RPO in seconds at the moment the event was recorded.

Querying replication health across all groups:

oc get rgs -n dr-<zone-name> \
  -o custom-columns=\
'NAME:.metadata.name,\
RPO-STATUS:.status.rpoStatus,\
CURRENT-RPO:.status.currentRPOSeconds,\
CONNECTION:.status.aggregated.connectionState,\
OUT-OF-SYNC:.status.aggregated.outOfSyncBytes'

Querying per-volume replication detail:

oc get rgs <pg-name>-replication -n dr-<zone-name> \
  -o jsonpath='{.status.volumeReplicationStatuses}' | jq .

Querying RPO violation history:

oc get rpoe -n dr-<zone-name> \
  --sort-by=.spec.timestamp \
  -o custom-columns=\
'PG:.spec.protectionGroupRef.name,\
TYPE:.spec.eventType,\
RPO:.spec.rpoAtEvent,\
REASON:.spec.violationReason,\
TIMESTAMP:.spec.timestamp'

Checking failover operation outcomes:

Failover results are tracked in FailoverRequest status. The status.state field (Pending / InProgress / Completed / Failed) and per-VM detail in status.vmStatuses are the primary signals:

oc get fr -n dr-<zone-name> \
  -o custom-columns=\
'NAME:.metadata.name,\
PG:.spec.protectionGroupRef.name,\
TYPE:.spec.failoverType,\
STATE:.status.state,\
PHASE:.status.phase'

Checking test failover outcomes:

oc get tfo -n dr-<zone-name> \
  -o custom-columns=\
'NAME:.metadata.name,\
PHASE:.status.phase,\
RESULT:.status.result,\
VERIFIED:.status.verificationResults.passed,\
SCHEDULING:.status.schedulingCheck.verdict'

Example 1: Check overall replication health for all protection groups

oc get rgs -n dr-prod \
  -o custom-columns=\
'NAME:.metadata.name,\
RPO-STATUS:.status.rpoStatus,\
CURRENT-RPO-S:.status.currentRPOSeconds,\
OUT-OF-SYNC:.status.aggregated.outOfSyncBytes,\
CONNECTION:.status.aggregated.connectionState,\
LAST-SYNC:.status.lastFullySyncedAt'

Example 2: Inspect per-volume sync progress for a specific protection group

oc get rgs web-tier-replication -n dr-prod -o yaml

The status.volumeReplicationStatuses array contains an entry per replicated volume with fields including pvcName, vmName, outOfSyncBytes, estimatedRPOSeconds, replicationState, and lastSyncedAt.

Example 3: List RPO violation events for a protection group, ordered by time

oc get rpoe -n dr-prod \
  -l siterecovery.trilio.io/protection-group=web-tier \
  --sort-by=.spec.timestamp \
  -o custom-columns=\
'TYPE:.spec.eventType,\
RPO-S:.spec.rpoAtEvent,\
OUT-OF-SYNC:.spec.outOfSyncBytesAtEvent,\
REASON:.spec.violationReason,\
TIMESTAMP:.spec.timestamp'

Example 4: Check the DRBDResource sync progress for a specific VM on the primary cluster

# Run against the primary cluster kubeconfig
oc get dres -n <vm-namespace> \
  -o custom-columns=\
'NAME:.metadata.name,\
SIDE:.spec.side,\
ROLE:.status.role,\
STATE:.status.connectionState,\
PHASE:.status.phase,\
SYNC-PCT:.status.syncProgress.percentage'

DRBDResource CRs live on workload clusters (not on the TSR-Manager cluster). Running this command against the TSR-Manager cluster returns server doesn't have a resource type.

Example 5: Verify test failover scheduling and verification results

oc get tfo -n dr-prod web-tier-tfo-20260101 -o yaml

Key fields to review:

• status.phase — current phase (e.g., Verifying, Succeeded, Failed)
• status.result — Succeeded or Failed
• status.verificationResults.passed — boolean overall pass/fail
• status.verificationResults.checks — per-check detail with type, vm, passed, and message
• status.schedulingCheck.verdict — Passed / Failed / Warned / Skipped
• status.testNamespace — the generated namespace where test VMs ran (system-assigned, never user-provided)

Example 6: Collect a diagnostic bundle including all metrics-relevant CRs

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  --dest-dir=./tsr-must-gather

The gathered bundle includes ReplicationGroupStatus, RPOEvent, ProtectionGroup, FailoverRequest, and TestFailover objects, along with controller logs from all clusters in the DR estate.

Issue: ReplicationGroupStatus shows Degraded or Critical unexpectedly

Symptom: The health field on a ReplicationGroupStatus object transitions to Degraded or Critical without a user-initiated failover.

Likely cause: The observed replication lag has exceeded spec.rpoObjectiveSeconds, or the DRBD connection between primary and DR nodes has dropped. An RPOEvent will be present in the same namespace.

Fix:

1. List recent RPOEvent records for the affected group:
   Copy

   oc get rpoevent -n dr-<name> \
     -o jsonpath='{range .items[*]}{.spec.eventType}\t{.spec.violationReason}\t{.spec.rpoAtEvent}s\n{end}'
   

2. Check DRBD node agent logs on the affected worker nodes:
   Copy

   oc logs -n <workload-namespace> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
     --tail=200
   

3. Verify TCP port 7000–8000 is open between primary and DR worker nodes.
4. If using Protocol C (synchronous), confirm round-trip latency is below 50ms; elevated latency causes write stalls that manifest as lag.

Issue: No metrics appear in Prometheus after configuring the ServiceMonitor

Symptom: The Prometheus target for site-recovery-quorum shows DOWN or never appears.

Likely cause: The Service selector labels do not match the pods, or the metrics port in the ServiceMonitor does not match the port exposed by the Service.

Fix:

1. Confirm the quorum control plane pods are running and 1/1:
   Copy

   oc get pods -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
   

2. Verify the Service endpoint resolves to a pod IP:
   Copy

   oc get endpoints site-recovery-quorum-metrics -n dr-<name>
   

3. Check that the port name in the ServiceMonitor (spec.endpoints[].port) exactly matches the port name defined in the Service (spec.ports[].name).
4. If TLS is enforced on the metrics endpoint, ensure tlsConfig is set correctly in the ServiceMonitor.

Issue: RPOEvent records accumulate but no alerts fire

Symptom: oc get rpoevent -n dr-<name> shows many critical events, but no Prometheus alerts trigger.

Likely cause: PrometheusRule or alerting rules have not been created for RPOEvent-based signals, or the Prometheus instance cannot reach the quorum cluster metrics endpoint.

Fix:

1. Confirm the Prometheus target is UP (see previous issue).
2. Verify that any alerting rules reference the correct metric names and label matchers as exposed by the controller. If the rules were written against invented metric names, they will never match.
3. Use tsr-gather to collect a full diagnostic bundle and share it with support:
   Copy

   oc adm must-gather \
     --image=<tsr-gather-image> \
     --dest-dir=./tsr-bundle
   

Issue: FailoverRequest remains in InProgress indefinitely

Symptom: A FailoverRequest does not transition to Completed or Failed after several minutes.

Likely cause: The failover-controller on the quorum cluster cannot reach the primary or DR API server, DRBD volume promotion is stalled, or a VM failed to start on the target cluster.

Fix:

1. Check quorum control plane logs for failover reconciler output:
   Copy

   oc logs -f deployment/<quorum-release-name> -n dr-<name> -c manager
   

2. Inspect per-VM status for the stuck request:
   Copy

   oc get failoverrequest <name> -n dr-<name> \
     -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool
   

3. Confirm TCP port 6443 is open from the quorum cluster to both workload cluster API servers.
4. Check DRBDResource status on the DR cluster to verify volume promotion succeeded:
   Copy

   oc --kubeconfig <dr-kubeconfig> get drbdresource -n <namespace>
   

Issue: TestFailover is stuck in CleaningUp and never reaches Cleaned

Symptom: A TestFailover object remains in the CleaningUp phase for an extended period.

Likely cause: The test-failover-controller is unable to delete resources in the isolated test namespace, often due to finalizers on test PVCs or VMs that cannot be released.

Fix:

1. Identify the test namespace from the CR status (do not construct it manually):
   Copy

   oc get testfailover <name> -n dr-<name> \
     -o jsonpath='{.status.testNamespace}'
   

2. Check for stuck resources in that namespace:
   Copy

   oc get all -n <test-namespace>
   

3. Review test-failover-controller logs for the specific error:
   Copy

   oc logs deployment/<quorum-release-name> -n dr-<name> -c manager | grep -i "testfailover"
   

4. Do not patch finalizers directly. Contact Trilio support and provide the tsr-gather bundle for guidance on safe remediation.