Guide

Metrics

Prometheus metrics exposed by the controller, including recovery success rates and duration

Overview

Trilio Site Recovery exposes a set of Prometheus metrics from its controller components that run on the quorum cluster. These metrics give you real-time visibility into DR operation outcomes, replication health, and failover durations—enabling you to build alerting rules, SLO dashboards, and capacity reports without querying CRD state directly. This page describes every metric exposed by the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and the replication-monitor reconciler bundled inside the site-recovery-quorum-control-plane deployment, along with how to configure scraping, interpret the values, and troubleshoot collection problems.

Prerequisites

Before you begin:

Trilio Site Recovery deployed on OpenShift ≥ 4.14 with site-recovery-quorum-control-plane Helm chart installed on the quorum cluster
site-recovery-workload-control-plane Helm chart installed on both the primary and DR clusters
OpenShift built-in monitoring stack (Prometheus Operator) enabled, or an external Prometheus instance with network access to the quorum cluster
kubectl or oc CLI configured with a kubeconfig targeting the quorum cluster
RBAC permission to read ServiceMonitor and PodMonitor resources in the DR deployment namespace (dr-<name>)
Basic familiarity with PromQL and Prometheus scrape configuration

Installation

The site-recovery-quorum-control-plane Helm chart creates a Service that exposes the metrics endpoint on each controller pod. Prometheus scraping is enabled through a ServiceMonitor that the chart can optionally create for you.

Step 1 — Enable metrics during Helm installation

When installing or upgrading the quorum control plane chart, set metrics.enabled=true and, if you are using the OpenShift monitoring stack, metrics.serviceMonitor.enabled=true:

helm upgrade --install site-recovery-quorum \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --namespace dr-<deployment-name> \
  --set metrics.enabled=true \
  --set metrics.serviceMonitor.enabled=true \
  --set metrics.serviceMonitor.namespace=openshift-monitoring

Replace <deployment-name> with the name of your DR deployment (for example, prod, resulting in namespace dr-prod).

Step 2 — Verify the metrics Service exists

oc get svc -n dr-<deployment-name> -l app.kubernetes.io/component=metrics

Expected output:

NAME                                          TYPE        CLUSTER-IP      PORT(S)    AGE
site-recovery-quorum-control-plane-metrics    ClusterIP   10.96.45.12     8080/TCP   2m

Step 3 — Confirm the ServiceMonitor is registered

oc get servicemonitor -n dr-<deployment-name>

Expected output:

NAME                                        AGE
site-recovery-quorum-control-plane          2m

Step 4 — Verify Prometheus can reach the endpoint

Port-forward the metrics service to your local machine and curl the endpoint:

oc port-forward svc/site-recovery-quorum-control-plane-metrics \
  8080:8080 -n dr-<deployment-name>

In a second terminal:

curl -s http://localhost:8080/metrics | head -40

You should see # HELP and # TYPE lines followed by metric samples.

Configuration

The following Helm values control metric exposure. Set them under the metrics key in your values file or with --set flags.

Helm value	Default	Valid values	Effect
`metrics.enabled`	`false`	`true`, `false`	Enables the `/metrics` HTTP endpoint on port 8080 of the quorum control plane pods. Set to `true` to expose any metrics.
`metrics.port`	`8080`	Any valid port integer	The container port on which the metrics HTTP server listens. Change only if 8080 conflicts with another workload.
`metrics.serviceMonitor.enabled`	`false`	`true`, `false`	Creates an OpenShift/Prometheus Operator `ServiceMonitor` resource so that the cluster Prometheus automatically discovers and scrapes the endpoint.
`metrics.serviceMonitor.namespace`	`""` (same namespace as the chart)	Any namespace string	The namespace where the `ServiceMonitor` is created. Set to `openshift-monitoring` when using the OpenShift built-in monitoring stack.
`metrics.serviceMonitor.interval`	`"30s"`	Any valid Prometheus duration string	How frequently Prometheus scrapes the metrics endpoint. Reduce to `"15s"` for higher-resolution RPO and RTO tracking.
`metrics.serviceMonitor.scrapeTimeout`	`"10s"`	Any valid Prometheus duration string	Maximum time Prometheus waits for a scrape response before marking it as failed.

Example values file snippet:

metrics:
  enabled: true
  port: 8080
  serviceMonitor:
    enabled: true
    namespace: openshift-monitoring
    interval: "30s"
    scrapeTimeout: "10s"

Apply the updated values:

helm upgrade site-recovery-quorum \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --namespace dr-<deployment-name> \
  -f values.yaml

Usage

Once Prometheus is scraping the quorum control plane, you can query metrics across four operational domains: replication health, protection operations, failover operations, and test failover operations. The sections below describe the most important metrics in each domain and show representative PromQL queries.

Replication health metrics

The replication-monitor reconciler inside site-recovery-quorum-control-plane continuously polls ReplicationGroupStatus CRs and publishes per-Protection-Group gauges.

Metric	Type	Labels	Description
`tsr_replication_group_health`	Gauge	`protection_group`, `namespace`, `deployment`	Current health of a Protection Group: `0` = Healthy, `1` = Degraded, `2` = Critical.
`tsr_replication_lag_seconds`	Gauge	`protection_group`, `namespace`, `volume`	Current replication lag in seconds for a single volume. Zero when using Protocol C (synchronous).
`tsr_rpo_violation_total`	Counter	`protection_group`, `namespace`, `severity`	Total number of RPO violations recorded as `RPOEvent` CRs since the controller started.
`tsr_last_sync_timestamp_seconds`	Gauge	`protection_group`, `namespace`	Unix timestamp of the last successful full synchronization for a Protection Group.

Alert on a degraded Protection Group:

tsr_replication_group_health{namespace=~"dr-.*"} > 0

Alert when replication lag exceeds your RPO objective:

tsr_replication_lag_seconds{namespace=~"dr-.*"} > 30

Protection operation metrics

The protection-controller publishes metrics as it processes ProtectionRequest CRs.

Metric	Type	Labels	Description
`tsr_protection_requests_total`	Counter	`namespace`, `deployment`, `result`	Total `ProtectionRequest` CRs processed, labelled by result (`success`, `failure`).
`tsr_protection_duration_seconds`	Histogram	`namespace`, `deployment`	Time in seconds from `ProtectionRequest` creation to reaching the `Protected` phase.
`tsr_protection_requests_pending`	Gauge	`namespace`, `deployment`	Number of `ProtectionRequest` CRs currently in a non-terminal phase.

Success rate over the last hour:

rate(tsr_protection_requests_total{result="success"}[1h])
/ on(namespace, deployment)
(rate(tsr_protection_requests_total[1h]))

95th-percentile protection duration:

histogram_quantile(0.95,
  rate(tsr_protection_duration_seconds_bucket{namespace=~"dr-.*"}[1h])
)

Failover operation metrics

The failover-controller publishes metrics each time it processes a FailoverRequest CR.

Metric	Type	Labels	Description
`tsr_failover_requests_total`	Counter	`namespace`, `deployment`, `failover_type`, `result`	Total failover operations completed, labelled by type (`planned`, `unplanned`) and result (`completed`, `failed`).
`tsr_failover_duration_seconds`	Histogram	`namespace`, `deployment`, `failover_type`	End-to-end failover duration in seconds from `FailoverRequest` creation to `Completed` status. Tracks your actual RTO.
`tsr_failover_retry_count`	Histogram	`namespace`, `deployment`	Number of reconciliation attempts required before a failover reached a terminal state. High values indicate intermittent infrastructure issues.
`tsr_failover_requests_active`	Gauge	`namespace`, `deployment`	Number of `FailoverRequest` CRs currently in `Pending` or `InProgress` state. Should normally be 0 or 1.

RTO tracking — median failover duration over the last 7 days:

histogram_quantile(0.50,
  rate(tsr_failover_duration_seconds_bucket{namespace=~"dr-.*"}[7d])
)

Planned vs unplanned failover success rate:

rate(tsr_failover_requests_total{result="completed"}[24h])
/ on(namespace, deployment, failover_type)
(rate(tsr_failover_requests_total[24h]))

Test failover metrics

The test-failover-controller publishes metrics when processing TestFailover CRs.

Metric	Type	Labels	Description
`tsr_test_failover_total`	Counter	`namespace`, `deployment`, `result`	Total `TestFailover` operations completed, labelled by result (`succeeded`, `failed`).
`tsr_test_failover_duration_seconds`	Histogram	`namespace`, `deployment`	Duration of the full test failover cycle including snapshot creation, VM startup, verification, and cleanup.

Detect failing DR tests:

rate(tsr_test_failover_total{result="failed"}[7d]) > 0

Examples

Example 1 — View all site recovery metrics from the quorum cluster

Port-forward the metrics service and list all tsr_ prefixed metrics:

oc port-forward svc/site-recovery-quorum-control-plane-metrics \
  8080:8080 -n dr-prod &

curl -s http://localhost:8080/metrics | grep '^tsr_'

Expected output (abbreviated):

tsr_replication_group_health{deployment="prod",namespace="dr-prod",protection_group="pg-frontend"} 0
tsr_replication_lag_seconds{namespace="dr-prod",protection_group="pg-frontend",volume="pvc-frontend-data"} 0
tsr_rpo_violation_total{namespace="dr-prod",protection_group="pg-frontend",severity="warning"} 0
tsr_last_sync_timestamp_seconds{namespace="dr-prod",protection_group="pg-frontend"} 1.718200000e+09
tsr_failover_requests_total{deployment="prod",failover_type="planned",namespace="dr-prod",result="completed"} 3
tsr_failover_duration_seconds_bucket{deployment="prod",failover_type="planned",namespace="dr-prod",le="60"} 1
tsr_failover_duration_seconds_bucket{deployment="prod",failover_type="planned",namespace="dr-prod",le="300"} 3
tsr_test_failover_total{deployment="prod",namespace="dr-prod",result="succeeded"} 5

Example 2 — Prometheus alert rule for replication degradation

Create a PrometheusRule on the quorum cluster to fire when any Protection Group enters a Degraded or Critical state:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: site-recovery-replication-alerts
  namespace: openshift-monitoring
spec:
  groups:
    - name: site-recovery.replication
      interval: 30s
      rules:
        - alert: SiteRecoveryProtectionGroupDegraded
          expr: tsr_replication_group_health{namespace=~"dr-.*"} == 1
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "Protection Group {{ $labels.protection_group }} is Degraded"
            description: >-
              Replication health for Protection Group {{ $labels.protection_group }}
              in deployment {{ $labels.deployment }} has been Degraded for more than
              2 minutes. Check DRBDResource status and network connectivity between
              clusters.

        - alert: SiteRecoveryProtectionGroupCritical
          expr: tsr_replication_group_health{namespace=~"dr-.*"} == 2
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Protection Group {{ $labels.protection_group }} is Critical"
            description: >-
              Replication health for Protection Group {{ $labels.protection_group }}
              in deployment {{ $labels.deployment }} is Critical. Immediate
              investigation required — RPO guarantee may be violated.

Apply the rule:

oc apply -f site-recovery-replication-alerts.yaml

Example 3 — Alert rule for RTO SLO breach

Fire when the 95th-percentile planned failover duration exceeds 8 minutes (the upper bound of the product's target RTO):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: site-recovery-rto-slo
  namespace: openshift-monitoring
spec:
  groups:
    - name: site-recovery.rto
      rules:
        - alert: SiteRecoveryRTOSLOBreach
          expr: >
            histogram_quantile(0.95,
              rate(tsr_failover_duration_seconds_bucket{
                namespace=~"dr-.*",
                failover_type="planned"
              }[7d])
            ) > 480
          labels:
            severity: warning
          annotations:
            summary: "Planned failover p95 RTO exceeds 8 minutes"
            description: >-
              The 95th-percentile planned failover duration over the last 7 days
              has exceeded 480 seconds. Review failover controller logs and
              FailoverRequest CR history for slow operations.

Example 4 — Inspect RPO violation history using RPOEvent CRs

While tsr_rpo_violation_total gives a running count, you can correlate it with the full RPOEvent record:

# List all RPO events for a deployment
oc get rpoevents -n dr-prod \
  -o custom-columns=\
NAME:.metadata.name,\
PG:.spec.protectionGroupRef.name,\
SEVERITY:.spec.eventType,\
LAG:.spec.rpoAtEvent,\
TIMESTAMP:.spec.timestamp

Expected output:

NAME                          PG             SEVERITY   LAG   TIMESTAMP
pg-frontend-rpo-1718201234    pg-frontend    warning    12    2024-06-12T14:27:14Z
pg-frontend-rpo-1718201890    pg-frontend    critical   47    2024-06-12T14:38:10Z

Correlate with the Prometheus counter to verify no events are missing from your alert pipeline:

tsr_rpo_violation_total{protection_group="pg-frontend",namespace="dr-prod"}

Troubleshooting

Issue 1 — No `tsr_` metrics appear in Prometheus

Symptom: Querying tsr_replication_group_health or any other tsr_ metric in the Prometheus UI returns an empty result.

Likely causes:

metrics.enabled was not set to true during Helm installation.
The ServiceMonitor was not created, so Prometheus never discovered the endpoint.
The ServiceMonitor was created in the wrong namespace and the Prometheus instance does not watch that namespace.

Fix:

# 1. Confirm metrics are enabled on the running pods
oc get deployment site-recovery-quorum-control-plane \
  -n dr-<deployment-name> \
  -o jsonpath='{.spec.template.spec.containers[0].args}'
# Look for --metrics-bind-address or --metrics-port flag

# 2. Check the Service exists
oc get svc -n dr-<deployment-name> -l app.kubernetes.io/component=metrics

# 3. Check the ServiceMonitor exists
oc get servicemonitor -n dr-<deployment-name>
oc get servicemonitor -n openshift-monitoring

# 4. If the ServiceMonitor is missing, upgrade the chart with the correct flags
helm upgrade site-recovery-quorum \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --namespace dr-<deployment-name> \
  --set metrics.enabled=true \
  --set metrics.serviceMonitor.enabled=true \
  --set metrics.serviceMonitor.namespace=openshift-monitoring

Issue 2 — Metrics endpoint returns connection refused

Symptom: curl http://localhost:8080/metrics after port-forwarding returns connection refused or times out.

Likely cause: The quorum control plane pod is not running, or it crashed after startup.

Fix:

# Check pod status
oc get pods -n dr-<deployment-name> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane

# If the pod is in CrashLoopBackOff, inspect logs
oc logs -n dr-<deployment-name> \
  deployment/site-recovery-quorum-control-plane --tail=100

# Check events for admission webhook or CRD errors
oc get events -n dr-<deployment-name> --sort-by=.lastTimestamp | tail -20

Issue 3 — `tsr_replication_group_health` is missing for some Protection Groups

Symptom: The metric exists for some Protection Groups but not all, even though ReplicationGroupStatus CRs exist for the missing groups.

Likely cause: The ReplicationGroupStatus CR for the affected Protection Group has a missing or malformed spec.protectionGroupRef, so the replication-monitor reconciler cannot associate it with a Protection Group label.

Fix:

# List all ReplicationGroupStatus CRs and check protectionGroupRef
oc get replicationgroupstatus -n dr-<deployment-name> -o yaml \
  | grep -A3 protectionGroupRef

# If the ref is missing, patch it
oc patch replicationgroupstatus <name> \
  -n dr-<deployment-name> \
  --type merge \
  -p '{"spec":{"protectionGroupRef":{"name":"<pg-name>","namespace":"<pg-namespace>"}}}'

# Restart the controller to force a reconcile
oc rollout restart deployment/site-recovery-quorum-control-plane \
  -n dr-<deployment-name>

Issue 4 — `tsr_failover_duration_seconds` shows unexpectedly high values

Symptom: Failover durations are consistently above 8 minutes, indicating an RTO SLO risk.

Likely causes:

DRBD volumes are not fully synchronized at the time of failover, causing the failover-controller to wait for sync completion.
The protection-group-controller is slow to stop or start VMs due to node pressure on the primary or DR cluster.
Network latency between the quorum cluster and workload cluster API servers exceeds acceptable thresholds.

Fix:

# Inspect the FailoverRequest CR for phase durations
oc get failoverrequest -n dr-<deployment-name> -o yaml | grep -A10 status

# Check replication lag before the next failover test
oc get replicationgroupstatus -n dr-<deployment-name> -o yaml \
  | grep -E 'health|lastSync|lagSeconds'

# Collect a full diagnostic bundle with tsr-gather
# Run tsr-gather against the quorum cluster
tsr-gather --kubeconfig ~/.kube/config-quorum \
  --output /tmp/tsr-bundle-$(date +%Y%m%d)

Issue 5 — `tsr_rpo_violation_total` counter is increasing but no alerts fire

Symptom: The counter increments in Prometheus, but the SiteRecoveryProtectionGroupDegraded alert never fires.

Likely cause: The tsr_replication_group_health gauge did not transition to 1 (Degraded) at the same time as the RPO violation, possibly because the violation was transient and resolved before the for: 2m period elapsed.

Fix:

Add a separate alert directly on the RPO violation counter:

- alert: SiteRecoveryRPOViolation
  expr: increase(tsr_rpo_violation_total{namespace=~"dr-.*"}[5m]) > 0
  labels:
    severity: warning
  annotations:
    summary: "RPO violation recorded for {{ $labels.protection_group }}"
    description: >-
      At least one RPO violation event was recorded for Protection Group
      {{ $labels.protection_group }} in the last 5 minutes.
      Check RPOEvent CRs in namespace {{ $labels.namespace }}.

Also inspect the RPOEvent CRs directly to understand the pattern:

oc get rpoevents -n dr-<deployment-name> \
  --sort-by=.spec.timestamp

Metrics

Replication health metrics

Protection operation metrics

Failover operation metrics

Test failover metrics

Example 1 — View all site recovery metrics from the quorum cluster

Example 2 — Prometheus alert rule for replication degradation

Example 3 — Alert rule for RTO SLO breach

Example 4 — Inspect RPO violation history using RPOEvent CRs

Issue 1 — No tsr_ metrics appear in Prometheus

Issue 2 — Metrics endpoint returns connection refused

Issue 3 — tsr_replication_group_health is missing for some Protection Groups

Issue 4 — tsr_failover_duration_seconds shows unexpectedly high values

Issue 5 — tsr_rpo_violation_total counter is increasing but no alerts fire

Issue 1 — No `tsr_` metrics appear in Prometheus

Issue 3 — `tsr_replication_group_health` is missing for some Protection Groups

Issue 4 — `tsr_failover_duration_seconds` shows unexpectedly high values

Issue 5 — `tsr_rpo_violation_total` counter is increasing but no alerts fire