Metrics

Site Recovery exposes Prometheus metrics from its controllers and agents to help you monitor reconciliation performance, replication health, and DR operation outcomes. This page describes the metrics emitted by the failover-controller, protection-group-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor components — covering reconciliation latency, error rates, RPO tracking, and failover operation status. Monitoring these metrics lets you detect replication degradation before it becomes a data loss risk, track SLO compliance for your RTO and RPO targets, and receive early warning of controller failures that could leave protection groups in an inconsistent state.

Before scraping or alerting on Site Recovery metrics, ensure you have:

• A working Site Recovery deployment with controllers running on the quorum cluster (failover-controller, protection-controller, pg-sync-controller) and on the primary and DR clusters (protection-group-controller, test-failover-controller)
• The replication-monitor agent deployed on both the primary and DR clusters (deployed automatically by the standard Ansible playbooks)
• Prometheus ≥ 2.30 with scrape access to the controller pods
• kubectl access to the quorum, primary, and DR clusters
• Familiarity with Kubernetes custom resources: ProtectionGroup, FailoverRequest, RPOEvent, and ReplicationGroupStatus

Site Recovery controllers expose metrics on a standard /metrics endpoint using the Prometheus exposition format. No additional installation step is required — the endpoint is active as soon as each controller pod is running.

Step 1 — Verify the metrics endpoint is reachable

Confirm the failover-controller metrics endpoint is live on the quorum cluster:

kubectl port-forward deployment/failover-controller 8080:8080 -n dr-<deployment-name>
curl -s http://localhost:8080/metrics | head -40

Repeat for the protection-group-controller on the primary cluster:

kubectl port-forward deployment/protection-group-controller 8080:8080 -n <namespace>
curl -s http://localhost:8080/metrics | head -40

Step 2 — Add a Prometheus scrape configuration

Add the following job definitions to your Prometheus scrape_configs. Adjust the namespaces list and kubeconfig context to match your deployment:

scrape_configs:
  - job_name: site-recovery-quorum
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - dr-prod
            - dr-staging
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: failover-controller|protection-controller|pg-sync-controller
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.*)
        replacement: ${1}

  - job_name: site-recovery-workload-clusters
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - <your-vm-namespace>
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: protection-group-controller|test-failover-controller|replication-monitor
        action: keep

Step 3 — Confirm metrics are ingested

Run a quick query in the Prometheus UI or via the API to confirm data is flowing:

curl -s 'http://<prometheus-host>:9090/api/v1/query?query=up{job="site-recovery-quorum"}' | jq '.data.result'

You should see one result per controller pod with value equal to 1.

Site Recovery controllers follow standard Kubebuilder/controller-runtime conventions for metrics. The options below affect how metrics are exposed and collected.

Metrics bind address

Each controller exposes metrics on port 8080 by default at the path /metrics. This is configurable via the --metrics-bind-address flag passed to the controller binary. You do not normally need to change this unless you are running multiple controllers on the same host or have a port conflict.

RPO threshold for RPOEvent generation

The replication-monitor creates an RPOEvent custom resource whenever observed replication lag exceeds the configured threshold. The threshold is set per DRBDReplicationPolicy and controls the sensitivity of RPO alerting. Lower thresholds produce more events and are appropriate for Protocol C (synchronous) deployments targeting RPO=0. Higher thresholds are suitable for Protocol A (asynchronous) deployments where a small RPO window is acceptable.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: prod-replication-policy
spec:
  protocol: C          # or A
  rpoThresholdSeconds: 30   # Lag above this creates an RPOEvent
  storageClassMappings:
    - primary: fast-ssd
      dr: fast-ssd-dr

Reconciliation timer interval (failover-controller)

The failover-controller uses a timer-based reconciliation loop that runs every 10 seconds (interval=10.0, idle=5.0). This is not user-configurable at runtime but determines the granularity of failover progress metrics — each reconciliation cycle updates the FailoverRequest status and emits corresponding metrics.

Retry limit

The failover-controller abandons a failover and marks the FailoverRequest as Failed after 30 reconciliation attempts (approximately 5 minutes). Metrics for failed failover operations reflect this limit. If you see consistently high retry counts in metrics, investigate DRBD sync state and quorum taint conditions before retrying.

Use Site Recovery metrics to answer three operational questions: Is replication healthy? Are controllers working correctly? Did a failover or failback complete within your RTO?

Checking replication health via ReplicationGroupStatus

The replication-monitor maintains ReplicationGroupStatus resources that aggregate per-volume sync state. Metrics derived from these resources show the overall health (Healthy, Degraded, or Critical) of each Protection Group. Query these to drive dashboards and alerts:

# Inspect ReplicationGroupStatus for a specific Protection Group
kubectl get replicationgroupstatus <pg-name> -n <namespace> -o yaml

You should see status.health and status.lastSuccessfulSyncTime in the output. A value of Degraded or Critical means the corresponding metrics will reflect elevated lag or sync errors.

Monitoring RPO violations

Whenever the replication-monitor detects lag above the configured rpoThresholdSeconds, it writes an RPOEvent resource. List active RPO violations:

kubectl get rpoevents -n <namespace> --sort-by=.metadata.creationTimestamp

In Prometheus, alert on the rate of new RPOEvent resources appearing in your namespace. A nonzero rate during steady-state operations indicates your replication link is under stress.

Tracking failover operation status

The failover-controller updates the FailoverRequest resource status at each reconciliation cycle. Metrics track transitions through Pending → InProgress → Completed (or Failed). Monitor the current retry count to detect stalled failovers before the 30-attempt limit is reached:

# Watch a FailoverRequest's status in real time
kubectl get failoverrequest <name> -n dr-<deployment-name> -w

Using pgctl to check Protection Group state

The pgctl CLI surfaces Protection Group status, which maps directly to the underlying metrics:

# Inspect all Protection Groups in a deployment context
pgctl pg list

# Validate configuration and replication readiness
pgctl pg validate <pg-name>

Use pgctl output alongside Prometheus metrics for a complete operational picture — the CLI gives you immediate human-readable state while Prometheus gives you trends and alerting.

Example 1 — Query reconciliation error rate for the failover-controller

This PromQL expression computes the per-second rate of reconciliation errors over a 5-minute window. A sustained nonzero value means the failover-controller is failing to reconcile FailoverRequest resources and requires investigation.

rate(controller_runtime_reconcile_errors_total{controller="failoverrequest"}[5m])

Expected output when healthy:

{controller="failoverrequest", namespace="dr-prod"} => 0

Example 2 — Query reconciliation latency percentiles

Use this expression to see the 99th-percentile reconciliation duration for the protection-group-controller. High latency here means the controller is taking longer than expected to stop or start VMs in response to spec.desiredState changes.

histogram_quantile(0.99,
  rate(controller_runtime_reconcile_time_seconds_bucket{
    controller="protectiongroup"
  }[10m])
)

Expected output at steady state (no active failovers):

{controller="protectiongroup", namespace="dr-prod"} => 0.45

Values above 30 seconds during a failover are expected. Values above 30 seconds during normal operation indicate a problem.

Example 3 — Inspect an RPOEvent resource after a replication lag alert

When the replication-monitor writes an RPOEvent, examine its contents to understand which Protection Group was affected, the observed lag, and severity.

kubectl get rpoevent <event-name> -n <namespace> -o yaml

Expected output:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: RPOEvent
metadata:
  name: rpo-pg-prod-db-1748900000
  namespace: dr-prod
spec:
  protectionGroup: prod-db
  observedLagSeconds: 47
  severity: Warning
status:
  recordedAt: "2025-06-03T10:23:17Z"

Example 4 — Check failover retry count for a stalled operation

If a failover is taking longer than expected, inspect the FailoverRequest status to see how many reconciliation attempts have been made. The controller abandons the operation after 30 attempts.

kubectl get failoverrequest <name> -n dr-prod -o jsonpath='{.status}' | jq .

Expected output for an in-progress failover:

{
  "state": "InProgress",
  "phase": "Retrying",
  "retryCount": 8,
  "vmStatuses": [
    {"name": "prod-vm-1", "state": "Running", "cluster": "dr-cluster"},
    {"name": "prod-vm-2", "state": "Starting", "cluster": "dr-cluster"}
  ]
}

If retryCount is approaching 30 and the state remains InProgress, check DRBD volume sync status and quorum taint conditions on the target cluster.

Example 5 — Query Protection Group replication health via ReplicationGroupStatus

kubectl get replicationgroupstatus -n <namespace> \
  -o custom-columns=\
NAME:.metadata.name,\
HEALTH:.status.health,\
LAST-SYNC:.status.lastSuccessfulSyncTime

Expected output:

NAME          HEALTH    LAST-SYNC
prod-db       Healthy   2025-06-03T10:20:00Z
prod-web      Healthy   2025-06-03T10:20:05Z
prod-cache    Degraded  2025-06-03T09:55:12Z

A Degraded or Critical health value means the replication-monitor has detected sync issues. Investigate the corresponding RPOEvent resources and DRBD volume status for that Protection Group.

Issue: Metrics endpoint returns connection refused

Symptom: curl http://localhost:8080/metrics returns Connection refused after port-forwarding to a controller pod.

Likely cause: The controller pod is not yet ready, or the port-forward target is a crashed or restarting pod.

Fix:

# Check pod status
kubectl get pods -n dr-<deployment-name> -l app=failover-controller

# If CrashLoopBackOff, view logs
kubectl logs deployment/failover-controller -n dr-<deployment-name> --tail=50

# Re-establish port-forward once pod is Running
kubectl port-forward deployment/failover-controller 8080:8080 -n dr-<deployment-name>

Issue: No metrics appear for protection-group-controller or replication-monitor

Symptom: Prometheus shows no series for protection-group-controller or replication-monitor jobs, even though the pods are running.

Likely cause: Prometheus scrape configuration targets the wrong namespace, or the pod labels used in relabel_configs do not match the deployed label values.

Fix:

# Verify the pod labels match your scrape relabel_configs
kubectl get pods -n <namespace> -l app=protection-group-controller --show-labels
kubectl get pods -n <namespace> -l app=replication-monitor --show-labels

# Confirm the metrics endpoint is live
kubectl port-forward deployment/replication-monitor 8080:8080 -n <namespace>
curl -s http://localhost:8080/metrics | grep replication

Issue: RPOEvent resources accumulating — replication lag alerts firing continuously

Symptom: kubectl get rpoevents -n <namespace> shows a growing list of events. Prometheus alerts for RPO violations are firing repeatedly.

Likely cause: DRBD replication is degraded. Common causes are network congestion between primary and DR worker nodes on TCP 7000–7999, a DRBD volume stuck in SyncTarget or Inconsistent state, or a node that has lost the DRBD kernel module.

Fix:

# Inspect the RPOEvent to identify the affected Protection Group
kubectl get rpoevents -n <namespace> -o yaml | grep protectionGroup

# Check DRBDVolume sync state for that Protection Group
kubectl get drbdvolume -n <namespace> -l protection-group=<pg-name> -o wide

# Review replication-monitor logs for detail
kubectl logs deployment/replication-monitor -n <namespace> --tail=100 | grep ERROR

If the DRBD volume is Inconsistent, do not initiate a failover until the volume is in Consistent or UpToDate state, as this risks data loss.

Issue: Reconciliation error rate nonzero for failover-controller

Symptom: rate(controller_runtime_reconcile_errors_total{controller="failoverrequest"}[5m]) is nonzero outside of active failover operations.

Likely cause: The failover-controller cannot reach one of the managed clusters, or a FailoverRequest resource is malformed.

Fix:

# Check controller logs for the specific error
kubectl logs deployment/failover-controller -n dr-<deployment-name> --tail=100

# Verify kubeconfig secrets are present and valid
kubectl get secrets -n dr-<deployment-name>
kubectl get secret cluster1-kubeconfig -n dr-<deployment-name> \
  -o jsonpath='{.data.kubeconfig}' | base64 -d | head -5

# List FailoverRequest resources to find any stuck in an error state
kubectl get failoverrequest -n dr-<deployment-name>

Issue: Failover-controller retry count reaches 30 and operation is marked Failed

Symptom: A FailoverRequest transitions to Failed with errorMessage: "Failover failed after 30 retry attempts".

Likely cause: DRBD quorum taints on the target cluster are blocking VM scheduling, PVCs cannot bind on the DR cluster, or the Protection Group currentState is stuck in mixed because some VMs failed to stop on the source cluster.

Fix:

# Inspect the FailoverRequest status
kubectl get failoverrequest <name> -n dr-<deployment-name> -o jsonpath='{.status}' | jq .

# Check Protection Group currentState on both clusters
kubectl get protectiongroup <pg-name> -n <namespace> \
  -o jsonpath='{.status.currentState}'

# Check for quorum taints on target cluster nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check controller logs for taint or PVC errors
kubectl logs deployment/failover-controller -n dr-<deployment-name> | grep -i "taint\|pvc\|SAFE"

Once you have resolved the underlying condition (cleared taints, bound PVCs, or manually stopped the remaining source VMs), create a new FailoverRequest resource. The controller is idempotent and will skip VMs that are already running on the target cluster.