--- title: Metrics product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/metrics --- # Metrics _Prometheus metrics exposed by the controller including reconciliation latency, error rates, and recovery status_ ## Overview Site Recovery exposes Prometheus metrics from its controllers and agents to help you monitor reconciliation performance, replication health, and DR operation outcomes. This page describes the metrics emitted by the failover-controller, protection-group-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor components — covering reconciliation latency, error rates, RPO tracking, and failover operation status. Monitoring these metrics lets you detect replication degradation before it becomes a data loss risk, track SLO compliance for your RTO and RPO targets, and receive early warning of controller failures that could leave protection groups in an inconsistent state. ## Prerequisites Before scraping or alerting on Site Recovery metrics, ensure you have: - A working Site Recovery deployment with controllers running on the quorum cluster (failover-controller, protection-controller, pg-sync-controller) and on the primary and DR clusters (protection-group-controller, test-failover-controller) - The replication-monitor agent deployed on both the primary and DR clusters (deployed automatically by the standard Ansible playbooks) - Prometheus ≥ 2.30 with scrape access to the controller pods - `kubectl` access to the quorum, primary, and DR clusters - Familiarity with Kubernetes custom resources: `ProtectionGroup`, `FailoverRequest`, `RPOEvent`, and `ReplicationGroupStatus` ## Installation Site Recovery controllers expose metrics on a standard `/metrics` endpoint using the Prometheus exposition format. No additional installation step is required — the endpoint is active as soon as each controller pod is running. **Step 1 — Verify the metrics endpoint is reachable** Confirm the failover-controller metrics endpoint is live on the quorum cluster: ```bash kubectl port-forward deployment/failover-controller 8080:8080 -n dr- curl -s http://localhost:8080/metrics | head -40 ``` Repeat for the protection-group-controller on the primary cluster: ```bash kubectl port-forward deployment/protection-group-controller 8080:8080 -n curl -s http://localhost:8080/metrics | head -40 ``` **Step 2 — Add a Prometheus scrape configuration** Add the following job definitions to your Prometheus `scrape_configs`. Adjust the `namespaces` list and kubeconfig context to match your deployment: ```yaml scrape_configs: - job_name: site-recovery-quorum kubernetes_sd_configs: - role: pod namespaces: names: - dr-prod - dr-staging relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app] regex: failover-controller|protection-controller|pg-sync-controller action: keep - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.*) replacement: ${1} - job_name: site-recovery-workload-clusters kubernetes_sd_configs: - role: pod namespaces: names: - relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app] regex: protection-group-controller|test-failover-controller|replication-monitor action: keep ``` **Step 3 — Confirm metrics are ingested** Run a quick query in the Prometheus UI or via the API to confirm data is flowing: ```bash curl -s 'http://:9090/api/v1/query?query=up{job="site-recovery-quorum"}' | jq '.data.result' ``` You should see one result per controller pod with `value` equal to `1`. ## Configuration Site Recovery controllers follow standard Kubebuilder/controller-runtime conventions for metrics. The options below affect how metrics are exposed and collected. **Metrics bind address** Each controller exposes metrics on port `8080` by default at the path `/metrics`. This is configurable via the `--metrics-bind-address` flag passed to the controller binary. You do not normally need to change this unless you are running multiple controllers on the same host or have a port conflict. | Flag | Default | Effect | |---|---|---| | `--metrics-bind-address` | `:8080` | Host and port for the Prometheus metrics endpoint | **RPO threshold for RPOEvent generation** The replication-monitor creates an `RPOEvent` custom resource whenever observed replication lag exceeds the configured threshold. The threshold is set per `DRBDReplicationPolicy` and controls the sensitivity of RPO alerting. Lower thresholds produce more events and are appropriate for Protocol C (synchronous) deployments targeting RPO=0. Higher thresholds are suitable for Protocol A (asynchronous) deployments where a small RPO window is acceptable. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: DRBDReplicationPolicy metadata: name: prod-replication-policy spec: protocol: C # or A rpoThresholdSeconds: 30 # Lag above this creates an RPOEvent storageClassMappings: - primary: fast-ssd dr: fast-ssd-dr ``` | Field | Default | Effect | |---|---|---| | `spec.rpoThresholdSeconds` | `30` | Seconds of lag before replication-monitor writes an RPOEvent | | `spec.protocol` | `C` | `C` = synchronous (RPO=0); `A` = asynchronous (near-zero RPO) | **Reconciliation timer interval (failover-controller)** The failover-controller uses a timer-based reconciliation loop that runs every 10 seconds (`interval=10.0`, `idle=5.0`). This is not user-configurable at runtime but determines the granularity of failover progress metrics — each reconciliation cycle updates the `FailoverRequest` status and emits corresponding metrics. **Retry limit** The failover-controller abandons a failover and marks the `FailoverRequest` as `Failed` after 30 reconciliation attempts (approximately 5 minutes). Metrics for failed failover operations reflect this limit. If you see consistently high retry counts in metrics, investigate DRBD sync state and quorum taint conditions before retrying. ## Usage Use Site Recovery metrics to answer three operational questions: Is replication healthy? Are controllers working correctly? Did a failover or failback complete within your RTO? **Checking replication health via ReplicationGroupStatus** The replication-monitor maintains `ReplicationGroupStatus` resources that aggregate per-volume sync state. Metrics derived from these resources show the overall health (`Healthy`, `Degraded`, or `Critical`) of each Protection Group. Query these to drive dashboards and alerts: ```bash # Inspect ReplicationGroupStatus for a specific Protection Group kubectl get replicationgroupstatus -n -o yaml ``` You should see `status.health` and `status.lastSuccessfulSyncTime` in the output. A value of `Degraded` or `Critical` means the corresponding metrics will reflect elevated lag or sync errors. **Monitoring RPO violations** Whenever the replication-monitor detects lag above the configured `rpoThresholdSeconds`, it writes an `RPOEvent` resource. List active RPO violations: ```bash kubectl get rpoevents -n --sort-by=.metadata.creationTimestamp ``` In Prometheus, alert on the rate of new `RPOEvent` resources appearing in your namespace. A nonzero rate during steady-state operations indicates your replication link is under stress. **Tracking failover operation status** The failover-controller updates the `FailoverRequest` resource status at each reconciliation cycle. Metrics track transitions through `Pending → InProgress → Completed` (or `Failed`). Monitor the current retry count to detect stalled failovers before the 30-attempt limit is reached: ```bash # Watch a FailoverRequest's status in real time kubectl get failoverrequest -n dr- -w ``` **Using pgctl to check Protection Group state** The `pgctl` CLI surfaces Protection Group status, which maps directly to the underlying metrics: ```bash # Inspect all Protection Groups in a deployment context pgctl pg list # Validate configuration and replication readiness pgctl pg validate ``` Use `pgctl` output alongside Prometheus metrics for a complete operational picture — the CLI gives you immediate human-readable state while Prometheus gives you trends and alerting. ## Examples **Example 1 — Query reconciliation error rate for the failover-controller** This PromQL expression computes the per-second rate of reconciliation errors over a 5-minute window. A sustained nonzero value means the failover-controller is failing to reconcile `FailoverRequest` resources and requires investigation. ```promql rate(controller_runtime_reconcile_errors_total{controller="failoverrequest"}[5m]) ``` Expected output when healthy: ``` {controller="failoverrequest", namespace="dr-prod"} => 0 ``` --- **Example 2 — Query reconciliation latency percentiles** Use this expression to see the 99th-percentile reconciliation duration for the protection-group-controller. High latency here means the controller is taking longer than expected to stop or start VMs in response to `spec.desiredState` changes. ```promql histogram_quantile(0.99, rate(controller_runtime_reconcile_time_seconds_bucket{ controller="protectiongroup" }[10m]) ) ``` Expected output at steady state (no active failovers): ``` {controller="protectiongroup", namespace="dr-prod"} => 0.45 ``` Values above 30 seconds during a failover are expected. Values above 30 seconds during normal operation indicate a problem. --- **Example 3 — Inspect an RPOEvent resource after a replication lag alert** When the replication-monitor writes an `RPOEvent`, examine its contents to understand which Protection Group was affected, the observed lag, and severity. ```bash kubectl get rpoevent -n -o yaml ``` Expected output: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: RPOEvent metadata: name: rpo-pg-prod-db-1748900000 namespace: dr-prod spec: protectionGroup: prod-db observedLagSeconds: 47 severity: Warning status: recordedAt: "2025-06-03T10:23:17Z" ``` --- **Example 4 — Check failover retry count for a stalled operation** If a failover is taking longer than expected, inspect the `FailoverRequest` status to see how many reconciliation attempts have been made. The controller abandons the operation after 30 attempts. ```bash kubectl get failoverrequest -n dr-prod -o jsonpath='{.status}' | jq . ``` Expected output for an in-progress failover: ```json { "state": "InProgress", "phase": "Retrying", "retryCount": 8, "vmStatuses": [ {"name": "prod-vm-1", "state": "Running", "cluster": "dr-cluster"}, {"name": "prod-vm-2", "state": "Starting", "cluster": "dr-cluster"} ] } ``` If `retryCount` is approaching 30 and the state remains `InProgress`, check DRBD volume sync status and quorum taint conditions on the target cluster. --- **Example 5 — Query Protection Group replication health via ReplicationGroupStatus** ```bash kubectl get replicationgroupstatus -n \ -o custom-columns=\ NAME:.metadata.name,\ HEALTH:.status.health,\ LAST-SYNC:.status.lastSuccessfulSyncTime ``` Expected output: ``` NAME HEALTH LAST-SYNC prod-db Healthy 2025-06-03T10:20:00Z prod-web Healthy 2025-06-03T10:20:05Z prod-cache Degraded 2025-06-03T09:55:12Z ``` A `Degraded` or `Critical` health value means the replication-monitor has detected sync issues. Investigate the corresponding `RPOEvent` resources and DRBD volume status for that Protection Group. ## Troubleshooting **Issue: Metrics endpoint returns connection refused** *Symptom:* `curl http://localhost:8080/metrics` returns `Connection refused` after port-forwarding to a controller pod. *Likely cause:* The controller pod is not yet ready, or the port-forward target is a crashed or restarting pod. *Fix:* ```bash # Check pod status kubectl get pods -n dr- -l app=failover-controller # If CrashLoopBackOff, view logs kubectl logs deployment/failover-controller -n dr- --tail=50 # Re-establish port-forward once pod is Running kubectl port-forward deployment/failover-controller 8080:8080 -n dr- ``` --- **Issue: No metrics appear for protection-group-controller or replication-monitor** *Symptom:* Prometheus shows no series for `protection-group-controller` or `replication-monitor` jobs, even though the pods are running. *Likely cause:* Prometheus scrape configuration targets the wrong namespace, or the pod labels used in `relabel_configs` do not match the deployed label values. *Fix:* ```bash # Verify the pod labels match your scrape relabel_configs kubectl get pods -n -l app=protection-group-controller --show-labels kubectl get pods -n -l app=replication-monitor --show-labels # Confirm the metrics endpoint is live kubectl port-forward deployment/replication-monitor 8080:8080 -n curl -s http://localhost:8080/metrics | grep replication ``` --- **Issue: RPOEvent resources accumulating — replication lag alerts firing continuously** *Symptom:* `kubectl get rpoevents -n ` shows a growing list of events. Prometheus alerts for RPO violations are firing repeatedly. *Likely cause:* DRBD replication is degraded. Common causes are network congestion between primary and DR worker nodes on TCP 7000–7999, a DRBD volume stuck in `SyncTarget` or `Inconsistent` state, or a node that has lost the DRBD kernel module. *Fix:* ```bash # Inspect the RPOEvent to identify the affected Protection Group kubectl get rpoevents -n -o yaml | grep protectionGroup # Check DRBDVolume sync state for that Protection Group kubectl get drbdvolume -n -l protection-group= -o wide # Review replication-monitor logs for detail kubectl logs deployment/replication-monitor -n --tail=100 | grep ERROR ``` If the DRBD volume is `Inconsistent`, do not initiate a failover until the volume is in `Consistent` or `UpToDate` state, as this risks data loss. --- **Issue: Reconciliation error rate nonzero for failover-controller** *Symptom:* `rate(controller_runtime_reconcile_errors_total{controller="failoverrequest"}[5m])` is nonzero outside of active failover operations. *Likely cause:* The failover-controller cannot reach one of the managed clusters, or a `FailoverRequest` resource is malformed. *Fix:* ```bash # Check controller logs for the specific error kubectl logs deployment/failover-controller -n dr- --tail=100 # Verify kubeconfig secrets are present and valid kubectl get secrets -n dr- kubectl get secret cluster1-kubeconfig -n dr- \ -o jsonpath='{.data.kubeconfig}' | base64 -d | head -5 # List FailoverRequest resources to find any stuck in an error state kubectl get failoverrequest -n dr- ``` --- **Issue: Failover-controller retry count reaches 30 and operation is marked Failed** *Symptom:* A `FailoverRequest` transitions to `Failed` with `errorMessage: "Failover failed after 30 retry attempts"`. *Likely cause:* DRBD quorum taints on the target cluster are blocking VM scheduling, PVCs cannot bind on the DR cluster, or the Protection Group `currentState` is stuck in `mixed` because some VMs failed to stop on the source cluster. *Fix:* ```bash # Inspect the FailoverRequest status kubectl get failoverrequest -n dr- -o jsonpath='{.status}' | jq . # Check Protection Group currentState on both clusters kubectl get protectiongroup -n \ -o jsonpath='{.status.currentState}' # Check for quorum taints on target cluster nodes kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints # Check controller logs for taint or PVC errors kubectl logs deployment/failover-controller -n dr- | grep -i "taint\|pvc\|SAFE" ``` Once you have resolved the underlying condition (cleared taints, bound PVCs, or manually stopped the remaining source VMs), create a new `FailoverRequest` resource. The controller is idempotent and will skip VMs that are already running on the target cluster.