--- title: Observability product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/observability --- # Observability _Metrics, structured logging, and monitoring for site recovery controller health and operation outcomes_ ## Overview This page explains how to monitor Site Recovery controller health and operation outcomes using Kubernetes-native observability patterns. Site Recovery surfaces its operational state through Custom Resource status fields, structured controller logs, and the Site Manager UI — giving you visibility into replication health, failover progress, Protection Group readiness, and RPO compliance without requiring external agents beyond what is deployed automatically. Understanding these signals is essential for SREs who need to detect degraded replication early, audit failover operations, and respond quickly when DR readiness is at risk. ## Prerequisites Before using the observability features described on this page, ensure you have the following in place: - **Kubernetes ≥ 1.28** on all clusters (quorum, primary, and DR) - **`kubectl`** configured with access to the quorum cluster and at minimum read access to primary and DR clusters - **`pgctl`** installed and a deployment context configured (see the DR deployment configuration guide) - **Site Recovery controllers deployed** on the quorum cluster: `failover-controller`, `protection-controller`, `pg-sync-controller`, and `replication-monitor` - **`protection-group-controller`** and **`test-failover-controller`** deployed on the primary and DR clusters - **`replication-monitor`** deployed and running — this agent is deployed automatically as part of the standard Ansible playbooks and is responsible for writing `RPOEvent` and `ReplicationGroupStatus` resources - **Site Manager UI** accessible on the quorum cluster if you prefer a dashboard view - Read access to the `dr-` namespace on the quorum cluster for your deployment ## Installation Site Recovery's observability components require no separate installation. The `replication-monitor` agent and all controllers that produce observable CRD state are deployed automatically when you run the standard Ansible playbooks during initial setup. To verify that all observability-relevant components are running after deployment: **Step 1: Confirm controllers are running on the quorum cluster.** ```bash kubectl get pods -n dr- -l app=failover-controller kubectl get pods -n dr- -l app=protection-controller kubectl get pods -n dr- -l app=pg-sync-controller kubectl get pods -n dr- -l app=replication-monitor ``` All pods should be in `Running` status with no crash loops. **Step 2: Confirm the protection-group-controller is running on the primary and DR clusters.** ```bash kubectl --kubeconfig ~/.kube/config-primary get pods -l app=protection-group-controller kubectl --kubeconfig ~/.kube/config-dr get pods -l app=protection-group-controller ``` **Step 3: Confirm CRD types are registered.** ```bash kubectl get crds | grep siterecovery.trilio.io ``` Expected output includes entries for `protectiongroups`, `protectionrequests`, `failoverrequests`, `testfailovers`, `drbdreplicationpolicies`, `drbdvolumes`, `rpoevents`, and `replicationgroupstatuses`. **Step 4: Verify the Site Manager UI is reachable.** ```bash kubectl get svc -n dr- -l app=site-manager-ui ``` Note the service address and confirm the dashboard loads in your browser. ## Configuration Site Recovery's observability behavior is shaped by thresholds and intervals embedded in controller configuration. The following settings are most relevant to monitoring outcomes. ### Replication Monitor: RPO Threshold The `replication-monitor` evaluates replication lag continuously and writes an `RPOEvent` resource whenever lag exceeds the configured threshold. The threshold is set per `DRBDReplicationPolicy`. | Field | Location | Default | Effect | |---|---|---|---| | `spec.rpoThresholdSeconds` | `DRBDReplicationPolicy` | Deployment-specific | Lag above this value triggers an `RPOEvent` with the observed lag and severity | | `spec.protocol` | `DRBDReplicationPolicy` | — | `C` (synchronous, RPO=0) or `A` (asynchronous, near-zero RPO); affects how frequently lag events are expected | For Protocol C deployments, any non-zero lag exceeding the threshold indicates a network or storage problem that warrants immediate investigation. For Protocol A deployments, small RPO windows are expected; set your threshold to match your SLA. ### Failover Controller: Reconciliation Interval and Retry Limit The `failover-controller` uses a timer-based reconciliation loop when processing a `FailoverRequest`. These intervals determine how quickly the controller retries after a partial failure. | Parameter | Value | Effect | |---|---|---| | Reconciliation interval | 10 seconds | How frequently the controller re-evaluates each VM's state during a failover | | Idle delay | 5 seconds | Initial delay before first reconciliation after a `FailoverRequest` is created | | Maximum retry attempts | 30 | After 30 attempts (~5 minutes), the `FailoverRequest` transitions to `Failed` | These values are fixed in the controller implementation. You cannot override them via CRD fields. If a failover reaches the `Failed` state due to a transient issue, you can delete the `FailoverRequest` resource and create a new one — the underlying VM and Protection Group state is reconciled idempotently. ### Protection Group Controller: Status Polling The `protection-group-controller` on each cluster reconciles VM state against the Protection Group's `spec.desiredState` and updates `status.currentState`. It polls VM state approximately every 60 seconds in steady state, and immediately on any spec change. | `status.currentState` value | Meaning | |---|---| | `running` | All VMs in the group are running | | `stopped` | All VMs in the group are stopped | | `mixed` | Some VMs are running, some are stopped | | `unknown` | VM states could not be determined | A `mixed` state persisting for more than a few minutes indicates a reconciliation problem and requires investigation. ## Usage ### Viewing Protection Group Replication Health The `ReplicationGroupStatus` resource provides an aggregated replication health view for each Protection Group, including overall health (`Healthy`, `Degraded`, or `Critical`), per-volume sync state, and the timestamp of the last successful sync. ```bash kubectl get replicationgroupstatus -n dr- ``` To inspect a specific group: ```bash kubectl describe replicationgroupstatus -n dr- ``` Look for `status.health` and `status.lastSyncTime`. A `Degraded` or `Critical` health value means one or more volumes are not replicating correctly. Cross-reference with `RPOEvent` resources to understand the history. ### Monitoring RPO Violations The `replication-monitor` writes an `RPOEvent` resource each time replication lag exceeds your configured threshold. List recent violations: ```bash kubectl get rpoevents -n dr- --sort-by='.metadata.creationTimestamp' ``` Each `RPOEvent` records which Protection Group was affected, the observed lag value, and the severity. A pattern of frequent `RPOEvent` resources for the same Protection Group may indicate network instability between primary and DR clusters or storage I/O pressure. ### Monitoring a FailoverRequest in Progress When a failover is triggered (planned or unplanned), the `failover-controller` updates the `FailoverRequest` status continuously. Watch the progression: ```bash kubectl get failoverrequest -n dr- -w ``` Expected `status.state` progression: `Pending` → `InProgress` → `Completed` (or `Failed` after 30 retry attempts). For per-VM detail during the operation: ```bash kubectl get failoverrequest -n dr- -o jsonpath='{.status.vmStatuses}' ``` ### Monitoring Individual Volume Replication In DRBD Operator deployments, each `DRBDVolume` resource exposes sync progress and frontend PVC references. Use this to verify per-volume replication state: ```bash kubectl get drbdvolumes -n --kubeconfig ~/.kube/config-primary ``` To inspect a specific volume: ```bash kubectl describe drbdvolume -n --kubeconfig ~/.kube/config-primary ``` Check `status.syncProgress` (should be `100%` for a fully synced volume) and `status.role` (should be `Primary` on the active cluster). ### Checking Controller Logs All controllers emit structured logs. Use these to trace specific operations: ```bash # Failover controller on the quorum cluster kubectl logs -f deployment/failover-controller -n dr- # Protection controller on the quorum cluster kubectl logs -f deployment/protection-controller -n dr- # Protection Group controller on the primary cluster kubectl logs -f deployment/protection-group-controller -n --kubeconfig ~/.kube/config-primary # Replication monitor on the quorum cluster kubectl logs -f deployment/replication-monitor -n dr- ``` Filter logs for a specific Protection Group name or failover operation name to reduce noise: ```bash kubectl logs deployment/failover-controller -n dr- | grep "production-protection-group" ``` ### Using pgctl for Status Checks The `pgctl` CLI provides a convenient view of Protection Group status across clusters: ```bash # Validate a Protection Group's configuration and replication readiness pgctl validate pg # Inspect a Protection Group's current state pgctl get pg ``` ### Using the Site Manager UI The Site Manager UI on the quorum cluster provides dashboards covering cluster health, VM protection status, Protection Group management, failover operation history, and replication status. Navigate to the UI in your browser and select your deployment from the deployments panel. The Protection Groups dashboard shows each group's replication health status and allows you to trigger operations directly. ## Examples ### Example 1: Inspect a Protection Group's aggregated replication health ```bash kubectl describe replicationgroupstatus production-protection-group -n dr-production ``` Expected output (abbreviated): ``` Name: production-protection-group Namespace: dr-production ... Status: Health: Healthy Last Sync Time: 2025-11-01T14:32:05Z Volume Statuses: Volume Name: pvc-12345 Sync State: Synced Volume Name: pvc-67890 Sync State: Synced ``` A `Health` value of `Healthy` with all volumes in `Synced` state confirms the Protection Group is ready for failover. --- ### Example 2: List recent RPO violations for all Protection Groups ```bash kubectl get rpoevents -n dr-production --sort-by='.metadata.creationTimestamp' -o wide ``` Expected output: ``` NAME PROTECTION-GROUP LAG SEVERITY AGE rpoevent-production-pg-1730462001 production-protection-group 12s Warning 5m rpoevent-production-pg-1730461800 production-protection-group 8s Warning 8m ``` Multiple `RPOEvent` resources in a short window for the same Protection Group indicates a persistent replication lag problem. Investigate network connectivity between primary and DR worker nodes on TCP 7000–7999. --- ### Example 3: Watch a FailoverRequest transition through its lifecycle ```bash kubectl get failoverrequest production-failover-1730462500 -n dr-production -w ``` Expected output progression: ``` NAME STATE PHASE RETRY-COUNT AGE production-failover-1730462500 Pending Initializing 0 2s production-failover-1730462500 InProgress StoppingVMs 1 12s production-failover-1730462500 InProgress StartingVMs 3 32s production-failover-1730462500 Completed Completed 4 52s ``` A `Completed` state confirms all VMs in the Protection Group have been successfully started on the target cluster. --- ### Example 4: Check per-VM status detail from a FailoverRequest ```bash kubectl get failoverrequest production-failover-1730462500 -n dr-production \ -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool ``` Expected output: ```json [ { "name": "prod-vm-1", "state": "Running", "cluster": "dr-cluster" }, { "name": "prod-vm-2", "state": "Running", "cluster": "dr-cluster" } ] ``` --- ### Example 5: Inspect a DRBDVolume for sync progress in a DRBD Operator deployment ```bash kubectl describe drbdvolume pvc-12345 -n production --kubeconfig ~/.kube/config-primary ``` Expected output (abbreviated): ``` Name: pvc-12345 Namespace: production ... Status: Sync Progress: 100% Role: Primary Frontend PVC: prod-vm-1-disk-frontend Health: Healthy ``` A `Sync Progress` of `100%` and `Role` of `Primary` on the primary cluster confirms this volume is fully replicated and serving the VM's disk I/O. --- ### Example 6: Filter failover-controller logs for a specific operation ```bash kubectl logs deployment/failover-controller -n dr-production | grep "production-failover-1730462500" ``` Expected output (representative lines): ``` [2025-11-01T14:41:40Z] [INFO] Acquired failover lock for production-protection-group [2025-11-01T14:41:40Z] [INFO] VM prod-vm-1 still Running on primary - stopping [2025-11-01T14:41:50Z] [INFO] VM prod-vm-1 already Running on dr-cluster - SUCCESS [2025-11-01T14:41:50Z] [INFO] VM prod-vm-2 stopped on source, starting on dr-cluster [2025-11-01T14:42:00Z] [INFO] Released failover lock for production-protection-group ``` --- ### Example 7: Check Protection Group state on both clusters after a failover ```bash # Primary cluster — should be stopped kubectl --kubeconfig ~/.kube/config-primary get protectiongroup production-protection-group \ -o jsonpath='{.status.currentState}' # Expected: stopped # DR cluster — should be running kubectl --kubeconfig ~/.kube/config-dr get protectiongroup production-protection-group \ -o jsonpath='{.status.currentState}' # Expected: running ``` ## Troubleshooting ### Issue 1: `ReplicationGroupStatus` shows `Degraded` or `Critical` **Symptom:** `kubectl get replicationgroupstatus` shows a Protection Group with `health: Degraded` or `health: Critical`. **Likely cause:** One or more DRBD volumes are not fully synced between the primary and DR clusters. This can be caused by network interruption on TCP 7000–7999, storage I/O pressure on worker nodes, or a recent node restart. **Fix:** 1. Identify which volumes are affected: `kubectl describe replicationgroupstatus -n dr-` and look at `Volume Statuses`. 2. Check `RPOEvent` resources for the same Protection Group to understand how long the lag has persisted. 3. Verify network connectivity between primary and DR worker nodes on TCP 7000–7999. 4. Check `replication-monitor` logs for error details: `kubectl logs deployment/replication-monitor -n dr-`. 5. In DRBD Operator deployments, describe the affected `DRBDVolume` resource for per-volume sync progress and role information. --- ### Issue 2: `RPOEvent` resources are being created frequently **Symptom:** Many `RPOEvent` resources are appearing in a short time window for the same Protection Group. **Likely cause:** Replication lag is consistently exceeding the threshold defined in `DRBDReplicationPolicy`. This is most common with Protocol A (asynchronous) deployments under high write workloads, or when network latency between clusters has increased. **Fix:** 1. List recent events: `kubectl get rpoevents -n dr- --sort-by='.metadata.creationTimestamp'`. 2. Check current network round-trip time between primary and DR clusters. Protocol C requires less than 50ms RTT; higher latency will cause persistent lag. 3. Review the `spec.rpoThresholdSeconds` in your `DRBDReplicationPolicy` — if the threshold is set too aggressively for an asynchronous deployment, you may need to adjust it to reflect your actual SLA. 4. If the lag is increasing over time (not just spikes), investigate storage I/O throughput on the primary cluster worker nodes. --- ### Issue 3: `FailoverRequest` is stuck in `InProgress` and not progressing **Symptom:** `kubectl get failoverrequest -n dr- -w` shows `state: InProgress` and `phase: Retrying` but does not advance after several minutes. **Likely cause:** The `failover-controller` is unable to reconcile one or more VMs — typically because the Protection Group resource is missing from the target cluster, a VM definition is missing, or there is a storage (PVC/DRBD) issue preventing VM startup. **Fix:** 1. Check the retry count: `kubectl get failoverrequest -n dr- -o jsonpath='{.status.retryCount}'`. After 30 retries the controller will mark it `Failed`. 2. Inspect the failover-controller logs for the specific error: `kubectl logs deployment/failover-controller -n dr- | grep ""`. 3. Verify the Protection Group exists on the target cluster: `kubectl --kubeconfig ~/.kube/config-dr get protectiongroup `. 4. Check that VM definitions exist on the target cluster and that PVCs are bound. 5. If the `FailoverRequest` reaches `Failed`, delete it and create a new one after resolving the underlying issue — the idempotent reconciliation logic will pick up from the current state without requiring you to manually reset VMs. --- ### Issue 4: Protection Group `status.currentState` is stuck in `mixed` **Symptom:** `kubectl get protectiongroup ` shows `status.currentState: mixed` persisting for more than a few minutes. **Likely cause:** The `protection-group-controller` reconciled some VMs successfully but encountered an error stopping or starting others. This may be due to a VM that is unresponsive, a scheduling failure on the target cluster, or a taint blocking VM placement. **Fix:** 1. Identify which VMs are in an unexpected state: `kubectl --kubeconfig ~/.kube/config- get vm -n `. 2. Check the `protection-group-controller` logs on the affected cluster: `kubectl logs deployment/protection-group-controller -n --kubeconfig ~/.kube/config-`. 3. For VMs that will not start on the DR cluster, check for `drbd.linbit.com/lost-quorum` taints on worker nodes: `kubectl --kubeconfig ~/.kube/config-dr get nodes -o json | jq '.items[].spec.taints'`. 4. If taints are blocking scheduling, review the failover procedure to ensure the `failover-controller` completed its taint-removal step. Check the `failover-controller` logs for taint removal messages. --- ### Issue 5: Controller logs show no output or the pod is restarting **Symptom:** `kubectl logs deployment/failover-controller -n dr-` returns no output, or the pod is in `CrashLoopBackOff`. **Likely cause:** The controller pod is not running due to a configuration error, missing credentials, or a resource conflict. **Fix:** 1. Check pod status and recent events: `kubectl describe pod -l app=failover-controller -n dr-`. 2. Confirm the deployment's kubeconfig Secrets exist in the `dr-` namespace. 3. Verify RBAC permissions allow the controller's service account to read and patch the relevant CRDs in the namespace. 4. If the pod keeps restarting, check the previous container logs: `kubectl logs deployment/failover-controller -n dr- --previous`. --- ### Issue 6: `pgctl validate pg` reports the Protection Group is not ready **Symptom:** `pgctl validate pg ` returns validation errors or reports the group as not ready for failover. **Likely cause:** One or more VMs listed in the Protection Group are not synced to the DR cluster, replication is not at 100% for one or more volumes, or the `ReplicationGroupStatus` health is not `Healthy`. **Fix:** 1. Review the validation output carefully — `pgctl` will report which check failed. 2. Sync any missing VM definitions: `pgctl sync vm `. 3. Check `ReplicationGroupStatus` to confirm all volumes are fully synced before retrying validation. 4. If replication lag is the issue, wait for the lag to resolve (visible in `ReplicationGroupStatus.status.lastSyncTime`) and re-run validation.