Observability

This page explains how to monitor Site Recovery controller health and operation outcomes using Kubernetes-native observability patterns. Site Recovery surfaces its operational state through Custom Resource status fields, structured controller logs, and the Site Manager UI — giving you visibility into replication health, failover progress, Protection Group readiness, and RPO compliance without requiring external agents beyond what is deployed automatically. Understanding these signals is essential for SREs who need to detect degraded replication early, audit failover operations, and respond quickly when DR readiness is at risk.

Before using the observability features described on this page, ensure you have the following in place:

• Kubernetes ≥ 1.28 on all clusters (quorum, primary, and DR)
• kubectl configured with access to the quorum cluster and at minimum read access to primary and DR clusters
• pgctl installed and a deployment context configured (see the DR deployment configuration guide)
• Site Recovery controllers deployed on the quorum cluster: failover-controller, protection-controller, pg-sync-controller, and replication-monitor
• protection-group-controller and test-failover-controller deployed on the primary and DR clusters
• replication-monitor deployed and running — this agent is deployed automatically as part of the standard Ansible playbooks and is responsible for writing RPOEvent and ReplicationGroupStatus resources
• Site Manager UI accessible on the quorum cluster if you prefer a dashboard view
• Read access to the dr-<name> namespace on the quorum cluster for your deployment

Site Recovery's observability components require no separate installation. The replication-monitor agent and all controllers that produce observable CRD state are deployed automatically when you run the standard Ansible playbooks during initial setup.

To verify that all observability-relevant components are running after deployment:

Step 1: Confirm controllers are running on the quorum cluster.

kubectl get pods -n dr-<deployment-name> -l app=failover-controller
kubectl get pods -n dr-<deployment-name> -l app=protection-controller
kubectl get pods -n dr-<deployment-name> -l app=pg-sync-controller
kubectl get pods -n dr-<deployment-name> -l app=replication-monitor

All pods should be in Running status with no crash loops.

Step 2: Confirm the protection-group-controller is running on the primary and DR clusters.

kubectl --kubeconfig ~/.kube/config-primary get pods -l app=protection-group-controller
kubectl --kubeconfig ~/.kube/config-dr get pods -l app=protection-group-controller

Step 3: Confirm CRD types are registered.

kubectl get crds | grep siterecovery.trilio.io

Expected output includes entries for protectiongroups, protectionrequests, failoverrequests, testfailovers, drbdreplicationpolicies, drbdvolumes, rpoevents, and replicationgroupstatuses.

Step 4: Verify the Site Manager UI is reachable.

kubectl get svc -n dr-<deployment-name> -l app=site-manager-ui

Note the service address and confirm the dashboard loads in your browser.

Site Recovery's observability behavior is shaped by thresholds and intervals embedded in controller configuration. The following settings are most relevant to monitoring outcomes.

Replication Monitor: RPO Threshold

The replication-monitor evaluates replication lag continuously and writes an RPOEvent resource whenever lag exceeds the configured threshold. The threshold is set per DRBDReplicationPolicy.

For Protocol C deployments, any non-zero lag exceeding the threshold indicates a network or storage problem that warrants immediate investigation. For Protocol A deployments, small RPO windows are expected; set your threshold to match your SLA.

Failover Controller: Reconciliation Interval and Retry Limit

The failover-controller uses a timer-based reconciliation loop when processing a FailoverRequest. These intervals determine how quickly the controller retries after a partial failure.

These values are fixed in the controller implementation. You cannot override them via CRD fields. If a failover reaches the Failed state due to a transient issue, you can delete the FailoverRequest resource and create a new one — the underlying VM and Protection Group state is reconciled idempotently.

Protection Group Controller: Status Polling

The protection-group-controller on each cluster reconciles VM state against the Protection Group's spec.desiredState and updates status.currentState. It polls VM state approximately every 60 seconds in steady state, and immediately on any spec change.

A mixed state persisting for more than a few minutes indicates a reconciliation problem and requires investigation.

Viewing Protection Group Replication Health

The ReplicationGroupStatus resource provides an aggregated replication health view for each Protection Group, including overall health (Healthy, Degraded, or Critical), per-volume sync state, and the timestamp of the last successful sync.

kubectl get replicationgroupstatus -n dr-<deployment-name>

To inspect a specific group:

kubectl describe replicationgroupstatus <pg-name> -n dr-<deployment-name>

Look for status.health and status.lastSyncTime. A Degraded or Critical health value means one or more volumes are not replicating correctly. Cross-reference with RPOEvent resources to understand the history.

Monitoring RPO Violations

The replication-monitor writes an RPOEvent resource each time replication lag exceeds your configured threshold. List recent violations:

kubectl get rpoevents -n dr-<deployment-name> --sort-by='.metadata.creationTimestamp'

Each RPOEvent records which Protection Group was affected, the observed lag value, and the severity. A pattern of frequent RPOEvent resources for the same Protection Group may indicate network instability between primary and DR clusters or storage I/O pressure.

Monitoring a FailoverRequest in Progress

When a failover is triggered (planned or unplanned), the failover-controller updates the FailoverRequest status continuously. Watch the progression:

kubectl get failoverrequest <name> -n dr-<deployment-name> -w

Expected status.state progression: Pending → InProgress → Completed (or Failed after 30 retry attempts).

For per-VM detail during the operation:

kubectl get failoverrequest <name> -n dr-<deployment-name> -o jsonpath='{.status.vmStatuses}'

Monitoring Individual Volume Replication

In DRBD Operator deployments, each DRBDVolume resource exposes sync progress and frontend PVC references. Use this to verify per-volume replication state:

kubectl get drbdvolumes -n <vm-namespace> --kubeconfig ~/.kube/config-primary

To inspect a specific volume:

kubectl describe drbdvolume <volume-name> -n <vm-namespace> --kubeconfig ~/.kube/config-primary

Check status.syncProgress (should be 100% for a fully synced volume) and status.role (should be Primary on the active cluster).

Checking Controller Logs

All controllers emit structured logs. Use these to trace specific operations:

# Failover controller on the quorum cluster
kubectl logs -f deployment/failover-controller -n dr-<deployment-name>

# Protection controller on the quorum cluster
kubectl logs -f deployment/protection-controller -n dr-<deployment-name>

# Protection Group controller on the primary cluster
kubectl logs -f deployment/protection-group-controller -n <namespace> --kubeconfig ~/.kube/config-primary

# Replication monitor on the quorum cluster
kubectl logs -f deployment/replication-monitor -n dr-<deployment-name>

Filter logs for a specific Protection Group name or failover operation name to reduce noise:

kubectl logs deployment/failover-controller -n dr-<deployment-name> | grep "production-protection-group"

Using pgctl for Status Checks

The pgctl CLI provides a convenient view of Protection Group status across clusters:

# Validate a Protection Group's configuration and replication readiness
pgctl validate pg <pg-name>

# Inspect a Protection Group's current state
pgctl get pg <pg-name>

Using the Site Manager UI

The Site Manager UI on the quorum cluster provides dashboards covering cluster health, VM protection status, Protection Group management, failover operation history, and replication status. Navigate to the UI in your browser and select your deployment from the deployments panel. The Protection Groups dashboard shows each group's replication health status and allows you to trigger operations directly.

Example 1: Inspect a Protection Group's aggregated replication health

kubectl describe replicationgroupstatus production-protection-group -n dr-production

Expected output (abbreviated):

Name:         production-protection-group
Namespace:    dr-production
...
Status:
  Health:          Healthy
  Last Sync Time:  2025-11-01T14:32:05Z
  Volume Statuses:
    Volume Name:    pvc-12345
    Sync State:     Synced
    Volume Name:    pvc-67890
    Sync State:     Synced

A Health value of Healthy with all volumes in Synced state confirms the Protection Group is ready for failover.

Example 2: List recent RPO violations for all Protection Groups

kubectl get rpoevents -n dr-production --sort-by='.metadata.creationTimestamp' -o wide

Expected output:

NAME                                  PROTECTION-GROUP              LAG      SEVERITY   AGE
rpoevent-production-pg-1730462001     production-protection-group   12s      Warning    5m
rpoevent-production-pg-1730461800     production-protection-group   8s       Warning    8m

Multiple RPOEvent resources in a short window for the same Protection Group indicates a persistent replication lag problem. Investigate network connectivity between primary and DR worker nodes on TCP 7000–7999.

Example 3: Watch a FailoverRequest transition through its lifecycle

kubectl get failoverrequest production-failover-1730462500 -n dr-production -w

Expected output progression:

NAME                                STATE       PHASE          RETRY-COUNT   AGE
production-failover-1730462500      Pending     Initializing   0             2s
production-failover-1730462500      InProgress  StoppingVMs    1             12s
production-failover-1730462500      InProgress  StartingVMs    3             32s
production-failover-1730462500      Completed   Completed      4             52s

A Completed state confirms all VMs in the Protection Group have been successfully started on the target cluster.

Example 4: Check per-VM status detail from a FailoverRequest

kubectl get failoverrequest production-failover-1730462500 -n dr-production \
  -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool

Expected output:

[
  {
    "name": "prod-vm-1",
    "state": "Running",
    "cluster": "dr-cluster"
  },
  {
    "name": "prod-vm-2",
    "state": "Running",
    "cluster": "dr-cluster"
  }
]

Example 5: Inspect a DRBDVolume for sync progress in a DRBD Operator deployment

kubectl describe drbdvolume pvc-12345 -n production --kubeconfig ~/.kube/config-primary

Expected output (abbreviated):

Name:         pvc-12345
Namespace:    production
...
Status:
  Sync Progress:   100%
  Role:            Primary
  Frontend PVC:    prod-vm-1-disk-frontend
  Health:          Healthy

A Sync Progress of 100% and Role of Primary on the primary cluster confirms this volume is fully replicated and serving the VM's disk I/O.

Example 6: Filter failover-controller logs for a specific operation

kubectl logs deployment/failover-controller -n dr-production | grep "production-failover-1730462500"

Expected output (representative lines):

[2025-11-01T14:41:40Z] [INFO] Acquired failover lock for production-protection-group
[2025-11-01T14:41:40Z] [INFO] VM prod-vm-1 still Running on primary - stopping
[2025-11-01T14:41:50Z] [INFO] VM prod-vm-1 already Running on dr-cluster - SUCCESS
[2025-11-01T14:41:50Z] [INFO] VM prod-vm-2 stopped on source, starting on dr-cluster
[2025-11-01T14:42:00Z] [INFO] Released failover lock for production-protection-group

Example 7: Check Protection Group state on both clusters after a failover

# Primary cluster — should be stopped
kubectl --kubeconfig ~/.kube/config-primary get protectiongroup production-protection-group \
  -o jsonpath='{.status.currentState}'
# Expected: stopped

# DR cluster — should be running
kubectl --kubeconfig ~/.kube/config-dr get protectiongroup production-protection-group \
  -o jsonpath='{.status.currentState}'
# Expected: running

Issue 1: ReplicationGroupStatus shows Degraded or Critical

Symptom: kubectl get replicationgroupstatus shows a Protection Group with health: Degraded or health: Critical.

Likely cause: One or more DRBD volumes are not fully synced between the primary and DR clusters. This can be caused by network interruption on TCP 7000–7999, storage I/O pressure on worker nodes, or a recent node restart.

Fix:

1. Identify which volumes are affected: kubectl describe replicationgroupstatus <pg-name> -n dr-<deployment-name> and look at Volume Statuses.
2. Check RPOEvent resources for the same Protection Group to understand how long the lag has persisted.
3. Verify network connectivity between primary and DR worker nodes on TCP 7000–7999.
4. Check replication-monitor logs for error details: kubectl logs deployment/replication-monitor -n dr-<deployment-name>.
5. In DRBD Operator deployments, describe the affected DRBDVolume resource for per-volume sync progress and role information.

Issue 2: RPOEvent resources are being created frequently

Symptom: Many RPOEvent resources are appearing in a short time window for the same Protection Group.

Likely cause: Replication lag is consistently exceeding the threshold defined in DRBDReplicationPolicy. This is most common with Protocol A (asynchronous) deployments under high write workloads, or when network latency between clusters has increased.

Fix:

1. List recent events: kubectl get rpoevents -n dr-<deployment-name> --sort-by='.metadata.creationTimestamp'.
2. Check current network round-trip time between primary and DR clusters. Protocol C requires less than 50ms RTT; higher latency will cause persistent lag.
3. Review the spec.rpoThresholdSeconds in your DRBDReplicationPolicy — if the threshold is set too aggressively for an asynchronous deployment, you may need to adjust it to reflect your actual SLA.
4. If the lag is increasing over time (not just spikes), investigate storage I/O throughput on the primary cluster worker nodes.

Issue 3: FailoverRequest is stuck in InProgress and not progressing

Symptom: kubectl get failoverrequest <name> -n dr-<deployment-name> -w shows state: InProgress and phase: Retrying but does not advance after several minutes.

Likely cause: The failover-controller is unable to reconcile one or more VMs — typically because the Protection Group resource is missing from the target cluster, a VM definition is missing, or there is a storage (PVC/DRBD) issue preventing VM startup.

Fix:

1. Check the retry count: kubectl get failoverrequest <name> -n dr-<deployment-name> -o jsonpath='{.status.retryCount}'. After 30 retries the controller will mark it Failed.
2. Inspect the failover-controller logs for the specific error: kubectl logs deployment/failover-controller -n dr-<deployment-name> | grep "<failoverrequest-name>".
3. Verify the Protection Group exists on the target cluster: kubectl --kubeconfig ~/.kube/config-dr get protectiongroup <pg-name>.
4. Check that VM definitions exist on the target cluster and that PVCs are bound.
5. If the FailoverRequest reaches Failed, delete it and create a new one after resolving the underlying issue — the idempotent reconciliation logic will pick up from the current state without requiring you to manually reset VMs.

Issue 4: Protection Group status.currentState is stuck in mixed

Symptom: kubectl get protectiongroup <pg-name> shows status.currentState: mixed persisting for more than a few minutes.

Likely cause: The protection-group-controller reconciled some VMs successfully but encountered an error stopping or starting others. This may be due to a VM that is unresponsive, a scheduling failure on the target cluster, or a taint blocking VM placement.

Fix:

1. Identify which VMs are in an unexpected state: kubectl --kubeconfig ~/.kube/config-<cluster> get vm -n <namespace>.
2. Check the protection-group-controller logs on the affected cluster: kubectl logs deployment/protection-group-controller -n <namespace> --kubeconfig ~/.kube/config-<cluster>.
3. For VMs that will not start on the DR cluster, check for drbd.linbit.com/lost-quorum taints on worker nodes: kubectl --kubeconfig ~/.kube/config-dr get nodes -o json | jq '.items[].spec.taints'.
4. If taints are blocking scheduling, review the failover procedure to ensure the failover-controller completed its taint-removal step. Check the failover-controller logs for taint removal messages.

Issue 5: Controller logs show no output or the pod is restarting

Symptom: kubectl logs deployment/failover-controller -n dr-<deployment-name> returns no output, or the pod is in CrashLoopBackOff.

Likely cause: The controller pod is not running due to a configuration error, missing credentials, or a resource conflict.

Fix:

1. Check pod status and recent events: kubectl describe pod -l app=failover-controller -n dr-<deployment-name>.
2. Confirm the deployment's kubeconfig Secrets exist in the dr-<deployment-name> namespace.
3. Verify RBAC permissions allow the controller's service account to read and patch the relevant CRDs in the namespace.
4. If the pod keeps restarting, check the previous container logs: kubectl logs deployment/failover-controller -n dr-<deployment-name> --previous.

Issue 6: pgctl validate pg reports the Protection Group is not ready

Symptom: pgctl validate pg <pg-name> returns validation errors or reports the group as not ready for failover.

Likely cause: One or more VMs listed in the Protection Group are not synced to the DR cluster, replication is not at 100% for one or more volumes, or the ReplicationGroupStatus health is not Healthy.

Fix:

1. Review the validation output carefully — pgctl will report which check failed.
2. Sync any missing VM definitions: pgctl sync vm <vm-name>.
3. Check ReplicationGroupStatus to confirm all volumes are fully synced before retrying validation.
4. If replication lag is the issue, wait for the lag to resolve (visible in ReplicationGroupStatus.status.lastSyncTime) and re-run validation.