Trilio Site Recovery for Kubernetes
Docs/site-recovery/Observability
Guide

Observability

Metrics, logging, and monitoring for site recovery operations

Troubleshooting

Use a consistent approach for each issue: identify the symptom, find the likely cause using the CRs and logs described in the Usage section, then apply the fix.

Issue 1 — ReplicationGroupStatus shows Degraded or Critical

Symptom: oc get replicationgroupstatuses shows Degraded or Critical for one or more ProtectionGroups.

Likely causes:

  • Network interruption between primary and DR worker nodes on TCP ports 7000–8000.
  • The DRBD node agent on a workload node has not converged. Look for a siterecovery.trilio.io/not-ready:NoSchedule taint on the node.
  • A workload node was rebooted and the DRBD kernel modules are reloading.

Fix:

  1. Check the DRBDResource CRs on both clusters for per-volume connection state: 
    oc get drbdresource -n <workload-namespace> -o wide
  2. Inspect node agent logs on the affected node: 
    oc logs -n <workload-namespace> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --follow
  3. Verify network connectivity on TCP ports 7000–8000 between the primary and DR worker nodes.
  4. Check whether the siterecovery.trilio.io/not-ready:NoSchedule taint is still applied to the node. If DRBD has converged, the node agent removes it automatically. If the taint persists after several minutes, the agent pod may itself be unhealthy.
  5. Review RPOEvent CRs for a timeline of when the degradation started: 
    oc get rpoevents -n dr-<name> --sort-by=.spec.timestamp

Issue 2 — No site-recovery-logging ConfigMap; log level change has no effect

Symptom: After patching spec.logging.runtimeLevel on the ProtectionZone, no ConfigMap appears and log verbosity does not change.

Likely cause: The ProtectionZone has not reached Available, so the ProtectionZone controller has not reconciled the zone. The ConfigMap is only rendered when runtimeLevel is non-empty and the zone is healthy.

Fix:

  1. Check the ProtectionZone status: 
    oc get protectionzone -n dr-<name>
  2. If the zone is not Available, describe it for conditions: 
    oc describe protectionzone my-zone -n dr-<name>
  3. Once the zone is Available, re-apply the patch. The ConfigMap should appear within one reconciliation cycle.

Issue 3 — Metrics endpoint returns a TLS error or connection refused

Symptom: curl https://localhost:8443/metrics from inside the manager pod fails with a certificate error or a connection refusal.

Likely causes:

  • The webhook-server-cert Secret does not exist in the release namespace. The manager will not start without it.
  • The metrics Service is not present (e.g., metrics.enabled was set to false internally).

Fix:

  1. Verify the Secret exists: 
    oc get secret webhook-server-cert -n dr-<name>
    If absent, create it with your CA-signed certificate and key: 
    oc create secret tls webhook-server-cert \
  --cert=tls.crt --key=tls.key \
  -n dr-<name>
  2. Check the manager pod is running (expected 1/1): 
    oc get pods -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
  3. Use -k to skip TLS verification for ad-hoc curl requests against the self-signed cert, or supply your CA bundle.

Issue 4 — ProtectionRequest stuck in Validating or moves to Failed

Symptom: A ProtectionRequest does not progress past Validating, or immediately transitions to Failed.

Likely cause: The protection controller found a configuration violation. Common causes include a VM disk that does not use volumeMode: Block, the VM not being in a running state at protection time, or a VM being on a node without a published InternalIP.

Fix:

  1. Describe the ProtectionRequest for conditions and events: 
    oc describe protectionrequest <name> -n <workload-namespace>
    Look for a Validated=False condition and the associated warning event message.
  2. Correct the VM configuration (disk mode, VM state, node configuration).
  3. The Failed phase is terminal. Create a new ProtectionRequest after fixing the underlying issue — you cannot retry the existing CR.

Issue 5 — TestFailover is stuck and the test namespace cannot be cleaned up

Symptom: A TestFailover CR is stuck in a phase (e.g., CreatingVMs) and you want to remove the test namespace.

Likely cause: The test-failover controller manages the lifecycle of the test namespace (status.testNamespace). If the controller is healthy, it will eventually transition to CleaningUp and then Cleaned.

Fix:

  1. Check the TestFailover phase and test namespace: 
    oc get testfailover <name> -n dr-<name> \
  -o jsonpath='{.status.phase} {.status.testNamespace}'
  2. Stream controller logs filtered to the test-failover reconciler to identify the blocking step: 
    oc logs -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager | jq 'select(.controller == "testfailover")'
  3. Do not attempt to manually delete the test namespace or patch finalizers — the test-failover controller owns the namespace lifecycle. Manual interference can leave the protection group in an inconsistent state.
  4. If the controller pod itself is unhealthy, restart it by deleting the pod (the Deployment will recreate it): 
    oc delete pod -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

Issue 6 — oc adm drain does not complete on a node hosting a protected VM

Symptom: oc adm drain <node> hangs and never completes.

Likely cause: The node hosts a running protected VM. Every KubeVirt migration path — including drain-triggered evacuation — is denied for protected VMs: live migration of VM "<name>" is not allowed: it uses DRBD-protected volumes with node-pinned PVs. The anchor pod for stopped protected VMs also blocks the drain because it has no controller owner and is node-pinned, so --force does not help.

Fix:

  • For in-place maintenance (reboot, kernel update) of the same node: proceed directly. DRBD modules reload on boot, the anchor pod returns to the same node, and DRBD reconnects and resynchronises the delta automatically. A drain is not needed.
  • To permanently remove the node: unprotect all VMs pinned to that node, perform node removal, then re-protect the VMs on a new node. Re-protection requires a full initial resynchronisation.
  • If the node was lost unexpectedly: follow the node-loss recovery runbook (docs/runbooks/drbdresource-node-loss.md). Remove only the agent finalizer as documented — removing the placement finalizer instead permanently leaks a PVC/PV pair.

Issue 7 — No metrics or log data in a third-party monitoring system

Symptom: Your Prometheus instance or log aggregator is not receiving data from TSR.

Likely cause: The metrics endpoint requires bearer-token authentication (TokenReview/SubjectAccessReview). An unauthenticated scrape job will be rejected. Log forwarding requires configuring your OpenShift log forwarding pipeline to target the dr-<name> namespace.

Fix:

  1. Configure your Prometheus scrape job to include a bearer token bound to the metrics-reader ClusterRole (which grants get and put on /debug/loglevel and get on /metrics).
  2. The metrics endpoint is on port 8443 with TLS. Supply the CA bundle or disable TLS verification in your scrape config.
  3. For log aggregation, configure your OpenShift log forwarding pipeline (e.g., ClusterLogForwarder) to forward logs from the dr-<name> namespace. Logs are emitted as structured JSON from the manager container.