Troubleshooting

Use this runbook to diagnose and resolve common controller errors, degraded replication states, and failed DR operations in a Trilio Site Recovery deployment running on OpenShift.

This runbook covers: interpreting controller logs from the quorum cluster controllers (failover-controller, protection-controller, pg-sync-controller, test-failover-controller); diagnosing degraded or failed ProtectionRequest, FailoverRequest, TestFailover, and ReplicationGroupStatus custom resources; checking drbd-node-agent health on workload cluster nodes; collecting a diagnostic bundle with tsr-gather; and common remediation steps.

This runbook does not cover: initial installation failures (see the deployment guide), DRBD kernel module installation, OpenShift cluster-level networking issues unrelated to Site Recovery, or KubeVirt / OpenShift Virtualization bugs.

Before using this runbook, ensure you have:

• oc CLI installed and configured with kubeconfig access to the quorum cluster and both workload clusters
• helm ≥ 3.0 installed
• Sufficient RBAC to get, list, describe, and logs across the dr-<name> namespace on the quorum cluster and the trilio-site-recovery-system namespace on workload clusters
• The deployment namespace names for your DR deployments (for example, dr-prod)
• The kubeconfig environment variables set for each cluster:
  Copy

  export KUBECONFIG_QUORUM=~/.kube/config-quorum
  export KUBECONFIG_CLUSTER1=~/.kube/config-cluster1
  export KUBECONFIG_CLUSTER2=~/.kube/config-cluster2
  

• OpenShift ≥ 4.14 on all clusters
• Trilio Site Recovery components deployed and previously healthy (if this is an initial deployment failure, consult the deployment guide instead)

Work through the sections below in order. Each section targets a specific symptom. Use the first section that matches your situation.

1. Establish a baseline — what is failing?

Before diving into logs, check the state of all Site Recovery custom resources on the quorum cluster. This tells you which controller to focus on.

Step 1.1 — List all Site Recovery CRDs on the quorum cluster

oc --kubeconfig $KUBECONFIG_QUORUM get \
  failoverrequest,testfailover,protectionrequest,replicationgroupstatus,rpoevents \
  -n <dr-namespace> -o wide

Look for resources with a Phase or Status of Failed, Degraded, or Critical. Note the resource name and kind — this determines which controller log to inspect.

Step 1.2 — Describe the failing resource

For any resource in a failed or stuck state, run:

oc --kubeconfig $KUBECONFIG_QUORUM describe <kind> <name> -n <dr-namespace>

The Events and Status.Conditions sections in the output almost always contain the first human-readable error message. Read these before going to logs.

Step 1.3 — Check the quorum control plane pod health

All quorum-side controllers (failover-controller, protection-controller, pg-sync-controller, test-failover-controller) run inside the site-recovery-quorum-control-plane deployment:

oc --kubeconfig $KUBECONFIG_QUORUM get pods -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane

If any pod is in CrashLoopBackOff, OOMKilled, or Error, address the pod health issue before troubleshooting individual CRDs. See Section 5 for pod restart procedures.

Step 1.4 — Check the workload control plane pod health on each workload cluster

# On the primary cluster
oc --kubeconfig $KUBECONFIG_CLUSTER1 get pods \
  -n trilio-site-recovery-system

# On the DR cluster
oc --kubeconfig $KUBECONFIG_CLUSTER2 get pods \
  -n trilio-site-recovery-system

Also check the drbd-node-agent DaemonSet on both clusters:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get daemonset -n trilio-site-recovery-system
oc --kubeconfig $KUBECONFIG_CLUSTER2 get daemonset -n trilio-site-recovery-system

If DaemonSet pods are not in Running state on every worker node, replication for VMs on those nodes will be degraded. Note which nodes have unhealthy agents.

2. Diagnosing a stuck or failed ProtectionRequest

A ProtectionRequest that stays in Pending or transitions to Failed means the protection-controller could not complete one of its three phases: VM validation, DRBDResource pair creation, or frontend PVC switchover.

Step 2.1 — Check the ProtectionRequest status

oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest <name> \
  -n <dr-namespace> -o yaml

Key fields to examine:

• status.phase — the last phase reached (Pending, Validating, CreatingDRBDResource, SwitchingPVCs, Protected, Failed)
• status.conditions — structured reason and message for each condition
• status.message — free-text error if present

Step 2.2 — Read the protection-controller logs

oc --kubeconfig $KUBECONFIG_QUORUM logs \
  -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
  --container manager \
  | grep -i 'protectionrequest\|protection-controller\|ERROR\|error' \
  | tail -100

Common error patterns and their meanings:

Step 2.3 — Remove a partial DRBDResource from a previous failed attempt

If the protection-controller left a partial DRBDResource CR, delete it from both clusters before re-applying the ProtectionRequest:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get drbdresource -n <vm-namespace>
oc --kubeconfig $KUBECONFIG_CLUSTER1 delete drbdresource <name> -n <vm-namespace>

oc --kubeconfig $KUBECONFIG_CLUSTER2 get drbdresource -n <vm-namespace>
oc --kubeconfig $KUBECONFIG_CLUSTER2 delete drbdresource <name> -n <vm-namespace>

Then delete and re-apply the ProtectionRequest on the quorum cluster.

Step 2.4 — Check the ProtectionZone webhook logs

oc --kubeconfig $KUBECONFIG_QUORUM logs \
  -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-protectionzone-controller \
  | tail -50

Webhook denials often indicate a misconfigured ProtectionZone CR or a missing cluster secret.

3. Diagnosing a stuck or failed FailoverRequest

A FailoverRequest that stays in InProgress or moves to Failed means the failover-controller could not complete one or more of its phases: VM shutdown on the source, DRBD volume promotion on the target, or VM startup on the target.

Step 3.1 — Check the FailoverRequest status

oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest <name> \
  -n <dr-namespace> -o yaml

Key fields:

• status.phase — Pending, InProgress, Completed, or Failed
• status.conditions — per-phase condition with reason and message
• status.failedVMs — list of VMs that could not be migrated if a partial failure occurred

Step 3.2 — Read the failover-controller logs

oc --kubeconfig $KUBECONFIG_QUORUM logs \
  -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
  --container manager \
  | grep -i 'failoverrequest\|failover-controller\|ERROR\|error' \
  | tail -200

Common error patterns:

Step 3.3 — Check VM events on the target cluster after a failover

oc --kubeconfig $KUBECONFIG_CLUSTER2 describe vm <vm-name> -n <vm-namespace>
oc --kubeconfig $KUBECONFIG_CLUSTER2 get events -n <vm-namespace> \
  --sort-by='.lastTimestamp' | tail -30

KubeVirt events will show why a VM failed to start (missing PVC, resource quota exceeded, scheduler pressure, and so on).

Step 3.4 — Check for a stuck FailoverRequest and how to recover

If a FailoverRequest is permanently stuck in InProgress and you need to recover:

1. Investigate the root cause using the log commands above before taking any action.
2. Do not delete the FailoverRequest while VMs are in an unknown state — this can leave the system split-brained.
3. If the primary cluster is available and VMs are still running there, a planned failover can be safely retried after resolving the underlying issue.
4. If the primary cluster is unavailable (unplanned failover scenario), promote DRBD resources manually only as a last resort — see Section 6.
5. Once the root cause is resolved, delete the failed FailoverRequest and apply a new one.

4. Diagnosing a failed or stuck TestFailover

The test-failover-controller manages the full lifecycle of TestFailover CRs. A failure in any phase leaves the system in a partially-built test environment on the DR cluster.

Step 4.1 — Check the TestFailover status and phase

oc --kubeconfig $KUBECONFIG_QUORUM get testfailover <name> \
  -n <dr-namespace> -o yaml

Phases in order: Pending → CreatingSnapshots → CreatingVolumes → CreatingVMs → VerifyingData → Succeeded or Failed → CleaningUp → Cleaned

Step 4.2 — Read the test-failover-controller logs

oc --kubeconfig $KUBECONFIG_QUORUM logs \
  -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
  --container manager \
  | grep -i 'testfailover\|test-failover\|ERROR\|error' \
  | tail -100

Common error patterns:

Step 4.3 — Manually clean up a stuck TestFailover

If the TestFailover is stuck in CleaningUp and the controller cannot proceed:

# Identify the test namespace (default is dr-test-<testfailover-name> or spec.testNamespace)
TEST_NS=<test-namespace>

# List all resources in the test namespace on the DR cluster
oc --kubeconfig $KUBECONFIG_CLUSTER2 get all,pvc,volumesnapshot -n $TEST_NS

# Delete the namespace (this removes all test resources)
oc --kubeconfig $KUBECONFIG_CLUSTER2 delete namespace $TEST_NS

# If the namespace is stuck due to finalizers, patch it:
oc --kubeconfig $KUBECONFIG_CLUSTER2 patch namespace $TEST_NS \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

Once the test namespace is cleaned up, delete the TestFailover CR so the controller can reset:

oc --kubeconfig $KUBECONFIG_QUORUM delete testfailover <name> -n <dr-namespace>

5. Diagnosing pg-sync-controller errors

The pg-sync-controller keeps Protection Group metadata synchronized between primary and DR clusters. Errors here manifest as stale VM specs on the DR side or missing standby VMs.

Step 5.1 — Read pg-sync-controller logs

oc --kubeconfig $KUBECONFIG_QUORUM logs \
  -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
  --container manager \
  | grep -i 'pg-sync\|pgsync\|synccontroller\|ERROR\|error' \
  | tail -100

Step 5.2 — Check for orphaned dataVolumeTemplates

The pg-sync-controller removes dataVolumeTemplates from standby VM specs on the DR cluster and replaces DataVolume references with PVC references. If you see VMs on the DR cluster attempting to import data rather than using existing PVCs, a sync failure has occurred.

# Inspect the standby VM spec on the DR cluster
oc --kubeconfig $KUBECONFIG_CLUSTER2 get vm <vm-name> -n <vm-namespace> -o yaml \
  | grep -A 20 dataVolumeTemplates

If dataVolumeTemplates is still present on the DR-side VM, the pg-sync-controller has not reconciled it yet. Check the controller logs for the specific error and ensure the controller pod is healthy.

Step 5.3 — Verify the controller can reach both workload cluster APIs

The pg-sync-controller reads from the primary cluster and writes to the DR cluster. If either cluster's API server is unreachable, syncs will fail. Check that the cluster credentials stored as secrets in the dr-<name> namespace are current:

oc --kubeconfig $KUBECONFIG_QUORUM get secrets -n <dr-namespace> \
  | grep -i cluster

If cluster credentials have rotated, update the relevant secrets and the controller will automatically retry.

6. Diagnosing degraded DRBD replication

DRBD replication health is surfaced through ReplicationGroupStatus and RPOEvent CRDs on the quorum cluster, and through the drbd-node-agent on each workload cluster node.

Step 6.1 — Check ReplicationGroupStatus

oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n <dr-namespace> -o wide

Overall status will be Healthy, Degraded, or Critical. For detail:

oc --kubeconfig $KUBECONFIG_QUORUM describe replicationgroupstatus <name> -n <dr-namespace>

Key fields:

• status.overallHealth — aggregate health
• status.volumes — per-volume sync state and out-of-sync byte count
• status.lastSyncTimestamp — when the last full sync completed

Step 6.2 — Check for RPOEvents

RPOEvents record replication lag violations. Review recent events to understand severity and duration:

oc --kubeconfig $KUBECONFIG_QUORUM get rpoevents -n <dr-namespace> \
  --sort-by='.spec.timestamp' | tail -20

oc --kubeconfig $KUBECONFIG_QUORUM describe rpoevent <name> -n <dr-namespace>

Key fields to note: spec.rpoAtEvent (observed lag in seconds), spec.violationReason, spec.outOfSyncBytesAtEvent.

Step 6.3 — Check drbd-node-agent logs on the affected node

First, identify which node is hosting the affected VM PVCs:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/name=drbd-node-agent -o wide

Then retrieve logs from the agent on the specific node:

oc --kubeconfig $KUBECONFIG_CLUSTER1 logs \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/name=drbd-node-agent \
  --field-selector spec.nodeName=<node-name> \
  | tail -150

Do the same on the DR cluster for the corresponding node.

Common drbd-node-agent error patterns:

Step 6.4 — Check DRBDResource status for a specific VM

# On the primary cluster
oc --kubeconfig $KUBECONFIG_CLUSTER1 get drbdresource -n <vm-namespace> -o wide
oc --kubeconfig $KUBECONFIG_CLUSTER1 describe drbdresource <name> -n <vm-namespace>

# On the DR cluster
oc --kubeconfig $KUBECONFIG_CLUSTER2 get drbdresource -n <vm-namespace> -o wide
oc --kubeconfig $KUBECONFIG_CLUSTER2 describe drbdresource <name> -n <vm-namespace>

The status.connectionState and status.volumes[].syncProgress fields show per-volume replication state.

7. Restarting a controller pod

If a quorum control plane pod is in CrashLoopBackOff or Error, restart it by deleting the pod (the deployment will recreate it):

# Identify the pod name
oc --kubeconfig $KUBECONFIG_QUORUM get pods -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane

# Delete the pod to trigger a restart
oc --kubeconfig $KUBECONFIG_QUORUM delete pod <pod-name> -n <dr-namespace>

For workload cluster control plane pods:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get pods -n trilio-site-recovery-system
oc --kubeconfig $KUBECONFIG_CLUSTER1 delete pod <pod-name> -n trilio-site-recovery-system

For drbd-node-agent pods (DaemonSet), delete the pod on the specific node:

oc --kubeconfig $KUBECONFIG_CLUSTER1 delete pod <drbd-agent-pod-name> \
  -n trilio-site-recovery-system

After deletion, verify the replacement pod reaches Running:

oc --kubeconfig $KUBECONFIG_QUORUM get pods -n <dr-namespace> -w

8. Collecting a diagnostic bundle with tsr-gather

When you cannot resolve an issue with the steps above, collect a support bundle before contacting Trilio support. Run tsr-gather against the quorum cluster — it automatically collects data from the quorum cluster and all associated workload clusters.

Step 8.1 — Collect the full estate bundle

oc adm must-gather \
  --image=<tsr-gather-image>:<tag> \
  --kubeconfig $KUBECONFIG_QUORUM

This collects 48 hours of logs by default.

Step 8.2 — Collect a narrower bundle (recent logs only, one zone)

# Last 2 hours of logs
oc adm must-gather \
  --image=<tsr-gather-image>:<tag> \
  --kubeconfig $KUBECONFIG_QUORUM \
  -- /usr/bin/gather --since 2h

# Specific protection zone only
oc adm must-gather \
  --image=<tsr-gather-image>:<tag> \
  --kubeconfig $KUBECONFIG_QUORUM \
  -- /usr/bin/gather --zone <zone-name>

# Quorum cluster only (if workload clusters are unreachable)
oc adm must-gather \
  --image=<tsr-gather-image>:<tag> \
  --kubeconfig $KUBECONFIG_QUORUM \
  -- /usr/bin/gather --local-only

Step 8.3 — Read the bundle

Once the bundle is written to your local directory:

1. Open SUMMARY.md first — it contains a human-readable summary of what is wrong.
2. Review manifest.yaml for the inventory of what was collected.
3. Log files, CRD state, and configuration are organized by cluster and namespace.

Note: tsr-gather never collects secrets, kubeconfig files, cloud-init payloads, or DRBD shared secrets.

After applying any remediation step, confirm the system has returned to a healthy state by checking each of the following:

Controllers are running

oc --kubeconfig $KUBECONFIG_QUORUM get pods -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane

All pods should be in Running state with no restarts in the last few minutes.

drbd-node-agent DaemonSet is fully scheduled on both clusters

oc --kubeconfig $KUBECONFIG_CLUSTER1 get daemonset -n trilio-site-recovery-system
oc --kubeconfig $KUBECONFIG_CLUSTER2 get daemonset -n trilio-site-recovery-system

DESIRED should equal READY on both clusters.

Target CRD has reached a terminal success state

# ProtectionRequest should be in Protected phase
oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest <name> -n <dr-namespace> \
  -o jsonpath='{.status.phase}'

# FailoverRequest should be in Completed phase
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest <name> -n <dr-namespace> \
  -o jsonpath='{.status.phase}'

# TestFailover should be in Succeeded phase
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover <name> -n <dr-namespace> \
  -o jsonpath='{.status.phase}'

Replication is healthy

oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n <dr-namespace> \
  -o jsonpath='{.items[*].status.overallHealth}'

All values should be Healthy. No new RPOEvent CRs should be appearing:

oc --kubeconfig $KUBECONFIG_QUORUM get rpoevents -n <dr-namespace> \
  --sort-by='.spec.timestamp' | tail -5

No error-level log entries in the last 5 minutes

oc --kubeconfig $KUBECONFIG_QUORUM logs \
  -n <dr-namespace> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
  --container manager --since 5m \
  | grep -i 'error\|ERROR\|panic' | wc -l

A count of 0 indicates no recent errors.

Site Recovery operations are CRD-driven; most rollback actions involve deleting the CR that triggered the failed operation and restoring the prior state.

Rolling back a failed ProtectionRequest

If protection partially applied and left the VM in an inconsistent state:

1. Stop the VM on the primary cluster if it is running:
   Copy

   oc --kubeconfig $KUBECONFIG_CLUSTER1 \
     patch vm <vm-name> -n <vm-namespace> \
     --type=merge -p '{"spec":{"running":false}}'
   

2. Delete the DRBDResource CRs from both clusters (see Step 2.3 above).
3. If the VM's PVCs were already switched to DRBD-backed frontend PVCs, restore the original PVC reference in the VM spec. Consult the VM configuration you captured before applying protection.
4. Delete the ProtectionRequest CR:
   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM delete protectionrequest <name> -n <dr-namespace>
   

5. Restart the VM after confirming its PVC is accessible.

Rolling back a failed FailoverRequest (planned failover only)

If a planned failover failed partway through and the primary cluster is still available:

1. Confirm VMs are still stopped on the primary (do not start them yet).
2. Verify DRBD volumes were not promoted on the DR side by checking DRBDResource role fields:
   Copy

   oc --kubeconfig $KUBECONFIG_CLUSTER2 get drbdresource -n <vm-namespace> \
     -o jsonpath='{.items[*].spec.role}'
   

   If the role is still Secondary on the DR side, the promotion did not complete and the primary side is still authoritative.
3. Restart the VMs on the primary cluster:
   Copy

   oc --kubeconfig $KUBECONFIG_CLUSTER1 \
     patch vm <vm-name> -n <vm-namespace> \
     --type=merge -p '{"spec":{"running":true}}'
   

4. Delete the failed FailoverRequest:
   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM delete failoverrequest <name> -n <dr-namespace>
   

Warning: Do not attempt to roll back an unplanned failover where the primary cluster is confirmed down. In that scenario, the DR cluster is the only authoritative copy and rollback is not applicable until the primary is restored and a failback is performed.

Rolling back a failed TestFailover

See Step 4.3 for cleaning up test resources. TestFailovers are non-disruptive and do not affect production VMs; no further rollback is required.

Rolling back a controller pod restart

Controller pod restarts are stateless with respect to the controllers themselves — the desired state is in etcd via the CRDs. If a pod restart causes an unexpected behavior, scale down the deployment to pause reconciliation while you investigate:

oc --kubeconfig $KUBECONFIG_QUORUM scale deployment \
  site-recovery-quorum-control-plane \
  -n <dr-namespace> --replicas=0

Restore it after the investigation:

oc --kubeconfig $KUBECONFIG_QUORUM scale deployment \
  site-recovery-quorum-control-plane \
  -n <dr-namespace> --replicas=1

If you have worked through this runbook and the issue is not resolved, escalate to Trilio support with the following information.

Information to gather before escalating

1. tsr-gather bundle — Collect a full diagnostic bundle as described in Section 8. This is the most important artifact for support:

   Copy

   oc adm must-gather \
     --image=<tsr-gather-image>:<tag> \
     --kubeconfig $KUBECONFIG_QUORUM
   

2. Symptom description — Include:

   • Which workflow failed (protection, failover, test failover, replication health)
   • The name and namespace of the failing CRD
   • The exact phase or status field value where it is stuck
   • A timeline of when the issue started and any changes made before it appeared (upgrades, configuration changes, network events)

3. CRD YAML snapshots — Capture the full YAML of the failing resource:

   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM get <kind> <name> -n <dr-namespace> \
     -o yaml > failing-resource.yaml
   

4. ReplicationGroupStatus and RPOEvents — If the issue involves replication degradation:

   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus,rpoevents \
     -n <dr-namespace> -o yaml > replication-status.yaml
   

5. Component versions — Capture the Helm release versions for all Site Recovery charts:

   Copy

   helm --kubeconfig $KUBECONFIG_QUORUM list -n <dr-namespace>
   helm --kubeconfig $KUBECONFIG_CLUSTER1 list -n trilio-site-recovery-system
   helm --kubeconfig $KUBECONFIG_CLUSTER2 list -n trilio-site-recovery-system
   

6. OpenShift versions — Run on each cluster:

   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM version
   oc --kubeconfig $KUBECONFIG_CLUSTER1 version
   oc --kubeconfig $KUBECONFIG_CLUSTER2 version
   

Contact

Submit a support case at the TrilioData support portal and attach the tsr-gather bundle and supplementary YAML files. Reference the specific runbook section where the troubleshooting stalled.