Recovery Operation Failures

This page helps you diagnose and recover from failed or partially completed disaster recovery operations in Trilio Site Recovery. Recovery operations—planned failovers, unplanned failovers, test failovers, and failback—are stateful, multi-step workflows coordinated across two or three clusters by the failover-controller and related operators running on the quorum cluster. When a failure occurs mid-workflow, the system leaves a persistent audit trail in FailoverRequest, TestFailover, and ReplicationGroupStatus custom resources, giving you a precise starting point for diagnosis. This guide walks through reading that state, identifying root causes, understanding rollback behavior, and recovering to a consistent operational state.

Before using this guide, ensure you have:

• OpenShift 4.14 or later on all clusters (quorum, primary, DR)
• OpenShift Virtualization (CNV) 1.0 or later
• kubectl or oc CLI installed and configured
• Kubeconfig files for all clusters in your DR estate (quorum, primary, and DR clusters)
• Read access to the DR namespace (dr-<name>) on the quorum cluster
• Sufficient RBAC permissions to read and patch FailoverRequest, TestFailover, ProtectionGroup, and DRBDResource custom resources across clusters
• The tsr-gather tool available if you need to collect a full diagnostic bundle for support

All commands that target a specific cluster use --kubeconfig flags explicitly. Substitute <quorum-kubeconfig>, <primary-kubeconfig>, and <dr-kubeconfig> with the actual paths to your kubeconfig files, and dr-<name> with your actual DR namespace.

No additional installation is required to diagnose recovery operation failures. The diagnostic tooling ships as part of the Site Recovery components you already deployed. If you need to collect a full support bundle, tsr-gather is available on the quorum cluster.

Step 1 — Verify controller health on the quorum cluster

Before diagnosing individual operation failures, confirm that the controllers responsible for orchestrating recovery are running:

kubectl --kubeconfig <quorum-kubeconfig> get pods \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane

Expected output shows all pods in Running state with a low restart count:

NAME                                              READY   STATUS    RESTARTS   AGE
site-recovery-quorum-control-plane-7d9f6c-xkp2l   3/3     Running   0          4d

If any pod is in CrashLoopBackOff or Error, retrieve its logs before proceeding with operation-level diagnosis:

kubectl --kubeconfig <quorum-kubeconfig> logs \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane \
  --all-containers \
  --tail=200

Step 2 — Verify workload control plane health on primary and DR clusters

Run the following on both the primary and DR clusters:

kubectl --kubeconfig <primary-kubeconfig> get pods \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery-workload-control-plane

kubectl --kubeconfig <dr-kubeconfig> get pods \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery-workload-control-plane

The drbd-node-agent DaemonSet must have one pod per worker node in Running state.

Step 3 — Collect a diagnostic bundle if needed

To collect logs, CRD state, and configuration from all clusters in a single bundle:

oc adm must-gather \
  --image=<tsr-gather-image> \
  -- /usr/bin/tsr-gather \
  --kubeconfig-quorum <quorum-kubeconfig> \
  --kubeconfig-primary <primary-kubeconfig> \
  --kubeconfig-dr <dr-kubeconfig>

The bundle is written to the current directory and contains no secrets or credentials.

Recovery operation behavior is governed by fields on the FailoverRequest and TestFailover custom resources. Understanding these fields helps you determine whether a failure was caused by a timeout, a misconfigured target, or an underlying infrastructure problem.

FailoverRequest spec fields relevant to failure diagnosis

TestFailover spec fields relevant to failure diagnosis

DRBDReplicationPolicy fields that affect recovery

ProtectionGroup spec fields that affect failover state

Reading FailoverRequest status after a failure

The FailoverRequest CR is the primary diagnostic artifact for failed planned or unplanned failovers. Its status subresource records the current phase and any error messages from the failover-controller.

Check overall status:

kubectl --kubeconfig <quorum-kubeconfig> get failoverrequest \
  -n dr-<name>

Example output showing a failed operation:

NAME                    TYPE       STATUS   PHASE        AGE
prod-failover-2025      Planned    Failed   DrainFailed   8m

Inspect the full status block:

kubectl --kubeconfig <quorum-kubeconfig> get failoverrequest prod-failover-2025 \
  -n dr-<name> \
  -o yaml

Look for the status.conditions array and status.message fields. The failover-controller populates these with the step that failed, the cluster it was operating on, and the underlying error.

Examine controller events on the FailoverRequest:

kubectl --kubeconfig <quorum-kubeconfig> describe failoverrequest prod-failover-2025 \
  -n dr-<name>

The Events section at the bottom shows the timeline of controller actions, which is useful for understanding how far the failover progressed before failing.

Reading TestFailover status after a failure

The TestFailover CR tracks the test workflow through phases: CreatingSnapshots, VerifyingData, Succeeded, Failed, and CleaningUp.

Check test failover status:

kubectl --kubeconfig <quorum-kubeconfig> get testfailover \
  -n dr-<name>

Inspect details:

kubectl --kubeconfig <quorum-kubeconfig> describe testfailover <test-name> \
  -n dr-<name>

If cleanupPolicy: Retain was set, the isolated test namespace and its VMs and PVCs remain available for inspection after failure:

kubectl --kubeconfig <dr-kubeconfig> get ns | grep testfailover
kubectl --kubeconfig <dr-kubeconfig> get vm,vmi,pvc -n <test-namespace>

Checking replication health before or after a failed operation

Many failover failures originate from a degraded replication state. Check the ReplicationGroupStatus CR on the quorum cluster:

kubectl --kubeconfig <quorum-kubeconfig> get replicationgroupstatus \
  -n dr-<name>

A Critical or Degraded status means the ProtectionGroup was not in a Consistent replication state when the failover was attempted. Check per-volume detail:

kubectl --kubeconfig <quorum-kubeconfig> get replicationgroupstatus <name> \
  -n dr-<name> \
  -o yaml

Also check for recent RPO violations that preceded the failure:

kubectl --kubeconfig <quorum-kubeconfig> get rpoevent \
  -n dr-<name> \
  --sort-by=.spec.timestamp

Checking DRBDResource state on both clusters

For DRBD Operator deployments, the DRBDResource CR on each cluster shows per-volume sync progress and connection state:

kubectl --kubeconfig <primary-kubeconfig> get drbdresource \
  -n <vm-namespace>

kubectl --kubeconfig <dr-kubeconfig> get drbdresource \
  -n <vm-namespace>

A resource stuck in Connecting or showing a stale sync percentage is a sign that the DRBD replication link between worker nodes has been interrupted. Verify that TCP ports 7000–7999 are open between primary and DR worker nodes.

Understanding rollback behavior

Trilio Site Recovery does not perform automatic rollback of a partially completed failover. The failover-controller is designed to be re-entrant: if you delete and recreate a FailoverRequest CR with the same parameters, the controller will re-evaluate the current actual state of VMs and DRBD resources on both clusters and continue from where it can safely proceed.

For a planned failover that failed during the drain phase (VMs not fully stopped on source), the VMs on the primary cluster may still be running. You can safely attempt the failover again after resolving the underlying cause—the controller checks actual VM state before taking action.

For an unplanned failover that failed after DRBD volume promotion on the DR cluster, the DR-side volumes may already be in Primary role. Do not attempt to start VMs on the primary cluster in this state. Verify DRBD resource roles before any manual intervention:

kubectl --kubeconfig <dr-kubeconfig> get drbdresource \
  -n <vm-namespace> \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.role}{"\n"}{end}'

Example 1 — Planned failover stuck in DrainFailed

A planned failover was created but reported DrainFailed after 8 minutes.

kubectl --kubeconfig <quorum-kubeconfig> get failoverrequest prod-pg-failover \
  -n dr-production \
  -o yaml

Expected output (abbreviated):

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: prod-pg-failover
  namespace: dr-production
spec:
  failoverType: Planned
  drainTimeoutSeconds: 300
  protectionGroupRef:
    name: production-pg
    namespace: vm-workloads
  targetCluster: dr-cluster
status:
  phase: DrainFailed
  message: "VM prod-vm-3 did not stop within 300 seconds on primary-cluster"
  conditions:
    - type: DrainComplete
      status: "False"
      reason: DrainTimeout
      message: "VM prod-vm-3 failed to stop within drainTimeoutSeconds=300"

The failover-controller stopped all VMs except prod-vm-3, which failed to stop within the configured drain timeout. Check the VM's state on the primary cluster:

kubectl --kubeconfig <primary-kubeconfig> get vm prod-vm-3 \
  -n vm-workloads \
  -o jsonpath='{.spec.running} {.status.printableStatus}'

If the VM is stuck in a bad state, investigate the VirtualMachineInstance:

kubectl --kubeconfig <primary-kubeconfig> describe vmi prod-vm-3 \
  -n vm-workloads

After resolving the VM issue, delete the failed FailoverRequest and create a new one. If the VM cannot be stopped gracefully, consider switching to failoverType: Unplanned:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: prod-pg-failover-retry
  namespace: dr-production
spec:
  failoverType: Unplanned
  protectionGroupRef:
    name: production-pg
    namespace: vm-workloads
  targetCluster: dr-cluster

kubectl --kubeconfig <quorum-kubeconfig> apply -f failoverrequest-retry.yaml

Example 2 — TestFailover stuck in Failed with snapshots retained

A TestFailover was configured with cleanupPolicy: Retain and failed during VerifyingData.

kubectl --kubeconfig <quorum-kubeconfig> describe testfailover prod-pg-test-1 \
  -n dr-production

Expected output (abbreviated):

Name:         prod-pg-test-1
Namespace:    dr-production
Status:
  Phase:    Failed
  Message:  Verification check failed: VM prod-vm-1 did not reach Running state within batchBootTimeoutSeconds=600
Events:
  Type     Reason              Age   Message
  ----     ------              ---   -------
  Normal   CreatingSnapshots   12m   Created volume snapshots for 3 PVCs
  Normal   ProvisioningPVCs    11m   Provisioned 3 test PVCs from snapshots
  Normal   StartingTestVMs     10m   Started test VMs in namespace testfailover-prod-pg-test-1
  Warning  VerificationFailed   2m   VM prod-vm-1 did not reach Running state

Because cleanupPolicy: Retain was set, the test namespace still exists. Inspect the test VM:

kubectl --kubeconfig <dr-kubeconfig> get vm,vmi,pvc \
  -n testfailover-prod-pg-test-1

kubectl --kubeconfig <dr-kubeconfig> describe vmi prod-vm-1 \
  -n testfailover-prod-pg-test-1

Common causes include a missing image, insufficient resource quota on the DR cluster, or a node selector that cannot be satisfied. After investigation, delete the TestFailover CR to trigger cleanup:

kubectl --kubeconfig <quorum-kubeconfig> delete testfailover prod-pg-test-1 \
  -n dr-production

The test-failover-controller will move to CleaningUp and remove the isolated namespace and all test resources.

Example 3 — Replication degraded before failover attempt

A FailoverRequest failed immediately with a replication-related message. Check the ReplicationGroupStatus:

kubectl --kubeconfig <quorum-kubeconfig> get replicationgroupstatus \
  -n dr-production \
  -o yaml

Expected output (abbreviated):

status:
  health: Degraded
  lastSyncTimestamp: "2025-11-01T03:14:00Z"
  volumes:
    - pvcName: prod-vm-1-disk
      syncState: Syncing
      syncPercent: 67
    - pvcName: prod-vm-2-disk
      syncState: Connected
      syncPercent: 100

prod-vm-1-disk is mid-resync. Check for accompanying RPO events:

kubectl --kubeconfig <quorum-kubeconfig> get rpoevent \
  -n dr-production \
  --sort-by=.spec.timestamp \
  -o custom-columns=NAME:.metadata.name,SEVERITY:.spec.eventType,LAG:.spec.rpoAtEvent,TIME:.spec.timestamp

Wait for the resync to complete (the syncPercent reaches 100 and health returns to Healthy) before retrying the failover. Monitor progress:

watch kubectl --kubeconfig <quorum-kubeconfig> get replicationgroupstatus \
  -n dr-production

Example 4 — Unplanned failover with partially promoted DRBD resources

An unplanned failover failed after DRBD volume promotion began. Check resource roles on the DR cluster:

kubectl --kubeconfig <dr-kubeconfig> get drbdresource \
  -n vm-workloads \
  -o custom-columns=NAME:.metadata.name,ROLE:.status.role,SYNC:.status.syncPercent

Expected output showing partial promotion:

NAME                          ROLE       SYNC
prod-vm-1-disk-dr             Primary    100
prod-vm-2-disk-dr             Secondary  100
prod-vm-3-disk-dr             Primary    100

prod-vm-2-disk-dr was not promoted. Do not start any VMs on the primary cluster. Delete the failed FailoverRequest and create a new one:

kubectl --kubeconfig <quorum-kubeconfig> delete failoverrequest prod-pg-failover-failed \
  -n dr-production

kubectl --kubeconfig <quorum-kubeconfig> apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: prod-pg-failover-retry
  namespace: dr-production
spec:
  failoverType: Unplanned
  protectionGroupRef:
    name: production-pg
    namespace: vm-workloads
  targetCluster: dr-cluster
EOF

The failover-controller will re-evaluate resource roles and complete the promotion for prod-vm-2-disk-dr.

Use the following reference to match the symptom you observe to its likely cause and resolution. Each issue uses a consistent format: Symptom, Likely cause, Fix.

FailoverRequest remains in Pending indefinitely

Symptom: You applied a FailoverRequest CR and it shows Pending phase for more than two minutes with no events.

Likely cause: The failover-controller pod on the quorum cluster is not running, or it cannot reach the API server of the primary or DR cluster.

Fix:

1. Verify the quorum control plane pods are running:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> get pods \
     -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane
   

2. Check controller logs for API connectivity errors:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> logs \
     -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane \
     --all-containers --tail=100
   

3. Verify TCP port 6443 is reachable from the quorum cluster to both primary and DR API servers.
4. Confirm the ProtectionZone CR references valid cluster endpoints:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> get protectionzone \
     -n dr-<name> -o yaml
   

FailoverRequest fails with DrainFailed

Symptom: A planned FailoverRequest reaches DrainFailed phase. The status message names a specific VM.

Likely cause: The named VM did not stop within spec.drainTimeoutSeconds. This can happen if the VM's guest OS is hung, if the VMI pod is stuck in Terminating, or if the VM is subject to a PodDisruptionBudget that prevents eviction.

Fix:

1. Check the VM and VMI state on the primary cluster:
   Copy

   kubectl --kubeconfig <primary-kubeconfig> get vm,vmi \
     -n <vm-namespace>
   

2. Describe the VMI to find scheduling or pod-level issues:
   Copy

   kubectl --kubeconfig <primary-kubeconfig> describe vmi <vm-name> \
     -n <vm-namespace>
   

3. If the VM is genuinely stuck, increase drainTimeoutSeconds in a new FailoverRequest, or switch failoverType to Unplanned if data loss is acceptable.
4. Delete the failed FailoverRequest before creating a new one.

FailoverRequest fails with VolumePromotionFailed

Symptom: An unplanned FailoverRequest fails during DRBD volume promotion on the DR cluster.

Likely cause: The drbd-node-agent DaemonSet on the DR cluster cannot promote one or more volumes because the DRBD connection is still open from the primary side (split-brain prevention), or the drbd-node-agent pod on the relevant worker node is not running.

Fix:

1. Check drbd-node-agent pods on the DR cluster:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get pods \
     -n dr-<name> \
     -l app.kubernetes.io/component=drbd-node-agent
   

2. Check DRBDResource status on the DR cluster:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get drbdresource \
     -n <vm-namespace> -o yaml
   

3. If the primary cluster is genuinely unreachable and DRBD is blocking promotion due to a stale connection, verify network isolation between clusters. For unplanned failovers where the primary is down, the controller is designed to force-promote; if it is not doing so, check the failover-controller logs for the specific rejection reason.
4. Do not manually start VMs on the primary cluster while the unplanned failover is in progress.

TestFailover stuck in CreatingSnapshots

Symptom: A TestFailover CR stays in CreatingSnapshots phase for more than five minutes.

Likely cause: The volume snapshot class is not configured on the DR cluster, or the CSI driver does not support snapshots for the storage class backing the protected PVCs.

Fix:

1. Check VolumeSnapshot resources on the DR cluster in the DR namespace:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get volumesnapshot \
     -n dr-<name>
   

2. Check VolumeSnapshotContent resources for errors:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get volumesnapshotcontent
   

3. Verify a VolumeSnapshotClass is configured and set as default on the DR cluster:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get volumesnapshotclass
   

4. Check test-failover-controller logs for specific snapshot errors:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> logs \
     -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane \
     --all-containers | grep -i testfailover
   

TestFailover fails in VerifyingData — test VMs do not start

Symptom: The TestFailover reaches VerifyingData but fails because test VMs in the isolated namespace do not reach Running state.

Likely cause: Insufficient compute resources or node selectors on the DR cluster that cannot be satisfied in the isolated test namespace, or a missing image pull secret.

Fix:

1. If cleanupPolicy: Retain was set, inspect the test namespace directly:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get vm,vmi,pvc \
     -n testfailover-<test-name>
   kubectl --kubeconfig <dr-kubeconfig> describe vmi <vm-name> \
     -n testfailover-<test-name>
   

2. Look for events indicating resource pressure or unschedulable pods:
   Copy

   kubectl --kubeconfig <dr-kubeconfig> get events \
     -n testfailover-<test-name> \
     --sort-by=.lastTimestamp
   

3. Increase spec.batchBootTimeoutSeconds if VMs are starting but slowly.
4. After investigation, delete the TestFailover CR to trigger cleanup:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> delete testfailover <test-name> \
     -n dr-<name>
   

ProtectionGroup shows Degraded replication health after failover

Symptom: After a completed failover, the ProtectionGroup status shows Degraded rather than Consistent, and ReplicationGroupStatus shows one or more volumes not yet synchronized.

Likely cause: After a failover, DRBD replication runs in reverse (DR cluster is now primary). Replication may not have been re-established because the original primary cluster is still unreachable, or because the DRBDReplicationPolicy was not updated to reflect the new primary.

Fix:

1. Check the ProtectionGroup status on the quorum cluster:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> get protectiongroup \
     -n <vm-namespace> -o yaml
   

2. Check the ReplicationGroupStatus for per-volume detail:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> get replicationgroupstatus \
     -n dr-<name> -o yaml
   

3. Verify that the drbd-node-agent on both clusters reports a connected state. If the original primary cluster is still offline, replication will remain Degraded until connectivity is restored—this is expected behavior.
4. Once the original primary is restored and network connectivity between clusters is re-established, the DRBD replication link should reconnect automatically and begin resync. Monitor progress:
   Copy

   watch kubectl --kubeconfig <quorum-kubeconfig> get replicationgroupstatus \
     -n dr-<name>
   

FailoverRequest fails immediately with InvalidTarget

Symptom: A FailoverRequest moves to Failed within seconds with a message referencing an invalid or unknown target cluster.

Likely cause: The value of spec.targetCluster does not match any cluster registered in the ProtectionZone CR on the quorum cluster.

Fix:

1. Check the ProtectionZone to confirm valid cluster names:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> get protectionzone \
     -n dr-<name> \
     -o jsonpath='{.spec.clusters[*].name}'
   

2. Correct the spec.targetCluster value in your FailoverRequest to exactly match one of the registered cluster names.
3. Delete the failed FailoverRequest and apply the corrected version.

Controller logs show repeated reconciliation errors but no CR status update

Symptom: The failover-controller or test-failover-controller logs show repeated errors, but the FailoverRequest or TestFailover CR status does not update.

Likely cause: The controller is losing its leader election lease, or it lacks RBAC permission to update the status subresource of the CR.

Fix:

1. Check for leader election errors in the controller logs:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> logs \
     -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane \
     --all-containers | grep -i 'leader\|election\|lease'
   

2. Verify RBAC for the controller's service account:
   Copy

   kubectl --kubeconfig <quorum-kubeconfig> auth can-i update \
     failoverrequests/status \
     --as=system:serviceaccount:dr-<name>:<controller-service-account> \
     -n dr-<name>
   

3. If RBAC is missing, re-run the helm upgrade for the site-recovery-quorum-control-plane chart with the current values to restore correct RBAC bindings:
   Copy

   helm upgrade site-recovery-quorum-control-plane \
     trilio/site-recovery-quorum-control-plane \
     -n dr-<name> \
     -f values.yaml