Recovery Operation Failures

Example 1: FailoverRequest stuck in InProgress

Check the phase and per-VM statuses:

oc get failoverrequest my-failover -n dr-prod -o yaml

Expected output when stuck:

status:
  phase: InProgress
  vmStatuses:
    - vmName: prod-vm-1
      status: Stopped
    - vmName: prod-vm-2
      status: Starting

Check quorum control plane logs to find why prod-vm-2 is not starting:

oc logs -n dr-prod \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  --since=30m | grep prod-vm-2

Check the DRBD resource for prod-vm-2 on the DR cluster to confirm volume promotion:

oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-dr | grep prod-vm-2
oc get drbdresource prod-vm-2 -n dr-prod --kubeconfig ~/.kube/config-dr -o yaml

If the DRBDResource shows role: Primary on the DR side and the VM is not starting, check node scheduling and DRBD node agent logs on the DR cluster:

oc logs -n dr-prod \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --kubeconfig ~/.kube/config-dr --since=30m

Example 2: ProtectionRequest stuck in CreatingDRBD

oc get protectionrequest vm-protection -n dr-prod -o jsonpath='{.status.phase}'

CreatingDRBD

This phase means the protection-controller has submitted the DRBDResource pair but it has not reached a replicated state. Inspect the DRBDResource:

# On primary cluster
oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-primary

# Describe to see connection state
oc describe drbdresource <resource-name> -n dr-prod --kubeconfig ~/.kube/config-primary

Check that TCP port 7000–8000 is open between the worker nodes of both clusters. A connection timeout in the DRBDResource events is a strong indicator of a firewall or routing issue.

Example 3: TestFailover stuck in CreatingVMs — inspect with Manual cleanup policy

If a TestFailover repeatedly fails at CreatingVMs, redeploy it with cleanupPolicy: Manual so the test resources remain available for inspection after failure:

cat <<EOF | oc apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: pg-test-debug
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: production-pg
  cleanupPolicy: Manual
  batchBootTimeoutSeconds: 300
EOF

After the TestFailover enters Failed, retrieve the test namespace and inspect resources there:

TEST_NS=$(oc get testfailover pg-test-debug -n dr-prod \
  -o jsonpath='{.status.testNamespace}')
echo "Test namespace: $TEST_NS"

# Inspect VMs in the isolated namespace
oc get vm,vmi,pvc -n "$TEST_NS" --kubeconfig ~/.kube/config-dr

# Check events
oc get events -n "$TEST_NS" --kubeconfig ~/.kube/config-dr --sort-by='.lastTimestamp'

Example 4: FailoverRequest in Failed with DRBD volumes already promoted

Confirm the DRBDResource roles on both sides:

# DR cluster — expect Primary after promotion
oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-dr \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.role}{"\n"}{end}'

# Primary cluster — expect Secondary after promotion
oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-primary \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.role}{"\n"}{end}'

If DR shows Primary and primary shows Secondary, volumes are promoted. Do not start VMs on the primary. Investigate why VMs did not start on the DR side by checking events and logs, then determine whether to retry the FailoverRequest or create a new one with adjusted batchBootTimeoutSeconds.

Example 5: Collecting a full diagnostic bundle

oc adm must-gather \
  --image=<tsr-gather-image> \
  -- /usr/bin/tsr-gather

The bundle captures logs, CRD state (FailoverRequest, TestFailover, ProtectionRequest, DRBDResource, ProtectionGroup, RPOEvent, ReplicationGroupStatus), and configuration from all clusters. Attach this bundle to any support case.

Use the following format to identify and resolve common failures. For each issue, collect quorum control plane logs and a tsr-gather bundle before escalating.

FailoverRequest stuck in InProgress indefinitely

Symptom: oc get failoverrequest shows Phase: InProgress and does not progress after 15+ minutes.

Likely causes:

• A VM on the target cluster failed to start within batchBootTimeoutSeconds.
• DRBD volume promotion did not complete because the replication link is still up and the source has not released the Primary role.
• The quorum control plane pod restarted mid-operation.

Fix:

1. Check status.vmStatuses for which VMs are stuck: oc get failoverrequest <name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}' | jq .
2. Check quorum controller logs: oc logs -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane --since=30m
3. Check the DRBDResource role on both clusters. If the source still shows Primary, the failover controller cannot promote the DR side safely—investigate DRBD connectivity.
4. If the primary is confirmed unreachable and the link is down, delete the current FailoverRequest only after verifying no VMs are running on the primary, then create a new FailoverRequest with spec.failoverType: unplanned.

Caution: Do not delete a stuck FailoverRequest while VMs are in an unknown state.

FailoverRequest enters Failed immediately on creation

Symptom: FailoverRequest transitions from Pending to Failed within seconds.

Likely causes:

• spec.targetCluster is not cluster1 or cluster2 (the only valid enum values).
• spec.protectionGroupRef references a ProtectionGroup that does not exist or is in Degraded state.
• The admission webhook rejected the resource.

Fix:

1. Validate the manifest: oc get failoverrequest <name> -n dr-<zone-name> -o yaml and confirm spec.targetCluster is exactly cluster1 or cluster2.
2. Confirm the ProtectionGroup exists and is Consistent: oc get protectiongroup <pg-name> -n dr-<zone-name>.
3. Check webhook logs and API server events: oc get events -n dr-<zone-name> --sort-by='.lastTimestamp'.

ProtectionRequest stuck in CreatingDRBD

Symptom: ProtectionRequest stays in CreatingDRBD for more than 10 minutes.

Likely causes:

• TCP ports 7000–8000 are blocked between primary and DR worker nodes, preventing DRBD from establishing the replication connection.
• The DRBD node agent DaemonSet is not running on one or more worker nodes.
• LVM thin pool is exhausted or misconfigured on one cluster.

Fix:

1. Check DRBD node agent pod status on both clusters: oc get pods -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane --kubeconfig ~/.kube/config-primary
2. Inspect DRBDResource events on both clusters: oc describe drbdresource <name> -n dr-<zone-name>
3. Verify network connectivity: from a worker node on the primary cluster, confirm TCP ports 7000–8000 reach worker nodes on the DR cluster.
4. Check LVM thin pool utilisation on affected nodes.

ProtectionRequest stuck in Activating

Symptom: ProtectionRequest is in Activating and does not reach Protected.

Likely causes:

• The VM failed to restart after its PVCs were switched to DRBD-backed frontend PVCs.
• A webhook rejected the updated VM spec.

Fix:

1. Check the VM's status on the primary cluster: oc get vm <vm-name> -n <vm-namespace> --kubeconfig ~/.kube/config-primary
2. Check VM events: oc describe vm <vm-name> -n <vm-namespace> --kubeconfig ~/.kube/config-primary
3. Review the protection-controller logs: oc logs -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane --since=30m | grep <vm-name>

TestFailover stuck in CreatingSnapshots

Symptom: TestFailover stays in CreatingSnapshots for more than 10 minutes.

Likely causes:

• The underlying storage class does not support volume snapshots, or the VolumeSnapshotClass is missing.
• CDI (Containerized Data Importer) is not installed or is not healthy on the DR cluster.

Fix:

1. Read the test namespace: oc get testfailover <name> -n dr-<zone-name> -o jsonpath='{.status.testNamespace}'
2. Check VolumeSnapshot objects in the test namespace on the DR cluster.
3. Verify the VolumeSnapshotClass exists: oc get volumesnapshotclass --kubeconfig ~/.kube/config-dr
4. Check CDI pods: oc get pods -n cdi --kubeconfig ~/.kube/config-dr

TestFailover enters Failed at StartingVMs

Symptom: TestFailover reaches StartingVMs then transitions to Failed.

Likely causes:

• Test VMs exceeded batchBootTimeoutSeconds before reaching Running state.
• Insufficient resources on DR cluster nodes for the test VM configuration.

Fix:

1. Set cleanupPolicy: Manual and re-run the TestFailover to preserve test resources.
2. Read the test namespace: oc get testfailover <name> -n dr-<zone-name> -o jsonpath='{.status.testNamespace}'
3. Inspect VM and VMI status in the test namespace: oc get vm,vmi -n <test-namespace> --kubeconfig ~/.kube/config-dr
4. Check node resource pressure: oc describe nodes --kubeconfig ~/.kube/config-dr | grep -A5 'Allocated resources'
5. Increase batchBootTimeoutSeconds in the TestFailover spec and reapply after cleaning up the previous attempt.

ProtectionGroup shows Degraded replication health after a failover

Symptom: After a failover completes, oc get protectiongroup shows HEALTH: Degraded.

Likely causes:

• The replication link between primary and DR clusters has not re-established after failover.
• DRBD resync is in progress (expected transiently after a failback).
• An RPOEvent has been raised indicating lag exceeds the configured RPO threshold.

Fix:

1. Check ReplicationGroupStatus: oc get replicationgroupstatus -n dr-<zone-name> and describe the relevant object for per-volume detail.
2. Check RPOEvents: oc get rpoevent -n dr-<zone-name> to see if lag violations are recorded.
3. Inspect DRBDResource sync progress on both clusters: oc get drbdresource -n dr-<zone-name> -o wide --kubeconfig ~/.kube/config-primary
4. Confirm DRBD node agents are running on all worker nodes of both clusters.
5. If the link is down, investigate network connectivity on TCP ports 7000–8000 between worker nodes.