Recovery Operation Failures

This page helps you diagnose and recover from failed or partially completed disaster recovery operations in Trilio Site Recovery. Recovery operations — planned failovers, unplanned failovers, and test failovers — are driven by FailoverRequest and TestFailover custom resources on the quorum cluster. When these operations fail or stall mid-execution, the controller is re-entrant rather than rolling back automatically, which means your intervention strategy depends on exactly which phase the operation reached and whether VMs are in an unknown state. This guide covers how to read operation status, identify the root cause, and safely resolve stuck or failed operations without creating data inconsistencies.

Before using this guide, ensure you have:

• Access to the quorum cluster with oc CLI and a kubeconfig that reaches all three clusters (quorum, primary, DR)
• The oc CLI installed — certain operations such as oc adm must-gather require it; kubectl-only operation is not a supported mode
• Sufficient RBAC to read and patch FailoverRequest, TestFailover, ProtectionGroup, ReplicationGroupStatus, and RPOEvent resources on the quorum cluster, and DRBDResource on both workload clusters
• Familiarity with the phase lifecycle of each CRD:
  • FailoverRequest phases: Initializing → Validating → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → Completed or Failed
  • TestFailover phases: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded or Failed → CleaningUp → Cleaned
  • ProtectionRequest phases: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected or Failed
• Network access to TCP ports 7000–8000 between primary and DR worker nodes (one port per replicated VM volume); a firewall gap here causes silent replication failures that surface as degraded failover preconditions

No additional installation is required to use this guide — all diagnostic operations use oc and kubectl against the existing clusters. To collect a full diagnostic bundle, use tsr-gather via oc adm must-gather:

# Run against the quorum cluster context.
# This collects logs, CRD state, and configuration from all clusters.
oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --context=<quorum-cluster-context>

The gather binary inside the image runs automatically as the ENTRYPOINT; no -- <command> override is needed. The resulting bundle contains logs from all three clusters and is suitable for sharing with Trilio support.

The following CRD fields govern how operations behave and what state they report. Understanding these helps you distinguish a transient failure from a permanent one.

FailoverRequest

Per-VM progress is reported in status.vmStatuses, not a top-level message or failedVMs field (those fields do not exist). Each entry reports name, phase, sourceStatus, targetStatus, and error.

status.state tracks the operation lifecycle: Pending → InProgress → Completed or Failed or Cancelled.

status.phase tracks the internal step: Initializing → Validating → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → Completed or Failed.

TestFailover

The test namespace is reported in status.testNamespace (generated as site-recovery-<protection-group>-<random>). Do not attempt to construct or guess this value — read it directly:

oc get tfo <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}'

ProtectionGroup replication state

ReplicationGroupStatus (short name rgs) aggregates replication health. Its name is always <protection-group-name>-replication, auto-created by the replication monitor. Filter by label rather than by field selector (field selectors on custom resource spec.* / status.* fields are not supported):

oc get rgs -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

RPOEvent (short name rpoe) records individual replication lag violations. These are immutable audit records written only by the replication-monitor service account.

Checking the status of a failed failover

All failover CRs live on the quorum cluster in the dr-<zone-name> namespace. To inspect a failed FailoverRequest:

# List all FailoverRequests in the zone namespace.
oc get fr -n dr-<zone-name>

Expected columns: NAME, Protection Group, Target, State, Phase, Age.

# Describe a specific FailoverRequest for full per-VM detail.
oc describe fr <name> -n dr-<zone-name>

Per-VM status is in status.vmStatuses. Look for entries where phase is Failed and read the error field for the specific failure reason.

# Read per-VM statuses as JSON for scripting.
oc get fr <name> -n dr-<zone-name> \
  -o jsonpath='{.status.vmStatuses}'

Checking the status of a failed test failover

oc get tfo -n dr-<zone-name>

Expected columns: Protection Group, Phase, Passed, Age.

# Get the controller-generated test namespace.
TEST_NS=$(oc get tfo <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')
echo "Test namespace: $TEST_NS"

# Inspect test VMs in the isolated namespace.
oc get vm -n "$TEST_NS"
oc get vmi -n "$TEST_NS"

Checking replication health before a failover

Before executing a failover, verify that replication is Healthy and Consistent:

# Check the ReplicationGroupStatus for the target ProtectionGroup.
oc get rgs -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

# Full detail including per-volume sync state.
oc describe rgs <pg-name>-replication -n dr-<zone-name>

Key fields to check:

• status.aggregated.connectionState: should be Connected
• status.rpoStatus: should be Met
• status.aggregated.outOfSyncBytes: should be 0 for a planned failover with zero data loss

# Check DRBDResource state on each workload cluster.
oc get dres -A --context=<primary-context>
oc get dres -A --context=<dr-context>

DRBDResource CRs live on workload clusters in the protected VM's namespace, not on the quorum cluster.

Checking the ProtectionGroup state

oc get pg -n dr-<zone-name>
oc describe pg <name> -n dr-<zone-name>

The status.state field reports Protected, Syncing, Degraded, or Failed. A Degraded group means replication is not fully healthy — investigate DRBDResource state on the workload clusters before proceeding with a failover.

Checking the DRBD node agent on a specific node

The node agent runs as a DaemonSet named <release>-site-recovery-workload-control-plane-agent on each workload cluster. To get logs for a specific node:

# Step 1: Find the agent pod on the target node.
AGENT_POD=$(oc get pods \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=<node-name> \
  -o jsonpath='{.items[0].metadata.name}' \
  --context=<workload-context>)

# Step 2: Stream logs from that pod.
oc logs "$AGENT_POD" \
  -n trilio-site-recovery-system \
  --context=<workload-context>

The node agent applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its own node until DRBD converges. If you see this taint persisting, check the agent log for DRBD connection errors.

Checking quorum control plane logs

The quorum control plane runs as a single manager container (pod output is 1/1) in the dr-<zone-name> namespace:

oc get pods \
  -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery

oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery \
  --all-containers

Do not use -l app.kubernetes.io/part-of=site-recovery-quorum-control-plane — that label does not exist and returns no pods.

Example 1: FailoverRequest stuck in InProgress

Check the phase and per-VM statuses:

oc get failoverrequest my-failover -n dr-prod -o yaml

Expected output when stuck:

status:
  phase: InProgress
  vmStatuses:
    - vmName: prod-vm-1
      status: Stopped
    - vmName: prod-vm-2
      status: Starting

Check quorum control plane logs to find why prod-vm-2 is not starting:

oc logs -n dr-prod \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  --since=30m | grep prod-vm-2

Check the DRBD resource for prod-vm-2 on the DR cluster to confirm volume promotion:

oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-dr | grep prod-vm-2
oc get drbdresource prod-vm-2 -n dr-prod --kubeconfig ~/.kube/config-dr -o yaml

If the DRBDResource shows role: Primary on the DR side and the VM is not starting, check node scheduling and DRBD node agent logs on the DR cluster:

oc logs -n dr-prod \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --kubeconfig ~/.kube/config-dr --since=30m

Example 2: ProtectionRequest stuck in CreatingDRBD

oc get protectionrequest vm-protection -n dr-prod -o jsonpath='{.status.phase}'

CreatingDRBD

This phase means the protection-controller has submitted the DRBDResource pair but it has not reached a replicated state. Inspect the DRBDResource:

# On primary cluster
oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-primary

# Describe to see connection state
oc describe drbdresource <resource-name> -n dr-prod --kubeconfig ~/.kube/config-primary

Check that TCP port 7000–8000 is open between the worker nodes of both clusters. A connection timeout in the DRBDResource events is a strong indicator of a firewall or routing issue.

Example 3: TestFailover stuck in CreatingVMs — inspect with Manual cleanup policy

If a TestFailover repeatedly fails at CreatingVMs, redeploy it with cleanupPolicy: Manual so the test resources remain available for inspection after failure:

cat <<EOF | oc apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: pg-test-debug
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: production-pg
  cleanupPolicy: Manual
  batchBootTimeoutSeconds: 300
EOF

After the TestFailover enters Failed, retrieve the test namespace and inspect resources there:

TEST_NS=$(oc get testfailover pg-test-debug -n dr-prod \
  -o jsonpath='{.status.testNamespace}')
echo "Test namespace: $TEST_NS"

# Inspect VMs in the isolated namespace
oc get vm,vmi,pvc -n "$TEST_NS" --kubeconfig ~/.kube/config-dr

# Check events
oc get events -n "$TEST_NS" --kubeconfig ~/.kube/config-dr --sort-by='.lastTimestamp'

Example 4: FailoverRequest in Failed with DRBD volumes already promoted

Confirm the DRBDResource roles on both sides:

# DR cluster — expect Primary after promotion
oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-dr \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.role}{"\n"}{end}'

# Primary cluster — expect Secondary after promotion
oc get drbdresource -n dr-prod --kubeconfig ~/.kube/config-primary \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.role}{"\n"}{end}'

If DR shows Primary and primary shows Secondary, volumes are promoted. Do not start VMs on the primary. Investigate why VMs did not start on the DR side by checking events and logs, then determine whether to retry the FailoverRequest or create a new one with adjusted batchBootTimeoutSeconds.

Example 5: Collecting a full diagnostic bundle

oc adm must-gather \
  --image=<tsr-gather-image> \
  -- /usr/bin/tsr-gather

The bundle captures logs, CRD state (FailoverRequest, TestFailover, ProtectionRequest, DRBDResource, ProtectionGroup, RPOEvent, ReplicationGroupStatus), and configuration from all clusters. Attach this bundle to any support case.

Use the following format to identify and resolve common failures. For each issue, collect quorum control plane logs and a tsr-gather bundle before escalating.

FailoverRequest stuck in InProgress indefinitely

Symptom: oc get failoverrequest shows Phase: InProgress and does not progress after 15+ minutes.

Likely causes:

• A VM on the target cluster failed to start within batchBootTimeoutSeconds.
• DRBD volume promotion did not complete because the replication link is still up and the source has not released the Primary role.
• The quorum control plane pod restarted mid-operation.

Fix:

1. Check status.vmStatuses for which VMs are stuck: oc get failoverrequest <name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}' | jq .
2. Check quorum controller logs: oc logs -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane --since=30m
3. Check the DRBDResource role on both clusters. If the source still shows Primary, the failover controller cannot promote the DR side safely—investigate DRBD connectivity.
4. If the primary is confirmed unreachable and the link is down, delete the current FailoverRequest only after verifying no VMs are running on the primary, then create a new FailoverRequest with spec.failoverType: unplanned.

Caution: Do not delete a stuck FailoverRequest while VMs are in an unknown state.

FailoverRequest enters Failed immediately on creation

Symptom: FailoverRequest transitions from Pending to Failed within seconds.

Likely causes:

• spec.targetCluster is not cluster1 or cluster2 (the only valid enum values).
• spec.protectionGroupRef references a ProtectionGroup that does not exist or is in Degraded state.
• The admission webhook rejected the resource.

Fix:

1. Validate the manifest: oc get failoverrequest <name> -n dr-<zone-name> -o yaml and confirm spec.targetCluster is exactly cluster1 or cluster2.
2. Confirm the ProtectionGroup exists and is Consistent: oc get protectiongroup <pg-name> -n dr-<zone-name>.
3. Check webhook logs and API server events: oc get events -n dr-<zone-name> --sort-by='.lastTimestamp'.

ProtectionRequest stuck in CreatingDRBD

Symptom: ProtectionRequest stays in CreatingDRBD for more than 10 minutes.

Likely causes:

• TCP ports 7000–8000 are blocked between primary and DR worker nodes, preventing DRBD from establishing the replication connection.
• The DRBD node agent DaemonSet is not running on one or more worker nodes.
• LVM thin pool is exhausted or misconfigured on one cluster.

Fix:

1. Check DRBD node agent pod status on both clusters: oc get pods -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane --kubeconfig ~/.kube/config-primary
2. Inspect DRBDResource events on both clusters: oc describe drbdresource <name> -n dr-<zone-name>
3. Verify network connectivity: from a worker node on the primary cluster, confirm TCP ports 7000–8000 reach worker nodes on the DR cluster.
4. Check LVM thin pool utilisation on affected nodes.

ProtectionRequest stuck in Activating

Symptom: ProtectionRequest is in Activating and does not reach Protected.

Likely causes:

• The VM failed to restart after its PVCs were switched to DRBD-backed frontend PVCs.
• A webhook rejected the updated VM spec.

Fix:

1. Check the VM's status on the primary cluster: oc get vm <vm-name> -n <vm-namespace> --kubeconfig ~/.kube/config-primary
2. Check VM events: oc describe vm <vm-name> -n <vm-namespace> --kubeconfig ~/.kube/config-primary
3. Review the protection-controller logs: oc logs -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane --since=30m | grep <vm-name>

TestFailover stuck in CreatingSnapshots

Symptom: TestFailover stays in CreatingSnapshots for more than 10 minutes.

Likely causes:

• The underlying storage class does not support volume snapshots, or the VolumeSnapshotClass is missing.
• CDI (Containerized Data Importer) is not installed or is not healthy on the DR cluster.

Fix:

1. Read the test namespace: oc get testfailover <name> -n dr-<zone-name> -o jsonpath='{.status.testNamespace}'
2. Check VolumeSnapshot objects in the test namespace on the DR cluster.
3. Verify the VolumeSnapshotClass exists: oc get volumesnapshotclass --kubeconfig ~/.kube/config-dr
4. Check CDI pods: oc get pods -n cdi --kubeconfig ~/.kube/config-dr

TestFailover enters Failed at StartingVMs

Symptom: TestFailover reaches StartingVMs then transitions to Failed.

Likely causes:

• Test VMs exceeded batchBootTimeoutSeconds before reaching Running state.
• Insufficient resources on DR cluster nodes for the test VM configuration.

Fix:

1. Set cleanupPolicy: Manual and re-run the TestFailover to preserve test resources.
2. Read the test namespace: oc get testfailover <name> -n dr-<zone-name> -o jsonpath='{.status.testNamespace}'
3. Inspect VM and VMI status in the test namespace: oc get vm,vmi -n <test-namespace> --kubeconfig ~/.kube/config-dr
4. Check node resource pressure: oc describe nodes --kubeconfig ~/.kube/config-dr | grep -A5 'Allocated resources'
5. Increase batchBootTimeoutSeconds in the TestFailover spec and reapply after cleaning up the previous attempt.

ProtectionGroup shows Degraded replication health after a failover

Symptom: After a failover completes, oc get protectiongroup shows HEALTH: Degraded.

Likely causes:

• The replication link between primary and DR clusters has not re-established after failover.
• DRBD resync is in progress (expected transiently after a failback).
• An RPOEvent has been raised indicating lag exceeds the configured RPO threshold.

Fix:

1. Check ReplicationGroupStatus: oc get replicationgroupstatus -n dr-<zone-name> and describe the relevant object for per-volume detail.
2. Check RPOEvents: oc get rpoevent -n dr-<zone-name> to see if lag violations are recorded.
3. Inspect DRBDResource sync progress on both clusters: oc get drbdresource -n dr-<zone-name> -o wide --kubeconfig ~/.kube/config-primary
4. Confirm DRBD node agents are running on all worker nodes of both clusters.
5. If the link is down, investigate network connectivity on TCP ports 7000–8000 between worker nodes.