Recovery Workflow

This page walks you through the complete recovery workflow for Trilio Site Recovery—from triggering a failover to verifying that your virtual machines are running on the DR cluster. It covers planned failover (graceful shutdown of source VMs before promotion), unplanned failover (force-promotion when the primary is unreachable), test failover (non-disruptive DR validation using snapshots), and failback. All operations are driven by Kubernetes Custom Resource Definitions, so every step is auditable, repeatable, and compatible with GitOps workflows. Understanding this flow helps you minimize both recovery time (RTO target: 3–8 minutes) and data loss (RPO=0 with Protocol C, seconds with Protocol A).

Before triggering any recovery operation, confirm the following:

• OpenShift ≥ 4.14 on all clusters
• OpenShift Virtualization (CNV) ≥ 1.0
• DRBD kernel module ≥ 9.x on all worker nodes
• kubectl or oc CLI available on your workstation
• Kubeconfig files for the quorum cluster, primary cluster, and DR cluster
• The site-recovery-quorum-control-plane Helm chart deployed to the quorum cluster
• The site-recovery-workload-control-plane Helm chart deployed to both the primary and DR clusters
• At least one ProtectionGroup (DRBD Operator deployments: ProtectionRequest) in a Healthy or Consistent replication state
• Network connectivity: TCP 7000–7999 open between primary and DR worker nodes, TCP 6443 open from the quorum cluster to both workload cluster API servers
• For Protocol C (synchronous) replication: round-trip latency < 50 ms between primary and DR clusters
• DRBDReplicationPolicy configured and bound to your protection groups (DRBD Operator model)

Tip: Run kubectl get replicationgroupstatus -n <dr-namespace> against the quorum cluster before any failover to confirm replication health is Healthy and that no RPOEvent records indicate an active lag violation.

The recovery workflow does not require additional software installation beyond the standard Site Recovery deployment. If you have not yet deployed the control planes, complete those steps first. The instructions below assume all Helm charts are already installed and your CRDs are registered.

Verify that the quorum control plane is ready:

kubectl --kubeconfig ~/.kube/config-quorum \
  get pods -n dr-<deployment-name> -l app.kubernetes.io/name=site-recovery-quorum-control-plane

Expected output: all pods in Running state.

Verify that the workload control plane is ready on both clusters:

# Primary cluster
kubectl --kubeconfig ~/.kube/config-primary \
  get pods -n site-recovery -l app.kubernetes.io/name=site-recovery-workload-control-plane

# DR cluster
kubectl --kubeconfig ~/.kube/config-dr \
  get pods -n site-recovery -l app.kubernetes.io/name=site-recovery-workload-control-plane

Verify the DRBD node agent DaemonSet is healthy on both clusters:

# Primary cluster
kubectl --kubeconfig ~/.kube/config-primary \
  get daemonset -n site-recovery -l app.kubernetes.io/name=drbd-node-agent

# DR cluster
kubectl --kubeconfig ~/.kube/config-dr \
  get daemonset -n site-recovery -l app.kubernetes.io/name=drbd-node-agent

All DESIRED, CURRENT, and READY counts should match before you proceed with any recovery operation.

Recovery behavior is controlled by the following Custom Resources. Set these fields before triggering a failover.

FailoverRequest

The FailoverRequest CR is what you create to trigger either a planned or unplanned failover. Key spec fields:

TestFailover

The TestFailover CR runs a non-disruptive DR validation without impacting production VMs.

DRBDReplicationPolicy (DRBD Operator model)

This CR governs how volumes replicate between clusters. Set drbdProtocol before protection is applied; changing it later requires re-protecting affected VMs.

ReplicationGroupStatus

This CR is read-only from an operational standpoint—the replication-monitor reconciler writes it. You configure its polling behavior:

Checking Replication Health Before Failover

Always confirm replication is healthy before triggering a recovery operation. A Degraded or Critical status may indicate incomplete data on the DR side.

kubectl --kubeconfig ~/.kube/config-quorum \
  get replicationgroupstatus -n dr-<deployment-name>

Check for any open RPO violations:

kubectl --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-<deployment-name> --sort-by=.spec.timestamp

Running a Test Failover (Non-Disruptive)

Before performing a real failover, validate DR readiness by running a TestFailover. This creates snapshot-backed copies of your VMs in an isolated namespace on the DR cluster, runs any configured verification checks, and then cleans up—without touching production workloads.

1. Create the TestFailover CR on the quorum cluster:

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-failover-production
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-<deployment-name>
  cleanupPolicy: automatic
  retentionTime: 2h
  timeout: 30m
EOF

2. Watch the phase progression:

kubectl --kubeconfig ~/.kube/config-quorum \
  get testfailover test-failover-production \
  -n dr-<deployment-name> -w

The test-failover-controller advances through: CreatingSnapshots → VerifyingData → Succeeded (then CleaningUp if cleanupPolicy: automatic).

3. Investigate failures if the phase reaches Failed:

kubectl --kubeconfig ~/.kube/config-quorum \
  describe testfailover test-failover-production \
  -n dr-<deployment-name>

Executing a Planned Failover

Use a planned failover during maintenance windows or scheduled migrations. The failover-controller gracefully stops VMs on the primary cluster, waits for DRBD volumes to flush, promotes volumes on the DR cluster, and starts VMs there.

1. Create the FailoverRequest CR on the quorum cluster:

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production-planned
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-<deployment-name>
  targetCluster: dr-cluster
  failoverType: planned
  drainTimeoutSeconds: 300
  batchBootTimeoutSeconds: 180
EOF

2. Monitor progress:

kubectl --kubeconfig ~/.kube/config-quorum \
  get failoverrequest failover-production-planned \
  -n dr-<deployment-name> -w

State transitions: Pending → InProgress → Completed (or Failed).

3. Verify VMs are running on the DR cluster:

kubectl --kubeconfig ~/.kube/config-dr \
  get vm -n <vm-namespace>

Executing an Unplanned Failover

Use an unplanned failover when the primary cluster is unreachable and you cannot wait for a graceful shutdown. The failover-controller force-promotes DRBD volumes on the DR cluster. If you were using Protocol A, there may be a small amount of data loss equal to the replication lag at the time of the outage.

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production-emergency
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-<deployment-name>
  targetCluster: dr-cluster
  failoverType: unplanned
EOF

Important: After an unplanned failover, do not attempt to restart VMs on the primary cluster manually. The replication link will be in a split-brain-prevention state until you complete failback or explicitly re-establish replication.

Performing Failback

Failback returns VMs from the DR cluster to the primary cluster after the primary is restored. Failback is a reverse FailoverRequest—you target the primary cluster.

1. Confirm the primary cluster API is reachable and worker nodes are healthy:

kubectl --kubeconfig ~/.kube/config-primary get nodes

2. Confirm replication has re-synchronized (the DR side must have finished pushing any divergent writes back to the primary):

kubectl --kubeconfig ~/.kube/config-quorum \
  get replicationgroupstatus -n dr-<deployment-name>

Wait until the status shows Healthy before proceeding.

3. Create the failback FailoverRequest targeting the primary cluster:

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failback-production
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-<deployment-name>
  targetCluster: primary-cluster
  failoverType: planned
  drainTimeoutSeconds: 300
  batchBootTimeoutSeconds: 180
EOF

4. Monitor until Completed:

kubectl --kubeconfig ~/.kube/config-quorum \
  get failoverrequest failback-production \
  -n dr-<deployment-name> -w

Example 1: End-to-End Planned Failover

This example shows the complete state transition sequence for a planned failover of a two-VM protection group.

Step 1 — Confirm replication is healthy

kubectl --kubeconfig ~/.kube/config-quorum \
  get replicationgroupstatus -n dr-prod -o wide

Expected output:

NAME                          HEALTH    PROTOCOL   LAST-SYNC             AGE
production-protection-group   Healthy   C          2024-01-15T10:30:00Z  5d

Step 2 — Create the FailoverRequest

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-prod-20240115
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-prod
  targetCluster: dr-cluster
  failoverType: planned
  drainTimeoutSeconds: 300
  batchBootTimeoutSeconds: 180
EOF

Expected output:

failoverrequest.siterecovery.trilio.io/failover-prod-20240115 created

Step 3 — Watch state transitions

kubectl --kubeconfig ~/.kube/config-quorum \
  get failoverrequest failover-prod-20240115 \
  -n dr-prod -w

Expected output (over approximately 4–6 minutes):

NAME                     STATUS      AGE
failover-prod-20240115   Pending     0s
failover-prod-20240115   InProgress  8s
failover-prod-20240115   InProgress  45s
failover-prod-20240115   InProgress  2m10s
failover-prod-20240115   Completed   4m33s

Step 4 — Verify VMs on the DR cluster

kubectl --kubeconfig ~/.kube/config-dr \
  get vm -n production

Expected output:

NAME         AGE   STATUS    READY
prod-vm-1    12d   Running   True
prod-vm-2    12d   Running   True

Example 2: Non-Disruptive Test Failover

This example runs a full DR validation, including post-boot verification, without affecting production VMs.

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: dr-test-q1
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-prod
  cleanupPolicy: automatic
  retentionTime: 1h
  timeout: 45m
  batchBootTimeoutSeconds: 180
EOF

Watch phase progression:

kubectl --kubeconfig ~/.kube/config-quorum \
  get testfailover dr-test-q1 -n dr-prod -w

Expected output:

NAME          PHASE               AGE
dr-test-q1    CreatingSnapshots   10s
dr-test-q1    VerifyingData       1m12s
dr-test-q1    Succeeded           3m45s
dr-test-q1    CleaningUp          3m46s

Example 3: Unplanned Failover After Primary Outage

The primary cluster API is unreachable. You force-promote DRBD volumes on the DR cluster.

kubectl --kubeconfig ~/.kube/config-quorum \
  apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover-20240115
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: dr-prod
  targetCluster: dr-cluster
  failoverType: unplanned
EOF

Expected output after monitoring:

NAME                         STATUS      AGE
emergency-failover-20240115  Pending     0s
emergency-failover-20240115  InProgress  6s
emergency-failover-20240115  Completed   2m18s

Check for any RPO violations that occurred before the failover:

kubectl --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-prod --sort-by=.spec.timestamp

Expected output:

NAME              SEVERITY   LAG(s)   PROTECTION-GROUP               TIMESTAMP
rpoevent-a1b2c3   warning    4        production-protection-group    2024-01-15T09:58:12Z

This record shows the replication lag at the time of the outage and serves as your data loss audit record.

Example 4: Collecting a Diagnostic Bundle After a Failed Failover

If a FailoverRequest reaches Failed state, collect a support bundle:

oc adm must-gather --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle \
  -- /usr/bin/tsr-gather \
  --kubeconfig-quorum ~/.kube/config-quorum \
  --kubeconfig-primary ~/.kube/config-primary \
  --kubeconfig-dr ~/.kube/config-dr

The tsr-gather tool collects logs, CRD state, and configuration from all clusters into ./tsr-bundle without capturing secrets or credentials.

Use the following patterns to diagnose common recovery failures. For each issue, check the FailoverRequest or TestFailover status first, then examine controller logs on the quorum cluster.

FailoverRequest Stuck in InProgress

Symptom: A FailoverRequest has been InProgress for more than 10 minutes with no state change.

Likely causes:

• VMs on the source cluster did not stop within drainTimeoutSeconds
• DRBD volumes are still syncing or have not released locks
• The failover-controller pod on the quorum cluster has crashed or restarted

Fix:

1. Check the failover-controller pod logs:

kubectl --kubeconfig ~/.kube/config-quorum \
  logs -n dr-<deployment-name> \
  -l app.kubernetes.io/component=failover-controller --tail=100

2. Check VM state on the source cluster:

kubectl --kubeconfig ~/.kube/config-primary \
  get vm -n <vm-namespace>

If VMs are still Running, the drain timed out. Investigate why VMs are not stopping (check KubeVirt events).

3. Check DRBD volume state via the drbd-node-agent DaemonSet:

kubectl --kubeconfig ~/.kube/config-dr \
  logs -n site-recovery \
  -l app.kubernetes.io/name=drbd-node-agent --tail=50

4. If the controller pod is crashed, restart it:

kubectl --kubeconfig ~/.kube/config-quorum \
  rollout restart deployment \
  -n dr-<deployment-name> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane

FailoverRequest Reaches Failed State

Symptom: The FailoverRequest status is Failed and VMs are not running on the DR cluster.

Likely causes:

• targetCluster name does not match a registered cluster in the ProtectionZone
• protectionGroupRef references a ProtectionGroup that does not exist or is in Degraded state
• DRBD volume promotion failed on the DR side (split-brain or missing replica)

Fix:

1. Describe the failed FailoverRequest for the detailed error message:

kubectl --kubeconfig ~/.kube/config-quorum \
  describe failoverrequest <name> -n dr-<deployment-name>

2. Verify the ProtectionGroup status:

kubectl --kubeconfig ~/.kube/config-quorum \
  get replicationgroupstatus -n dr-<deployment-name>

If the status is Degraded or Critical, resolve the replication issue before retrying. A Critical status means the DR volume does not have a consistent copy of the data.

3. Collect the full diagnostic bundle:

oc adm must-gather --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle \
  -- /usr/bin/tsr-gather

TestFailover Stuck in CreatingSnapshots

Symptom: A TestFailover has been in CreatingSnapshots phase for more than 5 minutes.

Likely causes:

• The VolumeSnapshot CSI driver is not installed or the VolumeSnapshotClass is not configured on the DR cluster
• The DR cluster worker nodes are under disk pressure

Fix:

1. Check the test-failover-controller logs:

kubectl --kubeconfig ~/.kube/config-quorum \
  logs -n dr-<deployment-name> \
  -l app.kubernetes.io/component=test-failover-controller --tail=100

2. Check for VolumeSnapshot objects in the test namespace on the DR cluster:

kubectl --kubeconfig ~/.kube/config-dr \
  get volumesnapshot -A

If snapshots are pending, check the CSI driver pod logs on the DR cluster.

VMs Do Not Start on DR Cluster After Failover

Symptom: The FailoverRequest shows Completed but VMs on the DR cluster remain in Stopped state.

Likely causes:

• Insufficient compute capacity on DR worker nodes
• KubeVirt admission webhook rejecting the VM spec on the DR cluster
• Node taints preventing VM pod scheduling

Fix:

1. Check VM events on the DR cluster:

kubectl --kubeconfig ~/.kube/config-dr \
  describe vm <vm-name> -n <vm-namespace>

2. Check node conditions and taints on DR worker nodes:

kubectl --kubeconfig ~/.kube/config-dr get nodes -o wide
kubectl --kubeconfig ~/.kube/config-dr describe node <node-name> | grep -A5 Taints

If DRBD quorum taints are still present, the failover-controller may not have cleared them. Check the controller logs as shown above and verify that DRBDResource status on the DR cluster shows the volumes as promoted.

3. Verify DRBDResource state on the DR cluster:

kubectl --kubeconfig ~/.kube/config-dr \
  get drbdresource -n <vm-namespace>

All DRBDResource objects for the failed-over VMs should show role: Primary.

ReplicationGroupStatus Shows Degraded Before Failover

Symptom: Pre-failover health check shows Degraded or Critical replication status.

Likely causes:

• Network interruption on the DRBD replication link (TCP 7000–7999)
• A worker node on the primary or DR cluster is offline
• The drbd-node-agent DaemonSet pod is not running on one or more nodes

Fix:

1. Check RPOEvent records for details on which volumes are affected:

kubectl --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-<deployment-name> -o yaml

2. Check drbd-node-agent pod status on both clusters:

kubectl --kubeconfig ~/.kube/config-primary \
  get pods -n site-recovery -l app.kubernetes.io/name=drbd-node-agent

kubectl --kubeconfig ~/.kube/config-dr \
  get pods -n site-recovery -l app.kubernetes.io/name=drbd-node-agent

3. Verify TCP port 7000–7999 connectivity between a primary worker node and a DR worker node (use nc or telnet from inside a node debug pod).

4. Do not proceed with a planned failover while replication is Degraded. If the situation is a genuine emergency requiring an unplanned failover, set failoverType: unplanned and accept the data loss risk documented in any open RPOEvent records.