Recovery Workflow

This page walks you through every phase of a Trilio Site Recovery (TSR) recovery workflow — from the moment you decide to trigger a failover through to a fully operational workload on the target cluster. It covers planned failovers (graceful, both clusters reachable), unplanned failovers (emergency, source cluster unavailable), non-disruptive test failovers, and the state transitions you will observe on each Custom Resource along the way. Understanding this end-to-end flow lets you confidently execute and verify recoveries, diagnose stuck phases, and roll back safely.

Before triggering any recovery operation, confirm the following:

• A healthy TSR deployment: a ProtectionZone in status.phase: Available on the quorum cluster (oc get pz -n dr-<name>).
• The ProtectionGroup you intend to fail over is in status.state: Protected or Syncing (not Failed or Degraded without investigation).
• DRBD replication is healthy: the ReplicationGroupStatus for the protection group shows status.aggregated.connectionState: Connected.
• No FailoverRequest or TestFailover is already in flight for the same protection group — the admission webhook will reject a new one if a Pending or InProgress operation exists.
• You have oc access to the quorum cluster and kubeconfig context set to it. All FailoverRequest, TestFailover, and ProtectionGroup CRs live on the quorum cluster in the dr-<name> namespace.
• Port 6443 (Kubernetes API) is open from the quorum cluster to both workload clusters' API servers.
• For an unplanned failover specifically: confirm that the source cluster is genuinely unreachable — not merely slow — before triggering, because the controller will skip the graceful VM shutdown.
• OpenShift 4.17 or later on all three clusters (quorum, primary, DR). Earlier 4.x versions are not supported.

Recovery operations are driven entirely by creating Custom Resources on the quorum cluster. There is no separate binary to install. Ensure you are connected to the quorum cluster before running any command in this guide.

# Verify your context points at the quorum cluster
oc config current-context

# Confirm the ProtectionZone is Available
oc get pz -n dr-<zone-name>

Expected output:

NAME         DISPLAY NAME   BACKEND   CLUSTERS           STATUS      AGE
my-zone      primary-dr     DRBD      cluster1,cluster2  Available   5d

If the zone is Provisioning or Degraded, resolve those conditions first — a recovery launched against a degraded zone may not complete cleanly.

Recovery behaviour is controlled by fields on the FailoverRequest and TestFailover CRs. All fields below use their exact names from the API.

FailoverRequest fields

TestFailover fields

Key replication health fields to check before recovery

ReplicationGroupStatus (short name rgs) is auto-created by the replication monitor and named <protection-group>-replication. Check it before triggering a failover:

oc get rgs -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name>

Key status fields:

Planned failover

A planned failover gracefully stops the workload on the source cluster and starts it on the target. Both clusters must be reachable.

Step 1: Verify replication health

oc get rgs -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name> -o wide

Confirm status.aggregated.connectionState is Connected and status.aggregated.outOfSyncBytes is 0.

Step 2: Create the FailoverRequest

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-to-dr
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-<zone-name>
  targetCluster: cluster2
  failoverType: planned

oc apply -f failover-to-dr.yaml

Step 3: Monitor progress

# Watch phase transitions
oc get fr failover-to-dr -n dr-<zone-name> -w

# Check conditions and per-VM status
oc describe fr failover-to-dr -n dr-<zone-name>

Expected phase progression: Validating → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → Completed

The status.vmStatuses array shows per-VM progress. The failover has a 5-minute wall-clock budget; if it does not complete within that window it transitions to Failed.

Unplanned failover

An unplanned failover is triggered when the source cluster is unavailable. The controller skips the graceful VM shutdown and promotes DRBD volumes on the DR cluster without waiting for the source.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-<zone-name>
  targetCluster: cluster2
  failoverType: unplanned

oc apply -f emergency-failover.yaml

With failoverType: unplanned, if the source cluster API server is unreachable, the controller:

1. Treats source VMs as crash-stopped (skips the halt step).
2. Stamps SourceStopped=True with reason SourceClusterUnreachable.
3. Proceeds through WaitingForLocks → StartingTarget → Monitoring → Completed.

If replication was Healthy and Consistent at the time of failure, data loss is zero. If replication was Degraded, minimal data loss corresponding to the out-of-sync bytes may occur.

Important: If the source cluster IS reachable, an unplanned failover still performs the full graceful stop. The unplanned flag only changes behaviour when the source is genuinely unreachable.

Test failover (non-disruptive)

A test failover creates isolated clones of protected VMs on the DR cluster from volume snapshots, without touching production workloads.

Step 1: Create the TestFailover CR

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: dr-test-q4
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-<zone-name>
  cleanupPolicy: Automatic
  retentionTime: 4h

oc apply -f dr-test-q4.yaml

Step 2: Monitor progress

oc get tfo dr-test-q4 -n dr-<zone-name> -w

Expected phase progression: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded

The test namespace is generated by the controller and recorded on status.testNamespace (format: site-recovery-<pg-name>-<random>). You do not need to construct or guess this value — read it from the CR status:

oc get tfo dr-test-q4 -n dr-<zone-name> -o jsonpath='{.status.testNamespace}'

Step 3: Verify results

oc describe tfo dr-test-q4 -n dr-<zone-name>

Look at status.verificationResults.passed and the status.conditions array. The Verified=True condition with reason VerificationPassed confirms all automatic checks passed.

Step 4: Clean up

With cleanupPolicy: Automatic, cleanup starts automatically after retentionTime elapses. To clean up immediately, delete the CR:

oc delete tfo dr-test-q4 -n dr-<zone-name>

The controller's finalizer ensures all test resources (VMs, PVCs, snapshots, the isolated namespace) are removed from the DR cluster before the CR is garbage-collected.

Adding reachability probes to a test failover

Reachability probes run TCP or HTTP checks against test VMs from within the isolated test namespace. They are optional and defined in spec.verification.reachability.

spec:
  verification:
    reachability:
      # TCP probe against every test VM on port 22
      - type: tcp
        port: 22
        timeout: 30s
      # HTTP probe against only the web-vm clone
      - type: http
        port: 8080
        endpoint: /health
        expectedStatus: 200
        vmName: web-vm
        timeout: 60s

Constraints enforced by the admission webhook:

• endpoint and expectedStatus are only valid on http probes; they are rejected on tcp probes.
• vmName must match a VM listed in spec.virtualMachines of the referenced ProtectionGroup.
• Probe timeout must not exceed 10m.

Observing FailoverRequest state transitions

The status.phase field tracks the current step. The status.conditions array provides per-phase detail. Key conditions:

The status.vmStatuses array shows individual VM phases (Pending, Stopping, Stopped, Starting, Running, Failed).

Failback

Failback is a planned failover in the reverse direction. After the workload is running on cluster2 (DR), create a new FailoverRequest targeting cluster1:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failback-to-primary
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-<zone-name>
  targetCluster: cluster1
  failoverType: planned

Note: Re-protection always requires a full unprotect and a complete initial DRBD resynchronisation. There is no incremental or in-place re-protection path.

Example 1: Planned failover of a two-VM protection group

This example fails over a protection group named webapp-pg from cluster1 to cluster2.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: webapp-planned-fo
  namespace: dr-webapp
spec:
  protectionGroupRef:
    name: webapp-pg
    namespace: dr-webapp
  targetCluster: cluster2
  failoverType: planned

oc apply -f webapp-planned-fo.yaml
oc get fr webapp-planned-fo -n dr-webapp -w

Expected output while in progress:

NAME                 PROTECTION GROUP   TARGET     STATE        PHASE            AGE
webapp-planned-fo    webapp-pg          cluster2   InProgress   StoppingSource   12s
webapp-planned-fo    webapp-pg          cluster2   InProgress   WaitingForLocks  35s
webapp-planned-fo    webapp-pg          cluster2   InProgress   StartingTarget   68s
webapp-planned-fo    webapp-pg          cluster2   InProgress   Monitoring       75s
webapp-planned-fo    webapp-pg          cluster2   Completed    Completed        98s

Example 2: Unplanned failover when primary is down

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: webapp-emergency-fo
  namespace: dr-webapp
spec:
  protectionGroupRef:
    name: webapp-pg
    namespace: dr-webapp
  targetCluster: cluster2
  failoverType: unplanned

oc apply -f webapp-emergency-fo.yaml
oc describe fr webapp-emergency-fo -n dr-webapp

Expected condition after StoppingSource completes against a dead source:

Conditions:
  Type            Status  Reason                    Message
  ----            ------  ------                    -------
  SourceStopped   True    SourceClusterUnreachable  Source cluster API server unreachable; treating source VMs as stopped (unplanned failover)

Example 3: Test failover with HTTP and TCP probes

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: webapp-drtest
  namespace: dr-webapp
spec:
  protectionGroupRef:
    name: webapp-pg
    namespace: dr-webapp
  cleanupPolicy: Automatic
  retentionTime: 2h
  verification:
    reachability:
      - type: tcp
        port: 22
        timeout: 30s
      - type: http
        port: 80
        endpoint: /healthz
        expectedStatus: 200
        timeout: 30s

oc apply -f webapp-drtest.yaml
oc get tfo webapp-drtest -n dr-webapp -w

Expected output on success:

NAME            PROTECTION GROUP   PHASE       PASSED   AGE
webapp-drtest   webapp-pg          Succeeded   true     8m

Read the test namespace and check test VM status:

TEST_NS=$(oc get tfo webapp-drtest -n dr-webapp -o jsonpath='{.status.testNamespace}')
echo "Test namespace: $TEST_NS"

Example 4: Querying per-VM failover status

# Show per-VM status during or after a failover
oc get fr webapp-planned-fo -n dr-webapp -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool

Expected output:

[
  {"name": "frontend-vm", "phase": "Running"},
  {"name": "backend-vm", "phase": "Running"}
]

Example 5: Boot-ordered failover

When a ProtectionGroup assigns bootOrder to VMs, the failover controller starts them in ascending batches. The database must be Running before the application tier starts.

ProtectionGroup spec:

spec:
  virtualMachines:
    - name: db-vm
      namespace: app-ns
      bootOrder: 1
    - name: web-vm
      namespace: app-ns
      bootOrder: 2

FailoverRequest with a 5-minute per-batch timeout:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: ordered-failover
  namespace: dr-webapp
spec:
  protectionGroupRef:
    name: webapp-pg
    namespace: dr-webapp
  targetCluster: cluster2
  failoverType: planned
  batchBootTimeoutSeconds: 300

Monitor the current batch:

oc get fr ordered-failover -n dr-webapp -o jsonpath='{.status.currentBootBatch}'

The field increments as each batch's VMs reach Running. If a batch times out, the controller records a BatchBootTimeout warning event and proceeds to the next batch.

Use the following structured format: Symptom → Likely cause → Fix.

FR stuck in Validating phase

Symptom: oc get fr shows Phase: Validating and State: InProgress for more than a minute with no progress.

Likely causes:

1. The ProtectionGroup referenced by spec.protectionGroupRef does not exist in the specified namespace on the quorum cluster.
2. A split-brain condition: VMs in the protection group are running on both clusters simultaneously.
3. For a planned failover: the source workload cluster API server is unreachable from the quorum cluster.

Fix:

# Check conditions on the FR
oc describe fr <name> -n dr-<zone-name>

# Verify the ProtectionGroup exists on the quorum cluster
oc get pg -n dr-<zone-name>

# Check quorum→workload cluster connectivity via the ProtectionZone
oc get pz -n dr-<zone-name> -o jsonpath='{.status.clusters}'

If cond.Reason is SplitBrainDetected, stop the VM on the incorrect cluster before retrying. If cond.Reason is ProtectionGroupNotFound, correct the spec.protectionGroupRef.name or namespace.

FR stuck in WaitingForLocks phase

Symptom: The failover advances past StoppingSource but parks at WaitingForLocks with condition Ready=False, Reason: DRBDPromotionPending.

Likely cause: The DRBD volume promotion on the DR cluster has not been confirmed. The ProtectionRequest controller (running on the quorum cluster) has not yet observed the promoted copy reporting as kernel Primary. This can also indicate that no ProtectionRequest exists for the protected VMs.

Fix:

# Check member ProtectionRequests
oc get pr -n dr-<zone-name>

# For each member PR, verify status.activeCluster matches the FR's targetCluster
oc get pr <pr-name> -n dr-<zone-name> -o jsonpath='{.status.activeCluster}'

# Check the quorum manager logs
oc logs -n dr-<zone-name> -l app.kubernetes.io/component=quorum-control-plane -c manager --since=5m

If status.conditions on the FR contains message: no ProtectionRequests found, the VMs in the protection group are not protected via ProtectionRequest CRs. Ensure the VMs were protected before triggering the failover.

The failover will fail automatically after its 5-minute wall-clock budget if confirmation never arrives.

FR failed with FailoverTimeout

Symptom: status.state: Failed, status.phase: Failed, condition Ready=False, Reason: FailoverTimeout.

Likely cause: The failover did not complete within the 5-minute wall-clock budget. Common sub-causes:

• VMs on the target cluster would not start (resource constraints, SCC issues, missing PVCs).
• The WaitingForLocks gate never opened (see above).
• A VMControlConflict: another controller held the VM control annotation.

Fix:

# Check per-VM status for which VMs failed
oc get fr <name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}'

# Check events on the FR
oc describe fr <name> -n dr-<zone-name>

# Collect a full diagnostic bundle
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag>

A failed FailoverRequest automatically releases the vm-control-owner annotation it stamped on VMs. You can create a new FailoverRequest after diagnosing the root cause.

Test failover stuck in CreatingSnapshots

Symptom: tfo CR shows Phase: CreatingSnapshots for several minutes with no progress.

Likely causes:

1. No VolumeSnapshotClass on the DR cluster has a driver matching the provisioner of the protected VM's StorageClass.
2. The external-snapshotter controller on the DR cluster is not running or is misconfigured.
3. A transient snapshot error (the controller retries for up to 5 minutes before failing).

Fix:

# Check the TestFailover conditions
oc describe tfo <name> -n dr-<zone-name>

# If Reason is SnapshotClassNotFound, list VolumeSnapshotClasses on the DR cluster
oc --kubeconfig <dr-kubeconfig> get volumesnapshotclasses

# Check the condition message — it names the PVC, StorageClass, and provisioner that have no match

If the condition reason is SnapshotClassNotFound, install a VolumeSnapshotClass on the DR cluster whose driver matches the provisioner of the affected StorageClass. If the reason is SnapshotError, check the external-snapshotter controller logs on the DR cluster.

Test failover failed with NetworkIsolationFailed

Symptom: tfo shows Phase: Failed, condition EnvironmentReady=False, Reason: NetworkIsolationFailed.

Likely cause: The workload service account on the DR cluster lacks permission to create NetworkPolicy objects in the test namespace. This is an RBAC gap — the workload chart's ClusterRole does not cover networkpolicies in the test namespace, or the workload control plane was installed into the wrong namespace.

Fix:

# Check RBAC for the workload service account on the DR cluster
oc --kubeconfig <dr-kubeconfig> auth can-i create networkpolicies -n <test-namespace>

# Verify the workload control plane is installed in trilio-site-recovery-system
oc --kubeconfig <dr-kubeconfig> get pods -n trilio-site-recovery-system -l app.kubernetes.io/component=workload-control-plane

Note: NetworkPolicy isolation for test failovers is only fully supported on OpenShift with OVN-Kubernetes. On a non-OVN CNI, the DNS isolation rule targets openshift-dns and will match nothing, so test VM name resolution will silently fail.

oc adm drain of a node hosting a protected VM does not complete

Symptom: oc adm drain <node> hangs or returns errors related to VirtualMachine objects.

Likely cause: A protected VM is pinned to this node. Live migration is denied by the admission webhook, and drain is implemented as a migration request, so it is refused. If the VM is stopped, its anchor pod is pinned through spec.nodeName and also blocks drain.

Fix: Draining a node that hosts a protected VM is not a supported operation. Options:

• In-place maintenance (kernel update, reboot): reboot the node directly. The DRBD node agent reloads the kernel modules on restart and replication resynchronises the delta automatically.
• Permanently replace the node: unprotect the VM, decommission the node, re-protect the VM. This costs a full initial DRBD resynchronisation.
• Failover first: fail the workload over to the DR cluster, maintain the primary node, then fail back.

Collecting a diagnostic bundle

When a phase is stuck or a recovery fails unexpectedly, collect a support bundle immediately — pod logs are bounded and restarted containers lose their previous log:

# Collect from the quorum cluster (covers all workload clusters in the zone)
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag>

# Narrow to the affected zone only
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> -- /usr/bin/gather --zone <zone-name>

# Narrow the log window to the last 2 hours
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> -- /usr/bin/gather --since 2h

The bundle is written to ./must-gather.local.<hash>/. Open SUMMARY.md first — it provides a cross-cluster health summary, recent failover history, and any detected warnings.