Deployment Guide

This guide walks you through deploying Trilio Site Recovery on OpenShift clusters: installing Helm charts on your quorum, primary, and DR clusters; configuring RBAC; setting resource limits; and wiring up cross-cluster connectivity. By the end you will have a fully operational disaster recovery management plane with the failover, protection, pg-sync, and test-failover controllers running on your quorum cluster, and the DRBD node agent DaemonSet running on every worker node of your primary and DR clusters. All subsequent DR operations—protecting VMs, triggering failovers, and monitoring replication health—are driven by Custom Resource Definitions and performed with kubectl.

Before you begin, ensure the following are in place across all clusters:

OpenShift version

• OpenShift ≥ 4.14 on all three clusters (quorum, primary, DR)

Required operators and software

• OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary and DR clusters
• DRBD kernel module ≥ 9.x loaded on every worker node of the primary and DR clusters
• Helm ≥ 3.0 on your workstation
• kubectl or oc CLI on your workstation
• DRBD Operator installed and running on both the primary and DR clusters
• Containerized Data Importer (CDI) installed on the primary and DR clusters (required for DataVolume support)

Kubeconfig files

• A valid kubeconfig for each cluster, accessible from your workstation:
  • ~/.kube/quorum.kubeconfig
  • ~/.kube/primary.kubeconfig
  • ~/.kube/dr.kubeconfig

Network connectivity — verify the following ports are open before proceeding:

Protocol C (synchronous replication): If you intend to use synchronous replication (RPO=0), the round-trip latency between primary and DR worker nodes must be below 50 ms.

Storage

• LVM thin pool configured on each worker node of the primary and DR clusters, backed by a dedicated block device, for use by the DRBD Operator storage pools

Helm chart access

• Access to the Trilio Helm repository and valid pull credentials for the controller container images

Installation proceeds in four ordered phases: apply the ProtectionZone CRDs and webhook, deploy the quorum control plane, deploy the workload control plane on the primary cluster, and deploy the workload control plane on the DR cluster. Do not reorder these steps—the quorum control plane depends on the ProtectionZone webhook being present, and the workload clusters require the quorum API to register.

Phase 1 — Add the Trilio Helm repository

Run this once from your workstation:

helm repo add trilio https://charts.trilio.io/site-recovery
helm repo update

Phase 2 — Deploy the ProtectionZone controller (quorum cluster)

The site-recovery-protectionzone-controller chart installs the ProtectionZone admission webhook and its CRDs. It must be installed before the quorum control plane chart.

helm upgrade --install site-recovery-protectionzone-controller \
  trilio/site-recovery-protectionzone-controller \
  --namespace site-recovery-system \
  --create-namespace \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  --wait

Verify the webhook pod is running:

kubectl get pods -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -l app.kubernetes.io/name=site-recovery-protectionzone-controller

Expected output:

NAME                                                   READY   STATUS    RESTARTS   AGE
site-recovery-protectionzone-controller-xxxxxxx-xxx    1/1     Running   0          30s

Phase 3 — Deploy the quorum control plane (quorum cluster)

The site-recovery-quorum-control-plane chart deploys the unified Go controller manager. It packages the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers, together with their admission webhooks, into a single Deployment per DR namespace.

Create a values file for your environment:

# quorum-values.yaml
controllerManager:
  replicas: 2                    # Run two replicas for HA
  resources:
    requests:
      cpu: "500m"
      memory: "512Mi"
    limits:
      cpu: "2000m"
      memory: "2Gi"

# Cross-cluster API access: provide kubeconfig secrets for primary and DR
clusterAccess:
  primaryCluster:
    kubeconfigSecretName: primary-cluster-kubeconfig
  drCluster:
    kubeconfigSecretName: dr-cluster-kubeconfig

Create kubeconfig Secrets on the quorum cluster (the controller manager reads these to talk to the workload clusters):

kubectl create secret generic primary-cluster-kubeconfig \
  --from-file=kubeconfig=~/.kube/primary.kubeconfig \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

kubectl create secret generic dr-cluster-kubeconfig \
  --from-file=kubeconfig=~/.kube/dr.kubeconfig \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

Install the chart:

helm upgrade --install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  --namespace site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -f quorum-values.yaml \
  --wait

Verify all controllers are running:

kubectl get pods -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

Expected output (all pods 1/1 Running):

NAME                                                    READY   STATUS    RESTARTS   AGE
site-recovery-quorum-control-plane-xxxxxxx-xxx          1/1     Running   0          60s
site-recovery-quorum-control-plane-xxxxxxx-yyy          1/1     Running   0          60s

Phase 4 — Deploy the workload control plane (primary cluster)

The site-recovery-workload-control-plane chart installs the DRBD node agent DaemonSet, admission webhooks, and supporting RBAC on a workload cluster. Repeat this phase for both the primary and DR clusters.

Create a values file:

# workload-values-primary.yaml
drbdNodeAgent:
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"

# The quorum cluster's API server address, so the node agent can register
quorumCluster:
  apiServer: "https://<quorum-api-server>:6443"
  kubeconfigSecretName: quorum-cluster-kubeconfig

Create a kubeconfig Secret on the primary cluster so the workload control plane can call back to the quorum:

kubectl create secret generic quorum-cluster-kubeconfig \
  --from-file=kubeconfig=~/.kube/quorum.kubeconfig \
  -n site-recovery-system \
  --kubeconfig ~/.kube/primary.kubeconfig

Install the chart on the primary cluster:

helm upgrade --install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  --namespace site-recovery-system \
  --create-namespace \
  --kubeconfig ~/.kube/primary.kubeconfig \
  -f workload-values-primary.yaml \
  --wait

Verify the DaemonSet is fully scheduled (one Pod per worker node):

kubectl get daemonset drbd-node-agent \
  -n site-recovery-system \
  --kubeconfig ~/.kube/primary.kubeconfig

Expected output:

NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
drbd-node-agent   3         3         3       3            3

Phase 5 — Deploy the workload control plane (DR cluster)

Repeat Phase 4 using the DR cluster kubeconfig and a corresponding values file:

# workload-values-dr.yaml
drbdNodeAgent:
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"

quorumCluster:
  apiServer: "https://<quorum-api-server>:6443"
  kubeconfigSecretName: quorum-cluster-kubeconfig

kubectl create secret generic quorum-cluster-kubeconfig \
  --from-file=kubeconfig=~/.kube/quorum.kubeconfig \
  -n site-recovery-system \
  --kubeconfig ~/.kube/dr.kubeconfig

helm upgrade --install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  --namespace site-recovery-system \
  --create-namespace \
  --kubeconfig ~/.kube/dr.kubeconfig \
  -f workload-values-dr.yaml \
  --wait

Phase 6 — Apply the ProtectionZone custom resource

After all charts are running, register your DR topology by creating a ProtectionZone on the quorum cluster. This resource tells the admission webhook which clusters are paired and which storage backend mode is in use.

# protection-zone.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production-zone
  namespace: site-recovery-system
spec:
  displayName: "Production DR Zone"       # required
  storageBackendMode: "drbd-operator"     # required; use "drbd-operator" for DRBD Operator deployments
  clusters:
    - name: primary
      kubeConfigSecretRef: primary-cluster-kubeconfig
    - name: dr
      kubeConfigSecretRef: dr-cluster-kubeconfig
  description: "Primary-to-DR replication zone for production workloads"

kubectl apply -f protection-zone.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

Confirm the ProtectionZone is accepted:

kubectl get protectionzone production-zone \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

The sections below describe the key configuration surfaces for each chart and the primary Custom Resources. All CRD fields shown are from the siterecovery.trilio.io/v1alpha1 API group.

Quorum control plane (site-recovery-quorum-control-plane)

Workload control plane (site-recovery-workload-control-plane)

DRBDReplicationPolicy — cross-cluster replication parameters

The DRBDReplicationPolicy CRD (applied on the primary and DR clusters) controls how DRBD replicates volumes between clusters. The most important fields are:

Choosing a protocol:

• Use drbdProtocol: "C" when your primary and DR clusters are within the same data center or connected by a sub-50 ms link. Every write is acknowledged only after committing to both sides, giving you RPO=0.
• Use drbdProtocol: "A" for geographically separated sites. Writes complete after the local disk commit; replication to the DR side happens in the background with a typical RPO of seconds.

ProtectionZone — cluster topology declaration

RBAC requirements

The quorum control plane needs cluster-scoped read/write access on the quorum cluster and cross-cluster API access to the primary and DR clusters. The Helm charts create the required ClusterRole and ClusterRoleBinding resources automatically. The key permissions are:

Quorum cluster (created by site-recovery-quorum-control-plane):

• get, list, watch, create, update, patch, delete on all siterecovery.trilio.io CRD kinds
• get, list, watch on kubevirt.io VirtualMachine and VirtualMachineInstance resources
• get, list, watch, create, update, patch, delete on coordination.k8s.io Leases (for leader election and failover locking)
• get, list, watch on Nodes, PersistentVolumes, and PersistentVolumeClaims

Primary and DR clusters (created by site-recovery-workload-control-plane):

• get, list, watch, create, update, patch, delete on siterecovery.trilio.io ProtectionGroup, DRBDResource, and DRBDReplicationPolicy CRDs
• get, list, watch, update, patch on kubevirt.io VirtualMachine and VirtualMachineInstance resources
• get, list, watch, update, patch on Nodes (for quorum taint management)
• get, list, watch, create on PersistentVolumes, PersistentVolumeClaims, and VolumeSnapshots

The quorum control plane accesses the primary and DR clusters using the kubeconfig Secrets you created in Phases 4 and 5. The service account bound to those kubeconfigs must have the workload-cluster RBAC listed above.

Once all components are running, your day-to-day DR operations follow this sequence: define a replication policy, configure a Protection Group or submit ProtectionRequests to protect VMs, monitor replication health, and trigger failovers or test failovers using CRDs. All interactions use kubectl against the appropriate cluster.

Step 1 — Define a replication policy (primary cluster)

Before protecting any VMs, create a DRBDReplicationPolicy on the primary cluster that maps storage classes and sets the replication protocol.

# replication-policy.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: production-replication-policy
  namespace: vm-workloads
spec:
  clusters:
    - name: primary
      endpoint: "<primary-drbd-endpoint>:7000"
    - name: dr
      endpoint: "<dr-drbd-endpoint>:7000"
  drbdProtocol: "C"           # Synchronous replication, RPO=0
  diskConfig:
    storageClassMappings:
      - primaryStorageClass: fast-ssd
        drStorageClass: fast-ssd-dr
  rpo:
    objectiveSeconds: 0        # Enforce RPO=0 for Protocol C
  isDefault: true

kubectl apply -f replication-policy.yaml \
  --kubeconfig ~/.kube/primary.kubeconfig

Step 2 — Protect a VM (DRBD Operator model)

Submit a ProtectionRequest on the quorum cluster. The protection-controller validates the VM, creates the DRBDResource pair covering all VM disks, and switches the VM to DRBD-backed frontend PVCs.

# protect-vm.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-prod-vm-1
  namespace: site-recovery-system
spec:
  sourceCluster: primary               # required
  virtualMachine:                       # required
    name: prod-vm-1
    namespace: vm-workloads
  replicationConfig:
    policyRef:
      name: production-replication-policy

kubectl apply -f protect-vm.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

Watch the protection phase progress:

kubectl get protectionrequest protect-prod-vm-1 \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -w

The status.phase field transitions: Pending → Validating → ProvisioningDRBDResource → SwitchingPVCs → Protected.

Step 3 — Monitor replication health

Check the ReplicationGroupStatus for an aggregated health view:

kubectl get replicationgroupstatus \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

Check per-VM sync state via DRBDResource on the primary cluster:

kubectl get drbdresource \
  -n vm-workloads \
  --kubeconfig ~/.kube/primary.kubeconfig

List any RPO violations:

kubectl get rpoevent \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  --sort-by='.spec.timestamp'

Step 4 — Run a test failover

A TestFailover validates DR readiness non-disruptively: it snapshots volumes, provisions test PVCs, starts test VMs in an isolated namespace, runs verification checks, and cleans up. Production workloads are not affected.

# test-failover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-failover-production
  namespace: site-recovery-system
spec:
  protectionGroupRef:        # required
    name: production-protection-group
    namespace: site-recovery-system
  cleanupPolicy: "OnSuccess"  # Clean up test VMs only on success
  timeout: "30m"
  batchBootTimeoutSeconds: 300

kubectl apply -f test-failover.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

kubectl get testfailover test-failover-production \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -w

Phase sequence: CreatingSnapshots → ProvisioningTestPVCs → StartingTestVMs → VerifyingData → Succeeded (then CleaningUp).

Step 5 — Execute a planned failover

A FailoverRequest with failoverType: Planned gracefully shuts down VMs on the primary, waits for DRBD to confirm all data is flushed, promotes volumes on the DR cluster, and starts VMs there.

# planned-failover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-production
  namespace: site-recovery-system
spec:
  protectionGroupRef:        # required
    name: production-protection-group
    namespace: site-recovery-system
  targetCluster: dr           # required
  failoverType: Planned
  drainTimeoutSeconds: 300
  batchBootTimeoutSeconds: 300

kubectl apply -f planned-failover.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

kubectl get failoverrequest planned-failover-production \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -w

Phase sequence: Pending → InProgress → Completed (or Failed).

Step 6 — Execute an unplanned failover

When the primary cluster is unavailable, use failoverType: Unplanned. The failover-controller force-promotes DRBD volumes on the DR cluster without waiting for the primary to shut down.

# unplanned-failover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover-production
  namespace: site-recovery-system
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: site-recovery-system
  targetCluster: dr
  failoverType: Unplanned
  batchBootTimeoutSeconds: 180

kubectl apply -f unplanned-failover.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

If Protocol A was in use at the time of the disaster, a small amount of data (the replication lag in seconds) may not have reached the DR cluster.

Example 1 — DRBDReplicationPolicy: asynchronous replication for geographically separated sites

Use Protocol A when your primary and DR clusters are separated by more than 50 ms RTT. This example maps different storage classes between the two sides and sets a 30-second RPO objective.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: geo-async-policy
  namespace: vm-workloads
spec:
  clusters:
    - name: primary
      endpoint: "10.0.1.10:7000"
    - name: dr
      endpoint: "10.1.1.10:7000"
  drbdProtocol: "A"
  diskConfig:
    storageClassMappings:
      - primaryStorageClass: local-nvme
        drStorageClass: remote-sas
  rpo:
    objectiveSeconds: 30
  resyncConfig:
    rateKiBps: 102400          # 100 MiB/s resync rate
  isDefault: false

Expected result: After applying, new ProtectionRequest objects that reference this policy will replicate asynchronously. The replication-monitor will emit an RPOEvent with severity: warning if lag exceeds 30 seconds.

Example 2 — ProtectionRequest: protect a VM with an explicit replication policy

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-db-vm
  namespace: site-recovery-system
spec:
  sourceCluster: primary
  virtualMachine:
    name: database-vm
    namespace: production
  replicationConfig:
    policyRef:
      name: geo-async-policy

kubectl apply -f protect-db-vm.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

Expected output when watching the resource:

NAME             PHASE                      AGE
protect-db-vm    ProvisioningDRBDResource   8s
protect-db-vm    SwitchingPVCs              23s
protect-db-vm    Protected                  41s

Example 3 — ProtectionGroup: group multiple VMs for coordinated failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: production-protection-group
  namespace: site-recovery-system
spec:
  sourceCluster: primary
  desiredState: running
  resourceGroupName: production-workloads
  virtualMachines:            # required
    - name: web-vm-1
    - name: web-vm-2
    - name: database-vm
  sla:
    rpoSeconds: 0
    rtoSeconds: 480

kubectl apply -f production-protection-group.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

kubectl get pg production-protection-group \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

Expected output:

NAME                          STATE     REPLICATION   AGE
production-protection-group   running   Syncing       12s

After initial sync completes, REPLICATION transitions to Consistent.

Example 4 — TestFailover with extended retention for compliance review

This example keeps the test VMs running for 2 hours so a compliance team can verify them before automatic cleanup.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: compliance-test-failover
  namespace: site-recovery-system
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: site-recovery-system
  cleanupPolicy: "Manual"       # Do not clean up automatically
  retentionTime: "2h"
  timeout: "45m"
  batchBootTimeoutSeconds: 600
  verification:
    enabled: true

kubectl apply -f compliance-test-failover.yaml \
  --kubeconfig ~/.kube/quorum.kubeconfig

Expected phase progression:

Phase: CreatingSnapshots
Phase: ProvisioningTestPVCs
Phase: StartingTestVMs
Phase: VerifyingData
Phase: Succeeded

When the compliance review is complete, delete the TestFailover CR to trigger cleanup:

kubectl delete testfailover compliance-test-failover \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig

Example 5 — ReplicationGroupStatus: poll aggregated health

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ReplicationGroupStatus
metadata:
  name: production-rgs
  namespace: site-recovery-system
spec:
  protectionGroupRef:          # required
    name: production-protection-group
    namespace: site-recovery-system
  pollingIntervalSeconds: 30
  replicationProtocol: "C"
  rpoObjectiveSeconds: 0

kubectl get replicationgroupstatus production-rgs \
  -n site-recovery-system \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -o jsonpath='{.status.health}'

Expected output:

Healthy

Example 6 — Collecting a diagnostic bundle with tsr-gather

Run tsr-gather against the quorum cluster to collect logs, CRD state, and configuration from all clusters in the DR estate:

oc adm must-gather \
  --image=registry.trilio.io/site-recovery/tsr-gather:latest \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -- /usr/bin/tsr-gather

The tool produces a timestamped directory (e.g., must-gather.local.1234567890/) containing logs from the quorum, primary, and DR clusters, all CRD objects, and controller configuration—without capturing secrets or sensitive credentials.

Use the following patterns to diagnose and resolve common deployment and operational problems. For each issue, collect a diagnostic bundle first:

oc adm must-gather \
  --image=registry.trilio.io/site-recovery/tsr-gather:latest \
  --kubeconfig ~/.kube/quorum.kubeconfig \
  -- /usr/bin/tsr-gather

Issue 1 — Quorum control plane pods are not starting

Symptom: site-recovery-quorum-control-plane pods are in Pending or CrashLoopBackOff immediately after Helm install.

Likely causes and fixes:

1. ProtectionZone controller not installed first. Check whether the ProtectionZone webhook is running:

   Copy

   kubectl get pods -n site-recovery-system \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     -l app.kubernetes.io/name=site-recovery-protectionzone-controller
   

   If no pods are listed, install site-recovery-protectionzone-controller before the quorum control plane chart (see Phase 2 in the installation section).

2. Kubeconfig Secrets missing or malformed. Verify both Secrets exist and contain a valid kubeconfig key:

   Copy

   kubectl get secret primary-cluster-kubeconfig dr-cluster-kubeconfig \
     -n site-recovery-system \
     --kubeconfig ~/.kube/quorum.kubeconfig
   

   Re-create any missing Secret using the kubectl create secret generic command from Phase 3.

3. Inspect pod logs for the specific error:

   Copy

   kubectl logs -n site-recovery-system \
     -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     --previous
   

Issue 2 — drbd-node-agent DaemonSet pods are not scheduled on all nodes

Symptom: kubectl get daemonset drbd-node-agent shows DESIRED > READY.

Likely causes and fixes:

1. DRBD kernel module not loaded. SSH to the affected node and verify:

   Copy

   lsmod | grep drbd
   

   If no output, load the module:

   Copy

   modprobe drbd
   

   Consult your DRBD 9.x installation documentation to make the module load persistent across reboots.

2. Node taint preventing scheduling. Check for any unexpected node taints:

   Copy

   kubectl describe node <node-name> \
     --kubeconfig ~/.kube/primary.kubeconfig | grep -A5 Taints
   

   The drbd.linbit.com/lost-quorum taint will block all non-tolerating pods. This taint is removed automatically by the failover-controller during a failover; if it persists after a completed failover, check the failover-controller logs on the quorum cluster.

3. Resource constraints. Confirm the node has sufficient CPU and memory for the agent pod:

   Copy

   kubectl describe node <node-name> \
     --kubeconfig ~/.kube/primary.kubeconfig | grep -A10 "Allocated resources"
   

Issue 3 — ProtectionRequest is stuck in ProvisioningDRBDResource

Symptom: A ProtectionRequest does not advance past the ProvisioningDRBDResource phase after several minutes.

Likely causes and fixes:

1. TCP port 7000–7999 blocked between clusters. From a worker node on the primary cluster, test connectivity to a DR worker node:

   Copy

   nc -zv <dr-worker-node-ip> 7000
   

   If the connection is refused or times out, open the port range in your network policy or firewall.

2. DRBDReplicationPolicy not found or misconfigured. Verify the policy exists on the primary cluster and its spec.clusters endpoints are reachable:

   Copy

   kubectl get drbdreplicationpolicy \
     -n vm-workloads \
     --kubeconfig ~/.kube/primary.kubeconfig
   

3. Protection-controller logs on the quorum cluster:

   Copy

   kubectl logs -n site-recovery-system \
     -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     | grep -i protectionrequest
   

Issue 4 — ReplicationGroupStatus shows Degraded or Critical

Symptom: kubectl get replicationgroupstatus reports Degraded or Critical health.

Likely causes and fixes:

1. Check for RPOEvents to understand which volumes have fallen behind:

   Copy

   kubectl get rpoevent \
     -n site-recovery-system \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     -o wide
   

   Each RPOEvent shows the affected volumes, rpoAtEvent, and violationReason.

2. Check DRBDResource status on both clusters for per-volume sync state:

   Copy

   kubectl get drbdresource \
     -n vm-workloads \
     --kubeconfig ~/.kube/primary.kubeconfig \
     -o wide
   
   kubectl get drbdresource \
     -n vm-workloads \
     --kubeconfig ~/.kube/dr.kubeconfig \
     -o wide
   

   A volume reporting Disconnected or SyncSource/SyncTarget in its connection state indicates a network interruption or active resync.

3. Network interruption. Verify DRBD port connectivity from the primary to the DR worker nodes (TCP 7000–7999) as described in Issue 3. After restoring connectivity, DRBD will resync automatically; health will return to Healthy once resync completes.

Issue 5 — FailoverRequest is stuck in InProgress

Symptom: A FailoverRequest remains in InProgress for longer than the expected RTO window (typically 3–8 minutes).

Likely causes and fixes:

1. VM on source cluster is not shutting down within drainTimeoutSeconds. Check the VM status on the source cluster:

   Copy

   kubectl get vm -n <vm-namespace> \
     --kubeconfig ~/.kube/primary.kubeconfig
   

   If a VM is stuck in Stopping, check KubeVirt events:

   Copy

   kubectl describe vmi <vm-name> -n <vm-namespace> \
     --kubeconfig ~/.kube/primary.kubeconfig
   

2. Failover lock held by another operation. Check for a Lease resource on the quorum cluster:

   Copy

   kubectl get lease -n site-recovery-system \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     | grep failover-lock
   

   If a stale lock exists (holder is no longer running), delete it to unblock the current failover:

   Copy

   kubectl delete lease failover-lock-<pg-name> \
     -n site-recovery-system \
     --kubeconfig ~/.kube/quorum.kubeconfig
   

3. Failover-controller logs:

   Copy

   kubectl logs -n site-recovery-system \
     -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     | grep -i failoverrequest
   

Issue 6 — TestFailover is stuck in CreatingSnapshots

Symptom: A TestFailover does not advance past the CreatingSnapshots phase.

Likely causes and fixes:

1. VolumeSnapshot CRDs or the snapshot controller are not installed on the DR cluster. Verify:

   Copy

   kubectl get crd volumesnapshots.snapshot.storage.k8s.io \
     --kubeconfig ~/.kube/dr.kubeconfig
   

   Install the CSI snapshotter if missing, following the OpenShift documentation for your version.

2. Storage class on the DR cluster does not support snapshots. Check the storageClassMappings in your DRBDReplicationPolicy and confirm the mapped DR storage class has a VolumeSnapshotClass:

   Copy

   kubectl get volumesnapshotclass \
     --kubeconfig ~/.kube/dr.kubeconfig
   

3. Test-failover-controller logs on the quorum cluster:

   Copy

   kubectl logs -n site-recovery-system \
     -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
     --kubeconfig ~/.kube/quorum.kubeconfig \
     | grep -i testfailover