Trilio Site Recovery for Kubernetes
Guide

Prerequisites

Kubernetes version requirements, required permissions, storage dependencies, and network prerequisites


Overview

This page describes every requirement you must satisfy before installing Trilio Site Recovery on OpenShift. Meeting these requirements ensures that the DRBD kernel module can replicate VM-attached PVCs between clusters, that the quorum control plane can reach both workload clusters over the Kubernetes API, and that Helm can deploy the three charts that make up the management and workload planes. Review each section carefully—missing a single networking or storage requirement is the most common cause of failed installations and degraded replication.


Prerequisites

Collect and verify every item below before you run any helm install or kubectl apply command.

OpenShift version

Cluster roleRequired version
Primary clusterOpenShift ≥ 4.14
DR clusterOpenShift ≥ 4.14
Quorum clusterOpenShift ≥ 4.14

Required cluster components

Install these on the primary and DR clusters before deploying Site Recovery. The quorum cluster does not run VM workloads and does not require KubeVirt or the DRBD kernel module.

ComponentMinimum versionCluster(s)
OpenShift Virtualization (CNV)1.0Primary, DR
DRBD kernel module9.xEvery worker node on Primary and DR
DRBD OperatorcurrentPrimary, DR
Containerized Data Importer (CDI)currentPrimary, DR
Multus CNIcurrentPrimary, DR (if using multiple network interfaces for replication)

Note: The DRBD kernel module must be loaded on every worker node that will host VM workloads. Confirm the module is present with lsmod | grep drbd on each node before proceeding.

Client tooling

You need the following tools on the machine from which you run installation commands:

  • helm ≥ 3.0
  • oc CLI (OpenShift client) compatible with your cluster version
  • kubectl compatible with your cluster version
  • Valid kubeconfig files for all clusters (see Kubeconfig setup below)

Kubeconfig setup

You will switch kubeconfig context frequently. Export a named variable for each cluster kubeconfig before you begin:

export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr
export KUBECONFIG_QUORUM=~/.kube/config-quorum

Verify API server access for all three clusters:

oc --kubeconfig $KUBECONFIG_PRIMARY get nodes
oc --kubeconfig $KUBECONFIG_DR get nodes
oc --kubeconfig $KUBECONFIG_QUORUM get nodes

Network requirements

All port and latency requirements must be met before you apply any CRDs or start replication. Firewall rules that are added after DRBD resources are created will not cause an error at creation time but will cause the replication link to remain in a StandAlone or Connecting state.

TrafficProtocolPort rangeDirectionRequired for
DRBD block replicationTCP7000–7999Primary worker nodes ↔ DR worker nodesAll deployments
Kubernetes API accessTCP6443Quorum cluster → Primary and DR API serversAll deployments

Protocol C (synchronous replication) latency requirement

If you configure drbdProtocol: C (synchronous replication, RPO=0), the round-trip latency between primary and DR worker nodes must be below 50 ms. Exceeding this threshold causes write acknowledgement delays that will degrade application performance. Measure latency before installation:

# Run from a primary worker node toward a DR worker node
ping -c 100 <dr-worker-node-ip>

If latency consistently exceeds 50 ms, use drbdProtocol: A (asynchronous replication) instead, which accepts an RPO of seconds.

Storage requirements

Each worker node on the primary and DR clusters that will host protected VMs must have:

  • An LVM thin pool configured on a dedicated block device. This thin pool is used by DRBD-backed frontend PVCs that replace a VM's original PVCs after protection is applied.
  • Sufficient raw block capacity to hold all DRBD volumes that will be placed on that node.

Verify that the thin pool exists on each worker node before installation:

# Run on each worker node
lvs --select 'lv_attr=~t' -o lv_name,lv_size,pool_lv

Permissions and RBAC

The Helm charts create the necessary ClusterRole, ClusterRoleBinding, Role, and RoleBinding objects during installation. The identity used to run helm install must have sufficient permissions to create these objects. On OpenShift, use a service account or user with the cluster-admin role for the initial installation:

# Verify your current identity has cluster-admin on the quorum cluster
oc --kubeconfig $KUBECONFIG_QUORUM auth can-i create clusterroles --all-namespaces

The expected response is yes. Repeat this check for the primary and DR clusters.


Installation

Installation follows a fixed order across three phases. Deviating from this order—for example, deploying the quorum control plane before the ProtectionZone webhook—will cause admission webhook failures at CRD creation time.

Phase order:

  1. Prepare storage and network on primary and DR clusters
  2. Deploy the ProtectionZone webhook and CRDs on the quorum cluster
  3. Deploy the quorum control plane on the quorum cluster
  4. Deploy the workload control plane on the primary and DR clusters

Step 1 — Verify DRBD kernel module on worker nodes

Confirm the DRBD 9.x kernel module is loaded on every worker node of the primary and DR clusters before installing any Helm chart.

# Run on each worker node (example using oc debug)
oc --kubeconfig $KUBECONFIG_PRIMARY debug node/<node-name> -- chroot /host lsmod | grep drbd
oc --kubeconfig $KUBECONFIG_DR debug node/<node-name> -- chroot /host lsmod | grep drbd

Expected output includes a line such as:

drbd                  548864  0

If the module is not loaded, consult your DRBD kernel module installation documentation before continuing.


Step 2 — Deploy the ProtectionZone webhook on the quorum cluster

The site-recovery-protectionzone-controller chart installs the ProtectionZone admission webhook and its CRDs. It must be installed before the quorum control plane chart because the quorum control plane chart depends on the ProtectionZone CRD being present in the cluster.

helm install site-recovery-protectionzone \
  ./helm/site-recovery-protectionzone-controller \
  --kubeconfig $KUBECONFIG_QUORUM \
  --namespace trilio-site-recovery-system \
  --create-namespace

Wait for the webhook pod to reach Running status before proceeding:

oc --kubeconfig $KUBECONFIG_QUORUM get pods \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/name=site-recovery-protectionzone-controller \
  --watch

Step 3 — Create the DR deployment namespace on the quorum cluster

Each DR deployment is isolated in its own namespace named dr-<name> on the quorum cluster. Create the namespace before deploying the quorum control plane into it. Replace <deployment-name> with your chosen name (for example, prod).

oc --kubeconfig $KUBECONFIG_QUORUM create namespace dr-<deployment-name>

Step 4 — Deploy the quorum control plane

The site-recovery-quorum-control-plane chart deploys the failover controller, protection controller, pg-sync controller, and replication-monitor reconcilers as a single controller manager deployment in the DR deployment namespace.

helm install dr-<deployment-name> \
  ./helm/site-recovery-quorum-control-plane \
  --kubeconfig $KUBECONFIG_QUORUM \
  --namespace dr-<deployment-name>

Verify that the controller manager pod is running:

oc --kubeconfig $KUBECONFIG_QUORUM get pods \
  -n dr-<deployment-name> \
  --watch

All pods should reach Running status before you proceed.


Step 5 — Deploy the workload control plane on the primary cluster

The site-recovery-workload-control-plane chart installs the drbd-node-agent DaemonSet, admission webhooks, and supporting RBAC on the workload cluster. Deploy it to both the primary and DR clusters.

# Primary cluster
helm install dr-<deployment-name> \
  ./helm/site-recovery-workload-control-plane \
  --kubeconfig $KUBECONFIG_PRIMARY \
  --namespace trilio-site-recovery-system \
  --create-namespace

Verify that the DaemonSet pod is running on every worker node:

oc --kubeconfig $KUBECONFIG_PRIMARY get daemonset \
  -n trilio-site-recovery-system

The DESIRED and READY counts must match before you proceed.


Step 6 — Deploy the workload control plane on the DR cluster

# DR cluster
helm install dr-<deployment-name> \
  ./helm/site-recovery-workload-control-plane \
  --kubeconfig $KUBECONFIG_DR \
  --namespace trilio-site-recovery-system \
  --create-namespace

Repeat the DaemonSet verification on the DR cluster:

oc --kubeconfig $KUBECONFIG_DR get daemonset \
  -n trilio-site-recovery-system

Step 7 — Create the ProtectionZone custom resource

A ProtectionZone CR defines the logical DR topology: which clusters participate, which storage backend mode is in use, and display metadata. Apply it on the quorum cluster after all three Helm charts are running.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: <deployment-name>
  namespace: dr-<deployment-name>
spec:
  displayName: "<Human-readable name>"
  storageBackendMode: drbd-operator
  clusters:
    - name: primary
      kubeConfigSecretRef:
        name: primary-kubeconfig
    - name: dr
      kubeConfigSecretRef:
        name: dr-kubeconfig
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionzone.yaml

Verify the ProtectionZone status:

oc --kubeconfig $KUBECONFIG_QUORUM get protectionzone <deployment-name> \
  -n dr-<deployment-name> -o yaml

With all three charts running and the ProtectionZone accepted by the webhook, your installation is complete. Proceed to configure replication and protect your first VM.


Configuration

All Site Recovery configuration is expressed as Kubernetes Custom Resource specifications. The sections below document the key fields for each CR that affects deployment behavior. Apply every CR with oc --kubeconfig $KUBECONFIG_QUORUM apply -f <file> unless stated otherwise.


ProtectionZone

ProtectionZone (API group siterecovery.trilio.io/v1alpha1) defines the overall DR topology for a deployment namespace. It is the first CR you create after installation.

FieldTypeRequiredDescription
spec.displayNamestringYesHuman-readable name shown in the TSR web console.
spec.storageBackendModestringYesStorage backend in use. For DRBD Operator deployments, set drbd-operator.
spec.clustersarrayYesList of cluster references (primary and DR). Each entry identifies a cluster and references a Secret containing its kubeconfig.
spec.descriptionstringNoFree-text description of this deployment.
spec.loggingobjectNoLogging configuration for this ProtectionZone.

DRBDReplicationPolicy

DRBDReplicationPolicy (API group siterecovery.trilio.io/v1alpha1) defines how block replication operates between the primary and DR clusters. One policy applies to all VMs protected under a deployment, unless per-VM overrides are configured.

FieldTypeRequiredDescription
spec.clustersarrayYesReferences to the primary and DR cluster entries defined in ProtectionZone.
spec.drbdProtocolstringNoC for synchronous (RPO=0, requires <50 ms RTT) or A for asynchronous (seconds RPO). Defaults to C.
spec.replicationModestringNoReplication mode. Consult the DRBD Operator documentation for valid values.
spec.diskConfigobjectNoDisk configuration for DRBD volumes, including storage class mappings between primary and DR.
spec.networkConfigobjectNoNetwork configuration for DRBD replication endpoints (IP addresses and ports in the 7000–7999 range).
spec.rpoobjectNoRPO thresholds that trigger RPOEvent CRs when violated.
spec.resyncConfigobjectNoTuning parameters for DRBD resynchronization speed and concurrency.
spec.isDefaultbooleanNoWhen true, this policy is used for ProtectionRequests that do not reference a named policy.

Protocol choice guidance:

  • Use drbdProtocol: C when your primary-to-DR round-trip latency is consistently below 50 ms. This guarantees RPO=0 because writes are only acknowledged after being committed on both sides.
  • Use drbdProtocol: A for geographically distant clusters where latency exceeds 50 ms. Writes complete after local disk commit and are shipped asynchronously, resulting in an RPO of seconds.

ProtectionRequest

ProtectionRequest (API group siterecovery.trilio.io/v1alpha1) requests block-level replication protection for a single VM in DRBD Operator deployments. The protection controller on the quorum cluster processes it.

FieldTypeRequiredDescription
spec.virtualMachineobjectYesReference to the VirtualMachine CR to protect (name and namespace).
spec.sourceClusterstringYesName of the cluster where the VM currently runs. Must match a cluster entry in the ProtectionZone.
spec.replicationConfigobjectNoOptional per-VM replication overrides. If omitted, the default DRBDReplicationPolicy for the deployment is used.

FailoverRequest

FailoverRequest (API group siterecovery.trilio.io/v1alpha1) triggers a failover operation. Apply it on the quorum cluster.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the ProtectionGroup that defines the set of VMs to fail over.
spec.targetClusterstringYesName of the cluster where VMs should start after failover.
spec.failoverTypestringNoplanned (graceful VM shutdown before promotion) or unplanned (force-promote without waiting for source). Defaults to planned.
spec.drainTimeoutSecondsintegerNoMaximum seconds to wait for VM shutdown on the source cluster during a planned failover.
spec.batchBootTimeoutSecondsintegerNoMaximum seconds to wait for each batch of VMs to reach Running state on the target cluster.

TestFailover

TestFailover (API group siterecovery.trilio.io/v1alpha1) runs a non-disruptive DR validation. Apply it on the quorum cluster.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the ProtectionGroup to test.
spec.cleanupPolicystringNoAutomatic to delete test resources when the test completes, or Manual to retain them for inspection.
spec.retentionTimestringNoDuration to retain test resources before automatic cleanup (for example, 2h). Ignored when cleanupPolicy: Manual.
spec.timeoutstringNoMaximum duration for the entire test failover operation before it is marked Failed.
spec.batchBootTimeoutSecondsintegerNoMaximum seconds to wait for each batch of test VMs to reach Running state.
spec.verificationobjectNoVerification checks to run against test VMs after they start.

ReplicationGroupStatus

ReplicationGroupStatus (API group siterecovery.trilio.io/v1alpha1) configures how replication health is monitored and when RPO violations are recorded.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the ProtectionGroup being monitored.
spec.replicationProtocolstringNoMatches the DRBD protocol in use (C or A). Used to contextualise health reporting.
spec.rpoObjectiveSecondsintegerNoRPO target in seconds. Violations generate RPOEvent CRs.
spec.pollingIntervalSecondsintegerNoHow often the replication monitor checks volume sync state. Lower values increase monitoring granularity at the cost of additional API calls.

Usage

Once installation is complete and the ProtectionZone is accepted, your day-to-day workflow follows a consistent pattern: define replication policy → protect VMs → verify replication health → run test failovers → execute failovers when needed. All operations are performed with oc apply or oc get against Custom Resources.


Protect a virtual machine

Create a ProtectionRequest on the quorum cluster to bring a VM under block-level replication protection. The protection controller validates the VM, creates a DRBDResource pair (one per cluster side), and replaces the VM's PVCs with DRBD-backed frontend PVCs.

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-vm.yaml

Watch the protection lifecycle:

oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest <name> \
  -n dr-<deployment-name> -o yaml

The status.phase field progresses from PendingValidatingProvisioningDRBDResourceProtected.


Verify replication health

Check the DRBDResource CRs on both clusters to confirm replication is active and volumes are in sync:

# Primary cluster
oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A

# DR cluster
oc --kubeconfig $KUBECONFIG_DR get drbdresource -A

Check ReplicationGroupStatus for an aggregated view:

oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -A \
  -n dr-<deployment-name>

A healthy deployment shows status.health: Healthy. Degraded or Critical statuses indicate a replication problem—check for RPOEvent CRs:

oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -A \
  -n dr-<deployment-name>

Run a test failover

A TestFailover validates DR readiness without impacting production VMs. The test-failover controller creates volume snapshots, provisions test PVCs, starts test VMs in an isolated namespace, runs verification checks, and cleans up afterward.

oc --kubeconfig $KUBECONFIG_QUORUM apply -f test-failover.yaml

# Monitor progress
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover <name> \
  -n dr-<deployment-name> --watch

Phases: PendingCreatingSnapshotsCreatingVolumesCreatingVMsVerifyingDataSucceededCleaningUpCleaned

To inspect test VMs before cleanup (when cleanupPolicy: Manual):

oc --kubeconfig $KUBECONFIG_DR get vm -n <testNamespace>

Delete the TestFailover CR to trigger cleanup:

oc --kubeconfig $KUBECONFIG_QUORUM delete testfailover <name> \
  -n dr-<deployment-name>

Execute a planned failover

A planned failover gracefully shuts down VMs on the source cluster, promotes DRBD volumes on the target cluster, and starts VMs there. Use this for scheduled maintenance or DR drills.

oc --kubeconfig $KUBECONFIG_QUORUM apply -f failover-planned.yaml

# Watch progress
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest <name> \
  -n dr-<deployment-name> --watch

The status.phase field progresses from PendingInProgressCompleted.


Execute an unplanned failover

An unplanned failover force-promotes DRBD volumes on the DR cluster without waiting for the primary to shut down. Use this only when the primary cluster is unreachable.

Set spec.failoverType: unplanned in the FailoverRequest spec:

oc --kubeconfig $KUBECONFIG_QUORUM apply -f failover-unplanned.yaml

Warning: If Protocol A (asynchronous) replication was in use, some writes that had not yet been shipped to the DR cluster at the time of the disaster will be lost. This is expected behavior for Protocol A.


Monitor all active operations

# All protection requests
oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest -A

# All failover requests
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest -A

# All test failovers
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover -A

Examples

The examples below are complete and runnable. Replace placeholder values (angle-bracket tokens) with values from your environment before applying.


Example 1 — DRBDReplicationPolicy for synchronous replication

This policy configures Protocol C (synchronous, RPO=0) replication between two clusters. Apply it on the quorum cluster before protecting any VMs.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: prod-sync-policy
  namespace: dr-prod
spec:
  drbdProtocol: C
  isDefault: true
  clusters:
    - name: primary
    - name: dr
  networkConfig:
    primaryEndpoint: "<primary-worker-ip>:7000"
    drEndpoint: "<dr-worker-ip>:7000"
  diskConfig:
    storageClassMappings:
      - primaryStorageClass: ocs-storagecluster-ceph-rbd
        drStorageClass: ocs-storagecluster-ceph-rbd
  rpo:
    objectiveSeconds: 0
oc --kubeconfig $KUBECONFIG_QUORUM apply -f prod-sync-policy.yaml

Expected result: The policy CR is accepted by the ProtectionZone webhook and stored. The status section will be populated once the first ProtectionRequest references this policy.


Example 2 — ProtectionRequest for a single VM

Protect a VM named db-primary running in the production namespace on the primary cluster.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-db-primary
  namespace: dr-prod
spec:
  sourceCluster: primary
  virtualMachine:
    name: db-primary
    namespace: production
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-db-primary.yaml

Watch the protection controller process the request:

oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-db-primary \
  -n dr-prod -o jsonpath='{.status.phase}' --watch

Expected output (progression):

Pending
Validating
ProvisioningDRBDResource
Protected

Once the phase is Protected, verify that a DRBDResource exists on each cluster:

oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -n production
oc --kubeconfig $KUBECONFIG_DR get drbdresource -n production

Example 3 — TestFailover with manual cleanup

Run a non-disruptive DR validation for all VMs in the app-group ProtectionGroup. Test VMs will be started in the dr-test-20240101 namespace on the DR cluster and retained for two hours for manual inspection.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: quarterly-dr-test
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: app-group
    namespace: dr-prod
  cleanupPolicy: Manual
  retentionTime: 2h
  batchBootTimeoutSeconds: 300
oc --kubeconfig $KUBECONFIG_QUORUM apply -f quarterly-dr-test.yaml

# Monitor phase transitions
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover quarterly-dr-test \
  -n dr-prod --watch

Expected output (progression):

NAME                  PHASE              AGE
quarterly-dr-test     Pending            0s
quarterly-dr-test     CreatingSnapshots  5s
quarterly-dr-test     CreatingVolumes    30s
quarterly-dr-test     CreatingVMs        60s
quarterly-dr-test     VerifyingData      90s
quarterly-dr-test     Succeeded          120s

Inspect test VMs before cleanup:

oc --kubeconfig $KUBECONFIG_DR get vm -n dr-test-20240101

When satisfied, delete the TestFailover CR to trigger cleanup:

oc --kubeconfig $KUBECONFIG_QUORUM delete testfailover quarterly-dr-test \
  -n dr-prod

Example 4 — Planned FailoverRequest

Fail over all VMs in the app-group ProtectionGroup to the DR cluster with a graceful shutdown on the primary side.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-app-group
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: app-group
    namespace: dr-prod
  targetCluster: dr
  failoverType: planned
  drainTimeoutSeconds: 120
  batchBootTimeoutSeconds: 300
oc --kubeconfig $KUBECONFIG_QUORUM apply -f planned-failover-app-group.yaml

# Watch failover progress
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest planned-failover-app-group \
  -n dr-prod --watch

Expected output:

NAME                          PHASE        AGE
planned-failover-app-group    Pending      0s
planned-failover-app-group    InProgress   10s
planned-failover-app-group    Completed    4m32s

Example 5 — Collecting a support bundle with tsr-gather

Collect a full diagnostic bundle from all clusters in the DR estate. Run this on the quorum cluster.

oc adm must-gather \
  --image=$DOCKER_REGISTRY/site-recovery-must-gather:<tag> \
  --kubeconfig $KUBECONFIG_QUORUM

To limit the log window to the last two hours (useful immediately after an incident):

oc adm must-gather \
  --image=$DOCKER_REGISTRY/site-recovery-must-gather:<tag> \
  --kubeconfig $KUBECONFIG_QUORUM \
  -- /usr/bin/gather --since 2h

The bundle lands in a timestamped directory in the current working directory. Start with SUMMARY.md for a triage overview, then manifest.yaml for a full inventory of what was collected. The bundle never contains secrets, kubeconfigs, cloud-init payloads, or DRBD shared secrets.


Troubleshooting

Use the consistent format below for each issue: Symptom, Likely cause, Fix.


Issue 1 — ProtectionRequest stays in Pending phase

Symptom: A ProtectionRequest CR has been applied on the quorum cluster but its status.phase does not advance beyond Pending after several minutes.

Likely cause: The protection controller pod in the dr-<deployment-name> namespace is not running, or the quorum cluster cannot reach the primary cluster's Kubernetes API server on TCP port 6443.

Fix:

  1. Check the controller manager pod status:
    oc --kubeconfig $KUBECONFIG_QUORUM get pods \
      -n dr-<deployment-name>
    
  2. If the pod is not Running, inspect its logs:
    oc --kubeconfig $KUBECONFIG_QUORUM logs \
      -n dr-<deployment-name> \
      -l app.kubernetes.io/name=site-recovery-quorum-control-plane
    
  3. Verify API server connectivity from the quorum cluster to the primary API server:
    oc --kubeconfig $KUBECONFIG_QUORUM exec -it <controller-pod> \
      -n dr-<deployment-name> -- curl -k https://<primary-api-server>:6443/healthz
    
  4. If connectivity fails, check firewall rules for TCP 6443 between the quorum cluster and the primary API server.

Issue 2 — DRBD replication link is StandAlone or Connecting

Symptom: The DRBDResource CR on the primary or DR cluster shows a connection state of StandAlone or Connecting rather than Connected.

Likely cause: TCP ports 7000–7999 are not open between the primary and DR worker nodes, or the replication endpoints in the DRBDReplicationPolicy are incorrect.

Fix:

  1. Verify port connectivity from a primary worker node to a DR worker node:
    # Run from a primary worker node
    nc -zv <dr-worker-ip> 7000
    
  2. If the connection is refused or times out, open TCP 7000–7999 bidirectionally between primary and DR worker nodes in your firewall or security group rules.
  3. Confirm the networkConfig endpoints in your DRBDReplicationPolicy match the actual node IP addresses:
    oc --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy <name> \
      -n dr-<deployment-name> -o jsonpath='{.spec.networkConfig}'
    
  4. Check the drbd-node-agent DaemonSet logs on the affected node for connection error details:
    oc --kubeconfig $KUBECONFIG_PRIMARY logs -n trilio-site-recovery-system \
      -l app.kubernetes.io/name=drbd-node-agent \
      --field-selector spec.nodeName=<node-name>
    

Issue 3 — FailoverRequest stuck in InProgress

Symptom: A FailoverRequest remains in the InProgress phase beyond the expected RTO window (3–8 minutes).

Likely cause: VM shutdown on the source cluster is taking longer than spec.drainTimeoutSeconds, or VMs on the target cluster are failing to start within spec.batchBootTimeoutSeconds.

Fix:

  1. Check the failover controller logs for the specific blocking step:
    oc --kubeconfig $KUBECONFIG_QUORUM logs \
      -n dr-<deployment-name> \
      -l app.kubernetes.io/name=site-recovery-quorum-control-plane
    
  2. Check VM status on the target cluster:
    oc --kubeconfig $KUBECONFIG_DR get vm -A
    
  3. If VMs are not starting, inspect KubeVirt events on the DR cluster:
    oc --kubeconfig $KUBECONFIG_DR get events -A \
      --field-selector reason=FailedCreate
    
  4. For unplanned failovers where the primary is unreachable, confirm that spec.failoverType: unplanned is set—planned failovers will wait indefinitely for source VM shutdown if the primary is down.
  5. Collect a full diagnostic bundle and inspect the SUMMARY.md:
    oc adm must-gather \
      --image=$DOCKER_REGISTRY/site-recovery-must-gather:<tag> \
      --kubeconfig $KUBECONFIG_QUORUM \
      -- /usr/bin/gather --since 1h
    

Issue 4 — TestFailover phase is Failed

Symptom: A TestFailover CR shows status.phase: Failed.

Likely cause: Volume snapshots could not be created (missing VolumeSnapshotClass on the DR cluster), test VMs did not start within the boot timeout, or verification checks failed.

Fix:

  1. Check the TestFailover status for a failure message:
    oc --kubeconfig $KUBECONFIG_QUORUM get testfailover <name> \
      -n dr-<deployment-name> -o yaml
    
    Review status.message and status.conditions for the root cause.
  2. Verify that a VolumeSnapshotClass exists on the DR cluster:
    oc --kubeconfig $KUBECONFIG_DR get volumesnapshotclass
    
  3. Check whether test VMs were created in the test namespace:
    oc --kubeconfig $KUBECONFIG_DR get vm -n <testNamespace>
    
  4. If test VMs exist but are not running, inspect KubeVirt events in the test namespace:
    oc --kubeconfig $KUBECONFIG_DR get events -n <testNamespace>
    
  5. After resolving the root cause, delete the failed TestFailover CR and reapply it. The controller will clean up any partial test resources before starting a new test run.

Issue 5 — RPOEvent CRs are being created

Symptom: RPOEvent CRs appear in the deployment namespace on the quorum cluster, indicating replication lag violations.

Likely cause: Network congestion or high write throughput on protected VMs is causing the DRBD replication buffer to grow faster than the link can drain it, or the link latency has increased beyond the Protocol C threshold.

Fix:

  1. Inspect the RPOEvent details:
    oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -A \
      -n dr-<deployment-name> -o yaml
    
    Note the spec.rpoAtEvent (lag in seconds), spec.violationReason, and spec.affectedVolumes.
  2. Measure current round-trip latency between the affected primary and DR worker nodes:
    ping -c 30 <dr-worker-ip>
    
  3. If latency has increased above 50 ms and you are using Protocol C, consider switching the DRBDReplicationPolicy to drbdProtocol: A. This changes the RPO guarantee from zero to seconds but prevents write stalls.
  4. If latency is within acceptable bounds, the lag may be caused by a burst of write activity. Check spec.outOfSyncBytesAtEvent to gauge whether this is a transient burst or a sustained throughput issue.
  5. Check the drbd-node-agent DaemonSet logs on the affected nodes for I/O error or buffer-full messages:
    oc --kubeconfig $KUBECONFIG_PRIMARY logs -n trilio-site-recovery-system \
      -l app.kubernetes.io/name=drbd-node-agent