Prerequisites
Kubernetes version requirements, required permissions, storage dependencies, and network prerequisites
This page describes every requirement you must satisfy before installing Trilio Site Recovery on OpenShift. Meeting these requirements ensures that the DRBD kernel module can replicate VM-attached PVCs between clusters, that the quorum control plane can reach both workload clusters over the Kubernetes API, and that Helm can deploy the three charts that make up the management and workload planes. Review each section carefully—missing a single networking or storage requirement is the most common cause of failed installations and degraded replication.
Collect and verify every item below before you run any helm install or kubectl apply command.
OpenShift version
| Cluster role | Required version |
|---|---|
| Primary cluster | OpenShift ≥ 4.14 |
| DR cluster | OpenShift ≥ 4.14 |
| Quorum cluster | OpenShift ≥ 4.14 |
Required cluster components
Install these on the primary and DR clusters before deploying Site Recovery. The quorum cluster does not run VM workloads and does not require KubeVirt or the DRBD kernel module.
| Component | Minimum version | Cluster(s) |
|---|---|---|
| OpenShift Virtualization (CNV) | 1.0 | Primary, DR |
| DRBD kernel module | 9.x | Every worker node on Primary and DR |
| DRBD Operator | current | Primary, DR |
| Containerized Data Importer (CDI) | current | Primary, DR |
| Multus CNI | current | Primary, DR (if using multiple network interfaces for replication) |
Note: The DRBD kernel module must be loaded on every worker node that will host VM workloads. Confirm the module is present with
lsmod | grep drbdon each node before proceeding.
Client tooling
You need the following tools on the machine from which you run installation commands:
helm≥ 3.0ocCLI (OpenShift client) compatible with your cluster versionkubectlcompatible with your cluster version- Valid kubeconfig files for all clusters (see Kubeconfig setup below)
Kubeconfig setup
You will switch kubeconfig context frequently. Export a named variable for each cluster kubeconfig before you begin:
export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr
export KUBECONFIG_QUORUM=~/.kube/config-quorum
Verify API server access for all three clusters:
oc --kubeconfig $KUBECONFIG_PRIMARY get nodes
oc --kubeconfig $KUBECONFIG_DR get nodes
oc --kubeconfig $KUBECONFIG_QUORUM get nodes
Network requirements
All port and latency requirements must be met before you apply any CRDs or start replication. Firewall rules that are added after DRBD resources are created will not cause an error at creation time but will cause the replication link to remain in a StandAlone or Connecting state.
| Traffic | Protocol | Port range | Direction | Required for |
|---|---|---|---|---|
| DRBD block replication | TCP | 7000–7999 | Primary worker nodes ↔ DR worker nodes | All deployments |
| Kubernetes API access | TCP | 6443 | Quorum cluster → Primary and DR API servers | All deployments |
Protocol C (synchronous replication) latency requirement
If you configure drbdProtocol: C (synchronous replication, RPO=0), the round-trip latency between primary and DR worker nodes must be below 50 ms. Exceeding this threshold causes write acknowledgement delays that will degrade application performance. Measure latency before installation:
# Run from a primary worker node toward a DR worker node
ping -c 100 <dr-worker-node-ip>
If latency consistently exceeds 50 ms, use drbdProtocol: A (asynchronous replication) instead, which accepts an RPO of seconds.
Storage requirements
Each worker node on the primary and DR clusters that will host protected VMs must have:
- An LVM thin pool configured on a dedicated block device. This thin pool is used by DRBD-backed frontend PVCs that replace a VM's original PVCs after protection is applied.
- Sufficient raw block capacity to hold all DRBD volumes that will be placed on that node.
Verify that the thin pool exists on each worker node before installation:
# Run on each worker node
lvs --select 'lv_attr=~t' -o lv_name,lv_size,pool_lv
Permissions and RBAC
The Helm charts create the necessary ClusterRole, ClusterRoleBinding, Role, and RoleBinding objects during installation. The identity used to run helm install must have sufficient permissions to create these objects. On OpenShift, use a service account or user with the cluster-admin role for the initial installation:
# Verify your current identity has cluster-admin on the quorum cluster
oc --kubeconfig $KUBECONFIG_QUORUM auth can-i create clusterroles --all-namespaces
The expected response is yes. Repeat this check for the primary and DR clusters.
Installation follows a fixed order across three phases. Deviating from this order—for example, deploying the quorum control plane before the ProtectionZone webhook—will cause admission webhook failures at CRD creation time.
Phase order:
- Prepare storage and network on primary and DR clusters
- Deploy the ProtectionZone webhook and CRDs on the quorum cluster
- Deploy the quorum control plane on the quorum cluster
- Deploy the workload control plane on the primary and DR clusters
Step 1 — Verify DRBD kernel module on worker nodes
Confirm the DRBD 9.x kernel module is loaded on every worker node of the primary and DR clusters before installing any Helm chart.
# Run on each worker node (example using oc debug)
oc --kubeconfig $KUBECONFIG_PRIMARY debug node/<node-name> -- chroot /host lsmod | grep drbd
oc --kubeconfig $KUBECONFIG_DR debug node/<node-name> -- chroot /host lsmod | grep drbd
Expected output includes a line such as:
drbd 548864 0
If the module is not loaded, consult your DRBD kernel module installation documentation before continuing.
Step 2 — Deploy the ProtectionZone webhook on the quorum cluster
The site-recovery-protectionzone-controller chart installs the ProtectionZone admission webhook and its CRDs. It must be installed before the quorum control plane chart because the quorum control plane chart depends on the ProtectionZone CRD being present in the cluster.
helm install site-recovery-protectionzone \
./helm/site-recovery-protectionzone-controller \
--kubeconfig $KUBECONFIG_QUORUM \
--namespace trilio-site-recovery-system \
--create-namespace
Wait for the webhook pod to reach Running status before proceeding:
oc --kubeconfig $KUBECONFIG_QUORUM get pods \
-n trilio-site-recovery-system \
-l app.kubernetes.io/name=site-recovery-protectionzone-controller \
--watch
Step 3 — Create the DR deployment namespace on the quorum cluster
Each DR deployment is isolated in its own namespace named dr-<name> on the quorum cluster. Create the namespace before deploying the quorum control plane into it. Replace <deployment-name> with your chosen name (for example, prod).
oc --kubeconfig $KUBECONFIG_QUORUM create namespace dr-<deployment-name>
Step 4 — Deploy the quorum control plane
The site-recovery-quorum-control-plane chart deploys the failover controller, protection controller, pg-sync controller, and replication-monitor reconcilers as a single controller manager deployment in the DR deployment namespace.
helm install dr-<deployment-name> \
./helm/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
--namespace dr-<deployment-name>
Verify that the controller manager pod is running:
oc --kubeconfig $KUBECONFIG_QUORUM get pods \
-n dr-<deployment-name> \
--watch
All pods should reach Running status before you proceed.
Step 5 — Deploy the workload control plane on the primary cluster
The site-recovery-workload-control-plane chart installs the drbd-node-agent DaemonSet, admission webhooks, and supporting RBAC on the workload cluster. Deploy it to both the primary and DR clusters.
# Primary cluster
helm install dr-<deployment-name> \
./helm/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_PRIMARY \
--namespace trilio-site-recovery-system \
--create-namespace
Verify that the DaemonSet pod is running on every worker node:
oc --kubeconfig $KUBECONFIG_PRIMARY get daemonset \
-n trilio-site-recovery-system
The DESIRED and READY counts must match before you proceed.
Step 6 — Deploy the workload control plane on the DR cluster
# DR cluster
helm install dr-<deployment-name> \
./helm/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_DR \
--namespace trilio-site-recovery-system \
--create-namespace
Repeat the DaemonSet verification on the DR cluster:
oc --kubeconfig $KUBECONFIG_DR get daemonset \
-n trilio-site-recovery-system
Step 7 — Create the ProtectionZone custom resource
A ProtectionZone CR defines the logical DR topology: which clusters participate, which storage backend mode is in use, and display metadata. Apply it on the quorum cluster after all three Helm charts are running.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: <deployment-name>
namespace: dr-<deployment-name>
spec:
displayName: "<Human-readable name>"
storageBackendMode: drbd-operator
clusters:
- name: primary
kubeConfigSecretRef:
name: primary-kubeconfig
- name: dr
kubeConfigSecretRef:
name: dr-kubeconfig
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionzone.yaml
Verify the ProtectionZone status:
oc --kubeconfig $KUBECONFIG_QUORUM get protectionzone <deployment-name> \
-n dr-<deployment-name> -o yaml
With all three charts running and the ProtectionZone accepted by the webhook, your installation is complete. Proceed to configure replication and protect your first VM.
All Site Recovery configuration is expressed as Kubernetes Custom Resource specifications. The sections below document the key fields for each CR that affects deployment behavior. Apply every CR with oc --kubeconfig $KUBECONFIG_QUORUM apply -f <file> unless stated otherwise.
ProtectionZone
ProtectionZone (API group siterecovery.trilio.io/v1alpha1) defines the overall DR topology for a deployment namespace. It is the first CR you create after installation.
| Field | Type | Required | Description |
|---|---|---|---|
spec.displayName | string | Yes | Human-readable name shown in the TSR web console. |
spec.storageBackendMode | string | Yes | Storage backend in use. For DRBD Operator deployments, set drbd-operator. |
spec.clusters | array | Yes | List of cluster references (primary and DR). Each entry identifies a cluster and references a Secret containing its kubeconfig. |
spec.description | string | No | Free-text description of this deployment. |
spec.logging | object | No | Logging configuration for this ProtectionZone. |
DRBDReplicationPolicy
DRBDReplicationPolicy (API group siterecovery.trilio.io/v1alpha1) defines how block replication operates between the primary and DR clusters. One policy applies to all VMs protected under a deployment, unless per-VM overrides are configured.
| Field | Type | Required | Description |
|---|---|---|---|
spec.clusters | array | Yes | References to the primary and DR cluster entries defined in ProtectionZone. |
spec.drbdProtocol | string | No | C for synchronous (RPO=0, requires <50 ms RTT) or A for asynchronous (seconds RPO). Defaults to C. |
spec.replicationMode | string | No | Replication mode. Consult the DRBD Operator documentation for valid values. |
spec.diskConfig | object | No | Disk configuration for DRBD volumes, including storage class mappings between primary and DR. |
spec.networkConfig | object | No | Network configuration for DRBD replication endpoints (IP addresses and ports in the 7000–7999 range). |
spec.rpo | object | No | RPO thresholds that trigger RPOEvent CRs when violated. |
spec.resyncConfig | object | No | Tuning parameters for DRBD resynchronization speed and concurrency. |
spec.isDefault | boolean | No | When true, this policy is used for ProtectionRequests that do not reference a named policy. |
Protocol choice guidance:
- Use
drbdProtocol: Cwhen your primary-to-DR round-trip latency is consistently below 50 ms. This guarantees RPO=0 because writes are only acknowledged after being committed on both sides. - Use
drbdProtocol: Afor geographically distant clusters where latency exceeds 50 ms. Writes complete after local disk commit and are shipped asynchronously, resulting in an RPO of seconds.
ProtectionRequest
ProtectionRequest (API group siterecovery.trilio.io/v1alpha1) requests block-level replication protection for a single VM in DRBD Operator deployments. The protection controller on the quorum cluster processes it.
| Field | Type | Required | Description |
|---|---|---|---|
spec.virtualMachine | object | Yes | Reference to the VirtualMachine CR to protect (name and namespace). |
spec.sourceCluster | string | Yes | Name of the cluster where the VM currently runs. Must match a cluster entry in the ProtectionZone. |
spec.replicationConfig | object | No | Optional per-VM replication overrides. If omitted, the default DRBDReplicationPolicy for the deployment is used. |
FailoverRequest
FailoverRequest (API group siterecovery.trilio.io/v1alpha1) triggers a failover operation. Apply it on the quorum cluster.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the ProtectionGroup that defines the set of VMs to fail over. |
spec.targetCluster | string | Yes | Name of the cluster where VMs should start after failover. |
spec.failoverType | string | No | planned (graceful VM shutdown before promotion) or unplanned (force-promote without waiting for source). Defaults to planned. |
spec.drainTimeoutSeconds | integer | No | Maximum seconds to wait for VM shutdown on the source cluster during a planned failover. |
spec.batchBootTimeoutSeconds | integer | No | Maximum seconds to wait for each batch of VMs to reach Running state on the target cluster. |
TestFailover
TestFailover (API group siterecovery.trilio.io/v1alpha1) runs a non-disruptive DR validation. Apply it on the quorum cluster.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the ProtectionGroup to test. |
spec.cleanupPolicy | string | No | Automatic to delete test resources when the test completes, or Manual to retain them for inspection. |
spec.retentionTime | string | No | Duration to retain test resources before automatic cleanup (for example, 2h). Ignored when cleanupPolicy: Manual. |
spec.timeout | string | No | Maximum duration for the entire test failover operation before it is marked Failed. |
spec.batchBootTimeoutSeconds | integer | No | Maximum seconds to wait for each batch of test VMs to reach Running state. |
spec.verification | object | No | Verification checks to run against test VMs after they start. |
ReplicationGroupStatus
ReplicationGroupStatus (API group siterecovery.trilio.io/v1alpha1) configures how replication health is monitored and when RPO violations are recorded.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the ProtectionGroup being monitored. |
spec.replicationProtocol | string | No | Matches the DRBD protocol in use (C or A). Used to contextualise health reporting. |
spec.rpoObjectiveSeconds | integer | No | RPO target in seconds. Violations generate RPOEvent CRs. |
spec.pollingIntervalSeconds | integer | No | How often the replication monitor checks volume sync state. Lower values increase monitoring granularity at the cost of additional API calls. |
Once installation is complete and the ProtectionZone is accepted, your day-to-day workflow follows a consistent pattern: define replication policy → protect VMs → verify replication health → run test failovers → execute failovers when needed. All operations are performed with oc apply or oc get against Custom Resources.
Protect a virtual machine
Create a ProtectionRequest on the quorum cluster to bring a VM under block-level replication protection. The protection controller validates the VM, creates a DRBDResource pair (one per cluster side), and replaces the VM's PVCs with DRBD-backed frontend PVCs.
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-vm.yaml
Watch the protection lifecycle:
oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest <name> \
-n dr-<deployment-name> -o yaml
The status.phase field progresses from Pending → Validating → ProvisioningDRBDResource → Protected.
Verify replication health
Check the DRBDResource CRs on both clusters to confirm replication is active and volumes are in sync:
# Primary cluster
oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A
# DR cluster
oc --kubeconfig $KUBECONFIG_DR get drbdresource -A
Check ReplicationGroupStatus for an aggregated view:
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -A \
-n dr-<deployment-name>
A healthy deployment shows status.health: Healthy. Degraded or Critical statuses indicate a replication problem—check for RPOEvent CRs:
oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -A \
-n dr-<deployment-name>
Run a test failover
A TestFailover validates DR readiness without impacting production VMs. The test-failover controller creates volume snapshots, provisions test PVCs, starts test VMs in an isolated namespace, runs verification checks, and cleans up afterward.
oc --kubeconfig $KUBECONFIG_QUORUM apply -f test-failover.yaml
# Monitor progress
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover <name> \
-n dr-<deployment-name> --watch
Phases: Pending → CreatingSnapshots → CreatingVolumes → CreatingVMs → VerifyingData → Succeeded → CleaningUp → Cleaned
To inspect test VMs before cleanup (when cleanupPolicy: Manual):
oc --kubeconfig $KUBECONFIG_DR get vm -n <testNamespace>
Delete the TestFailover CR to trigger cleanup:
oc --kubeconfig $KUBECONFIG_QUORUM delete testfailover <name> \
-n dr-<deployment-name>
Execute a planned failover
A planned failover gracefully shuts down VMs on the source cluster, promotes DRBD volumes on the target cluster, and starts VMs there. Use this for scheduled maintenance or DR drills.
oc --kubeconfig $KUBECONFIG_QUORUM apply -f failover-planned.yaml
# Watch progress
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest <name> \
-n dr-<deployment-name> --watch
The status.phase field progresses from Pending → InProgress → Completed.
Execute an unplanned failover
An unplanned failover force-promotes DRBD volumes on the DR cluster without waiting for the primary to shut down. Use this only when the primary cluster is unreachable.
Set spec.failoverType: unplanned in the FailoverRequest spec:
oc --kubeconfig $KUBECONFIG_QUORUM apply -f failover-unplanned.yaml
Warning: If Protocol A (asynchronous) replication was in use, some writes that had not yet been shipped to the DR cluster at the time of the disaster will be lost. This is expected behavior for Protocol A.
Monitor all active operations
# All protection requests
oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest -A
# All failover requests
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest -A
# All test failovers
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover -A
The examples below are complete and runnable. Replace placeholder values (angle-bracket tokens) with values from your environment before applying.
Example 1 — DRBDReplicationPolicy for synchronous replication
This policy configures Protocol C (synchronous, RPO=0) replication between two clusters. Apply it on the quorum cluster before protecting any VMs.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
name: prod-sync-policy
namespace: dr-prod
spec:
drbdProtocol: C
isDefault: true
clusters:
- name: primary
- name: dr
networkConfig:
primaryEndpoint: "<primary-worker-ip>:7000"
drEndpoint: "<dr-worker-ip>:7000"
diskConfig:
storageClassMappings:
- primaryStorageClass: ocs-storagecluster-ceph-rbd
drStorageClass: ocs-storagecluster-ceph-rbd
rpo:
objectiveSeconds: 0
oc --kubeconfig $KUBECONFIG_QUORUM apply -f prod-sync-policy.yaml
Expected result: The policy CR is accepted by the ProtectionZone webhook and stored. The status section will be populated once the first ProtectionRequest references this policy.
Example 2 — ProtectionRequest for a single VM
Protect a VM named db-primary running in the production namespace on the primary cluster.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: protect-db-primary
namespace: dr-prod
spec:
sourceCluster: primary
virtualMachine:
name: db-primary
namespace: production
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-db-primary.yaml
Watch the protection controller process the request:
oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-db-primary \
-n dr-prod -o jsonpath='{.status.phase}' --watch
Expected output (progression):
Pending
Validating
ProvisioningDRBDResource
Protected
Once the phase is Protected, verify that a DRBDResource exists on each cluster:
oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -n production
oc --kubeconfig $KUBECONFIG_DR get drbdresource -n production
Example 3 — TestFailover with manual cleanup
Run a non-disruptive DR validation for all VMs in the app-group ProtectionGroup. Test VMs will be started in the dr-test-20240101 namespace on the DR cluster and retained for two hours for manual inspection.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
name: quarterly-dr-test
namespace: dr-prod
spec:
protectionGroupRef:
name: app-group
namespace: dr-prod
cleanupPolicy: Manual
retentionTime: 2h
batchBootTimeoutSeconds: 300
oc --kubeconfig $KUBECONFIG_QUORUM apply -f quarterly-dr-test.yaml
# Monitor phase transitions
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover quarterly-dr-test \
-n dr-prod --watch
Expected output (progression):
NAME PHASE AGE
quarterly-dr-test Pending 0s
quarterly-dr-test CreatingSnapshots 5s
quarterly-dr-test CreatingVolumes 30s
quarterly-dr-test CreatingVMs 60s
quarterly-dr-test VerifyingData 90s
quarterly-dr-test Succeeded 120s
Inspect test VMs before cleanup:
oc --kubeconfig $KUBECONFIG_DR get vm -n dr-test-20240101
When satisfied, delete the TestFailover CR to trigger cleanup:
oc --kubeconfig $KUBECONFIG_QUORUM delete testfailover quarterly-dr-test \
-n dr-prod
Example 4 — Planned FailoverRequest
Fail over all VMs in the app-group ProtectionGroup to the DR cluster with a graceful shutdown on the primary side.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
name: planned-failover-app-group
namespace: dr-prod
spec:
protectionGroupRef:
name: app-group
namespace: dr-prod
targetCluster: dr
failoverType: planned
drainTimeoutSeconds: 120
batchBootTimeoutSeconds: 300
oc --kubeconfig $KUBECONFIG_QUORUM apply -f planned-failover-app-group.yaml
# Watch failover progress
oc --kubeconfig $KUBECONFIG_QUORUM get failoverrequest planned-failover-app-group \
-n dr-prod --watch
Expected output:
NAME PHASE AGE
planned-failover-app-group Pending 0s
planned-failover-app-group InProgress 10s
planned-failover-app-group Completed 4m32s
Example 5 — Collecting a support bundle with tsr-gather
Collect a full diagnostic bundle from all clusters in the DR estate. Run this on the quorum cluster.
oc adm must-gather \
--image=$DOCKER_REGISTRY/site-recovery-must-gather:<tag> \
--kubeconfig $KUBECONFIG_QUORUM
To limit the log window to the last two hours (useful immediately after an incident):
oc adm must-gather \
--image=$DOCKER_REGISTRY/site-recovery-must-gather:<tag> \
--kubeconfig $KUBECONFIG_QUORUM \
-- /usr/bin/gather --since 2h
The bundle lands in a timestamped directory in the current working directory. Start with SUMMARY.md for a triage overview, then manifest.yaml for a full inventory of what was collected. The bundle never contains secrets, kubeconfigs, cloud-init payloads, or DRBD shared secrets.
Use the consistent format below for each issue: Symptom, Likely cause, Fix.
Issue 1 — ProtectionRequest stays in Pending phase
Symptom: A ProtectionRequest CR has been applied on the quorum cluster but its status.phase does not advance beyond Pending after several minutes.
Likely cause: The protection controller pod in the dr-<deployment-name> namespace is not running, or the quorum cluster cannot reach the primary cluster's Kubernetes API server on TCP port 6443.
Fix:
- Check the controller manager pod status:
oc --kubeconfig $KUBECONFIG_QUORUM get pods \ -n dr-<deployment-name> - If the pod is not
Running, inspect its logs:oc --kubeconfig $KUBECONFIG_QUORUM logs \ -n dr-<deployment-name> \ -l app.kubernetes.io/name=site-recovery-quorum-control-plane - Verify API server connectivity from the quorum cluster to the primary API server:
oc --kubeconfig $KUBECONFIG_QUORUM exec -it <controller-pod> \ -n dr-<deployment-name> -- curl -k https://<primary-api-server>:6443/healthz - If connectivity fails, check firewall rules for TCP 6443 between the quorum cluster and the primary API server.
Issue 2 — DRBD replication link is StandAlone or Connecting
Symptom: The DRBDResource CR on the primary or DR cluster shows a connection state of StandAlone or Connecting rather than Connected.
Likely cause: TCP ports 7000–7999 are not open between the primary and DR worker nodes, or the replication endpoints in the DRBDReplicationPolicy are incorrect.
Fix:
- Verify port connectivity from a primary worker node to a DR worker node:
# Run from a primary worker node nc -zv <dr-worker-ip> 7000 - If the connection is refused or times out, open TCP 7000–7999 bidirectionally between primary and DR worker nodes in your firewall or security group rules.
- Confirm the
networkConfigendpoints in yourDRBDReplicationPolicymatch the actual node IP addresses:oc --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy <name> \ -n dr-<deployment-name> -o jsonpath='{.spec.networkConfig}' - Check the
drbd-node-agentDaemonSet logs on the affected node for connection error details:oc --kubeconfig $KUBECONFIG_PRIMARY logs -n trilio-site-recovery-system \ -l app.kubernetes.io/name=drbd-node-agent \ --field-selector spec.nodeName=<node-name>
Issue 3 — FailoverRequest stuck in InProgress
Symptom: A FailoverRequest remains in the InProgress phase beyond the expected RTO window (3–8 minutes).
Likely cause: VM shutdown on the source cluster is taking longer than spec.drainTimeoutSeconds, or VMs on the target cluster are failing to start within spec.batchBootTimeoutSeconds.
Fix:
- Check the failover controller logs for the specific blocking step:
oc --kubeconfig $KUBECONFIG_QUORUM logs \ -n dr-<deployment-name> \ -l app.kubernetes.io/name=site-recovery-quorum-control-plane - Check VM status on the target cluster:
oc --kubeconfig $KUBECONFIG_DR get vm -A - If VMs are not starting, inspect KubeVirt events on the DR cluster:
oc --kubeconfig $KUBECONFIG_DR get events -A \ --field-selector reason=FailedCreate - For unplanned failovers where the primary is unreachable, confirm that
spec.failoverType: unplannedis set—planned failovers will wait indefinitely for source VM shutdown if the primary is down. - Collect a full diagnostic bundle and inspect the
SUMMARY.md:oc adm must-gather \ --image=$DOCKER_REGISTRY/site-recovery-must-gather:<tag> \ --kubeconfig $KUBECONFIG_QUORUM \ -- /usr/bin/gather --since 1h
Issue 4 — TestFailover phase is Failed
Symptom: A TestFailover CR shows status.phase: Failed.
Likely cause: Volume snapshots could not be created (missing VolumeSnapshotClass on the DR cluster), test VMs did not start within the boot timeout, or verification checks failed.
Fix:
- Check the TestFailover status for a failure message:
Review
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover <name> \ -n dr-<deployment-name> -o yamlstatus.messageandstatus.conditionsfor the root cause. - Verify that a VolumeSnapshotClass exists on the DR cluster:
oc --kubeconfig $KUBECONFIG_DR get volumesnapshotclass - Check whether test VMs were created in the test namespace:
oc --kubeconfig $KUBECONFIG_DR get vm -n <testNamespace> - If test VMs exist but are not running, inspect KubeVirt events in the test namespace:
oc --kubeconfig $KUBECONFIG_DR get events -n <testNamespace> - After resolving the root cause, delete the failed TestFailover CR and reapply it. The controller will clean up any partial test resources before starting a new test run.
Issue 5 — RPOEvent CRs are being created
Symptom: RPOEvent CRs appear in the deployment namespace on the quorum cluster, indicating replication lag violations.
Likely cause: Network congestion or high write throughput on protected VMs is causing the DRBD replication buffer to grow faster than the link can drain it, or the link latency has increased beyond the Protocol C threshold.
Fix:
- Inspect the RPOEvent details:
Note the
oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -A \ -n dr-<deployment-name> -o yamlspec.rpoAtEvent(lag in seconds),spec.violationReason, andspec.affectedVolumes. - Measure current round-trip latency between the affected primary and DR worker nodes:
ping -c 30 <dr-worker-ip> - If latency has increased above 50 ms and you are using Protocol C, consider switching the
DRBDReplicationPolicytodrbdProtocol: A. This changes the RPO guarantee from zero to seconds but prevents write stalls. - If latency is within acceptable bounds, the lag may be caused by a burst of write activity. Check
spec.outOfSyncBytesAtEventto gauge whether this is a transient burst or a sustained throughput issue. - Check the
drbd-node-agentDaemonSet logs on the affected nodes for I/O error or buffer-full messages:oc --kubeconfig $KUBECONFIG_PRIMARY logs -n trilio-site-recovery-system \ -l app.kubernetes.io/name=drbd-node-agent