Problem Being Solved

Enterprise virtual machine workloads running on Kubernetes face a fundamental gap between what traditional disaster recovery tools can deliver and what modern applications require. This page explains the operational problem Trilio Site Recovery addresses: the combination of unacceptable data loss windows, slow recovery times, and operational friction that characterizes conventional VM backup and recovery approaches on Kubernetes, and why a storage-layer, Kubernetes-native approach closes that gap.

The gap between what VMs need and what Kubernetes DR tools deliver

KubeVirt and OpenShift Virtualization bring virtual machine workloads into Kubernetes, but they do not bring enterprise disaster recovery with them. The tools that already exist for Kubernetes workloads—volume snapshot schedulers, backup controllers, and cloud-provider replication integrations—were designed for stateless or lightly stateful applications. When applied to VMs, they produce recovery characteristics that most enterprise VM workloads cannot accept:

• RPO of 15–60 minutes. Snapshot-based backup tools protect a VM's storage by taking periodic snapshots. Any write that occurs between snapshots is lost if the primary cluster fails before the next snapshot completes. For a database VM processing hundreds of transactions per second, even a 15-minute window represents an enormous and often unacceptable quantity of lost work.
• RTO of 1–4 hours. Restoring a VM from a backup means copying potentially large volumes of data from a backup target to a DR cluster before the VM can start. This data movement is sequential, bounded by network throughput, and has to complete before recovery can even begin. A one-to-four-hour window before workloads are back online is the norm, not an edge case.

These are not implementation deficiencies that can be tuned away. They are structural consequences of the snapshot-and-restore model: RPO is set by the snapshot interval, and RTO is set by the volume of data that must be transferred at recovery time.

Why the storage layer is the right place to solve this

Trilio Site Recovery addresses these constraints by moving replication from the backup layer to the storage layer. Instead of periodically copying completed state, it continuously mirrors every write from the primary cluster to the DR cluster at the block device level, using DRBD's Protocol C synchronous replication.

With Protocol C synchronous replication, a write is acknowledged to the application only after it has been committed to disk on both the primary cluster and the DR cluster. The application never sees a write succeed unless that write has already been durably recorded on the DR side. The consequence is that RPO is always zero: at any moment, the DR cluster's storage reflects exactly the same state as the primary cluster's storage, with no lag and no data-loss window.

This is a fundamentally different model from backup. There is no snapshot interval, no accumulation of unflushed writes, and no bulk data transfer at recovery time. The DR cluster's storage is already current before the failure occurs.

The specific operational problem: VM workloads on Kubernetes

The storage-layer approach is not new—DRBD has been used for Linux high availability for decades. The problem Trilio Site Recovery solves is integrating that capability into Kubernetes in a way that is operationally manageable at scale.

Without an operator to automate it, running DRBD-backed VMs across two Kubernetes clusters requires:

• Manual DRBD resource configuration per VM per cluster, updated every time a VM is protected, unprotected, or its disk topology changes.
• Custom orchestration to coordinate VM shutdown on the primary cluster, DRBD volume promotion on the DR cluster, and VM startup—in the right order, with the right timing, across two clusters that cannot directly communicate with each other during a failure.
• Manual tracking of replication health across potentially hundreds of VMs, each with multiple disks, with no Kubernetes-native view of which volumes are synchronized and which are lagging.
• Custom logic to run non-disruptive DR drills, since testing failover without a real operator to create isolated copies and clean them up afterward is a manual, error-prone process.
• Custom RBAC, SCC, and admission webhook configuration to ensure that the privileged storage operations the node agent needs are granted correctly without over-privileging the rest of the control plane.

None of this is impossible, but it is a significant ongoing operational burden that grows with the number of protected VMs and clusters.

What Trilio Site Recovery automates

Trilio Site Recovery encodes this entire operational surface into Kubernetes Custom Resource Definitions. The controller reconciliation loops that watch those CRDs handle the orchestration automatically:

Protecting a VM creates a ProtectionRequest on the TSR-Manager cluster. The protection controller validates the VM, provisions DRBD resources on both workload clusters, and switches the VM to DRBD-backed storage—without application changes or VM downtime beyond the brief cutover.

Monitoring replication health is continuous and Kubernetes-native. A ReplicationGroupStatus object per protection group aggregates per-volume sync state, connection state, write latency, and out-of-sync byte counts into a single queryable object. RPOEvent records capture replication lag violations with timestamps and severity, providing an audit trail without requiring a separate monitoring system.

Running a DR drill creates a TestFailover object. The test-failover controller takes volume snapshots of the current DR state, provisions test PVCs and VMs in an isolated namespace, runs reachability and custom verification checks, and cleans up—all without touching production workloads on either cluster. The test namespace is generated by the controller and surfaced on status.testNamespace; you never need to construct or manage it.

Executing a planned failover creates a FailoverRequest with failoverType: planned. The failover controller coordinates VM shutdown on the source cluster, waits for DRBD volumes to reach a consistent state, promotes them on the target cluster, and starts the VMs there—within a five-minute wall-clock budget.

Executing an unplanned failover creates a FailoverRequest with failoverType: unplanned. Because the primary cluster may be unreachable, the failover controller force-promotes DRBD volumes on the DR cluster without waiting for the primary to shut down gracefully. If replication was healthy and consistent at the moment of failure, data loss is zero. If replication was degraded at that moment, data loss is bounded by the out-of-sync bytes recorded in ReplicationGroupStatus status at the time.

Because all of these operations are CRD-driven, they compose naturally with GitOps workflows, can be triggered by oc commands in scripts or CI pipelines, and leave a complete audit trail in Kubernetes event history and object status.

The three-cluster topology

The design requires three OpenShift clusters:

• Primary cluster (cluster1): Runs the active VM workloads. Hosts the DRBD node agent DaemonSet and the workload control plane.
• DR cluster (cluster2): Hosts standby VMs and DRBD replicas of the primary cluster's volumes. Also runs the DRBD node agent DaemonSet and workload control plane.
• TSR-Manager cluster: The management plane. Hosts the ProtectionZone controller, the unified manager control plane (failover, protection, pg-sync, replication-monitor, and test-failover reconcilers), and the TSR web console. Holds no VM workloads and no DRBD storage.

The TSR-Manager cluster is not optional. It provides the neutral arbitration point that makes planned and unplanned failover safe: because neither workload cluster controls the failover decision, split-brain scenarios where both clusters believe they are primary are prevented at the orchestration layer.

Scale boundaries

The storage-layer approach imposes one concrete scale constraint that shapes planning: DRBD allocates one TCP port per replicated VM volume from the range 7000–8000 inclusive. With one port per VM (for VMs with a single disk), this supports up to approximately 1,000 protected VMs per protection zone. VMs with multiple disks consume proportionally more ports, up to a maximum of 15 disks per VM. These ports must be open between the worker nodes of the two workload clusters; a firewall rule that stops at 7999 silently prevents replication on the volume allocated to port 8000.

Protected VMs are also pinned to one node per cluster for the lifetime of the protection. Live migration and cold migration of a protected VM are denied by admission webhook. Planning node capacity on both workload clusters to accommodate the pinned placement of the VMs you intend to protect is a prerequisite step, not an afterthought.

The following examples illustrate the CRD-driven model. All operations against the TSR-Manager cluster use the oc CLI or kubectl with the TSR-Manager kubeconfig; operations against a workload cluster use the appropriate workload kubeconfig.

Check replication health for all protection groups in a zone

# Run against the TSR-Manager cluster
oc get replicationgroupstatuses -n dr-<zone-name>

The kubectl get short name for ReplicationGroupStatus is rgs:

oc get rgs -n dr-<zone-name>

Printer columns show Protection-Group, RPO-Status, Out-Of-Sync bytes, Synced volume count, Connection state, and Last-Sync timestamp — all derived from live controller observations, not from a backup schedule.

List RPO violation events for a protection group

# Filter by the protection-group label; field selectors on spec.* are not supported on custom resources
oc get rpoevents -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name>

Initiate a test failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-q1
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: <pg-name>
    namespace: dr-<zone-name>
  cleanupPolicy: Automatic
  retentionTime: 2h
  schedulingCheck: Fail

oc apply -f drill-q1.yaml

# Monitor progress — phases: Queued, PreparingEnvironment, CreatingSnapshots,
# CreatingVolumes, CreatingVMs, StartingVMs, Verifying, Succeeded, CleaningUp, Cleaned
oc get testfailover drill-q1 -n dr-<zone-name> -w

The controller generates the test namespace and writes it to status.testNamespace. You do not need to specify or look up this value.

Initiate a planned failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-to-dr
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: <pg-name>
    namespace: dr-<zone-name>
  targetCluster: cluster2
  failoverType: planned

oc apply -f failover-to-dr.yaml

# Watch per-VM and overall progress
oc get failoverrequest failover-to-dr -n dr-<zone-name> -w

Per-VM detail is available at status.vmStatuses. The short name for FailoverRequest is fr.

• Deploy the workload control plane — how the DRBD node agent DaemonSet and workload-side admission webhooks are deployed automatically to both workload clusters by the ProtectionZone controller.
• Plan and verify infrastructure — network, node, and DRBD kernel module prerequisites that must be satisfied before protection can begin, including the TCP 7000–8000 port range requirement between workload cluster worker nodes.
• Configure cross-cluster replication — how the DRBDReplicationPolicy created automatically per zone defines storage class mappings and Protocol C synchronous replication parameters.
• Protect virtual machines — how to create a ProtectionRequest and track the protection phases from Pending through to Protected.
• Verify replication health — how to read ReplicationGroupStatus and RPOEvent objects to confirm that synchronous replication is healthy before a real failover is needed.
• Run a test failover — non-disruptive DR validation using TestFailover, including reachability probe configuration and scheduling checks.
• Execute a planned failover and Execute an unplanned failover — the two FailoverRequest paths, their coordination model, and the five-minute wall-clock budget.
• Diagnose a degraded or failed node or replication link — how to interpret DRBDResource connection and phase states, and when node loss requires manual finalizer intervention.