Problem Being Solved

Trilio Site Recovery (TSR) addresses a specific and costly gap in enterprise Kubernetes operations: protecting virtual machine workloads against site failure without the data loss and multi-hour recovery windows that traditional backup tools impose. This page explains the operational problem TSR solves, why existing approaches fall short for VM workloads on Kubernetes, and how TSR's architecture closes those gaps through storage-layer replication and Kubernetes-native automation.

The gap between Kubernetes and enterprise DR

Platform teams running virtual machines on KubeVirt or OpenShift Virtualization face a challenge that predates Kubernetes itself: how do you recover those workloads at a remote site when the primary site fails, with minimal data loss and in a predictable amount of time?

The answer matters in two hard numbers:

• RPO (Recovery Point Objective): how much data you can afford to lose, measured as the maximum age of the last recoverable write at the moment of failure.
• RTO (Recovery Time Objective): how long recovery takes from the moment you decide to fail over to the moment workloads are running at the DR site.

For many regulated workloads and business-critical applications, the acceptable window for both is close to zero.

Why traditional backup tools fall short

Conventional backup and snapshot tools were designed around the assumption that some data loss and some downtime are acceptable. A typical implementation takes periodic snapshots of VM disks, replicates them to a secondary site on a schedule, and requires you to restore from the most recent snapshot at recovery time.

This model imposes structural limits:

• RPO of 15–60 minutes — writes that arrived after the last snapshot are lost at failover. For a database committing transactions continuously, that can mean thousands of lost records.
• RTO of 1–4 hours — restoring large VM disk images across a WAN, hydrating volumes, and restarting workloads in dependency order takes time that compounds under pressure during an actual incident.
• No non-disruptive testing — verifying that your DR plan actually works requires a maintenance window and usually disrupts the production workload.
• Manual orchestration — the sequence of steps to bring up a coherent set of VMs at the DR site (promote storage, start VMs in the right order, verify connectivity) is typically documented in runbooks and executed by hand.

These limitations are not bugs to be fixed; they are properties of the architecture. Periodic replication cannot close the RPO gap, and manual recovery procedures cannot compress the RTO gap, regardless of how carefully you tune them.

What TSR does differently

TSR operates at the storage layer, below the application and below the hypervisor. Rather than taking snapshots, it uses DRBD synchronous block replication to mirror every write to VM-attached PersistentVolumeClaims from the primary cluster to the DR cluster in real time.

Synchronous replication and RPO=0

DRBD Protocol C, the only replication mode TSR supports, requires that a write be committed to disk on both the primary and DR cluster before it is acknowledged to the application. From the VM's perspective, a write is not complete until it has landed in two places. This means that at any point in time, the DR copy is byte-for-byte identical to the primary copy — there is no window of data that exists only on the primary.

The practical consequence: if the primary site fails at 14:37:52, the DR site holds a complete and consistent copy of every write that had been acknowledged up to 14:37:52. RPO is zero by construction, not by tuning.

This guarantee has a physical constraint: Protocol C requires round-trip network latency below approximately 50ms between the primary and DR cluster worker nodes. At higher latency, the write acknowledgement delay becomes visible to applications and throughput degrades. TSR is designed for metropolitan or regional DR pairs, not for intercontinental replication.

Automated failover in minutes

Because replication is continuous and the DR copy is always current, there is no restore step at failover time. The failover-controller promotes the DRBD volumes on the DR cluster to primary role, starts the standby VMs in the correct boot order, and brings the workloads online. Planned failovers — where the primary site is gracefully quiesced first — complete within the 5-minute wall-clock budget TSR enforces.

For unplanned failovers, where the primary cluster is unreachable and there is no time for graceful shutdown, the DR cluster force-promotes its volumes and starts VMs without waiting for the primary. If replication was Healthy and Consistent at the moment of failure, data loss is still zero. If replication had already degraded before the failure, the loss is bounded by the lag that existed at the time.

Kubernetes-native automation

Every DR operation — protecting a VM, monitoring replication health, running a test failover, triggering a planned failover — is expressed as a Kubernetes Custom Resource in the siterecovery.trilio.io/v1alpha1 API group. There are no separate DR consoles to log into, no agents to invoke manually, and no runbooks to execute step by step during an incident.

This CRD-driven model has several consequences that matter in practice:

• GitOps integration — protection policies and failover configurations are Kubernetes objects that can be stored in version control, reviewed in pull requests, and applied through your existing CI/CD pipeline.
• kubectl-based operations — operators already familiar with Kubernetes can inspect replication health, check failover status, and trigger recovery operations using kubectl get, kubectl describe, and kubectl apply.
• Automated state management — the TSR controllers continuously reconcile the desired state expressed in your CRs against actual DRBD and VM state. If a replication link degrades, the ReplicationGroupStatus and RPOEvent CRs reflect it immediately. You do not need to poll a separate system.
• No application changes — TSR operates on PVCs at the storage layer. Your VMs are unaware of replication and require no modification.

Non-disruptive DR validation

TSR includes a test failover mechanism that creates volume snapshots of the replicated PVCs, provisions temporary test PVCs from those snapshots, starts test VMs in an isolated namespace, runs reachability and verification checks, and then cleans up — all without touching the production workload or interrupting replication. You can validate your DR capability on a schedule, after infrastructure changes, or before a planned maintenance window, without coordinating a maintenance window of your own.

The three-cluster model

TSR requires three clusters:

• Primary cluster — where your VMs run normally. The DRBD node agent DaemonSet and workload control plane run here.
• DR cluster — the recovery site. An identical workload control plane and node agent run here, maintaining standby DRBD volumes and standby VM definitions.
• Quorum cluster — a neutral management plane that hosts the TSR operator and all orchestration controllers (failover, protection, pg-sync, replication-monitor, test-failover). It holds kubeconfig credentials for both workload clusters and coordinates all cross-cluster operations.

The quorum cluster is not optional. It holds no VM storage and runs no DRBD state; its role is arbitration and orchestration. Separating it from both workload clusters means that the management plane survives a primary site failure and can drive failover to the DR site without split-brain risk.

All quorum-side CRs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent, DRBDReplicationPolicy) live in a dr-<name> namespace on the quorum cluster. The DRBDResource CRs that represent individual VM disks live on each workload cluster in the VM's own namespace.

What TSR does not solve

TSR is scoped to specific conditions worth understanding before you deploy:

• Latency requirement — synchronous replication is only viable at sub-50ms RTT. If your primary and DR sites are farther apart than that network budget allows, TSR is not the right tool.
• Node pinning — once a VM is protected, it is pinned to a specific node on each cluster for the lifetime of the protection. Live migration and node drain of a protected running VM are denied. This is a deliberate constraint of the DRBD node-to-node replication model, not a configuration option.
• Scale ceiling — DRBD allocates one TCP port per protected VM from the range 7000–8000 (inclusive), giving a ceiling of approximately 1,000 protected VMs per protection zone.
• OpenShift only — TSR is developed, tested, and supported on OpenShift 4.17 and later. The charts install on upstream Kubernetes but that configuration is untested, and three capabilities (DRBD node agent admission via the privileged SCC, test failover network isolation via OVN-Kubernetes, and the TSR web console Route) are OpenShift-specific.

Checking replication health for a Protection Group

Once a Protection Group is protected and replication is running, query the auto-created ReplicationGroupStatus to see aggregated health. The replication monitor names this object <protection-group-name>-replication in the zone namespace on the quorum cluster.

# Run against the quorum cluster
oc get replicationgroupstatus -n dr-prod pg-databases-replication -o wide

Expected output when replication is healthy:

NAME                        PROTECTION-GROUP   RPO-STATUS   OUT-OF-SYNC   SYNCED   CONNECTION   LAST-SYNC
pg-databases-replication    pg-databases        Met          0             4        Connected    2026-05-01T14:32:00Z

If the RPO-STATUS shows Violated or the CONNECTION column shows Degraded, query the corresponding RPOEvent objects for the audit trail:

oc get rpoevents -n dr-prod -l siterecovery.trilio.io/protection-group=pg-databases

Verifying that a VM is protected

The siterecovery.trilio.io/protected-vm=true label is stamped on the KubeVirt VirtualMachine object on the workload cluster — not on the ProtectionRequest or DRBDResource. Use it against the primary cluster to confirm which VMs are under protection:

# Run against the primary workload cluster
oc get vm -A -l siterecovery.trilio.io/protected-vm=true

To see the protection lifecycle phase, query the ProtectionRequest on the quorum cluster:

oc get pr -n dr-prod

Expected output for a fully protected VM:

NAME              VM          NAMESPACE   PHASE       SYNC   AGE
protect-vm-web    vm-web      app-ns      Protected   100    2d

Monitoring failover progress

After applying a FailoverRequest to the quorum cluster, watch the per-VM status:

oc get fr -n dr-prod pg-databases-failover -w

Expected output during a planned failover:

NAME                      PROTECTION GROUP   TARGET     STATE        PHASE
pg-databases-failover     pg-databases        cluster2   InProgress   StoppingSource

Once complete:

NAME                      PROTECTION GROUP   TARGET     STATE       PHASE
pg-databases-failover     pg-databases        cluster2   Completed   Completed

For per-VM detail during the transition:

oc get fr pg-databases-failover -n dr-prod -o jsonpath='{.status.vmStatuses}' | jq .

• Plan and verify infrastructure — prerequisites, network port requirements, and cluster topology validation before deploying TSR.
• Deploy the workload control plane — how the ProtectionZone controller automatically deploys the DRBD node agent DaemonSet and workload control plane to your primary and DR clusters.
• Protect virtual machines — creating a ProtectionRequest to enroll a VM in block-level replication and track it through to the Protected phase.
• Verify replication health — interpreting ReplicationGroupStatus, RPOEvent, and DRBDResource status to confirm your replication link is healthy before you need it.
• Run a test failover — using the TestFailover CRD to validate your DR capability non-disruptively using volume snapshots.
• Execute a planned failover — the step-by-step procedure for a coordinated failover when the primary site is still reachable.
• Execute an unplanned failover — force-promoting DR volumes and recovering workloads when the primary cluster is unavailable.
• Diagnose a degraded or failed node or replication link — runbooks for interpreting DRBDResource connection state, RPOEvent violation reasons, and node-loss recovery procedures.