Problem Being Solved

Trilio Site Recovery solves the problem of enterprise disaster recovery for virtual machine workloads running on OpenShift, where traditional approaches impose recovery windows that are too wide for business-critical systems. This page explains the operational challenge—slow, manual, error-prone DR—and how Site Recovery addresses it through storage-layer replication, Kubernetes-native automation, and CRD-driven orchestration. Understanding this context will help you evaluate whether Site Recovery fits your environment and make informed decisions when planning your DR deployment.

The challenge: DR for VMs on Kubernetes is operationally hard

Virtual machines running on OpenShift under OpenShift Virtualization present a specific class of disaster recovery problem. Unlike stateless containerized workloads—which Kubernetes already handles well through replication controllers and rolling deployments—VMs carry persistent disk state that must be replicated, tracked, and safely promoted on a secondary cluster if the primary fails. Getting that right under pressure, in a real disaster, is what most organizations cannot do today.

What traditional approaches get wrong

Most teams protecting VM workloads on Kubernetes rely on one of three approaches, all of which share the same fundamental flaw: they protect data at the wrong layer, at the wrong time, or both.

Backup-and-restore tools snapshot VM disks on a schedule—typically every 15 to 60 minutes—and write those snapshots to object storage. When a disaster occurs, you restore from the last snapshot, accept the data loss since that snapshot was taken, and wait for the restore to complete. Recovery times of 1 to 4 hours are common. For databases, message brokers, or any workload where losing tens of minutes of transactions is unacceptable, this does not meet the requirement.

Manual replication runbooks exist in many organizations as a middle ground: engineers maintain scripts that trigger rsync jobs, coordinate storage volume promotion, and bring VMs up in a defined order on the DR cluster. These runbooks work in drills. Under real disaster conditions—with partial cluster failures, split networks, and time pressure—they fail at exactly the moment you need them most. They also require human intervention, which means your RTO is bounded below by how quickly the right person can be reached and can execute the procedure correctly.

Persistent Volume Claim-level replication through storage vendor features can achieve lower RPO windows, but these solutions are typically not Kubernetes-aware. They replicate volumes without understanding VM structure, disk ordering, boot dependencies, or multi-VM consistency groups. Promoting a replicated volume on the DR side and getting a working VM out of it requires manual coordination that the storage system cannot perform.

The specific gaps Site Recovery closes

Site Recovery is built around three operational gaps that none of the above approaches close cleanly.

Gap 1: RPO is too wide. Traditional tools accept data loss measured in minutes because they operate above the storage layer—snapshotting at the filesystem or volume level on a schedule. Site Recovery operates at the block device layer using DRBD replication, which means every write committed on the primary cluster is replicated to the DR cluster before (Protocol C, synchronous) or immediately after (Protocol A, asynchronous) it is acknowledged. In synchronous mode, RPO is zero. In asynchronous mode, RPO is measured in seconds—not minutes—even across long-distance links.

Gap 2: RTO is too long and too manual. When a disaster occurs, someone has to decide to act, find the runbook, execute the right sequence of commands on the right cluster, and verify the result. Site Recovery replaces that with a single CRD: creating a FailoverRequest on the quorum cluster causes the failover-controller to stop VMs on the source side, promote DRBD volumes on the DR cluster, and start VMs there—all automatically, in the right order, with dependency management. Automated RTO is 3 to 8 minutes.

Gap 3: No non-disruptive way to validate DR readiness. Runbook-based DR cannot be tested without disrupting production. Backup-restore tools can be tested on isolated storage, but they don't validate the actual replication path. Site Recovery's TestFailover CRD creates snapshot-based copies of VM disks on the DR cluster in an isolated namespace, boots test VMs, runs verification checks, and cleans everything up—without touching production workloads or interrupting replication. You can run this monthly, weekly, or as part of a CI pipeline.

How Site Recovery addresses these gaps

Storage-layer replication via DRBD

Site Recovery uses DRBD—a proven Linux kernel module for block-level replication—to keep VM disks synchronized between the primary and DR clusters at the storage layer. The drbd-node-agent DaemonSet runs on every worker node and manages DRBD replication for each VM's volumes directly at the kernel level. This is below the filesystem, below the PVC, and below the VM—meaning no application changes are required, and no agent runs inside the guest.

DRBD supports two replication modes that map directly to your RPO requirements:

• Protocol C (synchronous): A write is acknowledged to the VM only after it has been committed to disk on both the primary and DR clusters. RPO is zero. This requires round-trip network latency below approximately 50ms between clusters.
• Protocol A (asynchronous): A write is acknowledged after local disk commit and replicated to the DR cluster in the background. RPO is typically seconds. There is no distance constraint, making this appropriate for geographically distant DR sites.

The DRBDReplicationPolicy CRD lets you select the protocol and map storage classes between clusters—without modifying the VMs or their PVCs.

CRD-driven automation

Every DR operation in Site Recovery is expressed as a Kubernetes custom resource. There are no proprietary CLI tools to install, no out-of-band APIs to call, and no SSH sessions required. This has two important consequences.

First, it makes DR operations auditable and repeatable. A FailoverRequest manifest checked into Git is a record of intent, an executable runbook, and a compliance artifact all at once. Creating it with kubectl apply is identical whether a human or an automation system does it.

Second, it makes DR operations safe. The controllers that watch these CRDs—failover-controller, protection-controller, test-failover-controller, and others—run continuously on the quorum cluster and reconcile desired state with actual state. If a failover is interrupted, the controller resumes. If a protection request fails validation, the controller reports why through standard Kubernetes status conditions rather than leaving a system in an unknown state.

The key CRDs and the operations they express:

A neutral quorum cluster as the management plane

Orchestrating failover across two clusters that may be in different failure domains requires a third party that is not subject to either cluster's failure. Site Recovery addresses this with the quorum cluster: a separate OpenShift cluster that hosts the management plane—the failover, protection, pg-sync, and test-failover controllers—but does not run application VMs and does not relay DRBD replication traffic.

The quorum cluster is where you apply FailoverRequest and ProtectionRequest CRDs, and where the site-recovery-quorum-control-plane Helm chart is deployed. Because it sits outside both the primary and DR failure domains, it can make authoritative decisions about which cluster is the target of a failover even when one side is unreachable.

Multi-tenant isolation

A single quorum cluster can manage multiple independent DR deployments—each representing a different primary/DR cluster pair, a different business unit, or a different application stack. Each deployment runs in its own dr-<name> namespace on the quorum cluster, with its own instance of the quorum control plane and its own isolated set of CRDs. This means a failure or misconfiguration in one deployment cannot affect another.

What this means operationally

For platform engineers and SREs, Site Recovery changes the DR operational model in the following concrete ways:

• Protecting a VM means applying a ProtectionRequest manifest. The protection-controller validates the VM, provisions a DRBDResource pair covering all VM disks, and switches the VM to DRBD-backed frontend PVCs—without downtime.
• Monitoring replication health means reading ReplicationGroupStatus and RPOEvent CRs, which can be scraped by any standard Kubernetes monitoring stack.
• Running a DR test means applying a TestFailover manifest and watching it progress through phases—CreatingSnapshots, CreatingVolumes, CreatingVMs, VerifyingData, Succeeded—then deleting it to trigger cleanup. Production is unaffected.
• Executing a planned failover means applying a FailoverRequest with failoverType: planned. The failover-controller gracefully shuts down primary VMs, promotes DR volumes, and starts VMs on the DR cluster. Total time is 3 to 8 minutes.
• Executing an unplanned failover means applying a FailoverRequest with failoverType: unplanned. The failover-controller force-promotes DR volumes without waiting for the primary to respond, accepting minimal data loss if Protocol A was in use.
• Failing back means repeating the protection and failover workflow in the reverse direction after the primary cluster is restored.

All of these operations are compatible with GitOps workflows: manifests can be stored in version control, applied through a CI pipeline, and reviewed through standard pull request processes.

Example: What a DR gap looks like without Site Recovery

The following illustrates a typical backup-based recovery timeline for a database VM, contrasted with what Site Recovery provides.

Without Site Recovery (backup/restore approach)
─────────────────────────────────────────────────
T+0:00   Primary cluster failure detected
T+0:15   On-call SRE paged and acknowledges
T+0:30   SRE identifies latest snapshot (taken 47 minutes ago)
T+0:45   Restore job submitted to object storage
T+2:30   Restore complete, VM starts on DR cluster
T+2:45   Application verified, traffic cut over

RPO: up to 47 minutes of data loss
RTO: ~2 hours 45 minutes
Manual steps: ~8

With Site Recovery (DRBD Protocol C + FailoverRequest)
───────────────────────────────────────────────────────
T+0:00   Primary cluster failure detected
T+0:02   FailoverRequest CR applied to quorum cluster
T+0:02   failover-controller begins orchestration
T+0:06   DRBD volumes promoted on DR cluster, VMs starting
T+0:08   VMs running on DR cluster, verified by controller

RPO: 0 (synchronous replication, no data loss)
RTO: ~8 minutes
Manual steps: 1 (apply one manifest)

Example: Expressing failover intent as a Kubernetes resource

The following is a FailoverRequest manifest for a planned failover. The failover-controller on the quorum cluster watches for this CR and begins orchestration immediately.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: pg-databases-failover
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: pg-databases
  targetCluster: dr-cluster
  failoverType: planned
  drainTimeoutSeconds: 120
  batchBootTimeoutSeconds: 300

Apply this to the quorum cluster and watch progress:

kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f failover.yaml
kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest pg-databases-failover -n dr-prod -w

Expected output as the controller works through the phases:

NAME                      STATUS       AGE
pg-databases-failover     Pending      2s
pg-databases-failover     InProgress   5s
pg-databases-failover     InProgress   42s
pg-databases-failover     Completed    7m18s

Example: Checking replication health before a scheduled DR test

Before running a test failover, confirm that replication is healthy by reading the ReplicationGroupStatus CR on the quorum cluster.

kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o wide

Expected output for a healthy group:

NAME           HEALTH    PROTOCOL   LAST-SYNC             AGE
pg-databases   Healthy   C          2025-10-14T09:22:01Z  14d

If the HEALTH column shows Degraded or Critical, check for RPOEvent records before proceeding:

kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod

• Deploy the workload control plane — Install the site-recovery-workload-control-plane Helm chart on your primary and DR clusters to enable DRBD replication and VM protection.
• Deploy the quorum control plane — Install the site-recovery-quorum-control-plane Helm chart on your quorum cluster to bring up the failover, protection, and sync controllers.
• Configure cross-cluster replication (DRBD Operator model) — Define a DRBDReplicationPolicy to set replication protocol, endpoints, and storage class mappings between clusters.
• Protect virtual machines — Apply a ProtectionRequest to enroll a VM in block-level replication and switch it to DRBD-backed frontend PVCs.
• Verify replication health — Read ReplicationGroupStatus and RPOEvent CRDs to assess sync state before a planned failover or DR test.
• Run a test failover — Use the TestFailover CRD to validate DR readiness non-disruptively using volume snapshots.
• Execute a planned failover — Apply a FailoverRequest with failoverType: planned to perform a controlled, zero-data-loss transition to the DR cluster.
• Execute an unplanned failover — Apply a FailoverRequest with failoverType: unplanned when the primary cluster is unavailable and immediate recovery is required.