Concept

Problem Being Solved

What site recovery and disaster recovery challenges this operator automates in Kubernetes environments

Overview

Site Recovery addresses a specific and difficult problem: protecting KubeVirt and OpenShift Virtualization workloads against site-level failures in Kubernetes environments, without requiring you to re-architect your applications or introduce proprietary recovery tooling. This page explains the operational challenges that motivated the operator, the gaps in existing Kubernetes-native tooling that leave VM workloads exposed, and the design choices Site Recovery makes to close those gaps. Understanding this context will help you evaluate whether Site Recovery fits your environment and set realistic expectations for recovery objectives.

Content

The gap between Kubernetes resilience and disaster recovery

Kubernetes provides strong primitives for self-healing within a single cluster: pods restart on failure, deployments maintain replica counts, and node-level failures trigger rescheduling. What Kubernetes does not provide is any built-in mechanism for recovering workloads when an entire cluster or data center becomes unavailable. That gap is widest for stateful virtual machine workloads, where simply rescheduling a pod onto another node is not an option — the VM's disk state must be present and consistent on the target cluster before the VM can start.

Platform engineering and SRE teams managing KubeVirt or OpenShift Virtualization deployments face a set of interlocking problems that standard Kubernetes tooling does not solve:

Storage replication across clusters is not handled by Kubernetes

Kubernetes PersistentVolumeClaims are cluster-scoped. A volume that exists on your primary cluster has no automatic representation on your DR cluster. If the primary cluster fails, the DR cluster has no access to the data. Replicating block storage across cluster boundaries requires a kernel-level replication layer — DRBD in Site Recovery's case — that operates beneath the Kubernetes storage abstraction. Configuring, monitoring, and coordinating that replication layer across two independent Kubernetes clusters is operationally complex and error-prone when done manually.

VM failover is not atomic or orchestrated

Even when storage replication is in place, recovering a group of virtual machines after a site failure involves a precise sequence of operations: verifying replication is consistent, promoting the DR-side volumes from secondary to primary, ensuring the primary-side VMs are stopped (or accepting that they cannot be stopped in a disaster), and starting the DR-side VMs in the correct order. Missing any step, or performing steps out of order, risks data corruption or a split-brain condition where both sites believe they hold the authoritative copy of a volume. This sequencing cannot be safely left to manual runbooks under the time pressure of an actual incident.

Protection intent has no Kubernetes-native expression

Without a purpose-built operator, teams typically manage DR through a combination of scripts, cron jobs, monitoring dashboards, and runbooks that exist outside the Kubernetes control plane. This means protection state — which VMs are protected, what their replication health is, whether they are ready to fail over — is not visible through standard Kubernetes tooling. It also means protection configuration is not subject to GitOps workflows, audit trails, or access controls that the rest of your infrastructure benefits from.

Validating DR readiness without disruption is practically difficult

A DR plan that has never been tested is not a DR plan. However, actually testing failover in most environments means either accepting production downtime or maintaining a parallel environment that is kept in sync with production. Neither option is acceptable for teams that need to demonstrate recovery capability regularly without impacting their SLAs.

Multi-tenant DR management multiplies complexity

Large platform teams often need to manage DR for multiple independent application groups or tenants from a single operational plane. Maintaining separate tooling, credentials, and monitoring for each group grows linearly with the number of tenants and creates coordination overhead that increases the risk of configuration drift.

What Site Recovery automates

Site Recovery addresses each of these problems through a Kubernetes-native operator that encodes DR workflows as reconciliation loops over custom resources.

Continuous storage replication is established automatically when you declare protection intent. For DRBD Operator deployments, the protection-controller on the quorum cluster processes a ProtectionRequest, creates the necessary DRBDVolume resources, and switches the VM's storage to DRBD-backed frontend PVCs — all without you needing to configure replication endpoints manually per volume. The replication-monitor agent continuously measures replication lag and records RPOEvent resources when lag exceeds acceptable thresholds, giving you persistent, queryable evidence of replication health.

Orchestrated failover is triggered by creating a FailoverRequest custom resource, either through kubectl, the Site Manager UI, or the pgctl CLI. The failover-controller on the quorum cluster drives the full sequence: stopping VMs on the primary, promoting volumes on the DR cluster, verifying promotion, and starting VMs on the DR cluster. The controller handles both planned failovers — where primary VMs are gracefully stopped before promotion — and unplanned failovers where the primary is unreachable. Failure at any step causes the controller to surface the error in the FailoverRequest status rather than leaving the environment in an ambiguous intermediate state.

Declarative protection state lives in Kubernetes custom resources. ProtectionGroup and ProtectionRequest resources describe which VMs you want protected and how. ReplicationGroupStatus resources provide a live aggregated health view. Because these are Kubernetes resources, they integrate naturally with kubectl, RBAC, GitOps pipelines, and existing monitoring infrastructure.

Non-disruptive DR validation is available through the TestFailover custom resource in DRBD Operator deployments. A TestFailover creates snapshots of production volumes, boots test VMs on the DR cluster from those snapshots, runs verification checks, and cleans up — without affecting production VMs or interrupting replication. You can schedule regular test failovers to continuously validate recovery capability and produce audit evidence.

Multi-tenant isolation is built into the deployment model. Each DR deployment gets its own namespace on the quorum cluster (following the dr-<name> naming convention), its own set of controllers, and its own credentials. A single quorum cluster can manage multiple independent primary–DR cluster pairs without any cross-tenant visibility or interference.

Recovery objectives the operator targets

Site Recovery supports two replication protocols with different RPO characteristics:

Protocol C (synchronous) writes are acknowledged to the VM only after data is committed to disk on both the primary and DR clusters. This achieves RPO=0 — no data loss — at the cost of requiring low-latency network connectivity (less than 10ms round-trip between clusters). This is appropriate for workloads where any data loss is unacceptable.
Protocol A (asynchronous) writes are acknowledged after local commit, with replication happening in the background. This supports higher-latency or geographically distant DR sites and accepts a small RPO window measured in seconds. The replication-monitor tracks whether actual lag stays within your configured threshold and raises RPOEvents when it does not.

For RTO, the operator targets 3–8 minutes for automated failover from the point a FailoverRequest is submitted to VMs running on the DR cluster. Actual RTO depends on the number of VMs in the Protection Group, their startup time, and network conditions between the quorum cluster and the primary and DR clusters.

What the operator does not change

Site Recovery is designed to protect VMs without requiring application changes. The DRBD replication layer operates beneath the Kubernetes storage abstraction: your VM sees a standard PVC and has no awareness that its writes are being replicated to another cluster. After a failover, the VM starts on the DR cluster with its disk state intact, and applications resume from where they left off. No agents inside the VM, no application-level replication configuration, and no changes to how VMs are defined or deployed are required.

Examples

Expressing protection intent as a Kubernetes resource

The following example shows how you declare DR protection for a virtual machine in a DRBD Operator deployment. Creating this resource is the only action you need to take — the protection-controller handles replication setup automatically.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-database
  namespace: dr-prod
spec:
  vmName: vm-database
  vmNamespace: default
  sourceCluster: cluster1

After applying this manifest on the quorum cluster, you can watch the controller progress through its lifecycle:

kubectl --kubeconfig $KUBECONFIG_QUORUM \
  get protectionrequest protect-vm-database -n dr-prod -w

Expected output as the controller works through validation, volume creation, and replication setup:

NAME                   STATUS       AGE
protect-vm-database    Validating   5s
protect-vm-database    CreatingVolumes   30s
protect-vm-database    WaitingForSync    90s
protect-vm-database    Protected         4m12s

Checking replication health across a Protection Group

The replication-monitor continuously publishes aggregated health through ReplicationGroupStatus resources. You can query these to get a snapshot of replication state without logging into either workload cluster:

kubectl --kubeconfig $KUBECONFIG_QUORUM \
  get replicationgroupstatus -n dr-prod -o wide

Expected output showing a healthy group and the timestamp of the last successful sync:

NAME            HEALTH    VOLUMES   LAST-SYNC              AGE
pg-web-tier     Healthy   3/3       2024-01-15T14:23:01Z   12d
pg-database     Healthy   2/2       2024-01-15T14:22:58Z   12d

Reviewing RPO violations for audit purposes

RPOEvent resources are created automatically when replication lag exceeds configured thresholds. You can list them to identify periods when your RPO commitment was at risk:

kubectl --kubeconfig $KUBECONFIG_QUORUM \
  get rpoevent -n dr-prod --sort-by=.metadata.creationTimestamp

Expected output:

NAME                        PROTECTION-GROUP   LAG       SEVERITY   AGE
rpoevent-pg-database-001    pg-database        12s       Warning    3d
rpoevent-pg-database-002    pg-database        45s       Critical   1d

Related concepts

Architecture overview — How the quorum cluster, primary cluster, and DR cluster relate to each other, and where each controller runs.
Deployment models — The differences between the DRBD Operator model and the centralized storage model, and how to choose between them.
ProtectionGroup — How to group VMs so they fail over together as a coordinated unit.
ProtectionRequest — How to request DR protection for a single VM in DRBD Operator deployments.
FailoverRequest — How the failover-controller orchestrates planned and unplanned failover sequences.
TestFailover — How to validate DR readiness without disrupting production workloads.
DRBDReplicationPolicy — How to configure replication protocols, storage class mappings, and endpoints between clusters.
ReplicationGroupStatus and RPOEvent — How to monitor replication health and track RPO compliance over time.
Multi-tenant deployments — How a single quorum cluster manages multiple independent DR deployments in isolation.