Architecture

This page describes the internal architecture of Trilio Site Recovery: the controllers that run on each cluster, how they communicate through Custom Resources, and how they coordinate to protect virtual machine workloads and execute disaster recovery operations. Understanding this architecture helps you reason about what happens during protection, failover, and failback—and where to look when something goes wrong. All orchestration is CRD-driven, meaning every state transition is observable via kubectl and auditable through Kubernetes events.

Trilio Site Recovery is composed of two groups of components: those that run on the quorum cluster and provide the management plane, and those that run on the primary and DR clusters and manage local storage and VM lifecycle.

Quorum Cluster Components

site-recovery-quorum-control-plane (Helm chart / controller manager)

The unified Go controller manager deployed to the quorum cluster via Helm. It packages the failover, protection, pg-sync, and replication-monitor reconcilers together with admission webhooks into a single Deployment per DR namespace (dr-<name>). You deploy one instance of this chart per DR namespace.

failover-controller (operator)

Watches FailoverRequest CRDs on the quorum cluster. When you create a FailoverRequest, this controller orchestrates the full failover sequence: it patches the ProtectionGroup spec.desiredState on the source cluster to stopped, waits for status.currentState to confirm VMs have stopped, coordinates DRBD volume promotion, and then patches the ProtectionGroup on the target cluster to running. It never touches VMs directly—VM lifecycle is always delegated to the protection-group-controller on each workload cluster.

protection-controller (operator)

Watches ProtectionRequest CRDs on the quorum cluster (DRBD Operator deployments). When you submit a ProtectionRequest for a VM, this controller validates the VM, provisions a DRBDResource pair covering all VM disks on both the primary and DR clusters, and switches the VM to DRBD-backed frontend PVCs. After this completes, the VM's storage is being synchronously or asynchronously replicated to the DR side.

pg-sync-controller (operator)

Keeps ProtectionGroup metadata synchronized between the primary and DR clusters. It converts DataVolume references to PVC references and removes orphaned dataVolumeTemplates when building standby VM specs on the DR side. Without this controller, the DR-side VM spec would diverge from the primary over time.

test-failover-controller (operator)

Watches TestFailover CRDs. When you trigger a test failover, this controller creates volume snapshots of all replicated PVCs, provisions test PVCs from those snapshots, starts the VMs in an isolated namespace on the DR cluster, runs any verification checks you have defined, and then cleans everything up. Production workloads on both clusters are not affected at any point.

site-recovery-protectionzone-controller (Helm chart / admission webhook)

Must be installed on the quorum cluster before the quorum control plane chart. It installs the ProtectionZone admission webhook and CRDs, which define the boundary of a DR deployment: which clusters participate, which storage backend mode to use, and how the deployment is named and described.

TSR Web Console (service)

A React/Next.js web interface with a Python Flask backend deployed on the quorum cluster. It provides a centralized dashboard for viewing replication health across all DR deployments, protecting VMs, and triggering failover and test failover operations. All actions it performs are translated into the same CRD operations you can perform with kubectl.

Workload Cluster Components (Primary and DR)

site-recovery-workload-control-plane (Helm chart)

Deployed to both the primary and DR clusters. It installs the drbd-node-agent DaemonSet, admission webhooks, and the RBAC required for VM protection and replication on the workload side. This is the Helm chart you deploy on every cluster that will host or protect VMs.

protection-group-controller (operator)

Runs on both the primary and DR clusters. It watches ProtectionGroup CRDs locally and is responsible for a single concern: reconciling the spec.desiredState field against the actual running state of every VM in the group. When desiredState is stopped, it patches each VM's spec.running field to false and updates status.currentState. When desiredState is running, it starts them. This controller operates on its local cluster only and never makes cross-cluster API calls.

drbd-node-agent (DaemonSet agent)

Runs on every worker node of the primary and DR clusters. It manages DRBD kernel-level block replication for individual VM volumes, maintains peer connections, and reports per-volume sync progress and connection state into DRBDResource status fields. The replication data path between clusters is entirely through this agent—no replication traffic flows through the quorum cluster.

Custom Resources

The following traces a complete planned failover end-to-end, showing how control passes through every layer of the architecture.

1. You submit a FailoverRequest

You apply a FailoverRequest manifest to the quorum cluster's DR namespace:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-prod
  namespace: dr-production
spec:
  protectionGroupRef:
    name: production-pg
    namespace: dr-production
  targetCluster: dr-cluster
  failoverType: planned
  drainTimeoutSeconds: 120
  batchBootTimeoutSeconds: 300

The failover-controller on the quorum cluster detects the new CR through its watch loop.

2. Failover controller acquires a per-ProtectionGroup lock

Before taking any action, the failover-controller acquires a Kubernetes Lease object scoped to this Protection Group. This prevents concurrent FailoverRequest objects from racing on the same group. The lock name follows the pattern failover-lock-<protection-group-name> in the DR namespace. If another failover is already in progress for the same group, the new request stays in Pending until the lock is released.

3. Failover controller patches ProtectionGroup.spec.desiredState = stopped on the primary cluster

The failover-controller reaches the primary cluster's API server (using the kubeconfig stored for that cluster) and patches the ProtectionGroup CR:

spec:
  desiredState: stopped

The controller updates the FailoverRequest status to StoppingOnSource and returns from this reconciliation loop iteration.

4. protection-group-controller on the primary cluster reconciles VM state

The protection-group-controller running locally on the primary cluster detects the spec.desiredState change via its update handler. It iterates over every VM listed in spec.virtualMachines and patches each VirtualMachine object's spec.running: false. As VMs stop, it updates status.currentState. When all VMs report Stopped, it sets status.currentState: stopped.

ProtectionGroup.spec.desiredState = stopped
  → protection-group-controller patches VM-1 spec.running: false
  → protection-group-controller patches VM-2 spec.running: false
  → protection-group-controller patches VM-N spec.running: false
  → ProtectionGroup.status.currentState = stopped

5. Failover controller detects source is stopped; coordinates DRBD promotion

On its next reconciliation tick (every ~10 seconds), the failover-controller reads the ProtectionGroup status on the primary cluster and confirms currentState: stopped. It then:

1. Verifies that no other Protection Groups with currentState: running exist on the target cluster that would be disrupted by node-level taint changes.
2. Removes any DRBD quorum taints (drbd.linbit.com/lost-quorum) from worker nodes on the DR cluster.
3. Waits for DRBD volumes on the DR cluster to be promotable (confirmed via DRBDResource status fields on the DR cluster).

The FailoverRequest status moves to WaitingForDRBD.

6. Failover controller patches ProtectionGroup.spec.desiredState = running on the DR cluster

Once DRBD volumes are promotable, the failover-controller patches the ProtectionGroup on the DR cluster:

spec:
  desiredState: running

The FailoverRequest status moves to StartingOnTarget.

7. protection-group-controller on the DR cluster starts VMs

The local protection-group-controller on the DR cluster detects the spec.desiredState: running change and patches each standby VM's spec.running: true. The pg-sync-controller on the quorum cluster has already ensured that the DR-side VM specs are correctly formed (PVC references resolved, orphaned dataVolumeTemplates removed). VMs start against the DRBD-promoted volumes.

When all VMs reach Running, the controller sets ProtectionGroup.status.currentState: running on the DR cluster.

8. Failover controller marks the FailoverRequest complete

On its next reconciliation tick, the failover-controller reads currentState: running on the DR cluster's ProtectionGroup. It sets the FailoverRequest status to Completed, releases the per-Protection Group lease, and records a final event. The entire audit trail—what happened, in what order, and when—is preserved in the FailoverRequest CR's status and events.

Replication Health Data Flow

In parallel with the above failover workflow, drbd-node-agent DaemonSet pods on each worker node continuously report block replication state into DRBDResource status. The replication-monitor reconciler (packaged in site-recovery-quorum-control-plane) aggregates these into ReplicationGroupStatus CRs on the quorum cluster. When replication lag exceeds the threshold defined in ReplicationGroupStatus.spec.rpoObjectiveSeconds, the monitor creates an RPOEvent CR for audit and alerting purposes.

1. Controllers are local-only; cross-cluster orchestration belongs to the quorum tier

Each protection-group-controller instance operates exclusively against its local Kubernetes API server. It has no kubeconfig for any other cluster and makes no cross-cluster API calls. All cross-cluster coordination—reading the state of one cluster and acting on another—is the responsibility of the controllers that run on the quorum cluster (failover-controller, pg-sync-controller, protection-controller). This boundary is intentional and enforced by design.

Rationale: If workload-cluster controllers made cross-cluster calls, a network partition between clusters would cause them to enter split-brain or deadlock states. By isolating cross-cluster logic on the quorum cluster, the blast radius of a network partition is contained: workload controllers continue to reconcile local state correctly regardless of inter-cluster connectivity, while the quorum controllers detect the partition and surface it through ReplicationGroupStatus and RPOEvent CRDs rather than taking unsafe autonomous action.

2. VM lifecycle is always delegated to protection-group-controller, never done directly by failover-controller

The failover-controller never patches a VirtualMachine resource directly. It only patches ProtectionGroup.spec.desiredState and then waits for status.currentState to converge. The protection-group-controller performs the actual VM start and stop operations.

Rationale: This separation of concerns means that the VM state reconciliation logic—including idempotency, per-VM retry, and status tracking—lives in exactly one place. If the failover-controller patched VMs directly, it would duplicate this logic and introduce the possibility of the two controllers racing on the same VM objects. It also means that features added to protection-group-controller (such as pre-stop hooks or health checks) are automatically available to all failover paths without changes to the failover controller.

3. Per-Protection Group locks using Kubernetes Leases

Before taking any action on a FailoverRequest, the failover-controller acquires a coordination.k8s.io/v1 Lease object named failover-lock-<protection-group-name> in the DR namespace. The lease has a 5-minute TTL so that a crashed controller process does not permanently block future failover operations.

Rationale: Without a lock, two concurrent FailoverRequest objects for the same Protection Group could drive the group into an undefined state—for example, one request starting VMs on the DR cluster at the same moment another is stopping them. Kubernetes Lease objects are the standard mechanism for distributed leader election in the Kubernetes ecosystem and require no external dependencies.

4. Safety check before removing DRBD quorum taints

DRBD quorum taints (drbd.linbit.com/lost-quorum) are applied at the node level, not at the per-PVC or per-Protection Group level. Before removing taints from nodes on the target cluster, the failover-controller checks whether any other Protection Groups on that cluster have status.currentState: running. If any do, taint removal is blocked and the FailoverRequest is held in its current phase until the conflict clears or an operator uses the force field.

Rationale: Removing a node taint affects the scheduling of every workload on that node, not just the Protection Group being failed over. Blindly removing taints during a failover could allow VMs from another Protection Group to be rescheduled onto nodes that are not yet ready, causing data corruption or unexpected VM restarts. The safety check is the minimum viable protection against this class of concurrent-operation failure.

5. Quorum cluster hosts management plane only; no application VMs, no replication traffic

The quorum cluster runs controllers, admission webhooks, and the TSR Web Console, but it does not host any application VMs and DRBD replication traffic does not pass through it. All block replication happens directly between drbd-node-agent instances on the primary and DR worker nodes over TCP ports 7000–7999.

Rationale: Routing replication traffic through the quorum cluster would make it a single point of failure for data replication, which would undermine the purpose of having a separate management cluster. Keeping the quorum cluster out of the data path means that quorum cluster maintenance or failure does not interrupt ongoing replication, and that the replication throughput is not bounded by the quorum cluster's network bandwidth.

Known Limitations

DRBD taints are node-scoped, not Protection Group-scoped

The drbd.linbit.com/lost-quorum taint is applied to Kubernetes nodes by the DRBD kernel module, scoped to the entire node rather than to individual PVCs or Protection Groups. This means that when you fail over one Protection Group, removing the taint affects every other workload on those nodes. The safety check in the failover-controller mitigates this by blocking taint removal when other Protection Groups are running, but it does not eliminate the constraint—it serializes failovers that share target nodes. If you need to fail over multiple Protection Groups simultaneously to the same cluster, you must use the force field on the FailoverRequest and accept the risk of disruption to other groups, or schedule failovers sequentially.

Cross-cluster connectivity is required from the quorum cluster at operation time

The failover-controller and pg-sync-controller require TCP port 6443 (Kubernetes API) to be reachable from the quorum cluster to both the primary and DR API servers at the time a FailoverRequest or ProtectionRequest is processed. If the primary cluster's API server is unreachable during an unplanned failover, the failover-controller cannot read the primary ProtectionGroup state to confirm VMs are stopped before promoting DRBD volumes. In this scenario, failoverType: unplanned in the FailoverRequest spec signals to the controller that it should proceed with DR-side promotion without waiting for source confirmation, accepting the possibility of minimal data loss under Protocol A.

Safety check scope is per-namespace, not cluster-wide

The concurrent failover safety check examines ProtectionGroup resources in the same namespace as the FailoverRequest. Protection Groups in other namespaces on the same cluster are not checked before taint removal. In multi-tenant deployments where multiple DR namespaces share the same workload clusters, operators must coordinate failover timing manually or accept that a failover in one namespace may affect node taints that impact another namespace.

Replication health is eventually consistent

ReplicationGroupStatus and RPOEvent CRs are updated by the replication-monitor reconciler on a polling interval configured via ReplicationGroupStatus.spec.pollingIntervalSeconds. Between polling intervals, the status in these CRs may not reflect the most recent state of DRBD replication. For operations where real-time replication state is critical—such as immediately before executing a planned failover—inspect DRBDResource status directly on the workload clusters rather than relying solely on ReplicationGroupStatus.

Alternatives Not Chosen

Direct VM manipulation from the failover controller

Earlier design iterations had the failover-controller patch VirtualMachine.spec.running directly, bypassing the protection-group-controller entirely. This was rejected because it duplicated VM lifecycle logic, created the possibility of two controllers racing on the same VM objects, and made it impossible for the Protection Group layer to add pre-stop or post-start hooks without modifying the failover controller. The current architecture—where the failover controller only patches ProtectionGroup.spec.desiredState—is strictly more composable.

Shell scripts as the primary failover mechanism

Operational runbooks in the scripts/ directory can create FailoverRequest CRDs, but they are not the authoritative path for failover. Shell scripts that directly manipulate VMs or taints without going through CRDs provide no audit trail, no status tracking, and no idempotency guarantees. The CRD-driven model is the supported production path; shell scripts exist for bootstrapping and for scenarios where a human operator needs to interact with the system outside of normal automation.

Centralized LINSTOR storage model

The LINSTOR deployment model (three-cluster topology with a central LINSTOR controller on the quorum cluster managing storage across all satellite nodes) is not supported in the current product release. All supported deployments use the DRBD Operator model, where storage is managed in a distributed fashion by drbd-node-agent DaemonSets on each workload cluster.

When to Use a Different Approach

• RPO = 0 is required and link latency exceeds 50ms RTT: Protocol C (synchronous replication) requires acknowledgment from the DR side before writes complete. Above approximately 50ms RTT, write latency becomes unacceptable for most VM workloads. In this case, use Protocol A (asynchronous) and accept a small RPO window rather than degrading application performance.
• You need to recover individual files or application-consistent snapshots rather than full VM failover: Trilio Site Recovery operates at the block storage layer and fails over entire VMs. It is not a backup tool and does not provide granular file-level restore. For file-level recovery or application-consistent backup, use a complementary tool from the TrilioData ecosystem.
• Your workloads are not VM-based: Site Recovery is designed specifically for VM workloads managed by OpenShift Virtualization (KubeVirt/CNV). It does not protect pod-based workloads or stateful sets backed by ordinary PVCs outside the DRBD replication path.