Architecture

This page describes the internal architecture of Trilio Site Recovery (TSR): how its controllers, agents, webhooks, and custom resources are organized across three Kubernetes clusters, how they communicate, and why they are split the way they are. Understanding this architecture helps you diagnose problems, interpret status conditions, and reason about failure modes during DR operations. TSR is a Kubernetes operator that drives DRBD block-level replication between a primary and a DR cluster, with all orchestration CRDs residing on a third, neutral quorum cluster.

TSR is composed of three process groups, each deployed to a specific cluster tier.

Quorum cluster components

All quorum components run in the dr-<name> namespace. Workload components run in trilio-site-recovery-system on each workload cluster. The two namespaces are distinct — querying quorum pods in trilio-site-recovery-system returns nothing.

protection-zone-controller

Type: Kubernetes operator (Deployment) Binary: cmd/protectionzone (/quorum-manager)

The bootstrap operator. Installed by OLM via the site-recovery bundle, it watches ProtectionZone CRs cluster-wide and drives the full provisioning lifecycle:

1. Validates both kubeconfig Secrets and probes each workload cluster's API.
2. Provisions per-zone TLS certificates for admission webhooks.
3. Deploys the site-recovery-quorum-control-plane Helm chart into the zone's dr-<name> namespace on the quorum cluster.
4. Deploys the site-recovery-workload-control-plane Helm chart into trilio-site-recovery-system on both workload clusters.
5. Creates the zone's default DRBDReplicationPolicy (<zoneName>-default, isDefault: true).

The Helm charts are embedded in the operator binary — you never run helm install directly. This controller is also responsible for creating the OpenShift Route, ConsoleLink, and oauth-proxy redirect annotation that make the TSR web console reachable.

Status conditions it reports on ProtectionZone: ClustersValidated, QuorumStackReady, WorkloadStacksReady, ReplicationPolicyReady, Ready.

site-recovery-quorum-control-plane

Type: Unified Go controller manager (Deployment) Binary: cmd/quorum-control-plane (/quorum-manager)

The per-zone control plane, deployed once per protection zone into dr-<name>. A single manager container (pod output 1/1) runs all of the following reconcilers:

• failover-controller — Watches FailoverRequest CRs; orchestrates planned and unplanned failover by coordinating VM shutdown, DRBD volume promotion on the DR cluster, and VM startup.
• protection-controller — Watches ProtectionRequest CRs; validates the source VM, provisions a DRBDResource pair covering all VM disks on both workload clusters, and switches the VM to DRBD-backed frontend PVCs. Phases: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.
• pg-sync-controller — Watches ProtectionGroup CRs; keeps group metadata synchronized between primary and DR clusters, converting DataVolume references to PVC references and maintaining standby VM specs.
• replication-monitor — Aggregates per-volume DRBD replication state into ReplicationGroupStatus CRs (auto-created as <pg-name>-replication) and writes RPOEvent records on violations.
• test-failover-controller — Watches TestFailover CRs; creates volume snapshots, provisions test PVCs, starts test VMs in an isolated namespace (status.testNamespace, system-generated as site-recovery-<pg>-<random>), runs verification probes, and cleans up.

This Deployment also serves all quorum-side admission webhooks (scoped to the zone's namespace by a namespaceSelector).

Reaches: Both workload clusters via mounted kubeconfig Secrets (kubeconfigs.cluster1 / kubeconfigs.cluster2 chart values). Never the reverse.

TSR web console

Type: Two-container web application

• site-recovery-frontend — Next.js/React UI.
• site-recovery-backend — Python Flask API under gunicorn.
• On OpenShift, the ProtectionZone controller creates an oauth-proxy Route named site-recovery and a ConsoleLink for browser access. Zone-scoped pages derive the dr-<zoneId> namespace; a zone in a differently named namespace lists but does not render correctly.
• On upstream Kubernetes the pods are deployed but no ingress is created. Browser access is not supported on upstream Kubernetes.

Workload cluster components (primary and DR)

All workload components run in trilio-site-recovery-system.

site-recovery-workload-control-plane

Type: Kubernetes controller manager (Deployment) Binary: cmd/workload-control-plane (/workload-manager)

Deployed to each workload cluster by the ProtectionZone controller. Hosts:

• The DRBDResource placement reconciler — handles node binding, backing-PVC provisioning, per-VM data anchor pods, and storage teardown. It talks only to its own cluster's API server.
• The DRBDResource admission webhook — validates DRBDResource CRs written by the quorum-side protection-controller. Rejects CRs outside a dr--prefixed namespace.
• KubeVirt VM guards — deny live migration of DRBD-backed VMs and freeze the disk topology of a protected VM for its protection lifetime.

The manager cache is unscoped (no WATCH_NAMESPACE restriction on informers) because DRBDResource CRs and their associated VMs, PVCs, and PVs live in whichever namespace each protected VM occupies — not a single fixed namespace.

drbd-node-agent

Type: DaemonSet (one pod per worker node) Component label: app.kubernetes.io/component: drbd-node-agent DaemonSet name: <release>-site-recovery-workload-control-plane-agent

Runs on every schedulable node of both workload clusters. Owns the DRBDResource CRs whose spec.nodeName matches its node. Responsibilities:

• Loads the DRBD kernel module via a privileged init container (selects the kmod by kernel-ABI matching, no exact uname -r required).
• Manages DRBD block devices for individual VM volumes and reports per-volume sync progress and connection state back to DRBDResource status.
• Runs a health sampler that pauses VMs when their backing DRBD device enters a StandAlone state.
• Applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its node at startup and removes it once DRBD converges, preventing workloads from scheduling on nodes with unconverged replication.
• Runs an orphan sweeper that detects and cleans up DRBD resources on its node with no owning DRBDResource CR.

Leader election is off — each pod owns exactly its own node.

Custom resources by cluster

Important: DRBDResource CRs are never registered on the quorum cluster. Running kubectl get dres against the quorum cluster returns server doesn't have a resource type.

The following traces the lifecycle of protecting a VM and triggering a test failover, showing exactly which component acts at each step.

Phase 1: Zone bootstrap

1. You create a ProtectionZone CR in a dr-<name> namespace on the quorum cluster, referencing kubeconfig Secrets for both workload clusters.
2. The protection-zone-controller reconciles it:
   • Reads each kubeconfig Secret, resolves a rest.Config, and calls the workload cluster's API server (authenticated GET of trilio-site-recovery-system) to verify reachability. If the credential is expired, ClustersValidated goes False with reason AuthenticationFailed.
   • Runs SelfSubjectAccessReview checks for every verb in the canonical permission table against each workload cluster. Missing verbs degrade the zone with InsufficientClusterPermissions and list the exact missing grants.
   • Claims a zone-exclusivity lease on trilio-site-recovery-system by annotating that namespace with siterecovery.trilio.io/owner-zone-namespace-uid. If another zone already holds the lease, the zone degrades with ClusterAlreadyClaimed.
   • Deploys site-recovery-quorum-control-plane (embedded chart) into dr-<name> on the quorum cluster, injecting the kubeconfig Secret references as chart values under kubeconfigs.cluster1 / kubeconfigs.cluster2.
   • Deploys site-recovery-workload-control-plane (embedded chart) into trilio-site-recovery-system on both workload clusters.
   • Creates the default DRBDReplicationPolicy named <zoneName>-default with isDefault: true and drbdProtocol: C.
3. The zone transitions through Provisioning → Available once all Helm rollouts complete.

Phase 2: Protecting a VM

1. You create a ProtectionRequest CR on the quorum cluster in the dr-<name> namespace, specifying spec.sourceCluster: cluster1 and spec.virtualMachine.name/namespace.
2. The ProtectionRequest admission webhook (running inside site-recovery-quorum-control-plane) validates at admission:
   • sourceCluster must be cluster1 or cluster2.
   • replicationProtocol: A is rejected (ReplicationProtocolA is not supported yet).
   • replicationProtocol: C with a non-zero rpo is rejected.
   • The default replication config (protocol C, rpo 0) is filled in by the schema if omitted.
3. The protection-controller reconciles the ProtectionRequest:
   • Validates the source VM exists and is running on the source workload cluster (reached via the mounted kubeconfig Secret).
   • Creates a DRBDResource CR in the VM's namespace on each workload cluster (named drbd-<namespace>-<vm>-<hash8>) — one per cluster side, covering all VM disks as volumes.
   • Transitions the ProtectionRequest through: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.
4. On each workload cluster, the DRBDResource admission webhook validates the incoming CR, then the workload-control-plane placement reconciler:
   • Pins the VM's anchor pod to the source node on the primary cluster and selects a DR-side node on the DR cluster (latching that binding immutably — nodeName cannot be changed after being set).
   • Provisions backing PVCs and LVM thin-pool volumes for DRBD.
   • Applies the siterecovery.trilio.io/not-ready:NoSchedule taint to the target node until DRBD converges.
5. The drbd-node-agent on each pinned node:
   • Configures DRBD kernel devices for each VM volume.
   • Establishes the TCP replication connection on ports 7000–8000 (one port per VM volume) between the primary and DR node-agent pods.
   • Uses Protocol C (synchronous): every write is acknowledged only after it has been committed to disk on both sides, guaranteeing RPO=0.
   • Reports per-volume syncProgress, connectionState, and diskState back to the DRBDResource status.
6. The replication-monitor reconciler auto-creates a ReplicationGroupStatus CR named <pg-name>-replication and polls DRBD state on the schedule defined by spec.pollingIntervalSeconds (default 15s). On RPO violations it writes RPOEvent CRs stamped with the label siterecovery.trilio.io/protection-group: <pg>.
7. The KubeVirt VM guard webhooks (on both workload clusters) now deny any VirtualMachineInstanceMigration for this VM with: live migration of VM "<name>" is not allowed: it uses DRBD-protected volumes with node-pinned PVs.

Phase 3: Test failover

1. You create a TestFailover CR on the quorum cluster in dr-<name>, referencing a ProtectionGroup by name and namespace. The TestFailover admission webhook:
   • Defaults spec.protectionGroupRef.namespace from the TFO's own namespace if omitted.
   • Requires the referenced ProtectionGroup to exist (referential integrity at admission).
   • Rejects creation if a FailoverRequest in Pending or InProgress state exists for the same protection group.
   • Validates reachability probe specs: TCP probes may not carry endpoint or expectedStatus; probe timeout must not exceed 10 minutes.
   • Validates spec.verification.reachability[].vmName against the ProtectionGroup's member VM list.
2. The test-failover-controller drives the TestFailover through phases: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded → CleaningUp → Cleaned.
   • Creates volume snapshots of the DR-side DRBD-backed PVCs.
   • Provisions test PVCs from those snapshots.
   • Starts test VMs in an isolated namespace (status.testNamespace, generated as site-recovery-<pg>-<random> — never user-provided).
   • Runs TCP or HTTP reachability probes against test VM IPs as configured in spec.verification.reachability.
   • On cleanupPolicy: Automatic (the default), cleans up all test resources after spec.retentionTime (default 2h).
3. The primary production VMs and their DRBD replication are not interrupted — the test uses snapshots only.

Phase 4: Planned failover

1. You create a FailoverRequest CR with failoverType: planned and targetCluster: cluster2.
2. The failover-controller receives the CR and:
   • Stops VMs gracefully on the source cluster.
   • Waits for DRBD to report all volumes Consistent (RPO=0 guarantee for planned failover).
   • Promotes DRBD volumes to Primary on the DR cluster.
   • Starts VMs on the DR cluster.
   • Reports per-VM progress in status.vmStatuses.
3. The ProtectionGroup status.activeClusterID transitions from cluster1 to cluster2.

Control-plane direction

All cross-cluster control flow is quorum → workload only. The quorum control plane reads kubeconfig Secrets to reach workload clusters. Workload clusters never initiate connections back to quorum. The drbd-node-agent talks only to its own cluster's API server.

Single binary per tier, not per reconciler

All quorum-side reconcilers (failover, protection, pg-sync, replication-monitor, test-failover) run inside a single site-recovery-quorum-control-plane Deployment. Similarly, the workload-side placement reconciler and admission webhooks run in a single site-recovery-workload-control-plane Deployment. This consolidation means:

• One set of RBAC rules per cluster tier.
• One TLS certificate and one webhook endpoint namespace selector per zone.
• Simpler rollout gating — the ProtectionZone controller waits for one Deployment rollout, not five.

The trade-off is that a panic in any reconciler restarts the entire manager pod. Leader election (leaderElect: true by default in the charts) serializes writes across replicas.

ProtectionZone controller is a separate binary from the quorum control plane

The ProtectionZone controller (cmd/protectionzone) is deliberately separated from the quorum control plane (cmd/quorum-control-plane). The quorum control plane is itself deployed per zone by the ProtectionZone controller — it cannot run inside the thing it is bootstrapping. This separation also means the ProtectionZone admission webhooks and the reconciler run independently from, and before, the per-zone control planes that depend on them.

Helm charts are embedded in the operator binary

The three Helm charts (site-recovery-quorum-control-plane, site-recovery-workload-control-plane, site-recovery-protectionzone-controller) are compiled into the ProtectionZone controller binary via go:embed. There is no external chart registry. This means:

• The operator is self-contained — no registry pull at reconcile time.
• Chart values are an internal surface; customers never run helm install or helm upgrade.
• The ProtectionZone controller server-side-applies chart CRDs on every reconcile using the field manager site-recovery-pz-controller with ForceOwnership, so out-of-band CRD edits are reverted within one reconcile.

Quorum cache is namespace-scoped; workload cache is unscoped

The quorum manager's cache is scoped to the zone namespace (watchNamespace = dr-<name>). All quorum CRDs live in that single namespace, so scoping is safe and keeps informer memory bounded per zone.

The workload manager's cache is unscoped (WATCH_NAMESPACE is validated but not applied to cache scoping). Protected VMs — and therefore their DRBDResource CRs, PVCs, and PVs — live in whichever application namespace each VM occupies, not a fixed namespace. Scoping the workload informer to a single namespace would leave the placement reconciler and the KubeVirt VM migration guard blind to every other VM namespace.

One port per VM, fixed range 7000–8000

DRBD replication uses one TCP port per replicated VM volume, allocated from the range 7000–8000 inclusive. This is a hard constraint: the range supports up to approximately 1,000 protected VMs. Port 8000 is not optional — a firewall rule that stops at 7999 silently prevents replication on one volume. The DRBDReplicationPolicy.spec.networkConfig.portRange field is accepted by the API but not consumed by the replication stack in Tech Preview; ports are always allocated from this fixed range.

DRBDReplicationPolicy is system-managed, not user-authored

When a ProtectionZone is created, the ProtectionZone controller automatically creates a DRBDReplicationPolicy named <zoneName>-default with isDefault: true. Only this default policy in the ProtectionRequest's namespace is ever consulted. Customers do not author or modify replication policies. This eliminates a class of configuration errors where a user-created policy disagrees with the zone's storage layout.

Protocol C only, RPO always 0

Only DRBD Protocol C (synchronous replication) is supported. The admission webhook rejects replicationProtocol: A with ReplicationProtocolA is not supported yet and rejects a non-zero rpo with replicationProtocol: C. This is a deliberate Tech Preview scope constraint: synchronous replication requires sub-50ms RTT between primary and DR clusters, but it guarantees zero data loss and eliminates the need for RPO configuration and compliance tooling during initial deployments.

VM node pinning is immutable

A protected VM is pinned to one node per cluster for the lifetime of the protection. nodeName on a DRBDResource is immutable once set. This is required because DRBD replication is node-to-node at the kernel level — the DRBD device on the primary node replicates directly to the DRBD device on the DR node. There is no cluster-level storage path. Moving a VM's node would require tearing down and recreating the DRBD resource, which requires a full initial resynchronization. Re-protection is always a full unprotect + protect cycle.

Synchronous replication requires low-latency interconnect

Protocol C guarantees RPO=0 by holding every write until the DR cluster acknowledges disk commit. This requires round-trip latency below approximately 50ms between primary and DR worker nodes. Deployments with higher latency — multi-region pairs with 100–200ms RTT — are not supported in Tech Preview. If your DR requirement allows non-zero RPO and higher latency, TSR's synchronous-only mode is not the right fit for that use case in the current release.

Protected VMs cannot be live-migrated or node-drained

Because DRBD is node-to-node and the VM's nodeName is immutable, oc adm drain of a node hosting a running protected VM will not complete — every KubeVirt migration path is denied at admission. Planned maintenance of a node hosting a protected VM must be done in-place (reboot is fine; modules reload automatically). Permanently replacing such a node requires unprotecting the VM first, at the cost of a full initial resynchronization. Plan your cluster topology and maintenance windows around this constraint before enabling protection.

Re-protection always costs a full resync

There is no incremental re-protect operation. Any change to a protected VM's protection — node, disk topology, or target cluster — requires a full unprotect followed by a full initial resynchronization. For large VM disks over a bandwidth-limited link this can take hours. Do not protect VMs whose disk layout is still evolving.

Scale ceiling of approximately 1,000 VMs per zone

The 7000–8000 TCP port range provides at most 1,001 ports. With 15 replicated disks per VM as the maximum, and one port per volume, the practical ceiling depends on your average disk count per VM. There is exactly one protection zone per namespace and exactly two workload clusters per zone. Multi-tenant scale-out requires multiple zones (each in its own dr-<name> namespace), each with its own quorum control plane Deployment.

No quorum cluster storage or VMs

The quorum cluster is a pure management plane — it holds no VM workloads, no DRBD state, and no storage. Its loss does not immediately impact running VMs or in-progress replication (DRBD replicates directly between workload nodes), but it does prevent new DR operations (failover, protection changes) until quorum is restored. This is an intentional trade-off: isolating the management plane avoids correlated failures between the infrastructure being protected and the controller that protects it.

OpenShift only; upstream Kubernetes is not supported

Three capabilities are OpenShift-specific and have no equivalent on upstream Kubernetes:

• The DRBD node agent relies on the stock privileged SCC. On upstream Kubernetes you must configure your own Pod Security policy.
• Test failover network isolation uses OVN-Kubernetes NetworkPolicy rules targeting the openshift-dns namespace. On a non-OVN CNI or a kube-dns cluster this rule silently fails to isolate test VMs.
• The TSR web console Route and ConsoleLink are created only on OpenShift. On upstream Kubernetes no ingress is created and browser access is not supported.

Web console limitations on upstream Kubernetes

The site-recovery-frontend and site-recovery-backend pods are deployed by the operator on all clusters including upstream Kubernetes, but no ingress, Route, or authentication path is provided. On upstream Kubernetes you can reach the backend via port-forward for development purposes, but this is not a supported use case.

LINSTOR backend is not supported

The ProtectionZone CRD schema accepts storageBackendMode: Linstor, but the admission webhook rejects it with Linstor backend is not supported yet; use DRBD. DRBD is the only supported storage backend in Tech Preview. Do not attempt to configure a LINSTOR backend.

Several API fields are accepted but not consumed

The following fields are accepted by the API and pass schema validation but have no effect in Tech Preview:

• DRBDReplicationPolicy.spec.networkConfig.portRange — ports are always allocated from 7000–8000.
• ProtectionGroup.spec.sla.rpoTargetSeconds and rtoTargetSeconds.
• ReplicationGroupStatus.spec.rpoObjectiveSeconds.
• FailoverRequest.spec.drainTimeoutSeconds — accepted and range-checked but not consumed by the failover controller.
• replicationMode: Asynchronous on DRBDReplicationPolicy — the webhook does not block it at the policy level, but ProtectionRequest rejects Protocol A at admission.

Do not build automation or SLA dashboards that depend on these fields producing an effect in the current release.