Controller Architecture

This page explains how the Trilio Site Recovery controllers watch and reconcile custom resources across your quorum, primary, and DR clusters. Understanding the controller architecture helps you reason about reconciliation timing, debug unexpected states, and design GitOps workflows that align with how the system converges toward desired state. Each controller has a defined scope—either a single cluster or multi-cluster—and communicates with other controllers exclusively through Kubernetes Custom Resource Definitions rather than direct API calls, preserving a clean separation of concerns and a full audit trail in etcd.

Before reading this guide, you should be familiar with:

• Kubernetes operator pattern — controllers that watch resources and reconcile actual state toward desired state
• OpenShift 4.14 or later — the only supported platform for this release
• KubeVirt / OpenShift Virtualization (CNV) ≥ 1.0 — VMs are managed as VirtualMachine and VirtualMachineInstance custom resources
• DRBD kernel module ≥ 9.x — block-level replication runs in the kernel on every worker node
• Three clusters — a quorum cluster (management plane), a primary cluster, and a DR cluster
• Helm ≥ 3.0 and oc CLI — used to deploy charts and apply manifests
• Kubeconfig files for all three clusters — controllers on the quorum cluster reach across to the primary and DR API servers
• TCP port 6443 open from the quorum cluster to the primary and DR API servers
• TCP ports 7000–7999 open between primary and DR worker nodes for DRBD replication traffic

The controller architecture is deployed in three Helm chart installations. Install them in the order shown below.

Step 1 — Install the ProtectionZone controller on the quorum cluster

The site-recovery-protectionzone-controller chart registers the ProtectionZone CRD and its admission webhook. It must exist before the quorum control plane chart is installed because the quorum control plane chart references those CRDs.

helm upgrade --install site-recovery-protectionzone-controller \
  oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
  --namespace dr-system \
  --create-namespace \
  --kubeconfig ~/.kube/quorum-cluster.kubeconfig

Verify the webhook is ready:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get deployment -n dr-system -l app=site-recovery-protectionzone-controller

Wait until READY shows 1/1 before continuing.

Step 2 — Install the quorum control plane on the quorum cluster

The site-recovery-quorum-control-plane chart deploys the unified Go controller manager that packages the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers into a single Deployment per DR namespace.

helm upgrade --install site-recovery-quorum-control-plane \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --namespace dr-<name> \
  --create-namespace \
  --set primaryCluster.kubeconfigSecret=primary-kubeconfig \
  --set drCluster.kubeconfigSecret=dr-kubeconfig \
  --kubeconfig ~/.kube/quorum-cluster.kubeconfig

Replace <name> with your deployment name. The quorum control plane namespace follows the convention dr-<name> and isolates each DR deployment from others on the same quorum cluster.

Verify all reconcilers are running:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get pods -n dr-<name> -l app=site-recovery-quorum-control-plane

Step 3 — Install the workload control plane on the primary cluster

The site-recovery-workload-control-plane chart installs the drbd-node-agent DaemonSet, admission webhooks, and supporting RBAC on the workload side.

helm upgrade --install site-recovery-workload-control-plane \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --namespace dr-<name> \
  --create-namespace \
  --kubeconfig ~/.kube/primary-cluster.kubeconfig

Verify the DaemonSet is fully scheduled — one pod per worker node:

oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
  get daemonset -n dr-<name> drbd-node-agent

Step 4 — Install the workload control plane on the DR cluster

Repeat Step 3 using the DR cluster kubeconfig:

helm upgrade --install site-recovery-workload-control-plane \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --namespace dr-<name> \
  --create-namespace \
  --kubeconfig ~/.kube/dr-cluster.kubeconfig

Step 5 — Apply the CRD manifests

All Site Recovery CRDs are bundled with the charts and applied automatically during Helm installation. To verify they are registered:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get crd | grep siterecovery.trilio.io

Expected output includes protectiongroups, protectionrequests, failoverrequests, testfailovers, drbdreplicationpolicies, drbdresources, rpoevents, replicationgroupstatuses, and protectionzones.

The controllers themselves are configured through the Helm values of each chart. The Custom Resources they watch are the primary configuration surface at runtime. Key options are described below.

Quorum control plane Helm values

Workload control plane Helm values

ProtectionZone — the deployment anchor CRD

A ProtectionZone resource must be created before protecting any VM. It defines which clusters participate in a DR deployment and which storage backend mode to use.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: my-zone
  namespace: dr-<name>
spec:
  displayName: "Production DR Zone"        # required — human-readable label
  storageBackendMode: drbd-operator         # required — "drbd-operator" for this release
  clusters:                                 # required — list of cluster references
    - name: primary
    - name: dr
  description: "Primary to DR protection"  # optional
  logging:                                  # optional
    level: info

storageBackendMode controls which storage integration the controllers use. Only drbd-operator is supported in this release.

DRBDReplicationPolicy — replication parameters

A DRBDReplicationPolicy defines how volumes replicate between clusters. One policy can be marked as the cluster default with isDefault: true.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-policy
  namespace: dr-<name>
spec:
  clusters:                     # required — references to primary and DR cluster entries
    - name: primary
    - name: dr
  drbdProtocol: C               # "C" = synchronous (RPO=0, requires <50ms RTT)
                                # "A" = asynchronous (seconds RPO, any distance)
  replicationMode: synchronous  # mirrors drbdProtocol choice for readability
  isDefault: false              # set true to apply this policy when no explicit policy is referenced
  diskConfig: {}                # optional disk-level DRBD tuning
  networkConfig: {}             # optional network endpoint overrides
  resyncConfig: {}              # optional resync rate limits
  rpo:                          # optional RPO alerting thresholds
    objectiveSeconds: 30

Choose drbdProtocol: C only when your primary-to-DR round-trip latency is reliably below 50 ms. Use drbdProtocol: A for geographically separated clusters.

ReplicationGroupStatus — polling configuration

ReplicationGroupStatus resources are created automatically by the replication-monitor reconciler, but you can adjust polling behavior:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ReplicationGroupStatus
metadata:
  name: my-pg-status
  namespace: dr-<name>
spec:
  protectionGroupRef:           # required
    name: my-pg
  replicationProtocol: C
  pollingIntervalSeconds: 30    # how often to refresh per-volume sync state
  rpoObjectiveSeconds: 30       # lag threshold before an RPOEvent is emitted

How controllers communicate

All controllers communicate through CRDs — never through direct API calls to other controllers or to VM objects. The communication flow is:

You (oc apply) → CRD on quorum cluster → quorum control plane reconcilers
                                             ↓
                                   CRD patches on primary/DR clusters
                                             ↓
                                   drbd-node-agent (DaemonSet) + KubeVirt

This means you interact with the system by creating or updating custom resources, and the controllers converge the actual state toward what those resources declare.

Controller scopes

Protecting a VM (DRBD Operator model)

Create a ProtectionRequest on the quorum cluster. The protection-controller validates the VM, provisions a DRBDResource pair covering all VM disks, and switches the VM to DRBD-backed frontend PVCs.

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f protection-request.yaml -n dr-<name>

Watch the protection phases:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get protectionrequest my-vm-pr -n dr-<name> -w

Phases progress from Pending → Validating → ProvisioningDRBDResource → SwitchingPVCs → Protected.

Triggering a planned failover

Create a FailoverRequest on the quorum cluster referencing the target Protection Group and cluster:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f failover-request.yaml -n dr-<name>

The failover-controller acquires a per-Protection-Group Lease, patches ProtectionGroup.spec.desiredState: stopped on the source cluster, waits for VMs to stop, promotes DRBD volumes on the DR cluster, then patches ProtectionGroup.spec.desiredState: running on the DR cluster.

Running a non-disruptive test failover

Create a TestFailover resource. The test-failover-controller creates volume snapshots, provisions test PVCs, and starts test VMs in an isolated namespace without touching production workloads:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f test-failover.yaml -n dr-<name>

Phases progress from CreatingSnapshots → ProvisioningTestPVCs → StartingTestVMs → VerifyingData → Succeeded, followed by CleaningUp.

Monitoring replication health

The replication-monitor reconciler continuously updates ReplicationGroupStatus and emits RPOEvent resources when replication lag exceeds the configured threshold.

# Overall health per Protection Group
oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get replicationgroupstatus -n dr-<name>

# Recent RPO violations
oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get rpoevent -n dr-<name> --sort-by=.spec.timestamp

ReplicationGroupStatus reports an aggregated status of Healthy, Degraded, or Critical with per-volume detail and a last-sync timestamp. RPOEvent records each violation with its severity, lag duration in seconds, and affected volumes for compliance auditing.

Example 1 — Create a ProtectionZone

This resource anchors a DR deployment and must be created first.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod-zone
  namespace: dr-prod
spec:
  displayName: "Production Zone"
  storageBackendMode: drbd-operator
  clusters:
    - name: primary
    - name: dr
  description: "Production workloads DR"

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f protectionzone.yaml

Expected output:

protectionzone.siterecovery.trilio.io/prod-zone created

Example 2 — Create a DRBDReplicationPolicy for synchronous replication

Use Protocol C when primary-to-DR latency is below 50 ms.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-policy
  namespace: dr-prod
spec:
  clusters:
    - name: primary
    - name: dr
  drbdProtocol: C
  replicationMode: synchronous
  isDefault: true
  rpo:
    objectiveSeconds: 0

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f drbd-replication-policy.yaml

Expected output:

drbdreplicationpolicy.siterecovery.trilio.io/sync-policy created

Example 3 — Protect a VM with a ProtectionRequest

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: web-vm-pr
  namespace: dr-prod
spec:
  sourceCluster: primary          # required
  virtualMachine:                  # required
    name: web-vm
    namespace: workloads
  replicationConfig:
    policyRef:
      name: sync-policy

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f protection-request.yaml

Watch protection progress:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get protectionrequest web-vm-pr -n dr-prod -w

Expected output (abbreviated):

NAME        PHASE                    AGE
web-vm-pr   Validating               5s
web-vm-pr   ProvisioningDRBDResource 12s
web-vm-pr   SwitchingPVCs            28s
web-vm-pr   Protected                45s

Example 4 — Trigger a planned FailoverRequest

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: prod-failover
  namespace: dr-prod
spec:
  protectionGroupRef:              # required
    name: prod-pg
  targetCluster: dr                # required
  failoverType: planned
  drainTimeoutSeconds: 120
  batchBootTimeoutSeconds: 300

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f failover-request.yaml

Monitor progress:

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get failoverrequest prod-failover -n dr-prod -w

Expected output:

NAME            PHASE              AGE
prod-failover   Pending            2s
prod-failover   StoppingOnSource   8s
prod-failover   PromotingVolumes   35s
prod-failover   StartingOnTarget   58s
prod-failover   Completed          3m12s

Example 5 — Run a TestFailover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: prod-tf-01
  namespace: dr-prod
spec:
  protectionGroupRef:              # required
    name: prod-pg
  cleanupPolicy: OnSuccess
  timeout: 30m
  batchBootTimeoutSeconds: 300
  verification:
    enabled: true

oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  apply -f test-failover.yaml

Expected output (watch):

NAME          PHASE             AGE
prod-tf-01    CreatingSnapshots  4s
prod-tf-01    VerifyingData      2m10s
prod-tf-01    Succeeded          4m55s
prod-tf-01    CleaningUp         5m01s

Example 6 — Check replication health and RPO events

# Aggregated health
oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get replicationgroupstatus -n dr-prod -o wide

NAME              HEALTH    PROTOCOL   LAST-SYNC              AGE
prod-pg-status    Healthy   C          2024-06-10T14:22:01Z   6h

# RPO violations in the last hour
oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  get rpoevent -n dr-prod \
  --sort-by=.spec.timestamp

NAME                  TYPE      SEVERITY   RPO-AT-EVENT   AGE
prod-pg-rpo-001       Lag       warning    12s            47m

Issue 1 — ProtectionRequest stuck in Validating

Symptom: oc get protectionrequest shows PHASE: Validating for more than two minutes.

Likely cause: The protection-controller on the quorum cluster cannot reach the primary cluster API server, or the VirtualMachine referenced in spec.virtualMachine does not exist in the specified namespace.

Fix:

1. Verify the primary cluster kubeconfig Secret is present and valid in the DR namespace:
   Copy

   oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
     get secret primary-kubeconfig -n dr-<name>
   

2. Confirm the VM exists on the primary cluster:
   Copy

   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     get vm <vm-name> -n <vm-namespace>
   

3. Check the quorum control plane logs:
   Copy

   oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
     logs -n dr-<name> -l app=site-recovery-quorum-control-plane \
     --since=10m | grep -i protection
   

Issue 2 — FailoverRequest stuck in StoppingOnSource

Symptom: oc get failoverrequest shows PHASE: StoppingOnSource beyond the drainTimeoutSeconds value.

Likely cause: One or more VMs in the Protection Group failed to stop cleanly within the drain window, or the Protection Group controller on the source cluster is not running.

Fix:

1. Check the ProtectionGroup status on the primary cluster to see which VMs have not stopped:
   Copy

   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     get protectiongroup <pg-name> -n dr-<name> -o yaml | grep -A 20 status:
   

2. Verify the workload control plane pods are healthy on the primary cluster:
   Copy

   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     get pods -n dr-<name> -l app=site-recovery-workload-control-plane
   

3. For a genuine disaster scenario where the primary is unavailable, delete the FailoverRequest and create a new one with failoverType: unplanned.

Issue 3 — drbd-node-agent DaemonSet has pods not ready

Symptom: oc get daemonset drbd-node-agent -n dr-<name> shows DESIRED greater than READY.

Likely cause: The DRBD kernel module is not loaded on one or more worker nodes, or the node lacks the required TCP port access (7000–7999).

Fix:

1. Identify which nodes have unready pods:
   Copy

   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     get pods -n dr-<name> -l app=drbd-node-agent -o wide | grep -v Running
   

2. On the affected node, verify the DRBD module:
   Copy

   # Via a debug pod on the node
   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     debug node/<node-name> -- lsmod | grep drbd
   

   If no output appears, load the module or ensure the DRBD kernel package (≥ 9.x) is installed on the node.
3. Check drbd-node-agent pod logs for the specific error:
   Copy

   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     logs -n dr-<name> <drbd-node-agent-pod-name> --previous
   

Issue 4 — ReplicationGroupStatus shows Degraded or Critical

Symptom: oc get replicationgroupstatus -n dr-<name> reports a status other than Healthy.

Likely cause: DRBD replication has lost connectivity between primary and DR worker nodes, or a volume has gone out of sync.

Fix:

1. Check for recent RPO events to identify which volumes are affected:
   Copy

   oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
     get rpoevent -n dr-<name> -o yaml
   

2. Verify network connectivity between primary and DR worker nodes on ports 7000–7999.
3. Inspect individual DRBDResource status on both clusters:
   Copy

   oc --kubeconfig ~/.kube/primary-cluster.kubeconfig \
     get drbdresource -n dr-<name> -o wide
   
   oc --kubeconfig ~/.kube/dr-cluster.kubeconfig \
     get drbdresource -n dr-<name> -o wide
   

4. If a volume is listed as OutOfSync, allow it to resync naturally once connectivity is restored. Do not trigger a failover while any volume is resyncing unless it is a genuine disaster scenario.

Issue 5 — TestFailover remains in CreatingSnapshots indefinitely

Symptom: A TestFailover resource does not advance past CreatingSnapshots.

Likely cause: The Containerized Data Importer (CDI) volume snapshot capability is not available on the DR cluster, or the VolumeSnapshotClass is not configured.

Fix:

1. Verify that a VolumeSnapshotClass exists on the DR cluster:
   Copy

   oc --kubeconfig ~/.kube/dr-cluster.kubeconfig \
     get volumesnapshotclass
   

2. Check test-failover-controller logs for snapshot errors:
   Copy

   oc --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
     logs -n dr-<name> -l app=site-recovery-quorum-control-plane \
     --since=15m | grep -i snapshot
   

3. Confirm CDI is installed and healthy on the DR cluster:
   Copy

   oc --kubeconfig ~/.kube/dr-cluster.kubeconfig \
     get pods -n cdi
   

Issue 6 — Collecting a diagnostic bundle

When none of the above steps resolve the issue, collect a full diagnostic bundle using the tsr-gather must-gather tool and attach it to your support case:

oc adm must-gather \
  --image=registry.trilio.io/tsr-gather:latest \
  --kubeconfig ~/.kube/quorum-cluster.kubeconfig \
  -- /usr/bin/tsr-gather

tsr-gather collects logs, CRD state, and configuration from the quorum cluster and all associated workload clusters into a single bundle. It does not capture secrets or sensitive credentials.