Controller Architecture

This page explains how the Trilio Site Recovery controllers watch Custom Resources and drive reconciliation toward your desired failover state. Each controller runs in-cluster on the quorum cluster (or on the workload clusters for placement and DRBD node agents), continuously comparing the desired state expressed in a CR against the actual state of VMs, storage, and replication, then taking the minimum necessary action to close the gap. Understanding the controller architecture helps you interpret status fields, diagnose stuck operations, and reason about the ordering guarantees that keep your VMs safe during planned and unplanned failovers.

Before reading this page you should already have:

• A running quorum cluster with site-recovery-protectionzone-controller and site-recovery-quorum-control-plane deployed via Helm
• site-recovery-workload-control-plane (including the drbd-node-agent DaemonSet) deployed on both your primary and DR clusters
• kubectl access to all three clusters
• Familiarity with Kubernetes operator concepts (reconcile loops, finalizers, status conditions)
• Kubernetes ≥ 1.28, OpenShift ≥ 4.14 (if using OpenShift), KubeVirt ≥ 1.0, DRBD kernel module ≥ 9.0

The controllers are not installed independently — they are delivered as part of two Helm charts. Install them in the following order.

Step 1 — Install the ProtectionZone controller (quorum cluster)

This must run before the quorum control plane chart because it installs the CRDs and manages the ProtectionZone lifecycle that provisions all other components.

helm install site-recovery-protectionzone-controller \
  oci://<registry>/site-recovery-protectionzone-controller \
  --namespace trilio-site-recovery-system \
  --create-namespace

Step 2 — Create a ProtectionZone CR

Once the ProtectionZone controller is running, create a ProtectionZone CR. The controller then installs site-recovery-quorum-control-plane on the quorum cluster and site-recovery-workload-control-plane on both workload clusters automatically.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: my-dr-zone
  namespace: dr-my-dr-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      kubeconfigSecretRef:
        name: primary-kubeconfig
        key: kubeconfig
      distributionName: RHEL-CoreOS-9
    - name: dr-site
      clusterID: cluster2
      kubeconfigSecretRef:
        name: dr-kubeconfig
        key: kubeconfig
      distributionName: RHEL-CoreOS-9

kubectl apply -f protection-zone.yaml

Step 3 — Verify the zone is Available

kubectl get pz my-dr-zone -n dr-my-dr-zone

Expected output when all controllers are up and both clusters are reachable:

NAME          DISPLAY NAME         BACKEND   CLUSTERS            STATUS
my-dr-zone    Production DR Zone   DRBD      primary,dr-site     Available

Step 4 — Confirm all controller pods are running

# Quorum cluster
kubectl get pods -n dr-my-dr-zone

# Primary and DR clusters
kubectl get pods -n trilio-site-recovery-system

All controllers read their configuration from the ProtectionZone CR and from CR specs — there are no separate ConfigMaps to edit in normal operation. The key configuration points per controller are described below.

ProtectionZone controller (site-recovery-protectionzone-controller)

The controller probes both workload clusters every 10 minutes while the zone is Available (the defaultReadyResync interval). This ensures that expired kubeconfig credentials are detected even when no spec change triggers a reconcile.

DRBDReplicationPolicy (site-recovery-quorum-control-plane)

The DRBDReplicationPolicy controller validates and acknowledges policies. The policy itself carries all replication tuning — the controller does not apply defaults; that is the admission webhook's responsibility.

ReplicationGroupStatus / replication monitor

FailoverRequest

TestFailover

How the controllers relate to each other

The controllers form a layered pipeline. Understanding their relationships helps you predict how a user action propagates through the system.

User applies CR
      │
      ▼
ProtectionZone controller  ──► deploys ──► site-recovery-quorum-control-plane
                                           site-recovery-workload-control-plane
                                           (including drbd-node-agent DaemonSet)
      │
      ▼
DRBDReplicationPolicy controller  ──► validates policy, sets Ready
      │
      ▼
ProtectionRequest controller  ──► provisions DRBDResource pairs, drives
      │                            Pending → Validating → CreatingDRBD
      │                            → Syncing → ReadyToActivate
      │                            → Activating → Protected
      ▼
pg-sync controller (pgsync)  ──► mirrors VM/PVC/PV definitions
      │                            from active → standby cluster
      │
      ▼
Replication monitor  ──► scrapes DRBD metrics, writes
                          ReplicationGroupStatus, emits RPOEvents
      │
      ▼
FailoverRequest controller  ──► orchestrates planned/unplanned
      │                          failover state machine
      ▼
TestFailover controller  ──► non-disruptive DR validation
                              (DRBD Operator model only)

Cluster topology of controllers:

Watching controller progress

Every controller writes progress to the CR's status.phase and status.conditions. The phase is the primary axis for diagnosing a stuck operation.

# Watch a ProtectionRequest progress through phases
kubectl get pr my-vm-protection -n dr-my-dr-zone -w

# Inspect conditions on a FailoverRequest
kubectl describe fr my-failover -n dr-my-dr-zone

# Check replication health for a ProtectionGroup
kubectl get rgs -n dr-my-dr-zone

# View RPO violation events
kubectl get rpoe -n dr-my-dr-zone

Event-driven reconciliation

All quorum-side controllers are event-driven rather than poll-based. The ProtectionRequest controller watches VM, VMI, PVC, and DRBDResource objects on both workload clusters via remote caches, and re-enqueues the owning ProtectionRequest when any of those objects change. The FailoverRequest controller watches VM and VMI objects on both workload clusters to detect source halt and target Running transitions. Each controller also carries a coarse backstop RequeueAfter (30 seconds for the FailoverRequest, 20 minutes for pgsync) so a missed watch event self-heals.

Status fields to monitor in production

Example 1 — Check the ProtectionZone controller status

After applying a ProtectionZone CR, verify the controller reconciled it successfully:

kubectl get pz my-dr-zone -n dr-my-dr-zone -o wide

Expected output:

NAME          DISPLAY NAME         BACKEND   CLUSTERS            STATUS
my-dr-zone    Production DR Zone   DRBD      primary,dr-site     Available

If the zone is not yet Available, describe it to read the conditions:

kubectl describe pz my-dr-zone -n dr-my-dr-zone

The ClustersValidated condition shows whether both workload clusters answered the reachability probe. The Ready condition shows whether the Helm releases rolled out.

Example 2 — Watch a ProtectionRequest advance through phases

The ProtectionRequest controller drives a VM from unprotected to fully replicated through a typed phase sequence. Create a ProtectionRequest and watch it progress:

kubectl apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-vm
  namespace: dr-my-dr-zone
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: production
  replicationConfig:
    replicationProtocol: C
EOF

kubectl get pr protect-web-vm -n dr-my-dr-zone -w

Expected phase progression:

NAME              VM       NAMESPACE    PHASE            SYNC   AGE
protect-web-vm    web-vm   production   Pending          0      2s
protect-web-vm    web-vm   production   Validating       0      4s
protect-web-vm    web-vm   production   CreatingDRBD     0      12s
protect-web-vm    web-vm   production   Syncing          34     45s
protect-web-vm    web-vm   production   ReadyToActivate  100    2m10s
protect-web-vm    web-vm   production   Activating       100    2m15s
protect-web-vm    web-vm   production   Protected        100    2m30s

Example 3 — Inspect replication health via ReplicationGroupStatus

The replication monitor automatically creates a ReplicationGroupStatus CR for each ProtectionGroup. Query it to get the current RPO health:

kubectl get rgs -n dr-my-dr-zone

Expected output:

NAME             PROTECTION-GROUP   RPO-STATUS   OUT-OF-SYNC   SYNCED   CONNECTION   LAST-SYNC
my-pg-rgs        my-pg              Met          0             3        Connected    10s

For per-VM detail:

kubectl get rgs my-pg-rgs -n dr-my-dr-zone \
  -o jsonpath='{.status.volumeReplicationStatuses[*].vmName}'

Example 4 — Trigger a planned failover and watch its phase machine

Create a FailoverRequest to perform a planned failover of a ProtectionGroup to cluster2:

kubectl apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-my-pg
  namespace: dr-my-dr-zone
spec:
  protectionGroupRef:
    name: my-pg
  targetCluster: cluster2
  failoverType: planned
  drainTimeoutSeconds: 300
EOF

kubectl get fr failover-my-pg -n dr-my-dr-zone -w

Expected phase progression (planned failover):

NAME              PROTECTION GROUP   TARGET     STATE        PHASE
failover-my-pg    my-pg              cluster2   InProgress   Initializing
failover-my-pg    my-pg              cluster2   InProgress   Validating
failover-my-pg    my-pg              cluster2   InProgress   StoppingSource
failover-my-pg    my-pg              cluster2   InProgress   WaitingForLocks
failover-my-pg    my-pg              cluster2   InProgress   RemovingTaints
failover-my-pg    my-pg              cluster2   InProgress   StartingTarget
failover-my-pg    my-pg              cluster2   InProgress   Monitoring
failover-my-pg    my-pg              cluster2   Completed    Completed

The controller enforces a 5-minute wall-clock deadline across all phases. If the failover does not complete within that budget, the state transitions to Failed.

Example 5 — Run a test failover (DRBD Operator model only)

Create a TestFailover CR to validate DR readiness non-disruptively. The controller selects the standby cluster automatically using a layered heuristic: VM runStrategy (most authoritative) → pg.status.activeCluster → pg.spec.sourceCluster.

kubectl apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-my-pg
  namespace: dr-my-dr-zone
spec:
  protectionGroupRef:
    name: my-pg
  cleanupPolicy: Automatic
  retentionTime: 2h
  verification:
    reachability:
      - type: tcp
        vmName: web-vm
        port: 80
        timeout: 30s
EOF

kubectl get tfo test-my-pg -n dr-my-dr-zone -w

Expected phase progression:

NAME         PROTECTION GROUP   PHASE                PASSED   AGE
test-my-pg   my-pg              PreparingEnvironment          5s
test-my-pg   my-pg              CreatingSnapshots             20s
test-my-pg   my-pg              CreatingVolumes               45s
test-my-pg   my-pg              CreatingVMs                   1m
test-my-pg   my-pg              StartingVMs                   1m30s
test-my-pg   my-pg              Verifying                     2m
test-my-pg   my-pg              Succeeded                true  2m45s

With cleanupPolicy: Automatic, the controller parks the TestFailover at Succeeded and deletes all test resources 2 hours after CompletionTime.

Example 6 — Query RPO violation events

The replication monitor emits an RPOEvent CR whenever RPO status transitions (for example, Met → Violated). Query these for auditing:

kubectl get rpoe -n dr-my-dr-zone \
  -o custom-columns=NAME:.metadata.name,PG:.protectionGroupRef.name,TYPE:.eventType,REASON:.violationReason,TIMESTAMP:.timestamp

Expected output when a violation occurred:

NAME                    PG       TYPE         REASON          TIMESTAMP
my-pg-1735000000        my-pg    RPOViolated  ConnectionLost  2025-01-23T...
my-pg-1735003600        my-pg    SyncCompleted                2025-01-23T...

ProtectionZone stuck in Provisioning or Unavailable

Symptom: kubectl get pz shows Provisioning or Unavailable for an extended time.

Likely causes and fixes:

1. Cluster unreachable — The ClustersValidated condition on the ProtectionZone will show Unreachable for one or both clusters with a message explaining the failure class (connectivity, credential rejection, or missing Secret).

   Copy

   kubectl describe pz <name> -n <namespace>
   

   Fix: verify the kubeconfig Secret exists and the API server is reachable from the quorum cluster on TCP 6443.

2. Helm rollout stalled — The ProtectionZone controller waits for the site-recovery-quorum-control-plane Deployment and the site-recovery-workload-control-plane Deployment and DaemonSet (site-recovery-workload-control-plane-agent) to roll out.

   Copy

   kubectl get deploy,ds -n trilio-site-recovery-system
   

   Fix: check pod events for image pull failures or resource quota exhaustion.

3. Namespace already owned by another zone — A namespace can host only one ProtectionZone. The ClustersValidated condition will show ZoneNamespaceConflict. Fix: create the new zone in a different namespace.

ProtectionRequest stuck in Validating

Symptom: A ProtectionRequest stays in Validating for longer than expected.

Likely cause: The source VM is mid-transition under another site recovery operation (for example, a concurrent ProtectionRequest reconcile is using the vm-control-owner mutex). The controller does not fail the request — it parks it at Validating with a condition explaining the wait and re-validates on a steady timer.

Fix: Check the Kubernetes Events on the ProtectionRequest:

kubectl describe pr <name> -n <namespace>

Wait for the competing operation to complete, or resolve the conflicting state.

Also check: The source VM's StorageClass must not use WaitForFirstConsumer binding mode — the DRBD metadata PVC has no pod consumer and will never bind with that mode. The controller rejects this during validation.

ProtectionRequest stuck in Syncing

Symptom: status.syncProgress does not advance.

Likely cause 1: The DR cluster (the replication peer) is unreachable. The controller sets an AwaitingSuspendedPeerRecovery condition and polls on a steady timer rather than failing.

Fix: Restore connectivity between primary and DR worker nodes on TCP ports 7000–7999 (DRBD replication) and 6443 (API server). Then verify the DRBDResource on the DR cluster is making progress:

kubectl get dres -n <vm-namespace> --context <dr-cluster>

Likely cause 2: The drbd-node-agent DaemonSet pod on the relevant worker node is not running.

kubectl get pods -n trilio-site-recovery-system -l app=site-recovery-workload-control-plane-agent

FailoverRequest stuck in StoppingSource or StartingTarget

Symptom: A FailoverRequest stays in StoppingSource or StartingTarget for many minutes.

Likely cause: Source VMs are not halting within drainTimeoutSeconds (default 300 s), or target VMs are not reaching Running within batchBootTimeoutSeconds (default 300 s). The per-VM status is visible in status.vmStatuses:

kubectl get fr <name> -n <namespace> -o jsonpath='{.status.vmStatuses}'

The FailoverRequest controller enforces a 5-minute wall-clock deadline across all phases. A FailoverRequest that exceeds this deadline transitions to Failed with a timeout reason.

Fix for StoppingSource: Check KubeVirt VMI events on the source cluster for VMs that refuse to stop. You may need to force-stop the VM manually.

Fix for StartingTarget: Check KubeVirt VMI events on the target cluster. Verify the DRBD node agent successfully promoted the DRBD volumes to Primary (inspect status.role on the DRBDResource CRs on the target cluster).

FailoverRequest reaches Failed immediately at Validating

Symptom: The FailoverRequest transitions Initializing → Validating → Failed within seconds.

Likely causes:

• The referenced ProtectionGroup does not exist on the quorum cluster.
• The ProtectionGroup has no virtualMachines.
• A split-brain condition was detected (both clusters claim the same VMs are active).
• The target cluster is unreachable.

Fix: Inspect conditions on the FailoverRequest:

kubectl describe fr <name> -n <namespace>

TestFailover stuck in CreatingSnapshots

Symptom: A TestFailover stays in CreatingSnapshots for longer than expected.

Likely cause: A VolumeSnapshot on the target cluster reached a terminal error state and is not progressing. The controller detects this and marks the TestFailover Failed rather than waiting indefinitely, but there is a short grace window first.

Fix: Check VolumeSnapshot status on the DR cluster:

kubectl get volumesnapshot -n <test-namespace> --context <dr-cluster>

Ensure your CSI driver supports VolumeSnapshots and that the snapshot class is configured.

Note: TestFailover is only supported in the DRBD Operator deployment model.

ReplicationGroupStatus shows RPOStatus: Violated

Symptom: kubectl get rgs shows Violated in the RPO-STATUS column.

Fix: First identify which VMs are affected and why:

kubectl get rpoe -n <namespace> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

The violationReason field classifies the cause:

kubectl get dres -n <vm-namespace> --context <primary-cluster>