Controller Architecture

This page explains how Site Recovery's controllers watch Kubernetes custom resources and continuously reconcile actual cluster state toward the desired failover state. Understanding this architecture helps you reason about what happens when you create or update a FailoverRequest, ProtectionGroup, or ProtectionRequest, and why the system is resilient to partial failures. The design follows the Kubernetes operator pattern: each controller runs in-cluster on the quorum cluster (or on the primary and DR clusters, depending on the controller), observes relevant custom resources through informers, and drives the system toward the declared desired state without requiring manual intervention.

Before reading this page, you should be familiar with the following:

• Kubernetes operator pattern — controllers, reconcile loops, and custom resource definitions
• Site Recovery deployment — a running quorum cluster with controllers deployed, and at least one configured DR deployment (namespace dr-<name>)
• Core CRDs installed — ProtectionGroup, FailoverRequest, ProtectionRequest, DRBDVolume, DRBDReplicationPolicy, TestFailover, RPOEvent, and ReplicationGroupStatus must be present on the relevant clusters
• Kubernetes ≥ 1.28 on all clusters
• KubeVirt ≥ 1.0 on primary and DR clusters
• kubectl configured with access to the quorum, primary, and DR clusters
• DRBD kernel module ≥ 9.0 on primary and DR worker nodes (for DRBD Operator deployments)

You do not need to understand DRBD internals to follow this page, but familiarity with PersistentVolumeClaims and VirtualMachine resources in KubeVirt is assumed.

Controller deployment is handled by the standard Ansible playbooks included with Site Recovery. You do not deploy individual controllers manually. The steps below describe what the playbooks install and where, so you can verify a correct deployment.

Step 1 — Deploy quorum-cluster controllers

The following controllers run as Deployments on the quorum cluster, each in the namespace corresponding to their DR deployment (dr-<name>):

• failover-controller — watches FailoverRequest resources
• protection-controller — watches ProtectionRequest resources (DRBD Operator model only)
• pg-sync-controller — keeps ProtectionGroup metadata synchronized between primary and DR clusters

Verify they are running after the Ansible playbook completes:

kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/quorum-config

Expected output (names may include a generated suffix):

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
failover-controller       1/1     1            1           5m
protection-controller     1/1     1            1           5m
pg-sync-controller        1/1     1            1           5m

Step 2 — Deploy per-cluster controllers

The following controllers run on both the primary cluster and the DR cluster:

• protection-group-controller — manages ProtectionGroup resources and VM lifecycle on that cluster
• test-failover-controller — manages TestFailover resources (DRBD Operator model only)

Verify on the primary cluster:

kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/primary-config

Verify on the DR cluster:

kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/dr-config

Step 3 — Verify the replication-monitor agent

The replication-monitor agent is deployed automatically by the Ansible playbooks on the primary and DR clusters. It writes RPOEvent and ReplicationGroupStatus resources. Confirm it is running:

kubectl get pods -n dr-<name> -l app=replication-monitor --kubeconfig ~/.kube/primary-config

Step 4 — Verify the DRBD Operator (DRBD Operator model only)

If you are using the DRBD Operator deployment model, confirm the DRBD Operator is installed on both the primary and DR clusters:

kubectl get deployments -n drbd-operator --kubeconfig ~/.kube/primary-config
kubectl get deployments -n drbd-operator --kubeconfig ~/.kube/dr-config

Step 5 — Verify the Site Manager UI

The Site Manager UI and its API backend are deployed on the quorum cluster:

kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/quorum-config | grep site-manager

All controllers and agents are managed by the Ansible playbooks. To redeploy or update them, re-run the relevant playbook or use quorum-deployments.sh for guided operational tasks.

Controllers are configured through a combination of their deployment environment variables and the custom resources they watch. The following describes the key behavioral settings you can influence.

Reconciliation interval

All controllers use event-driven reconciliation triggered by Kubernetes watch events. When a resource is created, updated, or deleted, the relevant controller's reconcile loop is invoked immediately. In addition, controllers use a periodic re-sync (typically every 10 seconds for the failover state machine) to handle cases where a watched resource changed on a remote cluster without generating a local event.

You do not configure the reconciliation interval directly; it is determined by the controller implementation.

FailoverRequest — controlling failover behavior

The FailoverRequest CRD is the primary way you influence failover-controller behavior:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production
  namespace: dr-<name>
spec:
  protectionGroup: production-protection-group   # Required: which ProtectionGroup to fail over
  targetCluster: cluster2                        # Required: which cluster should become active
  failoverType: planned                          # "planned" or "unplanned"
  force: false                                   # true bypasses concurrent-failover safety checks

ProtectionGroup — controlling VM group desired state

The protection-group-controller watches spec.desiredState on ProtectionGroup resources and reconciles all VMs in the group to match:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: production-protection-group
  namespace: default
spec:
  desiredState: running       # "running" or "stopped" — drives VM lifecycle
  virtualMachines:
    - name: prod-vm-1
    - name: prod-vm-2

The failover-controller patches spec.desiredState on the appropriate cluster's ProtectionGroup during failover orchestration — you should not change desiredState manually during an in-progress failover.

DRBDReplicationPolicy — controlling replication behavior

In DRBD Operator deployments, the DRBDReplicationPolicy resource configures how volumes replicate. The protection-controller reads this resource when setting up replication for a new ProtectionRequest:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-policy
  namespace: dr-<name>
spec:
  protocol: C                      # "C" (synchronous, RPO=0) or "A" (asynchronous)
  storageClassMappings:
    - primary: fast-ssd             # Storage class name on the primary cluster
      dr: fast-ssd-dr               # Corresponding storage class on the DR cluster
  replicationEndpoints:
    - primary: 10.0.1.10:7000
      dr: 10.0.2.10:7000

The controller architecture is designed so that you interact with it declaratively: you create or update custom resources, and the controllers reconcile the actual state of the clusters to match. You never call controller APIs directly.

How the failover-controller processes a FailoverRequest

When you create a FailoverRequest, the failover-controller on the quorum cluster picks it up and drives the following state machine:

1. Pending — The controller reads the request and identifies the source cluster (where the Protection Group's VMs are currently running) and the target cluster.
2. StoppingOnSource — The controller patches spec.desiredState: stopped on the ProtectionGroup resource on the source cluster. The protection-group-controller running on that cluster reconciles all VMs in the group to the stopped state and updates status.currentState.
3. WaitingForDRBD — The controller waits for DRBD volume promotion to complete on the DR cluster and removes quorum taints from target worker nodes (subject to concurrent-failover safety checks).
4. StartingOnTarget — The controller patches spec.desiredState: running on the ProtectionGroup on the target cluster. The protection-group-controller on that cluster starts all VMs.
5. Completed — All VMs are confirmed running on the target cluster. The FailoverRequest status reflects completion.

The failover-controller never patches VMs directly. It exclusively coordinates through ProtectionGroup desired state. This separation ensures the protection-group-controller's idempotent VM reconciliation logic is always exercised.

To monitor progress:

kubectl get failoverrequest failover-production -n dr-<name> \
  --kubeconfig ~/.kube/quorum-config -o yaml

How the protection-group-controller manages VM lifecycle

The protection-group-controller runs on each cluster (primary and DR) and reconciles ProtectionGroup resources local to that cluster. When spec.desiredState changes:

• It iterates over all VMs listed in spec.virtualMachines and patches each VM's spec.running field to match the desired state.
• It is idempotent: if a VM is already in the correct state, no patch is issued.
• It updates status.currentState to running, stopped, mixed (reconciliation in progress), or unknown.
• It tracks per-VM replication state and reports overall group health.

To inspect a Protection Group's current reconciled state on the primary cluster:

kubectl get protectiongroup production-protection-group -n default \
  --kubeconfig ~/.kube/primary-config -o yaml

How the protection-controller handles ProtectionRequests

In DRBD Operator deployments, you protect a single VM by creating a ProtectionRequest on the quorum cluster. The protection-controller drives the lifecycle:

1. Validation — Confirms the VM exists and that its PVCs are eligible for DRBD replication.
2. DRBDVolume creation — Creates a DRBDVolume resource for each PVC, which the DRBD Operator uses to set up replication.
3. Sync wait — Waits for initial synchronization to complete (visible in DRBDVolume status).
4. Frontend PVC switchover — Switches the VM to a DRBD-backed frontend PVC, routing all disk I/O through the replication layer.
5. Protected — The ProtectionRequest reaches terminal protected status.

How the pg-sync-controller keeps clusters consistent

The pg-sync-controller runs on the quorum cluster and continuously synchronizes ProtectionGroup metadata — including VM specifications — from the primary cluster to the DR cluster. This ensures that standby VMs on the DR cluster remain accurate mirrors of their primary counterparts even when VM specs change (such as memory or CPU updates). You do not interact with this controller directly; it operates continuously in the background.

How the replication-monitor reports health

The replication-monitor agent runs on the primary and DR clusters. It samples DRBD replication lag at regular intervals and:

• Creates RPOEvent resources when lag exceeds configured thresholds, providing an audit trail of data-loss risk events.
• Maintains ReplicationGroupStatus resources with aggregated health (Healthy, Degraded, or Critical), per-volume sync state, and last-successful-sync timestamp.

To check replication health for a Protection Group:

kubectl get replicationgroupstatus production-protection-group -n dr-<name> \
  --kubeconfig ~/.kube/primary-config -o yaml

To list recent RPO violations:

kubectl get rpoevents -n dr-<name> --kubeconfig ~/.kube/quorum-config

Example 1 — Observe a FailoverRequest progressing through the state machine

Create a planned FailoverRequest and watch its status progress:

kubectl apply -f - --kubeconfig ~/.kube/quorum-config <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production
  namespace: dr-production
spec:
  protectionGroup: production-protection-group
  targetCluster: cluster2
  failoverType: planned
  force: false
EOF

Watch the controller update the status:

kubectl get failoverrequest failover-production -n dr-production \
  --kubeconfig ~/.kube/quorum-config -w

Expected progression:

NAME                   PHASE               AGE
failover-production    Pending             2s
failover-production    StoppingOnSource    8s
failover-production    StartingOnTarget    45s
failover-production    Completed           72s

The RTO for an automated planned failover is typically 3–8 minutes depending on VM count and DRBD sync state.

Example 2 — Watch the protection-group-controller reconcile VM state

Observe status.currentState change on the primary cluster as the failover controller patches spec.desiredState:

kubectl get protectiongroup production-protection-group -n default \
  --kubeconfig ~/.kube/primary-config -w

Expected output during a planned failover:

NAME                          STATE    VMS   REPLICATION   HEALTH    AGE
production-protection-group   Active   2     synchronous   Healthy   22h
# failover-controller patches desiredState: stopped
production-protection-group   Active   2     synchronous   Healthy   22h
# protection-group-controller stops VMs; currentState transitions
production-protection-group   Active   2     synchronous   Healthy   22h

Inspect the full status including currentState:

kubectl get protectiongroup production-protection-group -n default \
  --kubeconfig ~/.kube/primary-config \
  -o jsonpath='{.status.currentState}'

Expected output while stopping:

mixed

Expected output after all VMs have stopped:

stopped

Example 3 — Inspect a ProtectionRequest lifecycle (DRBD Operator model)

Create a ProtectionRequest for a single VM:

kubectl apply -f - --kubeconfig ~/.kube/quorum-config <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-prod-vm-1
  namespace: dr-production
spec:
  virtualMachine:
    name: prod-vm-1
    namespace: default
  replicationPolicy: sync-policy
EOF

Watch the protection-controller drive the lifecycle:

kubectl get protectionrequest protect-prod-vm-1 -n dr-production \
  --kubeconfig ~/.kube/quorum-config -w

Expected output:

NAME                PHASE        AGE
protect-prod-vm-1   Validating   3s
protect-prod-vm-1   Replicating  12s
protect-prod-vm-1   Syncing      18s
protect-prod-vm-1   Protected    4m

Confirm the DRBDVolume was created:

kubectl get drbdvolumes -n dr-production --kubeconfig ~/.kube/quorum-config

Expected output:

NAME                                                SYNC   HEALTH   AGE
pvc-8f04b7f3-ab58-46a9-9721-508337d30d61           100%   Healthy  4m

Example 4 — Check ReplicationGroupStatus written by the replication-monitor

kubectl get replicationgroupstatus production-protection-group \
  -n dr-production \
  --kubeconfig ~/.kube/primary-config \
  -o yaml

Expected output:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ReplicationGroupStatus
metadata:
  name: production-protection-group
  namespace: dr-production
status:
  overallHealth: Healthy
  lastSuccessfulSync: "2025-10-31T14:22:07Z"
  volumes:
    - pvcName: prod-vm-1-disk
      syncState: Consistent
    - pvcName: prod-vm-2-disk
      syncState: Consistent

Example 5 — Use pgctl to inspect Protection Group and failover status

# List all Protection Groups across the deployment
pgctl get protectiongroups --context dr-production

# Inspect a specific Protection Group
pgctl describe protectiongroup production-protection-group --context dr-production

# Trigger a failover via pgctl
pgctl failover production-protection-group \
  --target cluster2 \
  --type planned \
  --context dr-production

# Monitor the FailoverRequest status
pgctl status failover failover-production --context dr-production

Issue 1 — FailoverRequest stuck in StoppingOnSource

Symptom: A FailoverRequest remains in the StoppingOnSource phase for more than 5 minutes. VMs on the source cluster have not stopped.

Likely cause: The protection-group-controller on the source cluster is not running, or it is unable to reach the Kubernetes API of the source cluster from the quorum.

Fix:

1. Check the protection-group-controller pod on the source cluster:
   Copy

   kubectl get pods -n dr-<name> --kubeconfig ~/.kube/primary-config | grep protection-group
   kubectl logs -n dr-<name> <protection-group-controller-pod> --kubeconfig ~/.kube/primary-config
   

2. Verify the quorum cluster can reach the primary cluster API on TCP 6443:
   Copy

   curl -k https://<primary-api-endpoint>:6443/healthz
   

3. Check that the ProtectionGroup on the source cluster shows desiredState: stopped (confirming the failover-controller patch was applied):
   Copy

   kubectl get protectiongroup <pg-name> -n default --kubeconfig ~/.kube/primary-config \
     -o jsonpath='{.spec.desiredState}'
   

4. If the protection-group-controller is healthy but VMs are not stopping, inspect individual VM status:
   Copy

   kubectl get vms -n default --kubeconfig ~/.kube/primary-config
   

Issue 2 — FailoverRequest fails with a concurrent-failover safety error

Symptom: The FailoverRequest transitions to Failed with a message indicating that another Protection Group has running VMs on the target cluster and taint removal was aborted.

Likely cause: Another Protection Group's VMs are currently running on the target cluster. Removing node-level quorum taints would affect those VMs.

Fix:

1. Identify which Protection Groups are running on the target cluster:
   Copy

   kubectl get protectiongroups -n default --kubeconfig ~/.kube/dr-config \
     -o custom-columns=NAME:.metadata.name,STATE:.status.currentState
   

2. Wait for those Protection Groups to complete their operations, or coordinate the failover sequence so only one Protection Group is active on the target cluster at a time.
3. If this is an emergency and you accept the risk, set spec.force: true on the FailoverRequest:
   Copy

   kubectl patch failoverrequest <name> -n dr-<name> --type merge \
     -p '{"spec":{"force":true}}' --kubeconfig ~/.kube/quorum-config
   

   Warning: Using force: true bypasses safety checks and may disrupt VMs belonging to other Protection Groups.

Issue 3 — ProtectionRequest stuck in Syncing for an extended period

Symptom: A ProtectionRequest stays in the Syncing phase indefinitely. The associated DRBDVolume shows sync percentage below 100%.

Likely cause: Initial DRBD block-level sync is still in progress (expected for large volumes), network bandwidth between clusters is saturated, or the DRBD Operator on one of the clusters is not functioning.

Fix:

1. Check DRBDVolume sync progress:
   Copy

   kubectl get drbdvolumes -n dr-<name> --kubeconfig ~/.kube/quorum-config
   

2. Check DRBD Operator logs on the primary cluster:
   Copy

   kubectl logs -n drbd-operator <drbd-operator-pod> --kubeconfig ~/.kube/primary-config
   

3. Verify TCP 7000–7999 is open between the primary and DR worker nodes.
4. If sync percentage is increasing (even slowly), the operation is proceeding normally — large volumes can take significant time for the initial sync.

Issue 4 — ReplicationGroupStatus shows Degraded or Critical

Symptom: kubectl get replicationgroupstatus shows overallHealth: Degraded or Critical, or RPOEvent resources are being created.

Likely cause: Replication lag has exceeded configured thresholds. This may be caused by network latency between clusters exceeding the Protocol C requirement (< 50 ms RTT), high write I/O from VMs, or a DRBD replication link interruption.

Fix:

1. List recent RPO violations:
   Copy

   kubectl get rpoevents -n dr-<name> --kubeconfig ~/.kube/quorum-config
   

2. Check per-volume sync state in the ReplicationGroupStatus:
   Copy

   kubectl get replicationgroupstatus <pg-name> -n dr-<name> \
     --kubeconfig ~/.kube/primary-config -o yaml
   

3. Verify network latency between clusters:
   Copy

   ping <dr-worker-node-ip>   # Run from a primary worker node
   

4. If latency is consistently above 50 ms and you are using Protocol C, consider switching to Protocol A by updating your DRBDReplicationPolicy.
5. Check the replication-monitor logs for details:
   Copy

   kubectl logs -n dr-<name> <replication-monitor-pod> --kubeconfig ~/.kube/primary-config
   

Issue 5 — pg-sync-controller is not updating standby VM specs on the DR cluster

Symptom: After modifying a VM spec on the primary cluster (such as increasing memory), the corresponding standby VM on the DR cluster does not reflect the change.

Likely cause: The pg-sync-controller on the quorum cluster has lost connectivity to one of the clusters, or its kubeconfig credentials for the DR cluster have expired.

Fix:

1. Check pg-sync-controller logs:
   Copy

   kubectl logs -n dr-<name> <pg-sync-controller-pod> --kubeconfig ~/.kube/quorum-config
   

2. Verify the quorum cluster kubeconfig for the DR cluster is valid:
   Copy

   kubectl get nodes --kubeconfig ~/.kube/dr-config
   

3. Restart the pg-sync-controller if credentials have been rotated and the secret updated:
   Copy

   kubectl rollout restart deployment pg-sync-controller -n dr-<name> \
     --kubeconfig ~/.kube/quorum-config