---
title: Controller Architecture
product: trilio-site-recovery-for-kubernetes-openshift-virtualization
doc_type: guide
version: feature-tsr-24
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/controller-architecture
---

# Controller Architecture

_How controllers watch recovery resources and drive reconciliation toward desired failover state_

## Overview

This page explains how Site Recovery's controllers watch Kubernetes custom resources and continuously reconcile actual cluster state toward the desired failover state. Understanding this architecture helps you reason about what happens when you create or update a `FailoverRequest`, `ProtectionGroup`, or `ProtectionRequest`, and why the system is resilient to partial failures. The design follows the Kubernetes operator pattern: each controller runs in-cluster on the quorum cluster (or on the primary and DR clusters, depending on the controller), observes relevant custom resources through informers, and drives the system toward the declared desired state without requiring manual intervention.

## Prerequisites

Before reading this page, you should be familiar with the following:

- **Kubernetes operator pattern** — controllers, reconcile loops, and custom resource definitions
- **Site Recovery deployment** — a running quorum cluster with controllers deployed, and at least one configured DR deployment (namespace `dr-<name>`)
- **Core CRDs installed** — `ProtectionGroup`, `FailoverRequest`, `ProtectionRequest`, `DRBDVolume`, `DRBDReplicationPolicy`, `TestFailover`, `RPOEvent`, and `ReplicationGroupStatus` must be present on the relevant clusters
- **Kubernetes ≥ 1.28** on all clusters
- **KubeVirt ≥ 1.0** on primary and DR clusters
- **kubectl** configured with access to the quorum, primary, and DR clusters
- **DRBD kernel module ≥ 9.0** on primary and DR worker nodes (for DRBD Operator deployments)

You do not need to understand DRBD internals to follow this page, but familiarity with PersistentVolumeClaims and VirtualMachine resources in KubeVirt is assumed.

## Installation

Controller deployment is handled by the standard Ansible playbooks included with Site Recovery. You do not deploy individual controllers manually. The steps below describe what the playbooks install and where, so you can verify a correct deployment.

**Step 1 — Deploy quorum-cluster controllers**

The following controllers run as Deployments on the quorum cluster, each in the namespace corresponding to their DR deployment (`dr-<name>`):

- `failover-controller` — watches `FailoverRequest` resources
- `protection-controller` — watches `ProtectionRequest` resources (DRBD Operator model only)
- `pg-sync-controller` — keeps `ProtectionGroup` metadata synchronized between primary and DR clusters

Verify they are running after the Ansible playbook completes:

```bash
kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/quorum-config
```

Expected output (names may include a generated suffix):

```
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
failover-controller       1/1     1            1           5m
protection-controller     1/1     1            1           5m
pg-sync-controller        1/1     1            1           5m
```

**Step 2 — Deploy per-cluster controllers**

The following controllers run on both the primary cluster and the DR cluster:

- `protection-group-controller` — manages `ProtectionGroup` resources and VM lifecycle on that cluster
- `test-failover-controller` — manages `TestFailover` resources (DRBD Operator model only)

Verify on the primary cluster:

```bash
kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/primary-config
```

Verify on the DR cluster:

```bash
kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/dr-config
```

**Step 3 — Verify the replication-monitor agent**

The `replication-monitor` agent is deployed automatically by the Ansible playbooks on the primary and DR clusters. It writes `RPOEvent` and `ReplicationGroupStatus` resources. Confirm it is running:

```bash
kubectl get pods -n dr-<name> -l app=replication-monitor --kubeconfig ~/.kube/primary-config
```

**Step 4 — Verify the DRBD Operator (DRBD Operator model only)**

If you are using the DRBD Operator deployment model, confirm the DRBD Operator is installed on both the primary and DR clusters:

```bash
kubectl get deployments -n drbd-operator --kubeconfig ~/.kube/primary-config
kubectl get deployments -n drbd-operator --kubeconfig ~/.kube/dr-config
```

**Step 5 — Verify the Site Manager UI**

The Site Manager UI and its API backend are deployed on the quorum cluster:

```bash
kubectl get deployments -n dr-<name> --kubeconfig ~/.kube/quorum-config | grep site-manager
```

All controllers and agents are managed by the Ansible playbooks. To redeploy or update them, re-run the relevant playbook or use `quorum-deployments.sh` for guided operational tasks.

## Configuration

Controllers are configured through a combination of their deployment environment variables and the custom resources they watch. The following describes the key behavioral settings you can influence.

---

### Reconciliation interval

All controllers use event-driven reconciliation triggered by Kubernetes watch events. When a resource is created, updated, or deleted, the relevant controller's reconcile loop is invoked immediately. In addition, controllers use a periodic re-sync (typically every 10 seconds for the failover state machine) to handle cases where a watched resource changed on a remote cluster without generating a local event.

You do not configure the reconciliation interval directly; it is determined by the controller implementation.

---

### FailoverRequest — controlling failover behavior

The `FailoverRequest` CRD is the primary way you influence `failover-controller` behavior:

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production
  namespace: dr-<name>
spec:
  protectionGroup: production-protection-group   # Required: which ProtectionGroup to fail over
  targetCluster: cluster2                        # Required: which cluster should become active
  failoverType: planned                          # "planned" or "unplanned"
  force: false                                   # true bypasses concurrent-failover safety checks
```

| Field | Default | Valid values | Effect |
|---|---|---|---|
| `spec.failoverType` | — | `planned`, `unplanned` | `planned` gracefully stops VMs before promoting DR volumes; `unplanned` promotes immediately without waiting for a clean shutdown |
| `spec.force` | `false` | `true`, `false` | When `true`, bypasses safety checks that would otherwise block taint removal if other Protection Groups have running VMs on the target cluster. Use only in emergencies. |

---

### ProtectionGroup — controlling VM group desired state

The `protection-group-controller` watches `spec.desiredState` on `ProtectionGroup` resources and reconciles all VMs in the group to match:

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: production-protection-group
  namespace: default
spec:
  desiredState: running       # "running" or "stopped" — drives VM lifecycle
  virtualMachines:
    - name: prod-vm-1
    - name: prod-vm-2
```

| Field | Default | Valid values | Effect |
|---|---|---|---|
| `spec.desiredState` | `running` | `running`, `stopped` | The controller patches each VM's `spec.running` field to match. `status.currentState` reflects the actual aggregate state (`running`, `stopped`, `mixed`, or `unknown`). |

The `failover-controller` patches `spec.desiredState` on the appropriate cluster's `ProtectionGroup` during failover orchestration — you should not change `desiredState` manually during an in-progress failover.

---

### DRBDReplicationPolicy — controlling replication behavior

In DRBD Operator deployments, the `DRBDReplicationPolicy` resource configures how volumes replicate. The `protection-controller` reads this resource when setting up replication for a new `ProtectionRequest`:

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-policy
  namespace: dr-<name>
spec:
  protocol: C                      # "C" (synchronous, RPO=0) or "A" (asynchronous)
  storageClassMappings:
    - primary: fast-ssd             # Storage class name on the primary cluster
      dr: fast-ssd-dr               # Corresponding storage class on the DR cluster
  replicationEndpoints:
    - primary: 10.0.1.10:7000
      dr: 10.0.2.10:7000
```

| Field | Valid values | Effect |
|---|---|---|
| `spec.protocol` | `C`, `A` | `C` requires ≤ 50 ms RTT and guarantees RPO=0; `A` supports any latency with near-zero RPO |
| `spec.storageClassMappings` | List of primary/dr pairs | Maps storage classes so the controller can provision matching volumes on both clusters |
| `spec.replicationEndpoints` | List of primary/dr IP:port pairs | DRBD replication traffic flows on TCP 7000–7999 between these addresses |

## Usage

The controller architecture is designed so that you interact with it declaratively: you create or update custom resources, and the controllers reconcile the actual state of the clusters to match. You never call controller APIs directly.

---

### How the failover-controller processes a FailoverRequest

When you create a `FailoverRequest`, the `failover-controller` on the quorum cluster picks it up and drives the following state machine:

1. **Pending** — The controller reads the request and identifies the source cluster (where the Protection Group's VMs are currently running) and the target cluster.
2. **StoppingOnSource** — The controller patches `spec.desiredState: stopped` on the `ProtectionGroup` resource on the source cluster. The `protection-group-controller` running on that cluster reconciles all VMs in the group to the stopped state and updates `status.currentState`.
3. **WaitingForDRBD** — The controller waits for DRBD volume promotion to complete on the DR cluster and removes quorum taints from target worker nodes (subject to concurrent-failover safety checks).
4. **StartingOnTarget** — The controller patches `spec.desiredState: running` on the `ProtectionGroup` on the target cluster. The `protection-group-controller` on that cluster starts all VMs.
5. **Completed** — All VMs are confirmed running on the target cluster. The `FailoverRequest` status reflects completion.

The `failover-controller` never patches VMs directly. It exclusively coordinates through `ProtectionGroup` desired state. This separation ensures the `protection-group-controller`'s idempotent VM reconciliation logic is always exercised.

To monitor progress:

```bash
kubectl get failoverrequest failover-production -n dr-<name> \
  --kubeconfig ~/.kube/quorum-config -o yaml
```

---

### How the protection-group-controller manages VM lifecycle

The `protection-group-controller` runs on each cluster (primary and DR) and reconciles `ProtectionGroup` resources local to that cluster. When `spec.desiredState` changes:

- It iterates over all VMs listed in `spec.virtualMachines` and patches each VM's `spec.running` field to match the desired state.
- It is idempotent: if a VM is already in the correct state, no patch is issued.
- It updates `status.currentState` to `running`, `stopped`, `mixed` (reconciliation in progress), or `unknown`.
- It tracks per-VM replication state and reports overall group health.

To inspect a Protection Group's current reconciled state on the primary cluster:

```bash
kubectl get protectiongroup production-protection-group -n default \
  --kubeconfig ~/.kube/primary-config -o yaml
```

---

### How the protection-controller handles ProtectionRequests

In DRBD Operator deployments, you protect a single VM by creating a `ProtectionRequest` on the quorum cluster. The `protection-controller` drives the lifecycle:

1. **Validation** — Confirms the VM exists and that its PVCs are eligible for DRBD replication.
2. **DRBDVolume creation** — Creates a `DRBDVolume` resource for each PVC, which the DRBD Operator uses to set up replication.
3. **Sync wait** — Waits for initial synchronization to complete (visible in `DRBDVolume` status).
4. **Frontend PVC switchover** — Switches the VM to a DRBD-backed frontend PVC, routing all disk I/O through the replication layer.
5. **Protected** — The `ProtectionRequest` reaches terminal protected status.

---

### How the pg-sync-controller keeps clusters consistent

The `pg-sync-controller` runs on the quorum cluster and continuously synchronizes `ProtectionGroup` metadata — including VM specifications — from the primary cluster to the DR cluster. This ensures that standby VMs on the DR cluster remain accurate mirrors of their primary counterparts even when VM specs change (such as memory or CPU updates). You do not interact with this controller directly; it operates continuously in the background.

---

### How the replication-monitor reports health

The `replication-monitor` agent runs on the primary and DR clusters. It samples DRBD replication lag at regular intervals and:

- Creates `RPOEvent` resources when lag exceeds configured thresholds, providing an audit trail of data-loss risk events.
- Maintains `ReplicationGroupStatus` resources with aggregated health (`Healthy`, `Degraded`, or `Critical`), per-volume sync state, and last-successful-sync timestamp.

To check replication health for a Protection Group:

```bash
kubectl get replicationgroupstatus production-protection-group -n dr-<name> \
  --kubeconfig ~/.kube/primary-config -o yaml
```

To list recent RPO violations:

```bash
kubectl get rpoevents -n dr-<name> --kubeconfig ~/.kube/quorum-config
```

## Examples

### Example 1 — Observe a FailoverRequest progressing through the state machine

Create a planned `FailoverRequest` and watch its status progress:

```bash
kubectl apply -f - --kubeconfig ~/.kube/quorum-config <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production
  namespace: dr-production
spec:
  protectionGroup: production-protection-group
  targetCluster: cluster2
  failoverType: planned
  force: false
EOF
```

Watch the controller update the status:

```bash
kubectl get failoverrequest failover-production -n dr-production \
  --kubeconfig ~/.kube/quorum-config -w
```

Expected progression:

```
NAME                   PHASE               AGE
failover-production    Pending             2s
failover-production    StoppingOnSource    8s
failover-production    StartingOnTarget    45s
failover-production    Completed           72s
```

The RTO for an automated planned failover is typically 3–8 minutes depending on VM count and DRBD sync state.

---

### Example 2 — Watch the protection-group-controller reconcile VM state

Observe `status.currentState` change on the primary cluster as the failover controller patches `spec.desiredState`:

```bash
kubectl get protectiongroup production-protection-group -n default \
  --kubeconfig ~/.kube/primary-config -w
```

Expected output during a planned failover:

```
NAME                          STATE    VMS   REPLICATION   HEALTH    AGE
production-protection-group   Active   2     synchronous   Healthy   22h
# failover-controller patches desiredState: stopped
production-protection-group   Active   2     synchronous   Healthy   22h
# protection-group-controller stops VMs; currentState transitions
production-protection-group   Active   2     synchronous   Healthy   22h
```

Inspect the full status including `currentState`:

```bash
kubectl get protectiongroup production-protection-group -n default \
  --kubeconfig ~/.kube/primary-config \
  -o jsonpath='{.status.currentState}'
```

Expected output while stopping:

```
mixed
```

Expected output after all VMs have stopped:

```
stopped
```

---

### Example 3 — Inspect a ProtectionRequest lifecycle (DRBD Operator model)

Create a `ProtectionRequest` for a single VM:

```bash
kubectl apply -f - --kubeconfig ~/.kube/quorum-config <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-prod-vm-1
  namespace: dr-production
spec:
  virtualMachine:
    name: prod-vm-1
    namespace: default
  replicationPolicy: sync-policy
EOF
```

Watch the protection-controller drive the lifecycle:

```bash
kubectl get protectionrequest protect-prod-vm-1 -n dr-production \
  --kubeconfig ~/.kube/quorum-config -w
```

Expected output:

```
NAME                PHASE        AGE
protect-prod-vm-1   Validating   3s
protect-prod-vm-1   Replicating  12s
protect-prod-vm-1   Syncing      18s
protect-prod-vm-1   Protected    4m
```

Confirm the `DRBDVolume` was created:

```bash
kubectl get drbdvolumes -n dr-production --kubeconfig ~/.kube/quorum-config
```

Expected output:

```
NAME                                                SYNC   HEALTH   AGE
pvc-8f04b7f3-ab58-46a9-9721-508337d30d61           100%   Healthy  4m
```

---

### Example 4 — Check ReplicationGroupStatus written by the replication-monitor

```bash
kubectl get replicationgroupstatus production-protection-group \
  -n dr-production \
  --kubeconfig ~/.kube/primary-config \
  -o yaml
```

Expected output:

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ReplicationGroupStatus
metadata:
  name: production-protection-group
  namespace: dr-production
status:
  overallHealth: Healthy
  lastSuccessfulSync: "2025-10-31T14:22:07Z"
  volumes:
    - pvcName: prod-vm-1-disk
      syncState: Consistent
    - pvcName: prod-vm-2-disk
      syncState: Consistent
```

---

### Example 5 — Use pgctl to inspect Protection Group and failover status

```bash
# List all Protection Groups across the deployment
pgctl get protectiongroups --context dr-production

# Inspect a specific Protection Group
pgctl describe protectiongroup production-protection-group --context dr-production

# Trigger a failover via pgctl
pgctl failover production-protection-group \
  --target cluster2 \
  --type planned \
  --context dr-production

# Monitor the FailoverRequest status
pgctl status failover failover-production --context dr-production
```

## Troubleshooting

### Issue 1 — FailoverRequest stuck in `StoppingOnSource`

**Symptom:** A `FailoverRequest` remains in the `StoppingOnSource` phase for more than 5 minutes. VMs on the source cluster have not stopped.

**Likely cause:** The `protection-group-controller` on the source cluster is not running, or it is unable to reach the Kubernetes API of the source cluster from the quorum.

**Fix:**
1. Check the `protection-group-controller` pod on the source cluster:
   ```bash
   kubectl get pods -n dr-<name> --kubeconfig ~/.kube/primary-config | grep protection-group
   kubectl logs -n dr-<name> <protection-group-controller-pod> --kubeconfig ~/.kube/primary-config
   ```
2. Verify the quorum cluster can reach the primary cluster API on TCP 6443:
   ```bash
   curl -k https://<primary-api-endpoint>:6443/healthz
   ```
3. Check that the `ProtectionGroup` on the source cluster shows `desiredState: stopped` (confirming the `failover-controller` patch was applied):
   ```bash
   kubectl get protectiongroup <pg-name> -n default --kubeconfig ~/.kube/primary-config \
     -o jsonpath='{.spec.desiredState}'
   ```
4. If the `protection-group-controller` is healthy but VMs are not stopping, inspect individual VM status:
   ```bash
   kubectl get vms -n default --kubeconfig ~/.kube/primary-config
   ```

---

### Issue 2 — FailoverRequest fails with a concurrent-failover safety error

**Symptom:** The `FailoverRequest` transitions to `Failed` with a message indicating that another Protection Group has running VMs on the target cluster and taint removal was aborted.

**Likely cause:** Another Protection Group's VMs are currently running on the target cluster. Removing node-level quorum taints would affect those VMs.

**Fix:**
1. Identify which Protection Groups are running on the target cluster:
   ```bash
   kubectl get protectiongroups -n default --kubeconfig ~/.kube/dr-config \
     -o custom-columns=NAME:.metadata.name,STATE:.status.currentState
   ```
2. Wait for those Protection Groups to complete their operations, or coordinate the failover sequence so only one Protection Group is active on the target cluster at a time.
3. If this is an emergency and you accept the risk, set `spec.force: true` on the `FailoverRequest`:
   ```bash
   kubectl patch failoverrequest <name> -n dr-<name> --type merge \
     -p '{"spec":{"force":true}}' --kubeconfig ~/.kube/quorum-config
   ```
   > **Warning:** Using `force: true` bypasses safety checks and may disrupt VMs belonging to other Protection Groups.

---

### Issue 3 — ProtectionRequest stuck in `Syncing` for an extended period

**Symptom:** A `ProtectionRequest` stays in the `Syncing` phase indefinitely. The associated `DRBDVolume` shows sync percentage below 100%.

**Likely cause:** Initial DRBD block-level sync is still in progress (expected for large volumes), network bandwidth between clusters is saturated, or the DRBD Operator on one of the clusters is not functioning.

**Fix:**
1. Check `DRBDVolume` sync progress:
   ```bash
   kubectl get drbdvolumes -n dr-<name> --kubeconfig ~/.kube/quorum-config
   ```
2. Check DRBD Operator logs on the primary cluster:
   ```bash
   kubectl logs -n drbd-operator <drbd-operator-pod> --kubeconfig ~/.kube/primary-config
   ```
3. Verify TCP 7000–7999 is open between the primary and DR worker nodes.
4. If sync percentage is increasing (even slowly), the operation is proceeding normally — large volumes can take significant time for the initial sync.

---

### Issue 4 — ReplicationGroupStatus shows `Degraded` or `Critical`

**Symptom:** `kubectl get replicationgroupstatus` shows `overallHealth: Degraded` or `Critical`, or `RPOEvent` resources are being created.

**Likely cause:** Replication lag has exceeded configured thresholds. This may be caused by network latency between clusters exceeding the Protocol C requirement (< 50 ms RTT), high write I/O from VMs, or a DRBD replication link interruption.

**Fix:**
1. List recent RPO violations:
   ```bash
   kubectl get rpoevents -n dr-<name> --kubeconfig ~/.kube/quorum-config
   ```
2. Check per-volume sync state in the `ReplicationGroupStatus`:
   ```bash
   kubectl get replicationgroupstatus <pg-name> -n dr-<name> \
     --kubeconfig ~/.kube/primary-config -o yaml
   ```
3. Verify network latency between clusters:
   ```bash
   ping <dr-worker-node-ip>   # Run from a primary worker node
   ```
4. If latency is consistently above 50 ms and you are using Protocol C, consider switching to Protocol A by updating your `DRBDReplicationPolicy`.
5. Check the `replication-monitor` logs for details:
   ```bash
   kubectl logs -n dr-<name> <replication-monitor-pod> --kubeconfig ~/.kube/primary-config
   ```

---

### Issue 5 — pg-sync-controller is not updating standby VM specs on the DR cluster

**Symptom:** After modifying a VM spec on the primary cluster (such as increasing memory), the corresponding standby VM on the DR cluster does not reflect the change.

**Likely cause:** The `pg-sync-controller` on the quorum cluster has lost connectivity to one of the clusters, or its kubeconfig credentials for the DR cluster have expired.

**Fix:**
1. Check `pg-sync-controller` logs:
   ```bash
   kubectl logs -n dr-<name> <pg-sync-controller-pod> --kubeconfig ~/.kube/quorum-config
   ```
2. Verify the quorum cluster kubeconfig for the DR cluster is valid:
   ```bash
   kubectl get nodes --kubeconfig ~/.kube/dr-config
   ```
3. Restart the `pg-sync-controller` if credentials have been rotated and the secret updated:
   ```bash
   kubectl rollout restart deployment pg-sync-controller -n dr-<name> \
     --kubeconfig ~/.kube/quorum-config
   ```