Controller Architecture

This page describes the internal architecture of the controllers that power Trilio Site Recovery (TSR): how they are deployed, which clusters they run on, what custom resources they watch and reconcile, and how the admission webhooks enforce invariants before changes reach the API server. Understanding this architecture helps you reason about reconcile loops, diagnose stuck phases, and interpret log output when troubleshooting replication or failover issues across your primary, DR, and quorum clusters.

Before reading this page you should be familiar with:

• Kubernetes controller-runtime reconciler patterns (watch → reconcile → patch)
• The three-cluster TSR topology: quorum (management plane), primary (cluster1), and DR (cluster2)
• OpenShift 4.17 or later on all three clusters (the supported floor per docs/tech-preview-supported-configurations.md)
• The TSR custom resources: ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, DRBDResource, DRBDReplicationPolicy, ReplicationGroupStatus, and RPOEvent
• oc CLI access to all three clusters (required for OpenShift-specific paths; kubectl-only is not a supported mode)
• Kubeconfig files for all three clusters

TSR controllers are not installed by running helm install or helm upgrade directly. The Helm charts are embedded inside the operator binary and are applied automatically by the ProtectionZone controller once the zone reaches Available. The customer installation path is:

1. Install the OLM operator bundle on the quorum cluster.

   Create a CatalogSource, OperatorGroup, and Subscription in the trilio-site-recovery-system namespace:

   Copy

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: site-recovery-catalog
     namespace: trilio-site-recovery-system
   spec:
     sourceType: grpc
     image: quay.io/triliovault/site-recovery-catalog:latest
     displayName: Trilio Site Recovery
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: trilio-site-recovery-og
     namespace: trilio-site-recovery-system
   spec:
     targetNamespaces:
       - trilio-site-recovery-system
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: site-recovery
     namespace: trilio-site-recovery-system
   spec:
     channel: tech-preview
     name: site-recovery
     source: site-recovery-catalog
     sourceNamespace: trilio-site-recovery-system
   

   Copy

   oc apply -f catalogsource.yaml
   

   The bundle installs the protection-zone-controller Deployment, the TSR web console (site-recovery-backend and site-recovery-frontend), and the ProtectionZone CRD on the quorum cluster.

2. Create kubeconfig Secrets for the primary and DR clusters in the zone namespace (dr-<name>):

   Copy

   oc create namespace dr-zone1
   
   oc create secret generic cluster1-kubeconfig \
     --from-file=kubeconfig=/path/to/cluster1-kubeconfig \
     -n dr-zone1
   
   oc create secret generic cluster2-kubeconfig \
     --from-file=kubeconfig=/path/to/cluster2-kubeconfig \
     -n dr-zone1
   

3. Create a ProtectionZone CR on the quorum cluster. The ProtectionZone controller reads this CR, validates connectivity to both workload clusters, then automatically deploys the site-recovery-quorum-control-plane chart into the zone namespace and the site-recovery-workload-control-plane chart (including the drbd-node-agent DaemonSet) into trilio-site-recovery-system on both the primary and DR clusters.

   Copy

   apiVersion: siterecovery.trilio.io/v1alpha1
   kind: ProtectionZone
   metadata:
     name: zone1
     namespace: dr-zone1
   spec:
     displayName: "primary-dr-zone"
     storageBackendMode: DRBD
     clusters:
       - name: cluster1
         clusterID: cluster1
         kubeconfigSecretRef:
           name: cluster1-kubeconfig
           key: kubeconfig
       - name: cluster2
         clusterID: cluster2
         kubeconfigSecretRef:
           name: cluster2-kubeconfig
           key: kubeconfig
   

   Copy

   oc apply -f protectionzone.yaml
   

4. Verify the zone reaches Available:

   Copy

   oc get pz -n dr-zone1
   

   Expected output:

   Copy

   NAME    DISPLAY NAME        BACKEND   CLUSTERS            STATUS      AGE
   zone1   primary-dr-zone     DRBD      cluster1,cluster2   Available   2m
   

   Once Available, the quorum control plane pods and workload control plane pods (including DRBD node agents) are running and the full CRD set is registered.

TSR controller behaviour is shaped by the ProtectionZone spec and by a per-zone tunable ConfigMap named site-recovery-zone-config in the zone namespace. You do not edit Helm chart values directly.

ProtectionZone spec fields

Per-zone tunable ConfigMap (site-recovery-zone-config)

The ProtectionZone controller creates this ConfigMap once per zone namespace. You can edit it in place; controllers re-read it on every reconcile without a restart.

Values are Go duration strings (e.g. "90s", "2m"). An unparseable value falls back to the compiled-in default and is logged.

DRBDReplicationPolicy

You do not create or modify DRBDReplicationPolicy resources. The ProtectionZone controller creates exactly one per zone named <zoneName>-default with isDefault: true and drbdProtocol: C (Protocol C synchronous replication, the only supported mode). The admission webhook rejects any ProtectionRequest that specifies replicationProtocol: A. RPO is always 0 and is not tunable.

Webhook certificate

Webhook serving certificates are not managed by the charts. You must create a Secret named webhook-server-cert (the value of webhook.certSecretName) in each zone namespace before the webhooks can serve. The webhook port is 443 (external) with container port 9443.

Runtime log-level adjustment

You can change the log level without restarting pods by editing the site-recovery-logging ConfigMap:

oc edit configmap site-recovery-logging -n dr-zone1

Set the level key to one of error, info, debug, or trace. Per-component overrides use keys of the form level.<component> (e.g., level.drbd-node-agent). The level watcher in each component polls this ConfigMap every 15 seconds and applies changes live.

How controllers are organized across clusters

TSR uses two distinct binary roles:

• /quorum-manager — runs on the quorum cluster inside the site-recovery-quorum-control-plane Deployment (a single manager container, 1/1). Hosts all reconcilers that reason about the global DR state: failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor. Also serves admission webhooks for every quorum-resident CRD.
• /workload-manager — runs on the primary and DR clusters inside the site-recovery-workload-control-plane Deployment. Hosts the DRBDResource placement reconciler and the KubeVirt VM guard webhooks.
• drbd-node-agent — runs as a DaemonSet (one pod per worker node) on the primary and DR clusters. Manages DRBD kernel-level block replication for the volumes bound to its node.

One control-plane image carries both /quorum-manager and /workload-manager binaries; each chart selects its own via the container command.

CRD residency

Understanding which cluster hosts which CRD is essential for kubectl/oc commands. Running a get command on the wrong cluster returns server doesn't have a resource type.

Namespace layout

Published documentation has previously used trilio-system and site-recovery-system for the workload namespace — both are incorrect.

Checking controller pod health

# Quorum control plane (single manager container)
oc get pods -n dr-zone1 -l app.kubernetes.io/component=quorum-control-plane

# Workload control plane on the primary cluster
oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane \
  --context=cluster1

# DRBD node agents on the primary cluster
oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent \
  --context=cluster1

The quorum control plane pod reports 1/1 (one container named manager).

Viewing reconciler logs

# All quorum control-plane logs (all reconcilers share one pod)
oc logs -n dr-zone1 \
  -l app.kubernetes.io/component=quorum-control-plane \
  --since=15m

# Node agent logs for a specific node
NODE=worker-1.example.com
POD=$(oc get pods -n trilio-site-recovery-system \
  --context=cluster1 \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=${NODE} \
  -o jsonpath='{.items[0].metadata.name}')
oc logs -n trilio-site-recovery-system $POD --context=cluster1

Leader election

Both the quorum manager and the ProtectionZone controller use leader election by default (configured in the charts). Leader-election timings are loosened from the controller-runtime defaults to survive transient node-to-API-server stalls: lease duration 137 s, renew deadline 107 s, retry period 26 s. These match OpenShift library-go timings and are not tunable by customers.

The DRBD node agent uses no leader election — it is a DaemonSet where each pod owns its own node exclusively.

How the quorum manager reaches workload clusters

The quorum manager builds two cluster.Cluster handles at startup from the kubeconfig Secrets referenced in the ProtectionZone. Each workload-watching reconciler (failoverrequest, protectionrequest, testfailover, pgsync) uses the same two shared cluster handles, so each workload object kind is watched and cached exactly once. If either workload cluster is unreachable at startup, the quorum manager pod exits (CrashLoopBackOff) rather than silently disable controllers — this is intentional.

Cache scoping

• Quorum manager cache: scoped to the single per-tenant namespace (dr-<name>). All quorum CRs live there.
• Workload manager cache: unscoped (cluster-wide). DRBDResource CRs and the VMs, PVCs, and PVs the placement reconciler touches live in whichever namespace each protected VM sits in, not a single fixed namespace. Scoping the workload cache to a single namespace was a known bug (TSR-128) that prevented protect from completing on real multi-namespace workload clusters.

Admission webhooks

Every admission webhook runs with failurePolicy: Fail unless noted otherwise. Webhook paths follow the pattern /validate-siterecovery-trilio-io-v1alpha1-<resource> and /mutate-siterecovery-trilio-io-v1alpha1-<resource>.

Quorum-cluster webhooks (served by the quorum manager):

Workload-cluster webhooks (served by the workload manager):

VM control mutex

When a controller needs to mutate a protected VM's RunStrategy or Running field, it stamps a siterecovery.trilio.io/vm-control-owner annotation on the VM with the value <kind>:<quorum-namespace>:<uid> (e.g. protectionrequest:dr-zone1:abc-123). Any other TSR controller observing this annotation with a different owner refuses to mutate the VM. The siterecovery.trilio.io/saved-spec annotation captures the pre-mutation RunStrategy/Running so the final restore step can return the VM to its original state.

Example 1: Inspect all controller pods and their labels

Confirm that the quorum controller manager is running and has the expected label selectors:

oc get pods -n dr-production \
  -l app.kubernetes.io/part-of=site-recovery \
  -o wide

Expected output:

NAME                                              READY   STATUS    RESTARTS   AGE   NODE
site-recovery-quorum-control-plane-7d9f8b-x4k2p   1/1     Running   0          2h    quorum-node-1
site-recovery-protectionzone-controller-6c5d-9jn   1/1     Running   0          2h    quorum-node-1

Example 2: Watch a ProtectionRequest advance through its phases

After protecting a VM, watch the protection-controller drive the ProtectionRequest through its lifecycle:

oc get protectionrequest my-vm-protection -n dr-production -w

Expected output (phases appear in order):

NAME                  PHASE           AGE
my-vm-protection      Pending         0s
my-vm-protection      Validating      3s
my-vm-protection      CreatingDRBD    8s
my-vm-protection      Syncing         15s
my-vm-protection      Protected       4m32s

Valid phase values (in order): Pending, Validating, CreatingDRBD, Syncing, ReadyToActivate, Activating, Protected. If the controller encounters an error it transitions to Failed.

Example 3: Trigger a planned failover and watch controller progress

Create a FailoverRequest that asks the failover-controller to move a Protection Group from cluster1 to cluster2:

cat <<EOF | oc apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-01
  namespace: dr-production
spec:
  failoverType: planned
  targetCluster: cluster2
  protectionGroupRef:
    name: production-pg
    namespace: dr-production
  batchBootTimeoutSeconds: 300
EOF

Watch the failover-controller update the status:

oc get failoverrequest planned-failover-01 -n dr-production -w

Expected output:

NAME                  AGE
planned-failover-01   0s
planned-failover-01   12s
planned-failover-01   45s
planned-failover-01   3m10s

For per-VM detail, inspect status.vmStatuses:

oc get failoverrequest planned-failover-01 -n dr-production \
  -o jsonpath='{.status.vmStatuses}' | oc neat

Example 4: Run a non-disruptive test failover and retrieve the test namespace

The test-failover-controller generates the test namespace name; you must not construct or guess it — read it from status.testNamespace:

cat <<EOF | oc apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: dr-drill-01
  namespace: dr-production
spec:
  protectionGroupRef:
    name: production-pg
    namespace: dr-production
  cleanupPolicy: Manual
  timeout: 30m
EOF

Wait for the controller to reach the Verifying phase, then read the generated namespace:

oc get testfailover dr-drill-01 -n dr-production \
  -o jsonpath='{.status.testNamespace}'

Expected output:

site-recovery-production-pg-a3f2c

Inspect test VMs inside that namespace:

oc get vm -n site-recovery-production-pg-a3f2c

Valid TestFailover phase values: Queued, PreparingEnvironment, CreatingSnapshots, CreatingVolumes, CreatingVMs, StartingVMs, Verifying, Succeeded, Failed, CleaningUp, Cleaned.

Example 5: Check replication health and RPO events

The replication-monitor reconciler continuously updates these CRs:

# Aggregated health per Protection Group
oc get replicationgroupstatus -n dr-production

# RPO violation events
oc get rpoevents -n dr-production

To see detail on an RPO violation:

oc describe rpoevent <event-name> -n dr-production

The spec.rpoAtEvent field contains the observed lag in seconds; spec.violationReason explains why the threshold was breached.

Example 6: View quorum controller logs for a specific reconciler

All reconcilers write to the shared manager container. Filter by reconciler name:

# View all quorum controller logs
oc logs -n dr-production \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager --tail=100

# Filter for failover reconciler activity
oc logs -n dr-production \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i failover

# Filter for protection reconciler activity
oc logs -n dr-production \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i protection

Issue 1: Quorum control plane pod is in CrashLoopBackOff

Symptom: The site-recovery-quorum-control-plane pod restarts repeatedly.

Likely cause: The webhook-server-cert secret is missing or contains an invalid certificate. Webhook serving certificates are not managed by the chart and must be created out of band before deployment.

Fix:

# Check whether the secret exists
oc get secret webhook-server-cert -n dr-<zone-name>

# Check the controller logs for the specific error
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager --tail=50

Create the webhook-server-cert secret with a valid TLS certificate and key for the webhook service hostname, then restart the deployment:

oc rollout restart deployment \
  -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane

Issue 2: ProtectionRequest is stuck in Validating phase

Symptom: oc get protectionrequest <name> -n dr-<zone-name> shows PHASE: Validating for more than a few minutes.

Likely cause: The protection-controller cannot reach the primary cluster's API server. This usually means the kubeconfig secret referenced by the ProtectionZone is missing, expired, or the API server is unreachable on TCP 6443.

Fix:

# Check the protection-controller logs
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i protection

# Verify the kubeconfig secret exists and is referenced correctly in the ProtectionZone
oc get protectionzone -n dr-<zone-name> -o yaml | grep kubeconfigSecretRef -A 2
oc get secret <secret-name> -n dr-<zone-name>

If the secret is correct, verify network connectivity from the quorum cluster to the primary cluster's API server on TCP port 6443.

Issue 3: FailoverRequest stays at InProgress indefinitely

Symptom: oc get failoverrequest <name> -n dr-<zone-name> shows the request has not reached Completed after several minutes.

Likely cause: One of:

• VMs on the source cluster have not finished shutting down. (Note: drainTimeoutSeconds is accepted by the API but not consumed in Tech Preview, so it is not the deadline being applied here.)
• DRBD replication has not reached a consistent state, and the failover-controller is waiting before promoting volumes.
• The DR cluster API server is unreachable.

Fix:

# Check failover-controller logs for the stall reason
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i failover

# Check per-VM status for detail
oc get failoverrequest <name> -n dr-<zone-name> \
  -o jsonpath='{.status.vmStatuses}'

# Check DRBDResource state on both clusters
oc get drbdresource -n <vm-namespace> \
  --kubeconfig <primary-kubeconfig>
oc get drbdresource -n <vm-namespace> \
  --kubeconfig <dr-kubeconfig>

Issue 4: TestFailover is stuck in CreatingVMs phase

Symptom: A TestFailover CR shows PHASE: CreatingVMs and does not progress.

Likely cause: The test namespace on the DR cluster has insufficient capacity, or the snapshot-based PVCs failed to provision. Check that the DR cluster has available storage and that CDI (Containerized Data Importer) is operational if it is in use.

Fix:

# Get the generated test namespace name from status
TEST_NS=$(oc get testfailover <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')

# Inspect PVCs in the test namespace on the DR cluster
oc get pvc -n "$TEST_NS" --kubeconfig <dr-kubeconfig>

# Inspect events in the test namespace
oc get events -n "$TEST_NS" --kubeconfig <dr-kubeconfig> \
  --sort-by='.lastTimestamp'

# Check test-failover-controller logs
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i testfailover

Do not attempt to patch finalizers or delete resources in the test namespace manually — use the cleanupPolicy field or delete the TestFailover CR itself and let the controller perform the cleanup.

Issue 5: drbd-node-agent pods are not running on some worker nodes

Symptom: oc get pods -n dr-<zone-name> -l app.kubernetes.io/component=workload-control-plane shows fewer pods than expected worker nodes.

Likely cause: The node does not have the DRBD kernel module (9.x or later) loaded, or the LVM thin pool is not configured on that node.

Fix:

# Identify which nodes are missing the agent
oc get pods -n dr-<zone-name> \
  -l app.kubernetes.io/component=workload-control-plane \
  -o wide

# Check the DaemonSet for scheduling issues
oc describe daemonset -n dr-<zone-name> \
  -l app.kubernetes.io/component=workload-control-plane

# On the affected node, verify the DRBD module is loaded
# (run via a debug pod or node shell)
lsmod | grep drbd

Load the DRBD kernel module and verify that the LVM thin pool backed by a dedicated block device is present on the node, then the DaemonSet will schedule the agent automatically.

Issue 6: ProtectionZone creation is rejected with a webhook error

Symptom: oc apply -f protectionzone.yaml returns an admission webhook error.

Likely cause: Either storageBackendMode is set to Linstor (not supported), or the clusters array does not have exactly two entries each with a valid clusterID of cluster1 or cluster2.

Fix: Ensure your ProtectionZone manifest uses storageBackendMode: DRBD and that the clusters array contains exactly two entries:

spec:
  storageBackendMode: DRBD   # Only valid value
  clusters:
    - name: primary
      clusterID: cluster1    # Must be 'cluster1' or 'cluster2'
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr-site
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Collecting a diagnostic bundle

For any persistent controller issue, collect a full diagnostic bundle from the quorum cluster using the tsr-gather must-gather tool:

oc adm must-gather \
  --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle

The bundle includes logs, CRD state, and configuration from all clusters in the DR estate without capturing secrets or credentials.