Controller Architecture

This page explains how Trilio Site Recovery's controllers watch and reconcile resources across the TSR-Manager cluster and the two workload clusters. Understanding the controller architecture helps you diagnose stuck operations, interpret status conditions, and predict how a change to a custom resource propagates through the system. The product is built entirely on Kubernetes controller-runtime: every DR operation—protecting a VM, running a failover, or executing a test failover—is initiated by creating a custom resource and completed by the controllers that reconcile it.

Before reading this page you should be familiar with:

• Kubernetes controller-runtime concepts: reconcile loops, informer caches, field indexes, and admission webhooks
• The three-cluster topology: one TSR-Manager cluster (management plane) and two workload clusters (primary and DR), all running OpenShift 4.17 or later
• The eight CRDs on the TSR-Manager cluster: ProtectionZone, ProtectionGroup, ProtectionRequest, DRBDReplicationPolicy, FailoverRequest, TestFailover, ReplicationGroupStatus, and RPOEvent
• The one CRD on each workload cluster: DRBDResource
• kubectl and oc: all diagnostic commands on this page use these CLIs; oc is required for OpenShift-specific paths

Required cluster access:

• kubeconfig for the TSR-Manager cluster (where all control-plane pods run)
• kubeconfig files for both workload clusters (to query DRBDResource objects)

The controller binaries are not installed by the user directly. The OLM bundle installs the protection-zone-controller Deployment on the TSR-Manager cluster, and the ProtectionZone controller automatically deploys the remaining control-plane components once the zone reaches Available. No helm install commands are needed.

Step 1 — Install the OLM bundle on the TSR-Manager cluster

Create the CatalogSource, OperatorGroup, and Subscription in the trilio-site-recovery-system namespace:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: site-recovery-catalog
  namespace: trilio-site-recovery-system
spec:
  sourceType: grpc
  image: quay.io/triliovault/site-recovery-catalog
  displayName: Trilio Site Recovery
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: site-recovery-og
  namespace: trilio-site-recovery-system
spec:
  targetNamespaces:
  - trilio-site-recovery-system
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: tech-preview
  name: site-recovery
  source: site-recovery-catalog
  sourceNamespace: trilio-site-recovery-system

Apply the manifest:

oc apply -f subscription.yaml

This installs the protection-zone-controller Deployment, the ProtectionZone CRD, and the TSR web console (site-recovery-backend and site-recovery-frontend) on the TSR-Manager cluster.

Step 2 — Create kubeconfig Secrets for the workload clusters

In the zone namespace (for example, dr-prod), create one Secret per workload cluster:

oc create namespace dr-prod

oc create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=/path/to/cluster1.kubeconfig \
  -n dr-prod

oc create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=/path/to/cluster2.kubeconfig \
  -n dr-prod

Step 3 — Create the ProtectionZone

Creating a ProtectionZone triggers the protection-zone-controller to deploy the site-recovery-manager-control-plane chart into dr-prod on the TSR-Manager cluster and the site-recovery-workload-control-plane chart into trilio-site-recovery-system on each workload cluster.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: Production DR Zone
  storageBackendMode: DRBD
  clusters:
  - name: cluster1
    clusterID: cluster1
    kubeconfigSecretRef:
      name: cluster1-kubeconfig
      key: kubeconfig
  - name: cluster2
    clusterID: cluster2
    kubeconfigSecretRef:
      name: cluster2-kubeconfig
      key: kubeconfig

oc apply -f protectionzone.yaml

Watch the zone reach Available:

oc get pz prod -n dr-prod -w

Once the zone is Available, all control-plane controllers are running and the drbd-node-agent DaemonSet is present on both workload clusters.

Controller behavior is governed by two runtime-editable ConfigMaps and by the ProtectionZone spec itself. Customers never edit Helm chart values or run helm upgrade.

Zone-config ConfigMap (site-recovery-zone-config)

The ProtectionZone controller creates one site-recovery-zone-config ConfigMap per zone namespace on the TSR-Manager cluster. Controllers re-read it on every reconcile; changes take effect without a pod restart.

Edit a value in place:

oc edit configmap site-recovery-zone-config -n dr-prod

Values must be Go duration strings (for example 90s, 2m, 10m). An unparseable value falls back to the compiled-in default; a value below the minimum is clamped up.

Logging ConfigMap (site-recovery-logging)

The ProtectionZone controller also creates a site-recovery-logging ConfigMap per zone namespace. The LevelWatcher Runnable inside each controller binary polls it every 15 seconds and applies the level without a pod restart.

Note: warn is not a valid level. The log model is error/info/debug/trace only.

oc patch configmap site-recovery-logging -n dr-prod \
  --type=merge -p '{"data":{"level":"debug"}}'

ProtectionZone spec fields that affect controller behavior

Webhook certificates

Webhook TLS certificates are not managed by the charts or cert-manager. You must create the webhook-server-cert Secret (or the name configured in webhook.certSecretName) in each release namespace before the webhooks can serve. The webhook container port is 9443; it is exposed as port 443 on the Service.

How CRDs map to controllers

Every DR operation is driven by creating a custom resource. The table below shows which controller reconciles each resource, where that controller runs, and which cluster it watches the resource on.

All TSR-Manager-side controllers run inside a single manager container (1/1) inside the site-recovery-manager-control-plane Deployment in the zone namespace (dr-<name>). The workload-side controllers run inside the site-recovery-workload-control-plane Deployment in trilio-site-recovery-system on each workload cluster, plus the drbd-node-agent DaemonSet.

Watching controller pods and logs

All TSR-Manager control-plane pods carry the label app.kubernetes.io/part-of: site-recovery and app.kubernetes.io/component: manager-control-plane.

# List all TSR-Manager control-plane pods in a zone
oc get pods \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane \
  -n dr-prod

# Stream logs from the manager pod (all reconcilers share one container)
oc logs -l app.kubernetes.io/component=manager-control-plane \
  -n dr-prod --follow

Workload control-plane pods:

# On cluster1
kubectl get pods \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  -n trilio-site-recovery-system --context cluster1

Node-agent pods (one per worker node on each workload cluster):

kubectl get pods \
  -l app.kubernetes.io/component=drbd-node-agent \
  -n trilio-site-recovery-system --context cluster1

Important: The node-agent DaemonSet object name is <release>-site-recovery-workload-control-plane-agent, not drbd-node-agent. Use the label selector above, not the DaemonSet name, to find pods.

Cache scoping and namespace placement

The TSR-Manager manager's informer cache is scoped to the zone namespace (dr-<name>). All TSR-Manager-resident CRs—ProtectionZone, ProtectionGroup, ProtectionRequest, DRBDReplicationPolicy, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent—live in this namespace. Querying them against the TSR-Manager cluster:

oc get protectionrequest,protectiongroup,failoverrequest,testfailover \
  -n dr-prod

The workload manager's cache is unscoped (cluster-wide). DRBDResource objects live in the protected VM's namespace on each workload cluster:

kubectl get dres -A --context cluster1

Common mistake: running kubectl get dres against the TSR-Manager cluster returns server doesn't have a resource type because DRBDResource is registered only on the workload clusters.

Field indexes

The controllers rely on controller-runtime field indexes for efficient lookups. The most important ones:

• spec.protectionGroupRef — indexes FailoverRequest and TestFailover by their PG reference; used by admission webhooks to detect concurrent operations
• protectionRequest.vm — indexes ProtectionRequest by VM namespace/name; used by the protection controller and the PG webhook
• protectionGroup.vm — indexes ProtectionGroup by member VM; used by the PG uniqueness check
• drbdResource.node — indexes DRBDResource by spec.nodeName; used by the node agent to own only its own node's resources

Field selectors on spec.* or status.* fields of these custom resources do not work at the API server level. Filter with label selectors instead:

# Correct: use label to find ReplicationGroupStatus for a protection group
kubectl get rgs -l siterecovery.trilio.io/protection-group=<pg-name> -n dr-prod

# Wrong: --field-selector on custom resource spec fields fails
kubectl get rgs --field-selector spec.protectionGroupRef.name=<pg-name>  # returns error

Cross-cluster coordination

The TSR-Manager manager builds two cluster.Cluster handles at startup—one per workload cluster—using the kubeconfig Secrets referenced in the ProtectionZone. These handles provide cached reads and direct-write clients to both workload clusters. The ProtectionRequest controller uses them to create and manage DRBDResource objects on the correct workload cluster. The pg-sync-controller uses them to mirror ProtectionGroup metadata and keep standby VM specs up to date. Neither workload cluster ever reaches back to the TSR-Manager API server; traffic is one-way.

If a workload cluster is unreachable at manager startup, the pod crashes and enters CrashLoopBackOff until connectivity is restored. This is intentional: a silently disabled controller is more dangerous than a crashing one.

Admission webhooks

Every custom resource type has at least a validating webhook. Key cross-field rules enforced at admission (not by CRD schema alone):

Webhooks run on port 9443 (container) / 443 (Service). Certificates must be provided out of band in the webhook-server-cert Secret.

Inspect the zone status and cluster reachability

oc get pz prod -n dr-prod -o yaml

Check status.phase (should be Available) and status.clusters[*].status (should be Reachable).

Find all protected VMs across both workload clusters

The label siterecovery.trilio.io/protected-vm=true is stamped on the KubeVirt VirtualMachine object on the workload cluster only.

# On cluster1
kubectl get vm -A \
  -l siterecovery.trilio.io/protected-vm=true \
  --context cluster1

# On cluster2
kubectl get vm -A \
  -l siterecovery.trilio.io/protected-vm=true \
  --context cluster2

Check replication health for a protection group

# The ReplicationGroupStatus is named <pg-name>-replication
oc get rgs prod-web-replication -n dr-prod -o yaml

Or use the label:

oc get rgs \
  -l siterecovery.trilio.io/protection-group=prod-web \
  -n dr-prod

View all DRBDResource objects for a VM on a workload cluster

DRBDResource names follow the pattern drbd-<namespace>-<vm>-<hash8>.

kubectl get dres -A --context cluster1 | grep my-vm

Monitor a ProtectionRequest lifecycle

oc get pr -n dr-prod -w

Expected phases in order: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

Check logs for a specific reconciler

All reconcilers share one manager container. Filter by the reconciler name in the structured JSON output:

oc logs \
  -l app.kubernetes.io/component=manager-control-plane \
  -n dr-prod \
  | grep '"reconciler":"protectionrequest"'

List RPO violation events for a protection group

oc get rpoe \
  -l siterecovery.trilio.io/protection-group=prod-web \
  -n dr-prod \
  -o custom-columns='TIME:.spec.timestamp,TYPE:.spec.eventType,RPO:.spec.rpoAtEvent,REASON:.spec.violationReason'

Adjust the runtime log level without restarting pods

oc patch configmap site-recovery-logging -n dr-prod \
  --type=merge \
  -p '{"data":{"level":"debug","level.drbd-node-agent":"trace"}}'

The LevelWatcher Runnable in each pod polls this ConfigMap every 15 seconds and applies the change. Remove the key to revert to the startup level:

oc patch configmap site-recovery-logging -n dr-prod \
  --type=json \
  -p '[{"op":"remove","path":"/data/level"}]'

Collect a diagnostic bundle

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  -n dr-prod

The gather binary inside the image collects logs, CRD state, and configuration from all clusters in the DR estate into a single bundle.

Find node-agent logs for a specific worker node

# Step 1: find the agent pod on the target node
NODE=worker-1.example.com
POD=$(kubectl get pods \
  --field-selector spec.nodeName=${NODE} \
  -l app.kubernetes.io/component=drbd-node-agent \
  -n trilio-site-recovery-system \
  --context cluster1 \
  -o jsonpath='{.items[0].metadata.name}')

# Step 2: stream its logs
kubectl logs ${POD} \
  -n trilio-site-recovery-system \
  --context cluster1 \
  --follow

Issue 1: Quorum control plane pod is in CrashLoopBackOff

Symptom: The site-recovery-quorum-control-plane pod restarts repeatedly.

Likely cause: The webhook-server-cert secret is missing or contains an invalid certificate. Webhook serving certificates are not managed by the chart and must be created out of band before deployment.

Fix:

# Check whether the secret exists
oc get secret webhook-server-cert -n dr-<zone-name>

# Check the controller logs for the specific error
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager --tail=50

Create the webhook-server-cert secret with a valid TLS certificate and key for the webhook service hostname, then restart the deployment:

oc rollout restart deployment \
  -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane

Issue 2: ProtectionRequest is stuck in Validating phase

Symptom: oc get protectionrequest <name> -n dr-<zone-name> shows PHASE: Validating for more than a few minutes.

Likely cause: The protection-controller cannot reach the primary cluster's API server. This usually means the kubeconfig secret referenced by the ProtectionZone is missing, expired, or the API server is unreachable on TCP 6443.

Fix:

# Check the protection-controller logs
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i protection

# Verify the kubeconfig secret exists and is referenced correctly in the ProtectionZone
oc get protectionzone -n dr-<zone-name> -o yaml | grep kubeconfigSecretRef -A 2
oc get secret <secret-name> -n dr-<zone-name>

If the secret is correct, verify network connectivity from the quorum cluster to the primary cluster's API server on TCP port 6443.

Issue 3: FailoverRequest stays at InProgress indefinitely

Symptom: oc get failoverrequest <name> -n dr-<zone-name> shows the request has not reached Completed after several minutes.

Likely cause: One of:

• VMs on the source cluster have not finished shutting down. (Note: drainTimeoutSeconds is accepted by the API but not consumed in Tech Preview, so it is not the deadline being applied here.)
• DRBD replication has not reached a consistent state, and the failover-controller is waiting before promoting volumes.
• The DR cluster API server is unreachable.

Fix:

# Check failover-controller logs for the stall reason
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i failover

# Check per-VM status for detail
oc get failoverrequest <name> -n dr-<zone-name> \
  -o jsonpath='{.status.vmStatuses}'

# Check DRBDResource state on both clusters
oc get drbdresource -n <vm-namespace> \
  --kubeconfig <primary-kubeconfig>
oc get drbdresource -n <vm-namespace> \
  --kubeconfig <dr-kubeconfig>

Issue 4: TestFailover is stuck in CreatingVMs phase

Symptom: A TestFailover CR shows PHASE: CreatingVMs and does not progress.

Likely cause: The test namespace on the DR cluster has insufficient capacity, or the snapshot-based PVCs failed to provision. Check that the DR cluster has available storage and that CDI (Containerized Data Importer) is operational if it is in use.

Fix:

# Get the generated test namespace name from status
TEST_NS=$(oc get testfailover <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')

# Inspect PVCs in the test namespace on the DR cluster
oc get pvc -n "$TEST_NS" --kubeconfig <dr-kubeconfig>

# Inspect events in the test namespace
oc get events -n "$TEST_NS" --kubeconfig <dr-kubeconfig> \
  --sort-by='.lastTimestamp'

# Check test-failover-controller logs
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -i testfailover

Do not attempt to patch finalizers or delete resources in the test namespace manually — use the cleanupPolicy field or delete the TestFailover CR itself and let the controller perform the cleanup.

Issue 5: drbd-node-agent pods are not running on some worker nodes

Symptom: oc get pods -n dr-<zone-name> -l app.kubernetes.io/component=workload-control-plane shows fewer pods than expected worker nodes.

Likely cause: The node does not have the DRBD kernel module (9.x or later) loaded, or the LVM thin pool is not configured on that node.

Fix:

# Identify which nodes are missing the agent
oc get pods -n dr-<zone-name> \
  -l app.kubernetes.io/component=workload-control-plane \
  -o wide

# Check the DaemonSet for scheduling issues
oc describe daemonset -n dr-<zone-name> \
  -l app.kubernetes.io/component=workload-control-plane

# On the affected node, verify the DRBD module is loaded
# (run via a debug pod or node shell)
lsmod | grep drbd

Load the DRBD kernel module and verify that the LVM thin pool backed by a dedicated block device is present on the node, then the DaemonSet will schedule the agent automatically.

Issue 6: ProtectionZone creation is rejected with a webhook error

Symptom: oc apply -f protectionzone.yaml returns an admission webhook error.

Likely cause: Either storageBackendMode is set to Linstor (not supported), or the clusters array does not have exactly two entries each with a valid clusterID of cluster1 or cluster2.

Fix: Ensure your ProtectionZone manifest uses storageBackendMode: DRBD and that the clusters array contains exactly two entries:

spec:
  storageBackendMode: DRBD   # Only valid value
  clusters:
    - name: primary
      clusterID: cluster1    # Must be 'cluster1' or 'cluster2'
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr-site
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Collecting a diagnostic bundle

For any persistent controller issue, collect a full diagnostic bundle from the quorum cluster using the tsr-gather must-gather tool:

oc adm must-gather \
  --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle

The bundle includes logs, CRD state, and configuration from all clusters in the DR estate without capturing secrets or credentials.