Installation
Installing CRDs and deploying the site recovery operator
This page walks you through installing Trilio Site Recovery (TSR) on OpenShift. TSR is a Kubernetes operator that automates disaster recovery for KubeVirt virtual machines using DRBD synchronous block replication (Protocol C, RPO=0) across three clusters: a quorum cluster that hosts the management plane, a primary workload cluster, and a DR workload cluster. Installation has two complementary parts: first you install the operator bundle on the quorum cluster via OLM, then you create a ProtectionZone custom resource that causes the operator to automatically deploy the workload control planes on both workload clusters. You never run helm install directly — the Helm charts are embedded in the operator binary and are applied by the ProtectionZone controller once the zone reaches Available.
Before you begin, ensure the following are in place across all three clusters.
Clusters
- Three OpenShift clusters at version 4.17 or later: one quorum (management plane), one primary, and one DR cluster. OpenShift 4.17 is the supported floor for all three roles.
- KubeVirt or OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary and DR clusters.
Storage
- DRBD kernel module ≥ 9.x loaded on every worker node of the primary and DR clusters.
- An LVM thin pool configured on each worker node of the primary and DR clusters, backed by a dedicated block device, to serve as DRBD backing storage.
Tooling
ocCLI (required;kubectl-only operation is not a supported mode for OpenShift paths includingoc adm must-gatherand Routes)- Kubeconfig files for all three clusters
Network
- TCP ports 7000–8000 inclusive (DRBD replication) open in both directions between every worker node on the primary and DR clusters. One port is consumed per replicated VM volume — a rule that stops at 7999 will leave one volume unable to replicate.
- TCP port 6443 (Kubernetes API) open from the quorum cluster to the primary and DR API servers.
- For synchronous replication (Protocol C, RPO=0): round-trip latency below 50 ms between primary and DR clusters.
Certificates
- Webhook serving certificates are not managed for you. You must create a Secret named
webhook-server-certin each release namespace on the quorum cluster before the control planes start. There is no cert-manager integration.
OLM
- Operator Lifecycle Manager (OLM) running on the quorum cluster (standard on OpenShift).
Installation proceeds in two stages:
- Install the operator bundle on the quorum cluster via OLM — this deploys the
protection-zone-controller, the TSR web console (site-recovery-backendandsite-recovery-frontend), and registers theProtectionZoneCRD. - Create a ProtectionZone CR — this triggers the operator to automatically deploy the quorum control plane and the workload control planes (including the DRBD node-agent DaemonSet) on both workload clusters.
Stage 1 — Install the operator on the quorum cluster
All commands in this stage target the quorum cluster.
Step 1: Create the operator namespace
oc --context=<quorum-context> create namespace trilio-site-recovery-system
Step 2: Create the CatalogSource
This points OLM at the Trilio Site Recovery operator catalog.
# catalogsource.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: trilio-site-recovery-catalog
namespace: trilio-site-recovery-system
spec:
sourceType: grpc
image: quay.io/triliovault/site-recovery-catalog:latest
displayName: Trilio Site Recovery
publisher: Trilio Data, Inc.
oc --context=<quorum-context> apply -f catalogsource.yaml
Wait for the catalog pod to reach Running:
oc --context=<quorum-context> -n trilio-site-recovery-system \
get pods -l olm.catalogSource=trilio-site-recovery-catalog
Step 3: Create the OperatorGroup
# operatorgroup.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: trilio-site-recovery
namespace: trilio-site-recovery-system
spec:
targetNamespaces:
- trilio-site-recovery-system
oc --context=<quorum-context> apply -f operatorgroup.yaml
Step 4: Create the Subscription
# subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: site-recovery
namespace: trilio-site-recovery-system
spec:
channel: tech-preview
name: site-recovery
source: trilio-site-recovery-catalog
sourceNamespace: trilio-site-recovery-system
oc --context=<quorum-context> apply -f subscription.yaml
Step 5: Verify operator installation
Wait for the CSV to reach Succeeded and the protection-zone-controller pod to be Running:
# Check CSV status
oc --context=<quorum-context> -n trilio-site-recovery-system \
get csv -l operators.coreos.com/site-recovery.trilio-site-recovery-system
# Check operator pods — expect protection-zone-controller, site-recovery-backend,
# and site-recovery-frontend all Running
oc --context=<quorum-context> -n trilio-site-recovery-system \
get pods -l app.kubernetes.io/part-of=site-recovery
Expected output shows three pods running:
NAME READY STATUS RESTARTS AGE
protection-zone-controller-<hash> 1/1 Running 0 2m
site-recovery-backend-<hash> 1/1 Running 0 2m
site-recovery-frontend-<hash> 2/2 Running 0 2m
The site-recovery-frontend pod runs two containers: the Next.js frontend and the OpenShift oauth-proxy sidecar. On OpenShift, the ProtectionZone controller will create a Route and console link for browser access once a zone reaches Available.
Stage 2 — Create the ProtectionZone
The ProtectionZone CR wires together the quorum and workload clusters. Creating it causes the operator to:
- Deploy the quorum control plane (
site-recovery-quorum-control-plane) in the zone namespace on the quorum cluster. - Deploy the workload control plane (
site-recovery-workload-control-plane) including the DRBD node-agent DaemonSet intrilio-site-recovery-systemon each workload cluster. - Create the default
DRBDReplicationPolicynamed<zoneName>-defaultin the zone namespace.
All zone CRs live in a namespace with the prefix dr-<name>. All workload-side components live in trilio-site-recovery-system.
Step 6: Create the zone namespace on the quorum cluster
oc --context=<quorum-context> create namespace dr-<zone-name>
Replace <zone-name> with a short, lowercase identifier for your DR deployment (for example, prod).
Step 7: Create webhook serving certificates
The admission webhooks require a TLS certificate. This chart does not manage certificate provisioning; you must create the Secret before the zone is created.
Create the certificate secret in the zone namespace on the quorum cluster:
# Generate a self-signed cert for the webhook service SAN
# The SAN must cover the webhook service in the zone namespace.
# Replace <zone-name> with your zone name.
oc --context=<quorum-context> -n dr-<zone-name> create secret tls webhook-server-cert \
--cert=tls.crt \
--key=tls.key
Also create the webhook certificate secret in trilio-site-recovery-system on each workload cluster:
oc --context=<primary-context> -n trilio-site-recovery-system create secret tls webhook \
--cert=tls.crt \
--key=tls.key
oc --context=<dr-context> -n trilio-site-recovery-system create secret tls webhook \
--cert=tls.crt \
--key=tls.key
Note: the quorum chart's default webhook.certSecretName is webhook-server-cert; the workload chart's default is webhook.
Step 8: Create kubeconfig Secrets for the workload clusters
The ProtectionZone controller uses these to reach the primary and DR clusters. Create them in the zone namespace on the quorum cluster.
# Secret for the primary cluster (cluster1)
oc --context=<quorum-context> -n dr-<zone-name> create secret generic cluster1-kubeconfig \
--from-file=kubeconfig=<path-to-primary-kubeconfig>
# Secret for the DR cluster (cluster2)
oc --context=<quorum-context> -n dr-<zone-name> create secret generic cluster2-kubeconfig \
--from-file=kubeconfig=<path-to-dr-kubeconfig>
Step 9: Create the ProtectionZone CR
# protectionzone.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: prod
namespace: dr-prod
spec:
displayName: "Production DR Zone"
description: "Primary-to-DR DRBD protection zone for production VMs."
storageBackendMode: DRBD
clusters:
- name: primary
clusterID: cluster1
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster1-kubeconfig
key: kubeconfig
- name: dr
clusterID: cluster2
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster2-kubeconfig
key: kubeconfig
oc --context=<quorum-context> apply -f protectionzone.yaml
Step 10: Monitor the ProtectionZone until it reaches Available
oc --context=<quorum-context> -n dr-prod get pz prod -w
Expected progression:
NAME DISPLAY NAME BACKEND CLUSTERS STATUS AGE
prod Production DR Zone DRBD primary,dr Provisioning 10s
prod Production DR Zone DRBD primary,dr Available 90s
Once Available, the operator has:
- Deployed the quorum control plane manager in
dr-prodon the quorum cluster. - Deployed the workload control plane and DRBD node-agent DaemonSet in
trilio-site-recovery-systemon both workload clusters. - Created the default replication policy
prod-defaultindr-prod. - Created an OpenShift Route and console link for browser access to the TSR web console.
Step 11: Verify workload cluster components
Verify the workload control planes are running on both clusters:
# Primary cluster — workload control plane manager
oc --context=<primary-context> -n trilio-site-recovery-system \
get pods -l app.kubernetes.io/component=workload-control-plane
# Primary cluster — DRBD node agents (one per worker node)
oc --context=<primary-context> -n trilio-site-recovery-system \
get pods -l app.kubernetes.io/component=drbd-node-agent
# DR cluster — same checks
oc --context=<dr-context> -n trilio-site-recovery-system \
get pods -l app.kubernetes.io/component=workload-control-plane
oc --context=<dr-context> -n trilio-site-recovery-system \
get pods -l app.kubernetes.io/component=drbd-node-agent
Verify the quorum control plane manager on the quorum cluster:
oc --context=<quorum-context> -n dr-prod \
get pods -l app.kubernetes.io/component=quorum-control-plane
The quorum manager pod runs a single container named manager and shows 1/1 Running.
Step 12: Verify CRD registration
The quorum cluster registers 8 CRDs; each workload cluster registers 1:
# On the quorum cluster — should list 8 site-recovery CRDs
oc --context=<quorum-context> get crd | grep siterecovery.trilio.io
# On a workload cluster — should list only drbdresources
oc --context=<primary-context> get crd | grep siterecovery.trilio.io
Note: drbdresources.siterecovery.trilio.io is registered only on workload clusters. Running oc get drbdresource against the quorum cluster returns server doesn't have a resource type.
You do not modify Helm chart values directly — the ProtectionZone controller manages chart deployment. However, the following configuration surfaces are available to you.
ProtectionZone spec fields
These are the fields you author in the ProtectionZone CR. All other configuration is managed by the operator.
| Field | Type | Required | Default | Notes |
|---|---|---|---|---|
spec.displayName | string | Yes | — | Human-friendly name shown in the UI. 5–253 characters. |
spec.description | string | No | — | Free-form text. Max 1024 characters. |
spec.storageBackendMode | string | Yes | — | Must be DRBD. The value Linstor is rejected by the admission webhook. |
spec.clusters[].name | string | Yes | — | Operator-facing cluster identifier within the zone. Max 63 characters. |
spec.clusters[].clusterID | string | Yes | — | Deployment slot: cluster1 or cluster2. Must be unique across the two entries. |
spec.clusters[].distributionName | string | No | RHEL-CoreOS-9 | OS family: RHEL-CoreOS-9 or RHEL-CoreOS-10. Selects the DRBD kernel-module loader image. RHEL-CoreOS-10 is not validated in Tech Preview. |
spec.clusters[].kubeconfigSecretRef.name | string | Yes | — | Name of the Secret in the zone namespace holding the cluster kubeconfig. |
spec.clusters[].kubeconfigSecretRef.key | string | Yes | — | Key within the Secret (typically kubeconfig). |
spec.logging.level | string | Conditional | — | Runtime log level for all zone components: error, info, debug, or trace. Required when spec.logging is set. Takes effect within ~15 seconds without pod restarts. |
spec.logging.controlPlaneLevel | string | No | — | Overrides level for control-plane managers only. |
spec.logging.nodeAgentLevel | string | No | — | Overrides level for DRBD node agents only. |
DRBDReplicationPolicy
You do not create or modify this resource. The ProtectionZone controller automatically creates one default policy named <zoneName>-default with isDefault: true in the zone namespace when the zone reaches Available. Only this default policy is ever consulted by the protection controllers. Protocol C synchronous replication (RPO=0) is the only supported mode; the admission webhook rejects Protocol A.
Webhook serving certificates
Certificate provisioning is not managed by the charts. You must create TLS secrets out of band (see Step 7 in Installation):
| Secret name | Namespace | Cluster | Chart default webhook.certSecretName |
|---|---|---|---|
webhook-server-cert | dr-<zone-name> | Quorum | webhook-server-cert |
webhook | trilio-site-recovery-system | Primary | webhook |
webhook | trilio-site-recovery-system | DR | webhook |
Runtime log levels
Set spec.logging on the ProtectionZone CR to change log levels without restarting pods. The operator propagates the change to a site-recovery-logging ConfigMap in each component namespace, which is polled every ~15 seconds.
# Raise all zone components to debug
oc --context=<quorum-context> -n dr-prod patch pz prod --type=merge -p '
{"spec":{"logging":{"level":"debug"}}}'
# Set control-plane managers to debug, node agents to info
oc --context=<quorum-context> -n dr-prod patch pz prod --type=merge -p '
{"spec":{"logging":{"level":"info","controlPlaneLevel":"debug","nodeAgentLevel":"info"}}}'
Accepted levels: error, info, debug, trace.
Node agent resource tuning
The DRBD node-agent DaemonSet is deployed by the ProtectionZone controller with these default resource limits (from helm/site-recovery-workload-control-plane/values.yaml):
| Resource | Request | Limit |
|---|---|---|
| CPU | 10m | 500m |
| Memory | 64Mi | 256Mi |
These are the tested values. If your nodes require more, file a support request — direct chart value modification is not a customer procedure.
Zone configuration deadlines
The workload chart deploys a site-recovery-zone-config ConfigMap in trilio-site-recovery-system on each workload cluster. The two keys relevant to DR sizing are:
| Key | Default | Effect |
|---|---|---|
drbdMetadataReadyGracePeriod | 2m | How long a metadata PVC may report not-ready before ConditionPlacementReady is held False. |
drbdMetadataProvisioningFailureDeadline | 10m | How long ConditionPlacementReady may stay continuously False before the placement reconciler marks the copy Failed. |
These values survive pod restarts but are overwritten on the next operator reconcile. Do not hand-edit this ConfigMap between operator upgrades.
Once the ProtectionZone reaches Available, you can protect VMs and manage DR operations entirely through oc and the TSR web console.
Checking overall zone health
# Summary view of the zone
oc --context=<quorum-context> -n dr-prod get pz prod
# Detailed status including per-cluster reachability
oc --context=<quorum-context> -n dr-prod get pz prod -o yaml
Protecting a virtual machine
To protect a VM, create a ProtectionRequest in the zone namespace on the quorum cluster. The VM must be running when you create the request.
oc --context=<quorum-context> -n dr-prod apply -f protect-my-vm.yaml
Watch protection progress:
oc --context=<quorum-context> -n dr-prod get pr protect-my-vm -w
The phase progression is: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.
The Sync column shows block-level resync progress. Initial resync of a large disk can take time; protection is not complete until the phase reaches Protected.
Grouping VMs for coordinated failover
A ProtectionGroup lets you fail over a set of VMs as a single atomic unit:
oc --context=<quorum-context> -n dr-prod get pg
oc --context=<quorum-context> -n dr-prod get pg <name> -o yaml
Monitoring replication health
The replication monitor auto-creates one ReplicationGroupStatus per ProtectionGroup named <protection-group-name>-replication:
oc --context=<quorum-context> -n dr-prod get rgs
oc --context=<quorum-context> -n dr-prod get rgs <pg-name>-replication -o yaml
Filter RPO audit events by protection group:
oc --context=<quorum-context> -n dr-prod get rpoe \
-l siterecovery.trilio.io/protection-group=<pg-name>
Checking DRBD replication state on workload clusters
DRBDResource CRs live on the workload clusters in the VM's namespace — not on the quorum cluster:
# List all protected VM replication state on the primary cluster
oc --context=<primary-context> get dres -A
# Detail for a specific VM's resource
oc --context=<primary-context> -n <vm-namespace> get dres <resource-name> -o yaml
Identify protected VMs by label (workload clusters only):
oc --context=<primary-context> get vm -A \
-l siterecovery.trilio.io/protected-vm=true
TSR web console (OpenShift)
On OpenShift, the ProtectionZone controller creates a Route and an OpenShift console link once the zone reaches Available. Access the console through the OpenShift web console's application launcher or by retrieving the Route URL:
oc --context=<quorum-context> -n trilio-site-recovery-system get route
Authentication uses the OpenShift oauth-proxy. The console is not accessible on upstream Kubernetes.
RBAC — assigning personas
Three ClusterRoles are installed by the OLM bundle:
| ClusterRole | Binding type | Capabilities |
|---|---|---|
trilio-site-recovery-admin | ClusterRoleBinding | Full control of all TSR resources and kubeconfig Secrets. Effectively cluster-admin. Consider using a namespace-scoped RoleBinding instead. |
trilio-site-recovery-viewer | ClusterRoleBinding | Read-only across all zones. No Secret access. |
trilio-site-recovery-operator | RoleBinding (per zone namespace) | Full control of TestFailover; read-only on everything else. Cannot trigger production failover. |
# Grant a user admin access (cluster-wide — see security note in the ClusterRole)
oc --context=<quorum-context> create clusterrolebinding tsr-admin-<user> \
--clusterrole=trilio-site-recovery-admin \
--user=<username>
# Grant a user operator access to one zone
oc --context=<quorum-context> -n dr-prod create rolebinding tsr-operator-<user> \
--clusterrole=trilio-site-recovery-operator \
--user=<username>
Example 1 — Minimal ProtectionZone CR
A complete, minimal zone definition pairing two RHCOS 9 clusters:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: prod
namespace: dr-prod
spec:
displayName: "Production DR Zone"
storageBackendMode: DRBD
clusters:
- name: primary
clusterID: cluster1
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster1-kubeconfig
key: kubeconfig
- name: dr
clusterID: cluster2
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster2-kubeconfig
key: kubeconfig
Apply and verify:
oc --context=<quorum-context> apply -f protectionzone.yaml
oc --context=<quorum-context> -n dr-prod get pz prod -w
Expected output (once stable):
NAME DISPLAY NAME BACKEND CLUSTERS STATUS AGE
prod Production DR Zone DRBD primary,dr Available 2m
Example 2 — Protecting a single VM
Protect a VM named web-server in namespace default that currently runs on cluster1:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: protect-web-server
namespace: dr-prod
spec:
sourceCluster: cluster1
virtualMachine:
name: web-server
namespace: default
replicationConfig:
replicationProtocol: C
rpo: 0
oc --context=<quorum-context> apply -f protect-web-server.yaml
oc --context=<quorum-context> -n dr-prod get pr protect-web-server -w
Expected output (once Protected):
NAME VM NAMESPACE PHASE SYNC AGE
protect-web-server web-server default Protected 100 8m
Example 3 — Creating a ProtectionGroup for coordinated failover
Group two VMs that must fail over together:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
name: web-tier
namespace: dr-prod
spec:
sourceCluster: cluster1
desiredState: running
virtualMachines:
- name: web-server
namespace: default
bootOrder: 1
- name: web-cache
namespace: default
bootOrder: 2
oc --context=<quorum-context> apply -f web-tier-pg.yaml
oc --context=<quorum-context> -n dr-prod get pg web-tier
Expected output (once all VMs are protected):
NAME VMs STATE ACTIVE AGE
web-tier web-server,web-cache Protected cluster1 5m
Example 4 — Running a test failover
Validate DR readiness non-disruptively. The controller creates an isolated test namespace automatically — you do not specify it.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
name: web-tier-test
namespace: dr-prod
spec:
protectionGroupRef:
name: web-tier
cleanupPolicy: Automatic
retentionTime: 2h
verification:
reachability:
- type: tcp
port: 80
vmName: web-server
timeout: 30s
oc --context=<quorum-context> apply -f web-tier-test.yaml
oc --context=<quorum-context> -n dr-prod get tfo web-tier-test -w
Expected output (on success):
NAME PROTECTION GROUP PHASE PASSED AGE
web-tier-test web-tier Succeeded true 12m
Check which namespace the test VMs ran in (controller-generated, never user-specified):
oc --context=<quorum-context> -n dr-prod get tfo web-tier-test \
-o jsonpath='{.status.testNamespace}'
With cleanupPolicy: Automatic, test resources are cleaned up retentionTime after completionTime without any manual action. To trigger immediate cleanup, delete the CR:
oc --context=<quorum-context> -n dr-prod delete tfo web-tier-test
Example 5 — Checking replication health
# View the auto-created ReplicationGroupStatus for the web-tier group
oc --context=<quorum-context> -n dr-prod get rgs web-tier-replication -o yaml
Key status fields to check:
status:
rpoStatus: Met
aggregated:
connectionState: Connected
outOfSyncBytes: 0
syncedVolumeCount: 2
totalVolumeCount: 2
lastFullySyncedAt: "2026-07-10T14:23:01Z"
Example 6 — Executing a planned failover
Fail the web-tier group over to the DR cluster (cluster2):
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
name: web-tier-failover
namespace: dr-prod
spec:
protectionGroupRef:
name: web-tier
targetCluster: cluster2
failoverType: planned
oc --context=<quorum-context> apply -f web-tier-failover.yaml
oc --context=<quorum-context> -n dr-prod get fr web-tier-failover -w
Expected output (on completion):
NAME PROTECTION GROUP TARGET STATE PHASE AGE
web-tier-failover web-tier cluster2 Completed Completed 4m
Per-VM status during failover:
oc --context=<quorum-context> -n dr-prod get fr web-tier-failover \
-o jsonpath='{.status.vmStatuses}'
Use the following format for each issue: Symptom, Likely cause, Fix.
ProtectionZone stuck in Provisioning
Symptom: oc get pz shows STATUS: Provisioning for more than 5 minutes.
Likely cause: The protection-zone-controller cannot reach one or both workload clusters, the kubeconfig Secrets are missing or malformed, or the workload cluster is not running trilio-site-recovery-system namespace.
Fix:
- Check zone conditions for the specific failure:
oc --context=<quorum-context> -n dr-prod get pz prod -o yaml | grep -A 10 conditions - Check protection-zone-controller logs:
oc --context=<quorum-context> -n trilio-site-recovery-system \ logs deploy/protection-zone-controller -c manager -f - Verify the kubeconfig Secrets exist in the zone namespace and contain a valid
kubeconfigkey:oc --context=<quorum-context> -n dr-prod get secret cluster1-kubeconfig cluster2-kubeconfig - Verify TCP 6443 is open from the quorum cluster to both workload API servers.
ProtectionRequest stuck in Syncing
Symptom: oc get pr shows PHASE: Syncing with a SYNC percentage that is not advancing.
Likely cause: DRBD replication traffic is blocked between worker nodes (ports 7000–8000), the round-trip latency exceeds 50ms, or a DRBD node agent is unhealthy on one side.
Fix:
- Check the
SYNCpercentage for progress:oc --context=<quorum-context> -n dr-prod get pr <name> -o yaml | grep -A 5 syncProgress - Check the DRBDResource status on each workload cluster:
oc --context=<primary-context> -n <vm-namespace> get dres -o wide - Confirm ports 7000–8000 are open between worker nodes. A firewall stopping at 7999 silently breaks replication for one volume.
- Check node agent logs on the affected node:
# Find the agent pod on the relevant node NODE=<node-name> POD=$(oc --context=<primary-context> -n trilio-site-recovery-system \ get pods --field-selector spec.nodeName=${NODE} \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') oc --context=<primary-context> -n trilio-site-recovery-system logs ${POD} -c agent
ProtectionRequest in Failed phase with reason RoleConflict
Symptom: oc get pr shows PHASE: Failed; the Ready condition has reason: RoleConflict.
Likely cause: Both sides of the DRBD resource are trying to be Primary simultaneously, typically after an unclean failover or a split-brain scenario.
Fix:
- Inspect the ProtectionRequest conditions:
oc --context=<quorum-context> -n dr-prod get pr <name> -o yaml | grep -A 10 conditions - Check the DRBDResource role on both clusters:
oc --context=<primary-context> -n <vm-namespace> get dres <resource-name> oc --context=<dr-context> -n <vm-namespace> get dres <resource-name> - Refer to
docs/runbooks/for the split-brain recovery procedure. Do not manually patch finalizers.
DRBD node agent pod in Init:Error or Init:CrashLoopBackOff
Symptom: Node agent DaemonSet pods on a workload cluster show Init:Error or Init:CrashLoopBackOff.
Likely cause: The DRBD kernel module init container cannot find a precompiled kmod matching the node's kernel. This happens if the node is running a kernel newer than any mirrored DRBD build, or if the node OS does not match the distributionName configured in the ProtectionZone.
Fix:
- Check init container logs:
POD=$(oc --context=<primary-context> -n trilio-site-recovery-system \ get pods -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') oc --context=<primary-context> -n trilio-site-recovery-system \ logs ${POD} -c drbd-module-loader - Verify the node's kernel version:
oc --context=<primary-context> get node <node-name> \ -o jsonpath='{.status.nodeInfo.kernelVersion}' - Confirm
spec.clusters[].distributionNamematches the actual node OS (RHEL-CoreOS-9forel9kernels). Patch the ProtectionZone if it is wrong. - Verify nodes can pull from
quay.io/triliovault(the module loader image pull policy isAlways). - RHEL 10-based nodes (
el10kernels) are not validated in Tech Preview.
Webhook call rejected — ReplicationProtocolA is not supported yet
Symptom: Applying a ProtectionRequest returns an admission error: ReplicationProtocolA is not supported yet.
Likely cause: The ProtectionRequest specifies replicationProtocol: A. Protocol A is not supported in Tech Preview.
Fix: Set replicationProtocol: C (or omit the field — the default is C). Protocol C synchronous replication with rpo: 0 is the only supported mode.
Webhook call rejected — Linstor backend is not supported yet; use DRBD
Symptom: Applying a ProtectionZone returns an admission error about Linstor.
Likely cause: spec.storageBackendMode is set to Linstor.
Fix: Set spec.storageBackendMode: DRBD. Linstor is not supported in Tech Preview.
oc get drbdresource on the quorum cluster returns server doesn't have a resource type
Symptom: Running oc --context=<quorum-context> get dres fails with a resource-type error.
Likely cause: DRBDResource CRs are registered and live only on workload clusters, not the quorum cluster.
Fix: Run the command against a workload cluster context:
oc --context=<primary-context> get dres -A
oc adm drain does not complete on a node hosting a protected VM
Symptom: oc adm drain <node> hangs or reports that it cannot evict pods.
Likely cause: The node hosts a running protected VM. Live migration is denied by the workload admission webhook, and the anchor pod is node-pinned and unowned so drain refuses to evict it without --force.
Fix: Protected VMs cannot be migrated or relocated for the lifetime of the protection. Options:
- Schedule the maintenance to drain around the protected VM (stop other workloads first).
- Unprotect the VM, drain the node, then re-protect (which costs a full initial resync).
- For in-place maintenance (reboot, kernel update), drain is not necessary — the DRBD module reloads on boot and the agent pod restarts automatically.
Collecting a diagnostic bundle
For any issue you cannot resolve with the above steps, collect a must-gather bundle and attach it to your support case. Run this against the quorum cluster — the tool collects logs, CRD state, and configuration from all clusters in the DR estate:
oc --context=<quorum-context> adm must-gather \
--image=quay.io/triliovault/site-recovery-must-gather:<version>
The image's entrypoint (/usr/bin/gather) runs automatically. No -- <command> override is needed. The resulting archive contains a SUMMARY.md (read this first) and a manifest.yaml listing what was collected. Secret data, kubeconfigs, and DRBD shared secrets are never written to the bundle.