Trilio Site Recovery for Kubernetes
Guide

Installation

Installing CRDs and deploying the site recovery operator


Overview

This page walks you through installing Trilio Site Recovery (TSR) on OpenShift. TSR is a Kubernetes operator that automates disaster recovery for KubeVirt virtual machines using DRBD synchronous block replication (Protocol C, RPO=0) across three clusters: a quorum cluster that hosts the management plane, a primary workload cluster, and a DR workload cluster. Installation has two complementary parts: first you install the operator bundle on the quorum cluster via OLM, then you create a ProtectionZone custom resource that causes the operator to automatically deploy the workload control planes on both workload clusters. You never run helm install directly — the Helm charts are embedded in the operator binary and are applied by the ProtectionZone controller once the zone reaches Available.


Prerequisites

Before you begin, ensure the following are in place across all three clusters.

Clusters

  • Three OpenShift clusters at version 4.17 or later: one quorum (management plane), one primary, and one DR cluster. OpenShift 4.17 is the supported floor for all three roles.
  • KubeVirt or OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary and DR clusters.

Storage

  • DRBD kernel module ≥ 9.x loaded on every worker node of the primary and DR clusters.
  • An LVM thin pool configured on each worker node of the primary and DR clusters, backed by a dedicated block device, to serve as DRBD backing storage.

Tooling

  • oc CLI (required; kubectl-only operation is not a supported mode for OpenShift paths including oc adm must-gather and Routes)
  • Kubeconfig files for all three clusters

Network

  • TCP ports 7000–8000 inclusive (DRBD replication) open in both directions between every worker node on the primary and DR clusters. One port is consumed per replicated VM volume — a rule that stops at 7999 will leave one volume unable to replicate.
  • TCP port 6443 (Kubernetes API) open from the quorum cluster to the primary and DR API servers.
  • For synchronous replication (Protocol C, RPO=0): round-trip latency below 50 ms between primary and DR clusters.

Certificates

  • Webhook serving certificates are not managed for you. You must create a Secret named webhook-server-cert in each release namespace on the quorum cluster before the control planes start. There is no cert-manager integration.

OLM

  • Operator Lifecycle Manager (OLM) running on the quorum cluster (standard on OpenShift).

Installation

Installation proceeds in two stages:

  1. Install the operator bundle on the quorum cluster via OLM — this deploys the protection-zone-controller, the TSR web console (site-recovery-backend and site-recovery-frontend), and registers the ProtectionZone CRD.
  2. Create a ProtectionZone CR — this triggers the operator to automatically deploy the quorum control plane and the workload control planes (including the DRBD node-agent DaemonSet) on both workload clusters.

Stage 1 — Install the operator on the quorum cluster

All commands in this stage target the quorum cluster.

Step 1: Create the operator namespace

oc --context=<quorum-context> create namespace trilio-site-recovery-system

Step 2: Create the CatalogSource

This points OLM at the Trilio Site Recovery operator catalog.

# catalogsource.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: trilio-site-recovery-catalog
  namespace: trilio-site-recovery-system
spec:
  sourceType: grpc
  image: quay.io/triliovault/site-recovery-catalog:latest
  displayName: Trilio Site Recovery
  publisher: Trilio Data, Inc.
oc --context=<quorum-context> apply -f catalogsource.yaml

Wait for the catalog pod to reach Running:

oc --context=<quorum-context> -n trilio-site-recovery-system \
  get pods -l olm.catalogSource=trilio-site-recovery-catalog

Step 3: Create the OperatorGroup

# operatorgroup.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trilio-site-recovery
  namespace: trilio-site-recovery-system
spec:
  targetNamespaces:
    - trilio-site-recovery-system
oc --context=<quorum-context> apply -f operatorgroup.yaml

Step 4: Create the Subscription

# subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: tech-preview
  name: site-recovery
  source: trilio-site-recovery-catalog
  sourceNamespace: trilio-site-recovery-system
oc --context=<quorum-context> apply -f subscription.yaml

Step 5: Verify operator installation

Wait for the CSV to reach Succeeded and the protection-zone-controller pod to be Running:

# Check CSV status
oc --context=<quorum-context> -n trilio-site-recovery-system \
  get csv -l operators.coreos.com/site-recovery.trilio-site-recovery-system

# Check operator pods — expect protection-zone-controller, site-recovery-backend,
# and site-recovery-frontend all Running
oc --context=<quorum-context> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/part-of=site-recovery

Expected output shows three pods running:

NAME                                           READY   STATUS    RESTARTS   AGE
protection-zone-controller-<hash>              1/1     Running   0          2m
site-recovery-backend-<hash>                   1/1     Running   0          2m
site-recovery-frontend-<hash>                  2/2     Running   0          2m

The site-recovery-frontend pod runs two containers: the Next.js frontend and the OpenShift oauth-proxy sidecar. On OpenShift, the ProtectionZone controller will create a Route and console link for browser access once a zone reaches Available.


Stage 2 — Create the ProtectionZone

The ProtectionZone CR wires together the quorum and workload clusters. Creating it causes the operator to:

  • Deploy the quorum control plane (site-recovery-quorum-control-plane) in the zone namespace on the quorum cluster.
  • Deploy the workload control plane (site-recovery-workload-control-plane) including the DRBD node-agent DaemonSet in trilio-site-recovery-system on each workload cluster.
  • Create the default DRBDReplicationPolicy named <zoneName>-default in the zone namespace.

All zone CRs live in a namespace with the prefix dr-<name>. All workload-side components live in trilio-site-recovery-system.

Step 6: Create the zone namespace on the quorum cluster

oc --context=<quorum-context> create namespace dr-<zone-name>

Replace <zone-name> with a short, lowercase identifier for your DR deployment (for example, prod).

Step 7: Create webhook serving certificates

The admission webhooks require a TLS certificate. This chart does not manage certificate provisioning; you must create the Secret before the zone is created.

Create the certificate secret in the zone namespace on the quorum cluster:

# Generate a self-signed cert for the webhook service SAN
# The SAN must cover the webhook service in the zone namespace.
# Replace <zone-name> with your zone name.

oc --context=<quorum-context> -n dr-<zone-name> create secret tls webhook-server-cert \
  --cert=tls.crt \
  --key=tls.key

Also create the webhook certificate secret in trilio-site-recovery-system on each workload cluster:

oc --context=<primary-context> -n trilio-site-recovery-system create secret tls webhook \
  --cert=tls.crt \
  --key=tls.key

oc --context=<dr-context> -n trilio-site-recovery-system create secret tls webhook \
  --cert=tls.crt \
  --key=tls.key

Note: the quorum chart's default webhook.certSecretName is webhook-server-cert; the workload chart's default is webhook.

Step 8: Create kubeconfig Secrets for the workload clusters

The ProtectionZone controller uses these to reach the primary and DR clusters. Create them in the zone namespace on the quorum cluster.

# Secret for the primary cluster (cluster1)
oc --context=<quorum-context> -n dr-<zone-name> create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=<path-to-primary-kubeconfig>

# Secret for the DR cluster (cluster2)
oc --context=<quorum-context> -n dr-<zone-name> create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=<path-to-dr-kubeconfig>

Step 9: Create the ProtectionZone CR

# protectionzone.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  description: "Primary-to-DR DRBD protection zone for production VMs."
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
oc --context=<quorum-context> apply -f protectionzone.yaml

Step 10: Monitor the ProtectionZone until it reaches Available

oc --context=<quorum-context> -n dr-prod get pz prod -w

Expected progression:

NAME   DISPLAY NAME          BACKEND   CLUSTERS        STATUS         AGE
prod   Production DR Zone    DRBD      primary,dr      Provisioning   10s
prod   Production DR Zone    DRBD      primary,dr      Available      90s

Once Available, the operator has:

  • Deployed the quorum control plane manager in dr-prod on the quorum cluster.
  • Deployed the workload control plane and DRBD node-agent DaemonSet in trilio-site-recovery-system on both workload clusters.
  • Created the default replication policy prod-default in dr-prod.
  • Created an OpenShift Route and console link for browser access to the TSR web console.

Step 11: Verify workload cluster components

Verify the workload control planes are running on both clusters:

# Primary cluster — workload control plane manager
oc --context=<primary-context> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=workload-control-plane

# Primary cluster — DRBD node agents (one per worker node)
oc --context=<primary-context> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=drbd-node-agent

# DR cluster — same checks
oc --context=<dr-context> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=workload-control-plane

oc --context=<dr-context> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=drbd-node-agent

Verify the quorum control plane manager on the quorum cluster:

oc --context=<quorum-context> -n dr-prod \
  get pods -l app.kubernetes.io/component=quorum-control-plane

The quorum manager pod runs a single container named manager and shows 1/1 Running.

Step 12: Verify CRD registration

The quorum cluster registers 8 CRDs; each workload cluster registers 1:

# On the quorum cluster — should list 8 site-recovery CRDs
oc --context=<quorum-context> get crd | grep siterecovery.trilio.io

# On a workload cluster — should list only drbdresources
oc --context=<primary-context> get crd | grep siterecovery.trilio.io

Note: drbdresources.siterecovery.trilio.io is registered only on workload clusters. Running oc get drbdresource against the quorum cluster returns server doesn't have a resource type.


Configuration

You do not modify Helm chart values directly — the ProtectionZone controller manages chart deployment. However, the following configuration surfaces are available to you.

ProtectionZone spec fields

These are the fields you author in the ProtectionZone CR. All other configuration is managed by the operator.

FieldTypeRequiredDefaultNotes
spec.displayNamestringYes—Human-friendly name shown in the UI. 5–253 characters.
spec.descriptionstringNo—Free-form text. Max 1024 characters.
spec.storageBackendModestringYes—Must be DRBD. The value Linstor is rejected by the admission webhook.
spec.clusters[].namestringYes—Operator-facing cluster identifier within the zone. Max 63 characters.
spec.clusters[].clusterIDstringYes—Deployment slot: cluster1 or cluster2. Must be unique across the two entries.
spec.clusters[].distributionNamestringNoRHEL-CoreOS-9OS family: RHEL-CoreOS-9 or RHEL-CoreOS-10. Selects the DRBD kernel-module loader image. RHEL-CoreOS-10 is not validated in Tech Preview.
spec.clusters[].kubeconfigSecretRef.namestringYes—Name of the Secret in the zone namespace holding the cluster kubeconfig.
spec.clusters[].kubeconfigSecretRef.keystringYes—Key within the Secret (typically kubeconfig).
spec.logging.levelstringConditional—Runtime log level for all zone components: error, info, debug, or trace. Required when spec.logging is set. Takes effect within ~15 seconds without pod restarts.
spec.logging.controlPlaneLevelstringNo—Overrides level for control-plane managers only.
spec.logging.nodeAgentLevelstringNo—Overrides level for DRBD node agents only.

DRBDReplicationPolicy

You do not create or modify this resource. The ProtectionZone controller automatically creates one default policy named <zoneName>-default with isDefault: true in the zone namespace when the zone reaches Available. Only this default policy is ever consulted by the protection controllers. Protocol C synchronous replication (RPO=0) is the only supported mode; the admission webhook rejects Protocol A.

Webhook serving certificates

Certificate provisioning is not managed by the charts. You must create TLS secrets out of band (see Step 7 in Installation):

Secret nameNamespaceClusterChart default webhook.certSecretName
webhook-server-certdr-<zone-name>Quorumwebhook-server-cert
webhooktrilio-site-recovery-systemPrimarywebhook
webhooktrilio-site-recovery-systemDRwebhook

Runtime log levels

Set spec.logging on the ProtectionZone CR to change log levels without restarting pods. The operator propagates the change to a site-recovery-logging ConfigMap in each component namespace, which is polled every ~15 seconds.

# Raise all zone components to debug
oc --context=<quorum-context> -n dr-prod patch pz prod --type=merge -p '
{"spec":{"logging":{"level":"debug"}}}'

# Set control-plane managers to debug, node agents to info
oc --context=<quorum-context> -n dr-prod patch pz prod --type=merge -p '
{"spec":{"logging":{"level":"info","controlPlaneLevel":"debug","nodeAgentLevel":"info"}}}'

Accepted levels: error, info, debug, trace.

Node agent resource tuning

The DRBD node-agent DaemonSet is deployed by the ProtectionZone controller with these default resource limits (from helm/site-recovery-workload-control-plane/values.yaml):

ResourceRequestLimit
CPU10m500m
Memory64Mi256Mi

These are the tested values. If your nodes require more, file a support request — direct chart value modification is not a customer procedure.

Zone configuration deadlines

The workload chart deploys a site-recovery-zone-config ConfigMap in trilio-site-recovery-system on each workload cluster. The two keys relevant to DR sizing are:

KeyDefaultEffect
drbdMetadataReadyGracePeriod2mHow long a metadata PVC may report not-ready before ConditionPlacementReady is held False.
drbdMetadataProvisioningFailureDeadline10mHow long ConditionPlacementReady may stay continuously False before the placement reconciler marks the copy Failed.

These values survive pod restarts but are overwritten on the next operator reconcile. Do not hand-edit this ConfigMap between operator upgrades.


Usage

Once the ProtectionZone reaches Available, you can protect VMs and manage DR operations entirely through oc and the TSR web console.

Checking overall zone health

# Summary view of the zone
oc --context=<quorum-context> -n dr-prod get pz prod

# Detailed status including per-cluster reachability
oc --context=<quorum-context> -n dr-prod get pz prod -o yaml

Protecting a virtual machine

To protect a VM, create a ProtectionRequest in the zone namespace on the quorum cluster. The VM must be running when you create the request.

oc --context=<quorum-context> -n dr-prod apply -f protect-my-vm.yaml

Watch protection progress:

oc --context=<quorum-context> -n dr-prod get pr protect-my-vm -w

The phase progression is: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

The Sync column shows block-level resync progress. Initial resync of a large disk can take time; protection is not complete until the phase reaches Protected.

Grouping VMs for coordinated failover

A ProtectionGroup lets you fail over a set of VMs as a single atomic unit:

oc --context=<quorum-context> -n dr-prod get pg
oc --context=<quorum-context> -n dr-prod get pg <name> -o yaml

Monitoring replication health

The replication monitor auto-creates one ReplicationGroupStatus per ProtectionGroup named <protection-group-name>-replication:

oc --context=<quorum-context> -n dr-prod get rgs
oc --context=<quorum-context> -n dr-prod get rgs <pg-name>-replication -o yaml

Filter RPO audit events by protection group:

oc --context=<quorum-context> -n dr-prod get rpoe \
  -l siterecovery.trilio.io/protection-group=<pg-name>

Checking DRBD replication state on workload clusters

DRBDResource CRs live on the workload clusters in the VM's namespace — not on the quorum cluster:

# List all protected VM replication state on the primary cluster
oc --context=<primary-context> get dres -A

# Detail for a specific VM's resource
oc --context=<primary-context> -n <vm-namespace> get dres <resource-name> -o yaml

Identify protected VMs by label (workload clusters only):

oc --context=<primary-context> get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

TSR web console (OpenShift)

On OpenShift, the ProtectionZone controller creates a Route and an OpenShift console link once the zone reaches Available. Access the console through the OpenShift web console's application launcher or by retrieving the Route URL:

oc --context=<quorum-context> -n trilio-site-recovery-system get route

Authentication uses the OpenShift oauth-proxy. The console is not accessible on upstream Kubernetes.

RBAC — assigning personas

Three ClusterRoles are installed by the OLM bundle:

ClusterRoleBinding typeCapabilities
trilio-site-recovery-adminClusterRoleBindingFull control of all TSR resources and kubeconfig Secrets. Effectively cluster-admin. Consider using a namespace-scoped RoleBinding instead.
trilio-site-recovery-viewerClusterRoleBindingRead-only across all zones. No Secret access.
trilio-site-recovery-operatorRoleBinding (per zone namespace)Full control of TestFailover; read-only on everything else. Cannot trigger production failover.
# Grant a user admin access (cluster-wide — see security note in the ClusterRole)
oc --context=<quorum-context> create clusterrolebinding tsr-admin-<user> \
  --clusterrole=trilio-site-recovery-admin \
  --user=<username>

# Grant a user operator access to one zone
oc --context=<quorum-context> -n dr-prod create rolebinding tsr-operator-<user> \
  --clusterrole=trilio-site-recovery-operator \
  --user=<username>

Examples

Example 1 — Minimal ProtectionZone CR

A complete, minimal zone definition pairing two RHCOS 9 clusters:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Apply and verify:

oc --context=<quorum-context> apply -f protectionzone.yaml
oc --context=<quorum-context> -n dr-prod get pz prod -w

Expected output (once stable):

NAME   DISPLAY NAME          BACKEND   CLUSTERS     STATUS      AGE
prod   Production DR Zone    DRBD      primary,dr   Available   2m

Example 2 — Protecting a single VM

Protect a VM named web-server in namespace default that currently runs on cluster1:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-server
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-server
    namespace: default
  replicationConfig:
    replicationProtocol: C
    rpo: 0
oc --context=<quorum-context> apply -f protect-web-server.yaml
oc --context=<quorum-context> -n dr-prod get pr protect-web-server -w

Expected output (once Protected):

NAME                  VM           NAMESPACE   PHASE       SYNC   AGE
protect-web-server    web-server   default     Protected   100    8m

Example 3 — Creating a ProtectionGroup for coordinated failover

Group two VMs that must fail over together:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: web-tier
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  desiredState: running
  virtualMachines:
    - name: web-server
      namespace: default
      bootOrder: 1
    - name: web-cache
      namespace: default
      bootOrder: 2
oc --context=<quorum-context> apply -f web-tier-pg.yaml
oc --context=<quorum-context> -n dr-prod get pg web-tier

Expected output (once all VMs are protected):

NAME       VMs                   STATE      ACTIVE     AGE
web-tier   web-server,web-cache  Protected  cluster1   5m

Example 4 — Running a test failover

Validate DR readiness non-disruptively. The controller creates an isolated test namespace automatically — you do not specify it.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: web-tier-test
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: web-tier
  cleanupPolicy: Automatic
  retentionTime: 2h
  verification:
    reachability:
      - type: tcp
        port: 80
        vmName: web-server
        timeout: 30s
oc --context=<quorum-context> apply -f web-tier-test.yaml
oc --context=<quorum-context> -n dr-prod get tfo web-tier-test -w

Expected output (on success):

NAME             PROTECTION GROUP   PHASE       PASSED   AGE
web-tier-test    web-tier           Succeeded   true     12m

Check which namespace the test VMs ran in (controller-generated, never user-specified):

oc --context=<quorum-context> -n dr-prod get tfo web-tier-test \
  -o jsonpath='{.status.testNamespace}'

With cleanupPolicy: Automatic, test resources are cleaned up retentionTime after completionTime without any manual action. To trigger immediate cleanup, delete the CR:

oc --context=<quorum-context> -n dr-prod delete tfo web-tier-test

Example 5 — Checking replication health

# View the auto-created ReplicationGroupStatus for the web-tier group
oc --context=<quorum-context> -n dr-prod get rgs web-tier-replication -o yaml

Key status fields to check:

status:
  rpoStatus: Met
  aggregated:
    connectionState: Connected
    outOfSyncBytes: 0
    syncedVolumeCount: 2
    totalVolumeCount: 2
  lastFullySyncedAt: "2026-07-10T14:23:01Z"

Example 6 — Executing a planned failover

Fail the web-tier group over to the DR cluster (cluster2):

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: web-tier-failover
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: web-tier
  targetCluster: cluster2
  failoverType: planned
oc --context=<quorum-context> apply -f web-tier-failover.yaml
oc --context=<quorum-context> -n dr-prod get fr web-tier-failover -w

Expected output (on completion):

NAME                PROTECTION GROUP   TARGET     STATE       PHASE       AGE
web-tier-failover   web-tier           cluster2   Completed   Completed   4m

Per-VM status during failover:

oc --context=<quorum-context> -n dr-prod get fr web-tier-failover \
  -o jsonpath='{.status.vmStatuses}'

Troubleshooting

Use the following format for each issue: Symptom, Likely cause, Fix.


ProtectionZone stuck in Provisioning

Symptom: oc get pz shows STATUS: Provisioning for more than 5 minutes.

Likely cause: The protection-zone-controller cannot reach one or both workload clusters, the kubeconfig Secrets are missing or malformed, or the workload cluster is not running trilio-site-recovery-system namespace.

Fix:

  1. Check zone conditions for the specific failure:
    oc --context=<quorum-context> -n dr-prod get pz prod -o yaml | grep -A 10 conditions
    
  2. Check protection-zone-controller logs:
    oc --context=<quorum-context> -n trilio-site-recovery-system \
      logs deploy/protection-zone-controller -c manager -f
    
  3. Verify the kubeconfig Secrets exist in the zone namespace and contain a valid kubeconfig key:
    oc --context=<quorum-context> -n dr-prod get secret cluster1-kubeconfig cluster2-kubeconfig
    
  4. Verify TCP 6443 is open from the quorum cluster to both workload API servers.

ProtectionRequest stuck in Syncing

Symptom: oc get pr shows PHASE: Syncing with a SYNC percentage that is not advancing.

Likely cause: DRBD replication traffic is blocked between worker nodes (ports 7000–8000), the round-trip latency exceeds 50ms, or a DRBD node agent is unhealthy on one side.

Fix:

  1. Check the SYNC percentage for progress:
    oc --context=<quorum-context> -n dr-prod get pr <name> -o yaml | grep -A 5 syncProgress
    
  2. Check the DRBDResource status on each workload cluster:
    oc --context=<primary-context> -n <vm-namespace> get dres -o wide
    
  3. Confirm ports 7000–8000 are open between worker nodes. A firewall stopping at 7999 silently breaks replication for one volume.
  4. Check node agent logs on the affected node:
    # Find the agent pod on the relevant node
    NODE=<node-name>
    POD=$(oc --context=<primary-context> -n trilio-site-recovery-system \
      get pods --field-selector spec.nodeName=${NODE} \
      -l app.kubernetes.io/component=drbd-node-agent \
      -o jsonpath='{.items[0].metadata.name}')
    oc --context=<primary-context> -n trilio-site-recovery-system logs ${POD} -c agent
    

ProtectionRequest in Failed phase with reason RoleConflict

Symptom: oc get pr shows PHASE: Failed; the Ready condition has reason: RoleConflict.

Likely cause: Both sides of the DRBD resource are trying to be Primary simultaneously, typically after an unclean failover or a split-brain scenario.

Fix:

  1. Inspect the ProtectionRequest conditions:
    oc --context=<quorum-context> -n dr-prod get pr <name> -o yaml | grep -A 10 conditions
    
  2. Check the DRBDResource role on both clusters:
    oc --context=<primary-context> -n <vm-namespace> get dres <resource-name>
    oc --context=<dr-context> -n <vm-namespace> get dres <resource-name>
    
  3. Refer to docs/runbooks/ for the split-brain recovery procedure. Do not manually patch finalizers.

DRBD node agent pod in Init:Error or Init:CrashLoopBackOff

Symptom: Node agent DaemonSet pods on a workload cluster show Init:Error or Init:CrashLoopBackOff.

Likely cause: The DRBD kernel module init container cannot find a precompiled kmod matching the node's kernel. This happens if the node is running a kernel newer than any mirrored DRBD build, or if the node OS does not match the distributionName configured in the ProtectionZone.

Fix:

  1. Check init container logs:
    POD=$(oc --context=<primary-context> -n trilio-site-recovery-system \
      get pods -l app.kubernetes.io/component=drbd-node-agent \
      -o jsonpath='{.items[0].metadata.name}')
    oc --context=<primary-context> -n trilio-site-recovery-system \
      logs ${POD} -c drbd-module-loader
    
  2. Verify the node's kernel version:
    oc --context=<primary-context> get node <node-name> \
      -o jsonpath='{.status.nodeInfo.kernelVersion}'
    
  3. Confirm spec.clusters[].distributionName matches the actual node OS (RHEL-CoreOS-9 for el9 kernels). Patch the ProtectionZone if it is wrong.
  4. Verify nodes can pull from quay.io/triliovault (the module loader image pull policy is Always).
  5. RHEL 10-based nodes (el10 kernels) are not validated in Tech Preview.

Webhook call rejected — ReplicationProtocolA is not supported yet

Symptom: Applying a ProtectionRequest returns an admission error: ReplicationProtocolA is not supported yet.

Likely cause: The ProtectionRequest specifies replicationProtocol: A. Protocol A is not supported in Tech Preview.

Fix: Set replicationProtocol: C (or omit the field — the default is C). Protocol C synchronous replication with rpo: 0 is the only supported mode.


Webhook call rejected — Linstor backend is not supported yet; use DRBD

Symptom: Applying a ProtectionZone returns an admission error about Linstor.

Likely cause: spec.storageBackendMode is set to Linstor.

Fix: Set spec.storageBackendMode: DRBD. Linstor is not supported in Tech Preview.


oc get drbdresource on the quorum cluster returns server doesn't have a resource type

Symptom: Running oc --context=<quorum-context> get dres fails with a resource-type error.

Likely cause: DRBDResource CRs are registered and live only on workload clusters, not the quorum cluster.

Fix: Run the command against a workload cluster context:

oc --context=<primary-context> get dres -A

oc adm drain does not complete on a node hosting a protected VM

Symptom: oc adm drain <node> hangs or reports that it cannot evict pods.

Likely cause: The node hosts a running protected VM. Live migration is denied by the workload admission webhook, and the anchor pod is node-pinned and unowned so drain refuses to evict it without --force.

Fix: Protected VMs cannot be migrated or relocated for the lifetime of the protection. Options:

  • Schedule the maintenance to drain around the protected VM (stop other workloads first).
  • Unprotect the VM, drain the node, then re-protect (which costs a full initial resync).
  • For in-place maintenance (reboot, kernel update), drain is not necessary — the DRBD module reloads on boot and the agent pod restarts automatically.

Collecting a diagnostic bundle

For any issue you cannot resolve with the above steps, collect a must-gather bundle and attach it to your support case. Run this against the quorum cluster — the tool collects logs, CRD state, and configuration from all clusters in the DR estate:

oc --context=<quorum-context> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<version>

The image's entrypoint (/usr/bin/gather) runs automatically. No -- <command> override is needed. The resulting archive contains a SUMMARY.md (read this first) and a manifest.yaml listing what was collected. Secret data, kubeconfigs, and DRBD shared secrets are never written to the bundle.