Prerequisites

This page describes every requirement that must be satisfied before you install Trilio Site Recovery (TSR). TSR is a Kubernetes operator for OpenShift that provides zero data loss (RPO=0) disaster recovery for KubeVirt virtual machines by replicating VM-attached storage block-by-block between a primary cluster and a DR cluster using DRBD synchronous replication (Protocol C). Because TSR operates at the storage layer and requires real-time coordination across three clusters, the platform, network, and storage prerequisites are strict — failing to meet any one of them will either prevent installation or cause silent replication failures at the worst possible moment. Read this page carefully before provisioning any infrastructure.

Prerequisites Cluster topology Trilio Site Recovery requires three OpenShift clusters: Role Purpose Quorum cluster (Site C) Hosts the management plane: the OLM-installed operator, the TSR web console, and the quorum control plane. Runs no application VMs. Primary cluster (cluster1) Runs the active VM workloads and the workload control plane. DR cluster (cluster2) Hosts standby VMs and a mirror workload control plane. Receives replicated data from the primary. All three clusters must be reachable from one another at the API and storage layers (see Network requirements below).

OpenShift version Two version ranges apply, and you should satisfy both: OLM-installable range — read from the com.redhat.openshift.versions annotation in deploy/olm-catalog/site-recovery/metadata/annotations.yaml, currently v4.14-v4.19. The operator bundle can be installed on clusters in this range. Validated range — OpenShift 4.17–4.20 (Kubernetes 1.30–1.33), per docs/tech-preview-supported-configurations.md. This is the range Tech Preview testing covers. OpenShift 4.14–4.16 sits within the supported floor but is not covered by Tech Preview testing. Versions beyond 4.20 are expected to work but have not been validated. Use 4.17–4.20 on all three clusters unless you have a specific reason not to.

Note: Upstream Kubernetes workload clusters (non-OpenShift) are not a supported configuration. The TSR web console is served behind the OpenShift oauth-proxy, and the Route and console link are created by the ProtectionZone controller. Neither an ingress path nor an authentication path exists on upstream Kubernetes.

Virtualization OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary cluster and the DR cluster. The quorum cluster does not run VMs and does not require OpenShift Virtualization.

Storage DRBD kernel module DRBD kernel module ≥ 9.x loaded on every worker node of the primary cluster and the DR cluster. The drbd-node-agent DaemonSet manages block replication at the kernel level. If the module is absent or at an unsupported version, the DaemonSet pods will enter a crash loop and no replication can be established. LVM thin pool A dedicated LVM thin pool backed by a dedicated block device must be configured on every worker node of the primary cluster and the DR cluster before you deploy the workload control plane. DRBD uses this thin pool as its backing storage for replicated VM volumes. The thin pool must be large enough to accommodate all VM disk images you intend to protect, plus snapshot overhead during test failovers.

CLI tools Tool Version Notes oc Matching your OpenShift version Required. oc is the only supported CLI for OpenShift-specific paths (oc adm must-gather, Routes). kubectl-only operation is not a supported mode. kubectl Compatible with OpenShift 4.17+ API server Used for CRD and custom resource operations alongside oc.

You do not need Helm. The Helm charts for the quorum control plane, the workload control planes and the DRBD node agents are embedded in the operator binary, and the ProtectionZone controller installs them for you once a ProtectionZone reaches Available. There is no chart repository to add and no helm install step in any supported customer procedure.

Kubeconfig files You need a valid kubeconfig file for each of the three clusters:

~/.kube/config-quorum    # Quorum cluster
~/.kube/config-cluster1  # Primary cluster
~/.kube/config-cluster2  # DR cluster

Have these ready before you create the ProtectionZone. You store the two workload kubeconfigs as Secrets in the dr-<name> namespace on the quorum cluster and reference them from the ProtectionZone's clusters[].kubeconfigSecretRef; the controller reads them from there.

Network requirements Between primary and DR worker nodes (DRBD replication traffic) TCP ports 7000–8000 must be open in both directions between every worker node on the primary cluster and every worker node on the DR cluster. DRBD replication uses one port per replicated volume from this range. Blocking even a subset of these ports will cause replication to stall and Protection Groups to enter a Degraded state. Note that the range is inclusive of port 8000 — a firewall rule that stops at 7999 will leave one volume in a large estate unable to replicate. Between the quorum cluster and workload cluster API servers TCP port 6443 (Kubernetes API) must be open from the quorum cluster to the API server of both the primary cluster and the DR cluster. The failover, protection, pg-sync, and replication-monitor controllers running on the quorum cluster communicate with workload clusters exclusively through their API servers. Latency requirements Average round-trip time between primary and DR worker nodes must be below 50 ms. Synchronous DRBD Protocol C is the only supported replication mode. RPO is always 0 and is not tunable. Asynchronous replication (Protocol A) is rejected by the admission webhook with ReplicationProtocolA is not supported yet, so there is no long-distance, higher-RPO configuration to trade off against. Because every write is acknowledged only after it has committed on both clusters, exceeding 50 ms RTT will cause write I/O to stall on the primary. This is a hard constraint on how far apart the two sites can be, not a tuning preference. Measure it before you deploy — see Example 5.

Permissions The accounts you use to install and operate Site Recovery need the following access levels: Action Required permission Install the OLM operator bundle on the quorum cluster cluster-admin on the quorum cluster Create the dr-<name> namespace and the kubeconfig Secrets within it Namespace admin or cluster-admin on the quorum cluster Apply CRD manifests (ProtectionZone, ProtectionRequest, FailoverRequest, TestFailover) At least edit on the target namespace; cluster-admin for CRD-level operations Run oc adm must-gather with tsr-gather cluster-admin on the quorum cluster The kubeconfig you supply for each workload cluster must itself carry cluster-admin on that cluster. The ProtectionZone controller uses it to deploy the workload control plane and the DRBD node agent DaemonSet, and the DaemonSet requires a privileged SCC. You do not install anything on the workload clusters by hand.

Optional components The following components are not required for a basic deployment but unlock specific capabilities: Component Purpose Containerized Data Importer (CDI) Required if your VMs use DataVolumes. The pg-sync-controller converts DataVolume references to PVC references when building standby VM specs on the DR cluster. Multus CNI Required if your VMs use secondary network interfaces that must be preserved across failover.

TSR installation has two complementary phases that both must complete:

1. OLM bundle installation (on the quorum cluster) — installs the protection-zone-controller, the TSR web console (site-recovery-backend and site-recovery-frontend), and the ProtectionZone CRD.
2. ProtectionZone CR creation (on the quorum cluster) — once the zone reaches Available, the ProtectionZone controller automatically deploys the quorum and workload control plane Helm charts (embedded in the operator binary) to all three clusters.

You never run helm install or helm upgrade directly. The Helm charts are embedded in the operator binary and deployed automatically.

Step 1 — Prepare kubeconfig Secrets

Create a namespace for your first protection zone on the quorum cluster. The namespace must be prefixed dr-:

oc --kubeconfig $KUBECONFIG_QUORUM create namespace dr-prod

Create a Secret containing the kubeconfig for the primary cluster (cluster1):

oc --kubeconfig $KUBECONFIG_QUORUM create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=$KUBECONFIG_CLUSTER1 \
  -n dr-prod

Create a Secret containing the kubeconfig for the DR cluster (cluster2):

oc --kubeconfig $KUBECONFIG_QUORUM create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=$KUBECONFIG_CLUSTER2 \
  -n dr-prod

Step 2 — Create the operator namespace

Create the namespace the OLM operator will install into:

oc --kubeconfig $KUBECONFIG_QUORUM create namespace trilio-site-recovery-system

Step 3 — Install the operator via OLM

Apply the CatalogSource, OperatorGroup, and Subscription to the quorum cluster.

CatalogSource:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: trilio-site-recovery-catalog
  namespace: trilio-site-recovery-system
spec:
  sourceType: grpc
  image: quay.io/triliovault/site-recovery-catalog:<tag>
  displayName: Trilio Site Recovery
  publisher: Trilio Data, Inc.

oc --kubeconfig $KUBECONFIG_QUORUM apply -f catalogsource.yaml

OperatorGroup:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trilio-site-recovery
  namespace: trilio-site-recovery-system
spec:
  targetNamespaces:
    - trilio-site-recovery-system

oc --kubeconfig $KUBECONFIG_QUORUM apply -f operatorgroup.yaml

Subscription:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: tech-preview
  name: site-recovery
  source: trilio-site-recovery-catalog
  sourceNamespace: trilio-site-recovery-system

oc --kubeconfig $KUBECONFIG_QUORUM apply -f subscription.yaml

Wait for the operator to reach a running state:

oc --kubeconfig $KUBECONFIG_QUORUM -n trilio-site-recovery-system \
  rollout status deployment/protection-zone-controller

Step 4 — Create a webhook serving certificate

The quorum control plane chart requires a TLS Secret named webhook-server-cert in the zone namespace before the manager will start. Create this out of band before creating the ProtectionZone. The certificate's SAN must cover the webhook Service hostname in the zone namespace.

The exact procedure depends on your certificate management tooling. At minimum the Secret must contain tls.crt and tls.key:

# Example using a self-signed cert — replace with your CA-signed cert in production
oc --kubeconfig $KUBECONFIG_QUORUM create secret tls webhook-server-cert \
  --cert=tls.crt \
  --key=tls.key \
  -n dr-prod

The workload clusters each need a corresponding webhook Secret in the trilio-site-recovery-system namespace (the default name from webhook.certSecretName):

oc --kubeconfig $KUBECONFIG_CLUSTER1 create secret tls webhook \
  --cert=tls.crt \
  --key=tls.key \
  -n trilio-site-recovery-system

oc --kubeconfig $KUBECONFIG_CLUSTER2 create secret tls webhook \
  --cert=tls.crt \
  --key=tls.key \
  -n trilio-site-recovery-system

Step 5 — Create the ProtectionZone CR

The ProtectionZone CR ties the two workload clusters together and drives automatic deployment of the quorum and workload control planes. Apply it to the quorum cluster in the dr-prod namespace:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionzone.yaml

Watch the zone reach Available:

oc --kubeconfig $KUBECONFIG_QUORUM get pz -n dr-prod -w

Expected output when ready:

NAME   DISPLAY NAME          BACKEND   CLUSTERS        STATUS      AGE
prod   Production DR Zone    DRBD      primary, dr     Available   2m

Once the zone reaches Available, the ProtectionZone controller automatically installs:

• The quorum control plane (site-recovery-quorum-control-plane) into dr-prod on the quorum cluster.
• The workload control plane (site-recovery-workload-control-plane) into trilio-site-recovery-system on both workload clusters.
• The DRBD node agent DaemonSet on every worker node of both workload clusters.
• The default DRBDReplicationPolicy named prod-default with isDefault: true in the dr-prod namespace.

Step 6 — Verify the deployment

Check the quorum control plane manager on the quorum cluster (single container named manager, output is 1/1):

oc --kubeconfig $KUBECONFIG_QUORUM -n dr-prod \
  get pods -l app.kubernetes.io/component=quorum-control-plane

Check the workload control plane on the primary cluster:

oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=workload-control-plane

Check the DRBD node agent DaemonSet on the primary cluster:

oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
  get daemonset -l app.kubernetes.io/component=drbd-node-agent

Repeat the last two commands for the DR cluster using $KUBECONFIG_CLUSTER2.

Check the default replication policy was created:

oc --kubeconfig $KUBECONFIG_QUORUM get drp -n dr-prod

Expected output:

NAME           CLUSTERS         MODE          PROTOCOL   DEFAULT   PHASE   AGE
prod-default   cluster1,...     Synchronous   C          true      Ready   1m

All day-to-day configuration in TSR is expressed through Kubernetes custom resources. The Helm charts that deploy the control planes are embedded in the operator binary and managed automatically by the ProtectionZone controller — you do not set chart values directly.

The key configuration surfaces are:

ProtectionZone fields

The ProtectionZone CR is your primary configuration object. The fields that matter most:

Mutable fields after creation: displayName, description, each cluster's kubeconfigSecretRef and distributionName, and logging. All other fields are immutable once set — attempting to change storageBackendMode, a cluster's name, or a cluster's clusterID is rejected by the admission webhook.

DRBDReplicationPolicy — managed, not authored

The ProtectionZone controller creates exactly one DRBDReplicationPolicy named <zoneName>-default with isDefault: true in the zone namespace. You do not create or modify this object. The policy is always Protocol C (synchronous, RPO=0). The policy is the only one ever consulted for ProtectionRequests in the zone namespace.

You can inspect it:

oc --kubeconfig $KUBECONFIG_QUORUM get drp -n dr-prod -o yaml

ProtectionRequest replication config

When protecting a VM, you can tune the per-VM replication configuration through spec.replicationConfig:

Runtime log level

Log levels can be changed without restarting pods by updating spec.logging on the ProtectionZone CR. The controller propagates the change via the site-recovery-logging ConfigMap within approximately 15 seconds.

Accepted values: error, info, debug, trace.

Example — enable debug logging for all components in a zone:

oc --kubeconfig $KUBECONFIG_QUORUM patch pz prod -n dr-prod \
  --type=merge \
  -p '{"spec":{"logging":{"level":"debug"}}}'

Enable trace logging for the node agent only:

oc --kubeconfig $KUBECONFIG_QUORUM patch pz prod -n dr-prod \
  --type=merge \
  -p '{"spec":{"logging":{"level":"info","nodeAgentLevel":"trace"}}}'

FailoverRequest configuration

TestFailover configuration

The test namespace is always generated by the controller as site-recovery-<protection-group>-<random> and reported in status.testNamespace. You cannot specify it.

Node agent DaemonSet tolerations

By default the node agent tolerates all taints (operator: Exists), which means it runs on every schedulable node including control-plane nodes. This is intentional — the agent must run wherever a protected VM might be pinned.

Fields accepted but not functional in Tech Preview

The following fields are accepted by the API but have no effect in Tech Preview. Do not rely on them:

• DRBDReplicationPolicy.spec.networkConfig.portRange — ports are always allocated from the fixed 7000–8000 range.
• DRBDReplicationPolicy.spec.clusters[].replicationEndpoint.port
• DRBDReplicationPolicy.spec.clusters[].nodeSelector
• DRBDReplicationPolicy.spec.replicationMode: Asynchronous
• ProtectionGroup.spec.sla.rpoTargetSeconds and rtoTargetSeconds
• ReplicationGroupStatus.spec.rpoObjectiveSeconds
• FailoverRequest.spec.drainTimeoutSeconds

All TSR operations are driven by creating, reading, and deleting Kubernetes custom resources on the appropriate cluster. There is no separate CLI — use oc or kubectl with the CRDs, or use the TSR web console on OpenShift.

Cluster context reminder: CRs live on specific clusters.

• Quorum cluster: ProtectionZone, ProtectionGroup, ProtectionRequest, DRBDReplicationPolicy, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent.
• Workload clusters (primary and DR): DRBDResource (in the VM's namespace).

Protect a virtual machine

Create a ProtectionRequest on the quorum cluster in the zone namespace. The VM must be running at the time of protection.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-server
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-server
    namespace: production

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-web-server.yaml

Watch protection progress:

oc --kubeconfig $KUBECONFIG_QUORUM get pr protect-web-server -n dr-prod -w

Phases in order: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

Once Protected, the VM is pinned to its current node for the lifetime of the protection. Live migration and node drain of the pinned node are denied for running protected VMs.

Group VMs for coordinated failover

Create a ProtectionGroup on the quorum cluster. All VMs in the group fail over together as an atomic unit.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: web-tier
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachines:
    - name: web-server
      namespace: production
    - name: web-cache
      namespace: production
      bootOrder: 1

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protection-group.yaml

Check group replication health:

oc --kubeconfig $KUBECONFIG_QUORUM get pg web-tier -n dr-prod
oc --kubeconfig $KUBECONFIG_QUORUM get rgs web-tier-replication -n dr-prod

Monitor replication health

The ReplicationGroupStatus CR aggregates replication health for a ProtectionGroup. The replication monitor auto-creates exactly one per ProtectionGroup, named <protection-group-name>-replication.

# List all replication statuses in a zone
oc --kubeconfig $KUBECONFIG_QUORUM get rgs -n dr-prod

# Detail for a specific group
oc --kubeconfig $KUBECONFIG_QUORUM get rgs web-tier-replication -n dr-prod -o yaml

Filter RPO events for a group using the label siterecovery.trilio.io/protection-group:

oc --kubeconfig $KUBECONFIG_QUORUM get rpoe -n dr-prod \
  -l siterecovery.trilio.io/protection-group=web-tier

Check DRBDResource state on the primary cluster (in the VM's namespace):

oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n production

Find protected VMs on a workload cluster:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

Run a test failover

A test failover validates DR capability without impacting production. It creates volume snapshots, provisions test PVCs, starts test VMs in an isolated namespace, runs verification checks, and cleans up.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-web-tier
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: web-tier
    namespace: dr-prod
  cleanupPolicy: Manual
  retentionTime: 2h

oc --kubeconfig $KUBECONFIG_QUORUM apply -f test-failover.yaml

Monitor progress:

oc --kubeconfig $KUBECONFIG_QUORUM get tfo test-web-tier -n dr-prod -w

Phases: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded → CleaningUp → Cleaned.

Check the generated test namespace:

oc --kubeconfig $KUBECONFIG_QUORUM get tfo test-web-tier -n dr-prod \
  -o jsonpath='{.status.testNamespace}'

Clean up manually (when cleanupPolicy: Manual):

oc --kubeconfig $KUBECONFIG_QUORUM delete tfo test-web-tier -n dr-prod

Execute a planned failover

A planned failover gracefully shuts down VMs on the source cluster, waits for DRBD synchronization, then promotes and starts VMs on the target cluster.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-web-tier
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: web-tier
    namespace: dr-prod
  targetCluster: cluster2
  failoverType: planned

oc --kubeconfig $KUBECONFIG_QUORUM apply -f failover.yaml

Monitor progress:

oc --kubeconfig $KUBECONFIG_QUORUM get fr failover-web-tier -n dr-prod -w

Phases: Initializing → Validating → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → Completed.

Execute an unplanned failover

Use failoverType: unplanned only when the primary cluster is unavailable. The DR cluster's DRBD volumes are force-promoted without waiting for the primary to shut down. If replication was Consistent at the moment of failure, zero data loss occurs.

spec:
  failoverType: unplanned
  targetCluster: cluster2

Collect a diagnostic bundle

Run tsr-gather via oc adm must-gather against the quorum cluster. It collects logs, CRD state, and configuration from all clusters in the DR estate:

oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag>

Scope to a specific zone or time window:

# Last 2 hours only
oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

# Single zone
oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --zone prod

Example 1 — Minimal ProtectionZone

The simplest valid ProtectionZone pairing two RHEL 9 OpenShift clusters:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Expected output after oc get pz -n dr-prod:

NAME   DISPLAY NAME          BACKEND   CLUSTERS        STATUS      AGE
prod   Production DR Zone    DRBD      primary, dr     Available   3m

Example 2 — Protect a VM

Protect a single VM named database running in the app-prod namespace on cluster1:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-database
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: database
    namespace: app-prod
  replicationConfig:
    replicationProtocol: C
    rpo: 0
    drbdMetadataConfig:
      size: 128Mi

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-database.yaml

# Watch until Protected
oc --kubeconfig $KUBECONFIG_QUORUM get pr protect-database -n dr-prod -w

Expected final output:

NAME               VM         NAMESPACE   PHASE       SYNC   AGE
protect-database   database   app-prod    Protected   100    8m

Verify the VM is labeled as protected on the primary cluster:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get vm database -n app-prod \
  -o jsonpath='{.metadata.labels.siterecovery\.trilio\.io/protected-vm}'
# Expected: true

Example 3 — ProtectionGroup with boot ordering

Group a web server and its database with ordered boot on failover (database boots first):

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: app-tier
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  desiredState: running
  virtualMachines:
    - name: database
      namespace: app-prod
      bootOrder: 1
    - name: web-server
      namespace: app-prod
      bootOrder: 2

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protection-group.yaml
oc --kubeconfig $KUBECONFIG_QUORUM get pg app-tier -n dr-prod

Expected output:

NAME       VMs                    STATE       ACTIVE    AGE
app-tier   database, web-server   Protected   cluster1  5m

Example 4 — Test failover with HTTP verification probe

Validate DR and probe the web server's health endpoint after test VMs start:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-app-tier
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-prod
  cleanupPolicy: Automatic
  retentionTime: 1h
  verification:
    reachability:
      - type: http
        vmName: web-server
        port: 8080
        endpoint: /health
        expectedStatus: 200
        timeout: 30s

oc --kubeconfig $KUBECONFIG_QUORUM apply -f test-failover.yaml
oc --kubeconfig $KUBECONFIG_QUORUM get tfo test-app-tier -n dr-prod -w

Check verification results after Succeeded:

oc --kubeconfig $KUBECONFIG_QUORUM get tfo test-app-tier -n dr-prod \
  -o jsonpath='{.status.verificationResults}'

Example 5 — Planned failover to cluster2

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-app-tier
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-prod
  targetCluster: cluster2
  failoverType: planned
  batchBootTimeoutSeconds: 300

oc --kubeconfig $KUBECONFIG_QUORUM apply -f failover.yaml
oc --kubeconfig $KUBECONFIG_QUORUM get fr failover-app-tier -n dr-prod -w

Expected final output:

NAME                PROTECTION GROUP   TARGET     STATE       PHASE       AGE
failover-app-tier   app-tier           cluster2   Completed   Completed   4m

Verify per-VM status:

oc --kubeconfig $KUBECONFIG_QUORUM get fr failover-app-tier -n dr-prod \
  -o jsonpath='{.status.vmStatuses}'

Example 6 — Check DRBDResource state on a workload cluster

# List all DRBDResources in the VM's namespace on the primary cluster
oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n app-prod

Expected output:

NAME                              SIDE     ROLE      NODE             STATE      PHASE       AGE
drbd-app-prod-database-a1b2c3d4   Source   Primary   worker-node-1    Connected   Consistent   10m

The DRBDResource name follows the pattern drbd-<namespace>-<vm>-<hash8>.

Example 7 — Inspect replication health

# ReplicationGroupStatus — auto-created, named <pg-name>-replication
oc --kubeconfig $KUBECONFIG_QUORUM get rgs app-tier-replication -n dr-prod -o yaml

Key fields to check:

status:
  aggregated:
    connectionState: Connected
    outOfSyncBytes: 0
    syncedVolumeCount: 2
    totalVolumeCount: 2
  rpoStatus: Met
  lastFullySyncedAt: "2026-07-10T14:32:00Z"

Use a consistent diagnostic approach: check the CR's status.phase and status.conditions, then check pod logs in the appropriate namespace.

Namespace reminder:

• Quorum CRs and quorum manager logs: dr-<zone-name> namespace on the quorum cluster.
• Workload CRs (DRBDResource) and node agent logs: trilio-site-recovery-system namespace on the workload cluster, in the VM's namespace for DRBDResources.

Issue: ProtectionZone stuck in Provisioning

Symptom: oc get pz -n dr-prod shows STATUS: Provisioning for more than a few minutes.

Likely causes and fixes:

1. Kubeconfig Secret is missing or invalid.

   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM get pz prod -n dr-prod -o yaml
   # Look at status.clusters[].status and status.clusters[].message
   

   Fix: ensure both Secrets named in spec.clusters[].kubeconfigSecretRef.name exist in dr-prod and contain a valid kubeconfig.

2. Workload cluster is unreachable. The ProtectionZone controller probes the trilio-site-recovery-system namespace on each workload cluster. If that namespace does not exist, the zone cannot confirm the cluster is ready.

   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \
     -l app.kubernetes.io/component=protectionzone-controller
   

3. Webhook serving certificate missing. The quorum manager will not start without webhook-server-cert in the zone namespace.

   Copy

   oc --kubeconfig $KUBECONFIG_QUORUM get secret webhook-server-cert -n dr-prod
   

   Fix: create the Secret as described in the installation steps.

Issue: ProtectionRequest stuck in CreatingDRBD or Syncing

Symptom: oc get pr -n dr-prod shows the request in CreatingDRBD or Syncing for an unexpectedly long time.

Likely causes and fixes:

1. DRBD replication port blocked. TCP ports 7000–8000 must be open between primary and DR worker nodes. A blocked port causes silent replication failure.

   Copy

   # Check DRBDResource connection state on the primary cluster
   oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n <vm-namespace>
   # Look at STATUS column — Disconnected means a network issue
   

   Fix: open TCP 7000–8000 (inclusive) in both directions between primary and DR worker nodes. Port 8000 is mandatory.

2. DRBD module not loaded (node agent init container failing).

   Copy

   oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
     get pods -l app.kubernetes.io/component=drbd-node-agent
   # Look for Init:Error or Init:CrashLoopBackOff
   POD=$(oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
     get pods -l app.kubernetes.io/component=drbd-node-agent \
     --field-selector spec.nodeName=<node-name> -o name | head -1)
   oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
     logs $POD -c drbd-module-loader
   

   Fix: ensure the node's kernel version has a matching precompiled kmod in the module-loader image. RHEL 10 nodes are not validated.

3. PlacementReady=False on the DR-side DRBDResource.

   Copy

   oc --kubeconfig $KUBECONFIG_CLUSTER2 get dres -n <vm-namespace> -o yaml
   # Check status.conditions for ConditionPlacementReady
   

   If reason is AwaitingAnchorSchedule, all candidate nodes on the DR cluster may be tainted with siterecovery.trilio.io/not-ready:NoSchedule. The DRBDResource will recover automatically once a node's DRBD converges. If this persists past the drbdMetadataProvisioningFailureDeadline (default 10 minutes), protection will fail.

4. VM was not running at protection time. The VM must be running when you create the ProtectionRequest. The protection controller validates this in the Validating phase.

Issue: ProtectionRequest in Failed with RoleConflict

Symptom: oc get pr -n dr-prod -o yaml shows condition Ready=False with reason RoleConflict.

Likely cause: A DRBD role conflict exists — both sides are reporting Primary, which should not happen under Protocol C. This can occur after an unplanned failover where the primary cluster recovered unexpectedly.

Fix: Follow docs/runbooks/ for the role-conflict recovery procedure. Do not attempt to manually promote or demote DRBD resources.

Issue: Node agent pod in CrashLoopBackOff

Symptom: Node agent pods on a workload cluster are not starting.

oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=drbd-node-agent
oc --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
  logs <pod-name> -c agent

Likely causes:

• DRBD module already loaded with usermode_helper enabled. The loader init container will fail by design. Reboot the node after unloading the conflicting module.
• No InternalIP on the node. The agent cannot start without a node internal IP. Check: oc get node <name> -o jsonpath='{.status.addresses}'.
• Host ports 18090 or 18091 already in use. Another process has bound those ports. The node agent uses host networking.

Issue: oc adm drain does not complete on a node hosting a protected VM

Symptom: oc adm drain <node> hangs or reports the VM cannot be evicted.

Cause: A running protected VM's migration is denied by the workload admission webhook. All KubeVirt migration paths — user-initiated, evacuation, and drain — are realised as migration objects, which are denied:

live migration of VM "<name>" is not allowed: it uses DRBD-protected volumes with node-pinned PVs

Fix: Draining never relocates a protected VM. DRBD replication runs between two specific pinned nodes. Options:

• In-place maintenance (reboot, kernel update) on the same node is safe — modules reload on boot and DRBD reconnects.
• Permanently removing the node requires unprotecting the VMs, draining, and re-protecting (full initial resync).
• Node loss recovery — follow docs/runbooks/drbdresource-node-loss.md exactly. Remove only the agent finalizer, not the placement finalizer.

Issue: TestFailover stuck or not cleaning up

Symptom: TestFailover CR remains in a non-terminal phase or in CleaningUp for a long time.

oc --kubeconfig $KUBECONFIG_QUORUM get tfo <name> -n dr-prod -o yaml
# Check status.phase and status.conditions
# Check status.testNamespace for the generated test namespace

Check the test namespace on the target workload cluster:

TEST_NS=$(oc --kubeconfig $KUBECONFIG_QUORUM get tfo <name> -n dr-prod \
  -o jsonpath='{.status.testNamespace}')
oc --kubeconfig $KUBECONFIG_CLUSTER2 get pods -n $TEST_NS

Do not manually patch finalizers to force deletion of a TestFailover. The controller manages the test namespace lifecycle. If the cleanup appears stuck, check the quorum manager logs:

oc --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \
  -l app.kubernetes.io/component=quorum-control-plane -c manager

Issue: kubectl get drbdresource returns server doesn't have a resource type

Symptom: Running oc get dres against the quorum cluster returns an error.

Cause: DRBDResource CRDs are installed only on workload clusters, not on the quorum cluster. This is by design.

Fix: Always query DRBDResources against the primary or DR cluster:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n <vm-namespace>

Issue: TSR web console not accessible

Symptom: Cannot reach the TSR web console in a browser.

Cause: Browser access to the TSR web console requires OpenShift and the oauth-proxy Route and console link created by the ProtectionZone controller. On upstream Kubernetes, no ingress is created and browser access is not a supported use case.

Fix on OpenShift: Verify the Route exists:

oc --kubeconfig $KUBECONFIG_QUORUM get route -n trilio-site-recovery-system

If the Route is missing, check that the ProtectionZone has reached Available — the Route is created by the ProtectionZone controller as part of zone provisioning.

Collecting a support bundle

For any issue not resolved by the above steps, collect a full diagnostic bundle:

oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 4h

Read SUMMARY.md first (what went wrong), then manifest.yaml (what was collected). Secret data, kubeconfigs, and DRBD shared secrets are never written to the bundle.