Deployment Guide

This page guides platform engineers and SREs through a complete production deployment of Trilio Site Recovery on OpenShift. It covers both installation paths — the OLM bundle that installs the operator, ProtectionZone CRD, and TSR web console on the quorum cluster, and the Helm-chart-based quorum and workload control planes that the ProtectionZone controller deploys automatically — plus RBAC setup, resource sizing, multi-cluster configuration, and day-2 operational tasks. By the end you will have a functioning protection zone with DRBD synchronous replication (Protocol C, RPO=0) active between a primary and a DR cluster, all managed through Kubernetes Custom Resources in the siterecovery.trilio.io/v1alpha1 API group.

Before you begin, verify the following:

Clusters

• Three OpenShift clusters: one quorum (management plane) and two workload clusters (primary and DR). The quorum cluster is required — it is not optional.
• Each cluster must run OpenShift 4.17 or later. The OLM bundle annotation (deploy/olm-catalog/site-recovery/metadata/annotations.yaml) states com.redhat.openshift.versions: v4.14-v4.19. The validated range per docs/tech-preview-supported-configurations.md is OpenShift 4.17–4.20. OpenShift 4.14–4.16 are within the installable range but are not validated for Tech Preview; 4.17 is the supported floor for the quorum cluster and both workload clusters.
• Single-Node OpenShift (SNO) is supported for both workload and quorum roles.
• Upstream Kubernetes workload clusters are not supported in Tech Preview.

OpenShift Virtualization

• OpenShift Virtualization (CNV) installed on both workload clusters, serving the kubevirt.io/v1 API with VirtualMachine, VirtualMachineInstance, and VirtualMachineInstanceMigration kinds. The quorum cluster does not need KubeVirt.

DRBD kernel module

• DRBD 9 (drbd, drbd_transport_tcp, drbd_transport_lb-tcp) on every schedulable worker node of both workload clusters. The DRBD node-agent DaemonSet loads the module automatically via a privileged init container using LINBIT's drbd9-rhel9 image; no manual insmod is needed. RHEL 9-based RHCOS (el9 kernels) is supported. RHEL 10-based nodes are not validated in Tech Preview.

Storage

• LVM thin pool configured on each worker node of both workload clusters, backed by a dedicated block device, for DRBD backing storage.

Network

• TCP ports 7000–8000 inclusive open between primary and DR worker nodes (one port per replicated VM volume). Port 8000 is mandatory; a firewall rule that stops at 7999 silently prevents replication on one volume.
• TCP port 6443 (Kubernetes API) open from the quorum cluster to both workload cluster API servers.
• Round-trip latency below 50 ms between primary and DR clusters. Protocol C synchronous replication requires this; higher latency will degrade write performance and can cause replication to fall behind.

CLI tools

• oc CLI (required). oc is needed for OpenShift-specific paths (oc adm must-gather, Routes, oauth-proxy). kubectl-only operation is not a supported mode.
• Kubeconfig files for all three clusters.

Registry access

• All clusters must be able to pull images from quay.io/triliovault. The OLM CSV substitutes IMAGEREGISTRY at build time and RELATED_IMAGE_* environment variables override chart defaults at runtime; shipped images resolve to quay.io/triliovault.

Clock synchronisation

• NTP/chrony running on every node of all three clusters. Clock skew causes subtle credential and timestamp failures that do not surface as TSR errors.

Installation has two complementary phases that must both complete:

1. OLM bundle — installs the protection-zone-controller Deployment, the TSR web console (site-recovery-backend and site-recovery-frontend), and the ProtectionZone CRD on the quorum cluster.
2. ProtectionZone CR — once created, the ProtectionZone controller automatically deploys the quorum control plane and both workload control planes (including the DRBD node-agent DaemonSet) by running embedded Helm charts from inside the operator binary. You never run helm install or helm upgrade directly.

Phase 1: Install the OLM bundle on the quorum cluster

All commands in this phase target the quorum cluster.

Step 1 — Create the operator namespace

oc --kubeconfig=<quorum-kubeconfig> create namespace trilio-site-recovery-system

Step 2 — Create the CatalogSource

This points OLM at the Trilio operator catalog on Quay.

# catalogsource.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: trilio-site-recovery-catalog
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/triliovault/site-recovery-catalog:latest
  displayName: Trilio Site Recovery
  publisher: Trilio Data, Inc.

oc --kubeconfig=<quorum-kubeconfig> apply -f catalogsource.yaml

Wait for the catalog pod to become ready:

oc --kubeconfig=<quorum-kubeconfig> -n openshift-marketplace \
  get pods -l olm.catalogSource=trilio-site-recovery-catalog

Step 3 — Create the OperatorGroup

# operatorgroup.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trilio-site-recovery-og
  namespace: trilio-site-recovery-system
spec:
  targetNamespaces:
    - trilio-site-recovery-system

oc --kubeconfig=<quorum-kubeconfig> apply -f operatorgroup.yaml

Step 4 — Create the Subscription

# subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: tech-preview
  name: site-recovery
  source: trilio-site-recovery-catalog
  sourceNamespace: openshift-marketplace

oc --kubeconfig=<quorum-kubeconfig> apply -f subscription.yaml

Step 5 — Verify the operator is running

oc --kubeconfig=<quorum-kubeconfig> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/part-of=site-recovery

Expect the protection-zone-controller pod, site-recovery-backend, and site-recovery-frontend pods to reach Running. The web console has no opt-out and no feature flag; all three deploy unconditionally.

Phase 2: Create the ProtectionZone

The ProtectionZone CR lives on the quorum cluster in a namespace prefixed dr-. All quorum-side CRs and components for a given deployment live in this namespace; workload-side components deploy into trilio-site-recovery-system on each workload cluster.

Step 6 — Create the zone namespace on the quorum cluster

oc --kubeconfig=<quorum-kubeconfig> create namespace dr-prod

Step 7 — Store kubeconfig Secrets for both workload clusters

The ProtectionZone controller reads these to build its cross-cluster clients. The Secrets must be in the zone namespace (dr-prod).

# Primary cluster kubeconfig
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=<path-to-primary-kubeconfig>

# DR cluster kubeconfig
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=<path-to-dr-kubeconfig>

Step 8 — Create the webhook TLS Secret

The charts do not manage certificate provisioning. Create a webhook-server-cert Secret containing tls.crt and tls.key in the zone namespace before the control plane deploys. The exact method (cert-manager, openssl, your PKI) is your choice; the secret name must match webhook.certSecretName (default: webhook-server-cert).

# Example using openssl (adjust SANs for your environment)
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout tls.key -out tls.crt -days 365 \
  -subj "/CN=webhook-service.dr-prod.svc"

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  create secret tls webhook-server-cert \
  --cert=tls.crt --key=tls.key

Repeat this step on both workload clusters in the trilio-site-recovery-system namespace, since the workload control plane also serves admission webhooks:

oc --kubeconfig=<primary-kubeconfig> -n trilio-site-recovery-system \
  create secret tls webhook-server-cert \
  --cert=tls.crt --key=tls.key

oc --kubeconfig=<dr-kubeconfig> -n trilio-site-recovery-system \
  create secret tls webhook-server-cert \
  --cert=tls.crt --key=tls.key

Step 9 — Apply the ProtectionZone CR

# protectionzone.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  description: "DRBD synchronous replication between primary and DR sites."
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

oc --kubeconfig=<quorum-kubeconfig> apply -f protectionzone.yaml

Important: storageBackendMode: Linstor is rejected by the admission webhook. DRBD is the only accepted value.

Step 10 — Watch the zone reach Available

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get pz prod -w

The STATUS column transitions: Provisioning → Available. Once Available, the controller has:

• Deployed the site-recovery-quorum-control-plane chart into dr-prod on the quorum cluster.
• Deployed the site-recovery-workload-control-plane chart into trilio-site-recovery-system on both workload clusters, including the DRBD node-agent DaemonSet.
• Auto-created the default DRBDReplicationPolicy named prod-default with isDefault: true and Protocol C. You do not create or modify replication policies.

Step 11 — Verify the workload control planes

On each workload cluster:

# Quorum control-plane pod (single container named 'manager', expected 1/1)
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  get pods -l app.kubernetes.io/component=quorum-control-plane

# Workload control-plane pod on primary
oc --kubeconfig=<primary-kubeconfig> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=workload-control-plane

# DRBD node-agent DaemonSet on primary (one pod per schedulable node)
oc --kubeconfig=<primary-kubeconfig> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=drbd-node-agent

# DRBD node-agent DaemonSet on DR cluster
oc --kubeconfig=<dr-kubeconfig> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=drbd-node-agent

The DaemonSet object name follows the pattern <release>-site-recovery-workload-control-plane-agent. The label app.kubernetes.io/component=drbd-node-agent is the correct selector; drbd-node-agent is only the component label value, not the object name.

Step 12 — Verify the default replication policy

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get drp

Expect one row named prod-default with Default: true and Protocol: C.

Step 13 — Access the TSR web console (OpenShift only)

The ProtectionZone controller creates an OpenShift Route and console link automatically. Retrieve the Route:

oc --kubeconfig=<quorum-kubeconfig> -n trilio-site-recovery-system \
  get routes -l app.kubernetes.io/part-of=site-recovery

Open the URL in a browser. Authentication is handled by the OpenShift oauth-proxy. On upstream Kubernetes no Route or ingress is created; browser access is not a supported use case on non-OpenShift clusters.

All configuration surfaces are documented here. Remember: you never edit Helm chart values directly — the ProtectionZone controller manages the embedded charts. The fields below reflect what the controller exposes through the ProtectionZone CR and what the operator reads from the OLM Subscription.

ProtectionZone spec fields

These are the primary configuration controls available to you.

Logging example — raise control-plane verbosity for a troubleshooting window without restarting pods:

spec:
  logging:
    level: info
    controlPlaneLevel: debug

Replication policy (managed, read-only)

The ProtectionZone controller creates one DRBDReplicationPolicy named <zoneName>-default with isDefault: true in the zone namespace. You do not create or modify this object. The policy is always Protocol C (synchronous, RPO=0). Protocol A is not supported and is rejected by the admission webhook at the ProtectionRequest level.

To inspect the policy the controller created:

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get drp prod-default -o yaml

Webhook TLS

All three charts (site-recovery-quorum-control-plane, site-recovery-workload-control-plane, site-recovery-protectionzone-controller) serve admission webhooks on container port 9443. Each chart expects a pre-created Secret named webhook-server-cert (the webhook.certSecretName default) containing tls.crt and tls.key in its release namespace:

The charts do not integrate with cert-manager. You are responsible for creating and rotating these Secrets out of band.

Zone-exclusivity lease

Each workload cluster can belong to at most one ProtectionZone. The controller stamps two annotations on the workload cluster's trilio-site-recovery-system Namespace:

If a zone shows phase: Degraded with reason ClusterAlreadyClaimed, see the troubleshooting section.

Node-agent resource sizing

The DRBD node agent defaults:

If your nodes need more headroom, update the ProtectionZone CR's logging field to trigger a reconcile and raise an issue with Trilio support to adjust agent sizing — direct Helm value edits are not a customer procedure.

Anchor pod priority

Anchor pods keep DRBD-backing PVCs attached under active replication. By default they have no priority class. To protect them from eviction under node pressure ahead of ordinary workloads, set anchorPod.priorityClassName (for example system-cluster-critical) via a support-assisted chart update.

Zone configuration tunables

Two provisioning deadlines control how long the workload control plane waits before marking a DRBDResource placement as failed:

These defaults are suitable for most environments. Do not raise metadataReadyGracePeriod independently of metadataProvisioningFailureDeadline; the grace period is always a floor for the failure deadline.

Once the ProtectionZone is Available, your day-to-day workflow is driven entirely by Custom Resources applied to the quorum cluster (for ProtectionZone-scoped objects) or to the workload clusters (for DRBDResource objects). This section covers the most common operational patterns.

Namespace placement quick reference

Always use the right cluster context:

kubectl get drbdresource against the quorum cluster returns server doesn't have a resource type — that CRD is registered on workload clusters only.

Protecting a VM

Create a ProtectionRequest in the zone namespace on the quorum cluster. The VM must be running at protection time (node pinning requires a live VM).

# protectionrequest.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: vm-webserver
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: webserver
    namespace: app-production
  replicationConfig:
    replicationProtocol: C
    rpo: 0

oc --kubeconfig=<quorum-kubeconfig> apply -f protectionrequest.yaml

Watch the phase progress:

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get pr vm-webserver -w

Expected phase sequence: Pending → Validating → CreatingDRBD → Syncing → Protected.

Once Protected, the VM is pinned to its current node on both clusters for the lifetime of the protection. Live migration is denied for protected VMs. Re-protection after unprotect costs a full initial resynchronisation.

Monitoring replication health

The replication monitor auto-creates one ReplicationGroupStatus per ProtectionGroup, named <protection-group-name>-replication:

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get rgs

Filter by protection group using the label (field selectors on spec.*/status.* do not work on custom resources):

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get rgs \
  -l siterecovery.trilio.io/protection-group=<pg-name>

Check per-VM replication state on the workload cluster:

# List all protected VMs
oc --kubeconfig=<primary-kubeconfig> get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

# Inspect the DRBDResource for a specific VM (workload cluster)
oc --kubeconfig=<primary-kubeconfig> -n app-production get dres

Healthy per-volume replication state is replicationState: Established. The resource-scoped healthy connection state is connectionState: Connected.

Running a test failover

A TestFailover performs a non-disruptive DR validation by creating volume snapshots, starting test VMs in an isolated namespace, running verification checks, and cleaning up:

# testfailover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: prod-pg
    namespace: dr-prod
  cleanupPolicy: Automatic
  retentionTime: 2h

oc --kubeconfig=<quorum-kubeconfig> apply -f testfailover.yaml

Watch the phase:

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get tfo test-prod-pg -w

Phase sequence: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded → CleaningUp → Cleaned.

The test namespace is generated by the controller as site-recovery-<protection-group>-<random> and reported in status.testNamespace. Never construct or guess the namespace name manually.

Executing a planned failover

A planned failover gracefully shuts down VMs on the source cluster, promotes DRBD volumes on the target, and starts VMs there:

# failoverrequest.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: prod-pg
    namespace: dr-prod
  failoverType: planned
  targetCluster: cluster2

oc --kubeconfig=<quorum-kubeconfig> apply -f failoverrequest.yaml

Watch progress:

oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get fr failover-prod-pg -w

Per-VM detail is in status.vmStatuses. There is no failedVMs field and no message field on FailoverRequest status.

Executing an unplanned failover

When the primary cluster is unavailable, force-promote DRBD volumes on the DR cluster without waiting for a clean shutdown:

spec:
  failoverType: unplanned
  targetCluster: cluster2

If replication was Healthy and Consistent at the moment of failure, zero data loss occurs. If replication was Degraded, minimal data loss may occur depending on out-of-sync bytes at the time.

Collecting a diagnostic bundle

Run tsr-gather on the quorum cluster to collect logs, CRD state, and configuration from all clusters in the DR estate:

oc --kubeconfig=<quorum-kubeconfig> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag>

On an OLM-installed cluster, oc adm must-gather --all-images picks up the image from the CSV annotation automatically. The must-gather entrypoint is /usr/bin/gather — no -- <command> override is needed.

Example 1: Minimal ProtectionZone

A two-cluster zone using the default RHEL 9 module loader and info-level logging.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Expected output after oc -n dr-prod get pz prod:

NAME   DISPLAY NAME          BACKEND   CLUSTERS         STATUS      AGE
prod   Production DR Zone    DRBD      primary,dr       Available   3m

Example 2: ProtectionZone with debug logging for control planes

Raise the quorum and workload control-plane log level to debug without restarting pods. The nodeAgentLevel stays at info to avoid verbose DRBD kernel output.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
  logging:
    level: info
    controlPlaneLevel: debug
    nodeAgentLevel: info

The change takes effect within ~15 seconds; no pod restart is required.

Example 3: Protecting a VM

Protect webserver in the app-production namespace, sourced from cluster1, using Protocol C (the only supported protocol).

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: pr-webserver
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: webserver
    namespace: app-production
  replicationConfig:
    replicationProtocol: C
    rpo: 0

oc --kubeconfig=<quorum-kubeconfig> apply -f protectionrequest.yaml
# Watch progress
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get pr pr-webserver -w

Expected column output when complete:

NAME           VM          NAMESPACE        PHASE       SYNC   AGE
pr-webserver   webserver   app-production   Protected   100    8m

Example 4: Creating a ProtectionGroup and TestFailover

Group two VMs that must fail over together, then validate with a non-disruptive test:

# protectiongroup.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: prod-pg
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachines:
    - name: webserver
      namespace: app-production
      bootOrder: 1
    - name: database
      namespace: app-production
      bootOrder: 2
  sla:
    rtoTargetSeconds: 300

oc --kubeconfig=<quorum-kubeconfig> apply -f protectiongroup.yaml
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get pg prod-pg

# testfailover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: tfo-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: prod-pg
    namespace: dr-prod
  cleanupPolicy: Automatic
  retentionTime: 2h
  verification:
    reachability:
      - vmName: webserver
        type: tcp
        port: 80
        timeout: 30s

oc --kubeconfig=<quorum-kubeconfig> apply -f testfailover.yaml
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get tfo tfo-prod-pg -w

Expected final output:

NAME          PROTECTION GROUP   PHASE     PASSED   AGE
tfo-prod-pg   prod-pg            Cleaned   true     12m

Example 5: Planned failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: fr-prod-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: prod-pg
    namespace: dr-prod
  failoverType: planned
  targetCluster: cluster2

oc --kubeconfig=<quorum-kubeconfig> apply -f failoverrequest.yaml
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get fr fr-prod-pg -w

Expected column output when complete:

NAME         PROTECTION GROUP   TARGET     STATE       PHASE       AGE
fr-prod-pg   prod-pg            cluster2   Completed   Completed   4m

Example 6: Inspecting replication health

# ReplicationGroupStatus (quorum cluster)
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get rgs

# DRBDResource per-volume detail (primary workload cluster)
oc --kubeconfig=<primary-kubeconfig> -n app-production get dres

# RPO violation events
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get rpoe

# Detailed volume state
oc --kubeconfig=<primary-kubeconfig> -n app-production \
  get dres -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.status.connectionState}{"  "}{range .status.volumes[*]}{.volumeNumber}:{.replicationState}{" "}{end}{"\n"}{end}'

Healthy per-volume state is replicationState: Established; resource-scoped healthy is connectionState: Connected.

Example 7: Collecting a support bundle

# Full estate bundle from the quorum cluster
oc --kubeconfig=<quorum-kubeconfig> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag>

# Narrow to the last 2 hours (faster)
oc --kubeconfig=<quorum-kubeconfig> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

# One zone only
oc --kubeconfig=<quorum-kubeconfig> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --zone prod

Use a consistent approach for each issue: check the relevant CR status, inspect pod logs, then apply the fix.

Log locations:

• Quorum control-plane: oc --kubeconfig=<quorum-kubeconfig> -n dr-prod logs -l app.kubernetes.io/component=quorum-control-plane -c manager
• Workload control-plane: oc --kubeconfig=<primary-kubeconfig> -n trilio-site-recovery-system logs -l app.kubernetes.io/component=workload-control-plane -c manager
• DRBD node agent on a specific node (two steps, because kubectl logs has no --field-selector):
  Copy

  AGENT_POD=$(oc --kubeconfig=<primary-kubeconfig> -n trilio-site-recovery-system \
    get pods --field-selector spec.nodeName=<node-name> \
    -l app.kubernetes.io/component=drbd-node-agent \
    -o jsonpath='{.items[0].metadata.name}')
  oc --kubeconfig=<primary-kubeconfig> -n trilio-site-recovery-system \
    logs $AGENT_POD -c agent
  

Issue 1: ProtectionZone stuck in Provisioning

Symptom: oc -n dr-prod get pz shows STATUS: Provisioning for more than 5 minutes.

Likely causes:

• Kubeconfig Secret missing, malformed, or pointing at an unreachable cluster.
• trilio-site-recovery-system namespace not yet created on a workload cluster (the reachability probe GETs this namespace by name).
• webhook-server-cert Secret missing in the zone namespace or on a workload cluster.

Fix:

# Check per-cluster reachability
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get pz prod -o jsonpath='{.status.clusters}' | jq .

# Check conditions for the reason
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  get pz prod -o jsonpath='{.status.conditions}' | jq .

# Verify the kubeconfig Secret exists
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get secret cluster1-kubeconfig

# Verify the webhook-server-cert exists on the zone namespace
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod get secret webhook-server-cert

If a workload cluster shows status: Unreachable, verify connectivity (port 6443 from quorum to that cluster's API server) and that the kubeconfig credential is valid.

Issue 2: ProtectionZone Degraded with reason ClusterAlreadyClaimed

Symptom: oc -n dr-prod get pz shows STATUS: Degraded; conditions show ClustersValidated=False with reason ClusterAlreadyClaimed.

Cause: One of the workload clusters is already leased to a different ProtectionZone.

Fix:

# 1. Find the current owner
kubectl --kubeconfig=<workload-kubeconfig> get ns trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations.siterecovery\.trilio\.io/owner-zone}{"\n"}'
kubectl --kubeconfig=<workload-kubeconfig> get ns trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations.siterecovery\.trilio\.io/owner-zone-namespace-uid}{"\n"}'

# 2. Verify the owner zone still exists on the quorum cluster by matching the UID
kubectl --kubeconfig=<quorum-kubeconfig> get ns \
  -o custom-columns=NAME:.metadata.name,UID:.metadata.uid

# 3. ONLY if the owning zone genuinely no longer exists, release the lease
kubectl --kubeconfig=<workload-kubeconfig> annotate ns trilio-site-recovery-system \
  siterecovery.trilio.io/owner-zone-namespace-uid- \
  siterecovery.trilio.io/owner-zone-

Warning: Never release a lease while the owning zone is still live. Doing so allows two zones to drive one workload cluster simultaneously.

A UID in the lease that matches no namespace on this quorum cluster was written by a different TSR deployment. Only release it after confirming that deployment is retired.

Issue 3: DRBD node-agent pod stuck in Init:Error or Init:CrashLoopBackOff

Symptom: Node-agent pods on a workload cluster show Init:Error.

Cause: The drbd-module-loader init container could not find a precompiled kmod for this node's kernel.

Fix:

# 1. Identify which pod is failing
oc --kubeconfig=<workload-kubeconfig> -n trilio-site-recovery-system \
  get pods -l app.kubernetes.io/component=drbd-node-agent

# 2. Read the loader log
oc --kubeconfig=<workload-kubeconfig> -n trilio-site-recovery-system \
  logs <pod-name> -c drbd-module-loader

# 3. Check the node kernel version
oc --kubeconfig=<workload-kubeconfig> get node <node-name> \
  -o jsonpath='{.status.nodeInfo.kernelVersion}'

If the log says "no matching package", the LINBIT mirror does not yet have a kmod for this kernel. Wait for the mirror job to rebuild the tag (the loader uses pullPolicy: Always and retags latest), then delete the stuck pod to trigger a fresh pull. If a foreign DRBD module is loaded with usermode_helper enabled, reboot the node after removing the external module loader.

Issue 4: ProtectionRequest stuck in Syncing with no progress

Symptom: oc -n dr-prod get pr <name> shows PHASE: Syncing and SYNC percentage is not increasing.

Likely causes:

• DRBD replication link is down (network issue between worker nodes).
• TCP ports 7000–8000 not fully open. A rule stopping at 7999 silently prevents one volume from replicating.
• The DR-side node is tainted with siterecovery.trilio.io/not-ready:NoSchedule and the anchor pod has not scheduled.

Fix:

# Check the DRBDResource on the source workload cluster
oc --kubeconfig=<primary-kubeconfig> -n app-production get dres -o wide

# Per-volume replication state
oc --kubeconfig=<primary-kubeconfig> -n app-production \
  get dres <resource-name> -o jsonpath='{.status.volumes[*].replicationState}'

# Connection state
oc --kubeconfig=<primary-kubeconfig> -n app-production \
  get dres <resource-name> -o jsonpath='{.status.connectionState}'

# Check the ProtectionRequest conditions for RoleConflict
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  get pr <name> -o jsonpath='{.status.conditions}' | jq .

# Verify node taints on the DR cluster
oc --kubeconfig=<dr-kubeconfig> get nodes \
  -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.spec.taints}{"\n"}{end}'

A Ready=False condition with reason RoleConflict on the ProtectionRequest means a DRBD volume on the workload side has an unexpected primary role. Check the DRBDResource status on both clusters and consult the node-agent logs.

Issue 5: oc adm drain does not complete on a node hosting a protected VM

Symptom: oc adm drain <node> hangs or fails on a node hosting a running protected VM.

Cause: Every KubeVirt migration path (including drain-triggered evacuation) creates a VirtualMachineInstanceMigration object. The TSR admission webhook denies live migration for protected VMs. Drain of a stopped protected VM is also blocked because the anchor pod is pinned via spec.nodeName and returns to the same node even after a forced delete.

Fix: There is no drain path for a protected VM. Plan maintenance around pinned nodes:

• In-place maintenance (reboot, kernel update): Safe. DRBD modules reload on boot and replication resynchronises the delta automatically.
• Permanently removing the node: Unprotect the VM first, perform node replacement, then re-protect (costs a full initial resync).
• Unexpected node loss: Follow docs/runbooks/drbdresource-node-loss.md — remove only the agent finalizer from the DRBDResource; removing the placement finalizer permanently leaks a PVC/PV pair.

Issue 6: oc get drbdresource against the quorum cluster returns an error

Symptom: server doesn't have a resource type 'drbdresource' when querying the quorum cluster.

Cause: DRBDResource CRDs are registered only on workload clusters, not on the quorum cluster. This is by design.

Fix: Run get dres against the appropriate workload cluster kubeconfig:

oc --kubeconfig=<primary-kubeconfig> -n <vm-namespace> get dres

Issue 7: TSR web console page shows no data for a zone

Symptom: After navigating to a zone's page in the web console, the page is blank or shows errors.

Cause: Zone-scoped console pages derive the dr-<zoneId> namespace from the zone ID in the URL. If the zone lives in a differently named namespace, the page cannot resolve its data.

Fix: Verify the zone namespace matches the dr-<zoneName> pattern:

oc --kubeconfig=<quorum-kubeconfig> get pz -A

The zone name in metadata.name must match the dr- prefix of its namespace. For example, a zone named prod must live in dr-prod.

Issue 8: FailoverRequest stuck in InProgress

Symptom: oc -n dr-prod get fr <name> shows STATE: InProgress for longer than 5 minutes.

Fix:

# Check per-VM statuses
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  get fr <name> -o jsonpath='{.status.vmStatuses}' | jq .

# Check the phase for more detail
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  get fr <name> -o jsonpath='{.status.phase}'

# Check failover-controller logs on the quorum cluster
oc --kubeconfig=<quorum-kubeconfig> -n dr-prod \
  logs -l app.kubernetes.io/component=quorum-control-plane -c manager \
  --since=10m | grep -i failover

# Collect a full bundle for support
oc --kubeconfig=<quorum-kubeconfig> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --zone prod --since 30m

Note: spec.drainTimeoutSeconds is accepted by the API but has no effect on failover behaviour in Tech Preview. Do not adjust it expecting to influence drain timing.