Trilio Site Recovery for Kubernetes
Tutorial

Getting Started

Installing the operator and creating your first site recovery resource


Overview

This page walks you through installing Trilio Site Recovery on OpenShift and creating your first protected VM workload. By the end, you will have the quorum control plane running on your management cluster, the workload control plane running on your primary and DR clusters, and a ProtectionRequest submitted to begin block-level replication of a virtual machine—all driven by Kubernetes custom resources with no application-layer changes required.


Prerequisites

Before you begin, confirm the following are in place across all three clusters:

Clusters and platforms

  • Three OpenShift clusters at version 4.14 or later: one quorum (management) cluster, one primary cluster, and one DR cluster
  • OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary and DR clusters; the quorum cluster does not run VMs
  • KubeVirt virtual machines already defined on the primary cluster

Storage and kernel

  • DRBD kernel module ≥ 9.x loaded on every worker node of the primary and DR clusters
  • DRBD Operator installed on the primary and DR clusters
  • Containerized Data Importer (CDI) installed on the primary and DR clusters

Networking

  • TCP ports 7000–7999 open between every primary worker node and every DR worker node (DRBD replication traffic)
  • TCP port 6443 open from the quorum cluster to the primary and DR API servers (Kubernetes API)
  • For synchronous replication (Protocol C, RPO=0): round-trip latency below 50 ms between primary and DR worker nodes

Tooling

  • Helm ≥ 3.0 installed on your workstation
  • oc CLI available and authenticated to all three clusters
  • A valid kubeconfig file for each cluster

Access

  • Cluster-admin privileges on all three clusters (the operator requires broad RBAC to watch and manage resources across namespaces)
  • The Trilio Site Recovery Helm chart repository and image registry credentials provided by TrilioData

Quick start

The steps below take you from a clean OpenShift environment to a protected VM in the shortest path. Each step is described in full detail in the Steps section.

  1. Export kubeconfig paths for all three clusters.
  2. Install the ProtectionZone webhook on the quorum cluster.
  3. Install the quorum control plane on the quorum cluster.
  4. Install the workload control plane on the primary cluster.
  5. Install the workload control plane on the DR cluster.
  6. Create a ProtectionZone custom resource to register your clusters.
  7. Create a DRBDReplicationPolicy to define cross-cluster replication parameters.
  8. Submit a ProtectionRequest to protect your first VM.
  9. Verify that replication is healthy.
# Step 1 — export kubeconfigs
export KUBECONFIG_QUORUM=~/.kube/config-quorum
export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr

# Step 2 — ProtectionZone webhook (must precede the quorum control plane)
helm install site-recovery-pz \
  oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
  --kubeconfig $KUBECONFIG_QUORUM \
  -n dr-prod --create-namespace

# Step 3 — quorum control plane
helm install site-recovery-quorum \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --kubeconfig $KUBECONFIG_QUORUM \
  -n dr-prod

# Step 4 — workload control plane on primary
helm install site-recovery-workload \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --kubeconfig $KUBECONFIG_PRIMARY \
  -n trilio-site-recovery-system --create-namespace

# Step 5 — workload control plane on DR
helm install site-recovery-workload \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --kubeconfig $KUBECONFIG_DR \
  -n trilio-site-recovery-system --create-namespace

# Steps 6-9 — see the Steps section for the CRD manifests

Steps

Step 1 — Set kubeconfig environment variables

All Site Recovery operations target specific clusters by kubeconfig context. Set these variables once in your shell session; every subsequent command references them.

export KUBECONFIG_QUORUM=~/.kube/config-quorum
export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr

Verify access to all three clusters before proceeding:

oc --kubeconfig $KUBECONFIG_QUORUM get nodes
oc --kubeconfig $KUBECONFIG_PRIMARY get nodes
oc --kubeconfig $KUBECONFIG_DR get nodes

Each command should return a list of Ready nodes. If any cluster is unreachable, resolve connectivity before continuing.


Step 2 — Install the ProtectionZone webhook on the quorum cluster

The site-recovery-protectionzone-controller chart installs the ProtectionZone admission webhook and CRDs. This chart must be installed before the quorum control plane chart, because the quorum control plane's admission webhooks depend on the ProtectionZone CRD being present.

helm install site-recovery-pz \
  oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
  --kubeconfig $KUBECONFIG_QUORUM \
  -n dr-prod --create-namespace

Confirm the webhook pod is running:

oc --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod

You should see the ProtectionZone controller pod in Running state before proceeding.


Step 3 — Install the quorum control plane

The site-recovery-quorum-control-plane chart deploys the unified Go controller manager to the quorum cluster. This single deployment packages the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers, along with admission webhooks, into one Deployment per DR namespace.

helm install site-recovery-quorum \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --kubeconfig $KUBECONFIG_QUORUM \
  -n dr-prod

Verify the controller manager is healthy:

oc --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod
oc --kubeconfig $KUBECONFIG_QUORUM get crds | grep siterecovery.trilio.io

You should see the controller manager pod in Running state and all Site Recovery CRDs registered.


Step 4 — Install the workload control plane on the primary cluster

The site-recovery-workload-control-plane chart installs the drbd-node-agent DaemonSet, admission webhooks, and supporting RBAC on the primary cluster. The drbd-node-agent runs on every worker node and manages DRBD kernel-level block replication for individual VM volumes.

helm install site-recovery-workload \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --kubeconfig $KUBECONFIG_PRIMARY \
  -n trilio-site-recovery-system --create-namespace

Verify the DaemonSet has a pod on every worker node:

oc --kubeconfig $KUBECONFIG_PRIMARY get daemonset -n trilio-site-recovery-system

The DESIRED and READY counts should match the number of worker nodes.


Step 5 — Install the workload control plane on the DR cluster

Repeat the workload control plane installation on the DR cluster. This is identical to Step 4 but targets your DR kubeconfig.

helm install site-recovery-workload \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --kubeconfig $KUBECONFIG_DR \
  -n trilio-site-recovery-system --create-namespace

Verify the same way:

oc --kubeconfig $KUBECONFIG_DR get daemonset -n trilio-site-recovery-system

Step 6 — Register your clusters with a ProtectionZone

A ProtectionZone CR on the quorum cluster binds your primary and DR clusters into a named DR deployment. The storageBackendMode field selects the DRBD Operator model. The displayName and clusters array are required fields.

Create a file named protectionzone.yaml:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod-zone
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: drbd-operator
  description: "Primary to DR replication for production VMs"
  clusters:
    - name: primary-cluster
      role: primary
      kubeConfigSecretRef:
        name: primary-cluster-kubeconfig
    - name: dr-cluster
      role: dr
      kubeConfigSecretRef:
        name: dr-cluster-kubeconfig

Apply it to the quorum cluster:

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionzone.yaml

Verify the ProtectionZone was accepted and its status reflects both clusters:

oc --kubeconfig $KUBECONFIG_QUORUM get protectionzone prod-zone -n dr-prod -o yaml

Note: Before applying the ProtectionZone, create the kubeconfig Secrets referenced under kubeConfigSecretRef in the dr-prod namespace on the quorum cluster. Each Secret should contain the kubeconfig for the respective workload cluster. Follow your organization's secret management practices; Site Recovery never writes these credentials to diagnostic bundles.


Step 7 — Create a DRBDReplicationPolicy

A DRBDReplicationPolicy defines how volumes replicate between your primary and DR clusters: which storage classes to map, the DRBD replication endpoints on each side, and whether to use synchronous (Protocol C, RPO=0) or asynchronous (Protocol A) replication.

Choose Protocol C if your primary-to-DR round-trip latency is below 50 ms. Choose Protocol A for longer distances where some RPO (typically seconds) is acceptable.

Create a file named replication-policy.yaml:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: prod-replication-policy
  namespace: dr-prod
spec:
  drbdProtocol: C
  clusters:
    - name: primary-cluster
      replicationEndpoint: "10.0.1.10:7000"
    - name: dr-cluster
      replicationEndpoint: "10.0.2.10:7000"
  diskConfig:
    storageClassMappings:
      - primaryStorageClass: ocs-storagecluster-ceph-rbd
        drStorageClass: ocs-storagecluster-ceph-rbd
  isDefault: true

Apply it to the quorum cluster:

oc --kubeconfig $KUBECONFIG_QUORUM apply -f replication-policy.yaml

Verify the policy was created:

oc --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy prod-replication-policy -n dr-prod -o yaml

Replace the replicationEndpoint IP addresses with the actual IPs of your DRBD-capable worker nodes. Ensure TCP ports 7000–7999 are open between these endpoints at the firewall level before submitting the first ProtectionRequest.


Step 8 — Protect your first VM

A ProtectionRequest tells the protection-controller on the quorum cluster to validate the named VM, create a DRBDResource pair covering all of the VM's disks, and switch the VM's PVCs to DRBD-backed frontend PVCs that replicate to the DR cluster.

The spec.virtualMachine and spec.sourceCluster fields are required.

Create a file named protect-vm.yaml:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-my-vm
  namespace: dr-prod
spec:
  sourceCluster: primary-cluster
  virtualMachine:
    name: my-vm
    namespace: default
  replicationConfig:
    policyRef:
      name: prod-replication-policy

Apply it to the quorum cluster:

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-vm.yaml

Watch the ProtectionRequest move through its phases:

oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-my-vm -n dr-prod -w

The protection-controller progresses the CR through Pending → Validating → Provisioning → Protected. When the status shows Protected, the VM's disks are actively replicating to the DR cluster.


Step 9 — Verify replication health

Once protection is active, confirm that the DRBDResource CRs on both clusters report a healthy sync state.

# Check the DRBDResource on the primary cluster
oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A

# Check the DRBDResource on the DR cluster
oc --kubeconfig $KUBECONFIG_DR get drbdresource -A

# Check the aggregated ReplicationGroupStatus on the quorum cluster
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o yaml

A healthy deployment shows status.health: Healthy on the ReplicationGroupStatus and no RPOEvent CRs with critical severity in the dr-prod namespace. If you see Degraded or Critical, consult the Troubleshooting section.


Examples

Example 1 — Minimal ProtectionRequest for a single VM

This is the smallest valid ProtectionRequest. It uses the default replication policy (isDefault: true) so no explicit replicationConfig.policyRef is needed, and targets a VM named web-frontend in the production namespace.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-frontend
  namespace: dr-prod
spec:
  sourceCluster: primary-cluster
  virtualMachine:
    name: web-frontend
    namespace: production
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-web-frontend.yaml

Expected output after a few minutes:

NAME                   STATUS      AGE
protect-web-frontend   Protected   4m12s

Example 2 — DRBDReplicationPolicy with asynchronous replication

Use Protocol A when your DR cluster is geographically distant and sub-50 ms latency is not achievable. This accepts a small RPO (typically seconds) in exchange for unlimited replication distance.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: async-replication-policy
  namespace: dr-prod
spec:
  drbdProtocol: A
  clusters:
    - name: primary-cluster
      replicationEndpoint: "10.0.1.10:7000"
    - name: dr-cluster
      replicationEndpoint: "203.0.113.50:7000"
  diskConfig:
    storageClassMappings:
      - primaryStorageClass: ocs-storagecluster-ceph-rbd
        drStorageClass: ocs-storagecluster-ceph-rbd
  isDefault: false

Expected output:

oc --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy -n dr-prod
NAME                       AGE
async-replication-policy   23s
prod-replication-policy    8m

Example 3 — Non-disruptive test failover

A TestFailover CR validates your DR readiness without touching production workloads. The test-failover-controller creates snapshot-based copies of the VM's volumes on the DR cluster in an isolated namespace, starts the test VMs, runs verification checks, and then cleans up.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: test-web-frontend
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: web-frontend-pg
    namespace: dr-prod
  cleanupPolicy: Automatic
  retentionTime: 1h
  batchBootTimeoutSeconds: 300
oc --kubeconfig $KUBECONFIG_QUORUM apply -f test-failover.yaml

# Watch progress through phases
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover test-web-frontend -n dr-prod -w

Expected phase progression:

NAME                 PHASE               AGE
test-web-frontend    CreatingSnapshots   0m10s
test-web-frontend    CreatingVolumes     0m45s
test-web-frontend    CreatingVMs         1m20s
test-web-frontend    VerifyingData       2m05s
test-web-frontend    Succeeded           2m50s
test-web-frontend    CleaningUp          3m10s

With cleanupPolicy: Automatic, all test resources are removed after the retentionTime expires. Switch to cleanupPolicy: Manual if you want to inspect the test VMs before cleanup.


Example 4 — Checking replication health with kubectl

# Aggregated health for all Protection Groups
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod

# Detailed view including per-volume sync state and last-sync timestamp
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o yaml

# Look for any RPO violation events
oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod

Example healthy output:

NAME                       HEALTH    LAST-SYNC
web-frontend-pg-status     Healthy   2024-01-15T10:32:01Z

Example RPO event (warning):

NAME                       TYPE        SEVERITY   AGE
web-frontend-rpo-evt-001   LagSpike    warning    5m

An RPO event at warning severity indicates a transient lag spike. A critical severity event requires immediate investigation of the DRBD replication link.


Troubleshooting

Use the following format for each issue: Symptom → Likely cause → Fix.


ProtectionRequest stays in Pending and does not advance

Symptom: oc get protectionrequest -n dr-prod shows STATUS: Pending for more than two minutes after creation.

Likely cause: The protection-controller on the quorum cluster cannot reach the primary cluster's API server, or the ProtectionZone CR is misconfigured or not yet reconciled.

Fix:

  1. Check the quorum control plane logs for reconciliation errors:
    oc --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \
      -l app.kubernetes.io/component=site-recovery-quorum-control-plane --tail=100
    
  2. Verify the ProtectionZone status and confirm both clusters are reachable:
    oc --kubeconfig $KUBECONFIG_QUORUM get protectionzone -n dr-prod -o yaml
    
  3. Confirm the kubeconfig Secrets referenced in the ProtectionZone exist in the dr-prod namespace:
    oc --kubeconfig $KUBECONFIG_QUORUM get secrets -n dr-prod
    

ProtectionRequest reaches Provisioning and then transitions to Failed

Symptom: The ProtectionRequest status shows Failed with an error message referencing DRBDResource creation.

Likely cause: The DRBD node agent DaemonSet on the primary or DR cluster is not running on all nodes, or TCP ports 7000–7999 are blocked between the clusters.

Fix:

  1. Verify the drbd-node-agent DaemonSet is fully deployed on both clusters:
    oc --kubeconfig $KUBECONFIG_PRIMARY get daemonset -n trilio-site-recovery-system
    oc --kubeconfig $KUBECONFIG_DR get daemonset -n trilio-site-recovery-system
    
    The DESIRED and READY counts must match. If they do not, check individual pod logs:
    oc --kubeconfig $KUBECONFIG_PRIMARY logs -n trilio-site-recovery-system \
      -l app.kubernetes.io/component=drbd-node-agent --tail=50
    
  2. From a worker node on the primary cluster, test TCP connectivity to a DR worker node on a DRBD port:
    nc -zv <dr-worker-ip> 7000
    
    If the connection is refused or times out, open ports 7000–7999 at your network firewall.

ReplicationGroupStatus shows Degraded or Critical

Symptom: oc get replicationgroupstatus -n dr-prod shows HEALTH: Degraded or HEALTH: Critical, and RPOEvent CRs are present.

Likely cause: The DRBD replication link between primary and DR clusters is experiencing packet loss, bandwidth saturation, or has gone down entirely. For Protocol C (synchronous), even a brief link interruption will degrade replication state.

Fix:

  1. Inspect the RPOEvent CRs for details on which volumes are affected and the observed lag:
    oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod -o yaml
    
  2. Check individual DRBDResource status on both clusters:
    oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A -o wide
    oc --kubeconfig $KUBECONFIG_DR get drbdresource -A -o wide
    
  3. Review drbd-node-agent logs on affected worker nodes for connection state messages.
  4. Once the network issue is resolved, DRBD will automatically begin resynchronizing. Monitor ReplicationGroupStatus until it returns to Healthy.

Helm install of the quorum control plane fails with a webhook admission error

Symptom: helm install site-recovery-quorum exits with an error such as failed calling webhook: protectionzone.siterecovery.trilio.io.

Likely cause: The site-recovery-protectionzone-controller chart was not installed first, or its webhook pod is not yet ready.

Fix:

  1. Confirm the ProtectionZone controller is installed and its pod is in Running state:
    oc --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod
    
  2. If the pod is present but not yet ready, wait for it to become ready and retry:
    oc --kubeconfig $KUBECONFIG_QUORUM wait --for=condition=Ready pod \
      -l app.kubernetes.io/component=site-recovery-protectionzone-controller \
      -n dr-prod --timeout=120s
    
  3. Then re-run the quorum control plane install.

TestFailover stays in CreatingSnapshots for a long time

Symptom: A TestFailover CR has been in CreatingSnapshots phase for more than five minutes.

Likely cause: The VolumeSnapshot class is not configured on the DR cluster, or CDI is not installed.

Fix:

  1. Confirm CDI is installed on the DR cluster:
    oc --kubeconfig $KUBECONFIG_DR get pods -n cdi
    
  2. Confirm that a VolumeSnapshotClass is available on the DR cluster:
    oc --kubeconfig $KUBECONFIG_DR get volumesnapshotclass
    
  3. Check the test-failover-controller logs on the quorum cluster for snapshot errors:
    oc --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \
      -l app.kubernetes.io/component=site-recovery-quorum-control-plane --tail=100 \
      | grep -i snapshot
    

Collecting a diagnostic bundle for support

If you cannot resolve an issue with the steps above, collect a full diagnostic bundle using the tsr-gather must-gather tool. Run it against the quorum cluster; it automatically collects logs and CRD state from all clusters in the DR estate.

oc adm must-gather \
  --image=registry.trilio.io/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

The bundle never includes secrets, kubeconfigs, cloud-init payloads, or DRBD shared secrets. Attach the resulting archive when opening a support case with TrilioData.