Getting Started
Installing the operator and creating your first site recovery resource
This page walks you through installing Trilio Site Recovery on OpenShift and creating your first protected VM workload. By the end, you will have the quorum control plane running on your management cluster, the workload control plane running on your primary and DR clusters, and a ProtectionRequest submitted to begin block-level replication of a virtual machine—all driven by Kubernetes custom resources with no application-layer changes required.
Before you begin, confirm the following are in place across all three clusters:
Clusters and platforms
- Three OpenShift clusters at version 4.14 or later: one quorum (management) cluster, one primary cluster, and one DR cluster
- OpenShift Virtualization (CNV) ≥ 1.0 installed on the primary and DR clusters; the quorum cluster does not run VMs
- KubeVirt virtual machines already defined on the primary cluster
Storage and kernel
- DRBD kernel module ≥ 9.x loaded on every worker node of the primary and DR clusters
- DRBD Operator installed on the primary and DR clusters
- Containerized Data Importer (CDI) installed on the primary and DR clusters
Networking
- TCP ports 7000–7999 open between every primary worker node and every DR worker node (DRBD replication traffic)
- TCP port 6443 open from the quorum cluster to the primary and DR API servers (Kubernetes API)
- For synchronous replication (Protocol C, RPO=0): round-trip latency below 50 ms between primary and DR worker nodes
Tooling
- Helm ≥ 3.0 installed on your workstation
ocCLI available and authenticated to all three clusters- A valid kubeconfig file for each cluster
Access
- Cluster-admin privileges on all three clusters (the operator requires broad RBAC to watch and manage resources across namespaces)
- The Trilio Site Recovery Helm chart repository and image registry credentials provided by TrilioData
The steps below take you from a clean OpenShift environment to a protected VM in the shortest path. Each step is described in full detail in the Steps section.
- Export kubeconfig paths for all three clusters.
- Install the ProtectionZone webhook on the quorum cluster.
- Install the quorum control plane on the quorum cluster.
- Install the workload control plane on the primary cluster.
- Install the workload control plane on the DR cluster.
- Create a
ProtectionZonecustom resource to register your clusters. - Create a
DRBDReplicationPolicyto define cross-cluster replication parameters. - Submit a
ProtectionRequestto protect your first VM. - Verify that replication is healthy.
# Step 1 — export kubeconfigs
export KUBECONFIG_QUORUM=~/.kube/config-quorum
export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr
# Step 2 — ProtectionZone webhook (must precede the quorum control plane)
helm install site-recovery-pz \
oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace
# Step 3 — quorum control plane
helm install site-recovery-quorum \
oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod
# Step 4 — workload control plane on primary
helm install site-recovery-workload \
oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_PRIMARY \
-n trilio-site-recovery-system --create-namespace
# Step 5 — workload control plane on DR
helm install site-recovery-workload \
oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_DR \
-n trilio-site-recovery-system --create-namespace
# Steps 6-9 — see the Steps section for the CRD manifests
Step 1 — Set kubeconfig environment variables
All Site Recovery operations target specific clusters by kubeconfig context. Set these variables once in your shell session; every subsequent command references them.
export KUBECONFIG_QUORUM=~/.kube/config-quorum
export KUBECONFIG_PRIMARY=~/.kube/config-primary
export KUBECONFIG_DR=~/.kube/config-dr
Verify access to all three clusters before proceeding:
oc --kubeconfig $KUBECONFIG_QUORUM get nodes
oc --kubeconfig $KUBECONFIG_PRIMARY get nodes
oc --kubeconfig $KUBECONFIG_DR get nodes
Each command should return a list of Ready nodes. If any cluster is unreachable, resolve connectivity before continuing.
Step 2 — Install the ProtectionZone webhook on the quorum cluster
The site-recovery-protectionzone-controller chart installs the ProtectionZone admission webhook and CRDs. This chart must be installed before the quorum control plane chart, because the quorum control plane's admission webhooks depend on the ProtectionZone CRD being present.
helm install site-recovery-pz \
oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace
Confirm the webhook pod is running:
oc --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod
You should see the ProtectionZone controller pod in Running state before proceeding.
Step 3 — Install the quorum control plane
The site-recovery-quorum-control-plane chart deploys the unified Go controller manager to the quorum cluster. This single deployment packages the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers, along with admission webhooks, into one Deployment per DR namespace.
helm install site-recovery-quorum \
oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod
Verify the controller manager is healthy:
oc --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod
oc --kubeconfig $KUBECONFIG_QUORUM get crds | grep siterecovery.trilio.io
You should see the controller manager pod in Running state and all Site Recovery CRDs registered.
Step 4 — Install the workload control plane on the primary cluster
The site-recovery-workload-control-plane chart installs the drbd-node-agent DaemonSet, admission webhooks, and supporting RBAC on the primary cluster. The drbd-node-agent runs on every worker node and manages DRBD kernel-level block replication for individual VM volumes.
helm install site-recovery-workload \
oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_PRIMARY \
-n trilio-site-recovery-system --create-namespace
Verify the DaemonSet has a pod on every worker node:
oc --kubeconfig $KUBECONFIG_PRIMARY get daemonset -n trilio-site-recovery-system
The DESIRED and READY counts should match the number of worker nodes.
Step 5 — Install the workload control plane on the DR cluster
Repeat the workload control plane installation on the DR cluster. This is identical to Step 4 but targets your DR kubeconfig.
helm install site-recovery-workload \
oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_DR \
-n trilio-site-recovery-system --create-namespace
Verify the same way:
oc --kubeconfig $KUBECONFIG_DR get daemonset -n trilio-site-recovery-system
Step 6 — Register your clusters with a ProtectionZone
A ProtectionZone CR on the quorum cluster binds your primary and DR clusters into a named DR deployment. The storageBackendMode field selects the DRBD Operator model. The displayName and clusters array are required fields.
Create a file named protectionzone.yaml:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: prod-zone
namespace: dr-prod
spec:
displayName: "Production DR Zone"
storageBackendMode: drbd-operator
description: "Primary to DR replication for production VMs"
clusters:
- name: primary-cluster
role: primary
kubeConfigSecretRef:
name: primary-cluster-kubeconfig
- name: dr-cluster
role: dr
kubeConfigSecretRef:
name: dr-cluster-kubeconfig
Apply it to the quorum cluster:
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionzone.yaml
Verify the ProtectionZone was accepted and its status reflects both clusters:
oc --kubeconfig $KUBECONFIG_QUORUM get protectionzone prod-zone -n dr-prod -o yaml
Note: Before applying the ProtectionZone, create the kubeconfig Secrets referenced under
kubeConfigSecretRefin thedr-prodnamespace on the quorum cluster. Each Secret should contain the kubeconfig for the respective workload cluster. Follow your organization's secret management practices; Site Recovery never writes these credentials to diagnostic bundles.
Step 7 — Create a DRBDReplicationPolicy
A DRBDReplicationPolicy defines how volumes replicate between your primary and DR clusters: which storage classes to map, the DRBD replication endpoints on each side, and whether to use synchronous (Protocol C, RPO=0) or asynchronous (Protocol A) replication.
Choose Protocol C if your primary-to-DR round-trip latency is below 50 ms. Choose Protocol A for longer distances where some RPO (typically seconds) is acceptable.
Create a file named replication-policy.yaml:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
name: prod-replication-policy
namespace: dr-prod
spec:
drbdProtocol: C
clusters:
- name: primary-cluster
replicationEndpoint: "10.0.1.10:7000"
- name: dr-cluster
replicationEndpoint: "10.0.2.10:7000"
diskConfig:
storageClassMappings:
- primaryStorageClass: ocs-storagecluster-ceph-rbd
drStorageClass: ocs-storagecluster-ceph-rbd
isDefault: true
Apply it to the quorum cluster:
oc --kubeconfig $KUBECONFIG_QUORUM apply -f replication-policy.yaml
Verify the policy was created:
oc --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy prod-replication-policy -n dr-prod -o yaml
Replace the
replicationEndpointIP addresses with the actual IPs of your DRBD-capable worker nodes. Ensure TCP ports 7000–7999 are open between these endpoints at the firewall level before submitting the first ProtectionRequest.
Step 8 — Protect your first VM
A ProtectionRequest tells the protection-controller on the quorum cluster to validate the named VM, create a DRBDResource pair covering all of the VM's disks, and switch the VM's PVCs to DRBD-backed frontend PVCs that replicate to the DR cluster.
The spec.virtualMachine and spec.sourceCluster fields are required.
Create a file named protect-vm.yaml:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: protect-my-vm
namespace: dr-prod
spec:
sourceCluster: primary-cluster
virtualMachine:
name: my-vm
namespace: default
replicationConfig:
policyRef:
name: prod-replication-policy
Apply it to the quorum cluster:
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-vm.yaml
Watch the ProtectionRequest move through its phases:
oc --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-my-vm -n dr-prod -w
The protection-controller progresses the CR through Pending → Validating → Provisioning → Protected. When the status shows Protected, the VM's disks are actively replicating to the DR cluster.
Step 9 — Verify replication health
Once protection is active, confirm that the DRBDResource CRs on both clusters report a healthy sync state.
# Check the DRBDResource on the primary cluster
oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A
# Check the DRBDResource on the DR cluster
oc --kubeconfig $KUBECONFIG_DR get drbdresource -A
# Check the aggregated ReplicationGroupStatus on the quorum cluster
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o yaml
A healthy deployment shows status.health: Healthy on the ReplicationGroupStatus and no RPOEvent CRs with critical severity in the dr-prod namespace. If you see Degraded or Critical, consult the Troubleshooting section.
Example 1 — Minimal ProtectionRequest for a single VM
This is the smallest valid ProtectionRequest. It uses the default replication policy (isDefault: true) so no explicit replicationConfig.policyRef is needed, and targets a VM named web-frontend in the production namespace.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: protect-web-frontend
namespace: dr-prod
spec:
sourceCluster: primary-cluster
virtualMachine:
name: web-frontend
namespace: production
oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-web-frontend.yaml
Expected output after a few minutes:
NAME STATUS AGE
protect-web-frontend Protected 4m12s
Example 2 — DRBDReplicationPolicy with asynchronous replication
Use Protocol A when your DR cluster is geographically distant and sub-50 ms latency is not achievable. This accepts a small RPO (typically seconds) in exchange for unlimited replication distance.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
name: async-replication-policy
namespace: dr-prod
spec:
drbdProtocol: A
clusters:
- name: primary-cluster
replicationEndpoint: "10.0.1.10:7000"
- name: dr-cluster
replicationEndpoint: "203.0.113.50:7000"
diskConfig:
storageClassMappings:
- primaryStorageClass: ocs-storagecluster-ceph-rbd
drStorageClass: ocs-storagecluster-ceph-rbd
isDefault: false
Expected output:
oc --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy -n dr-prod
NAME AGE
async-replication-policy 23s
prod-replication-policy 8m
Example 3 — Non-disruptive test failover
A TestFailover CR validates your DR readiness without touching production workloads. The test-failover-controller creates snapshot-based copies of the VM's volumes on the DR cluster in an isolated namespace, starts the test VMs, runs verification checks, and then cleans up.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
name: test-web-frontend
namespace: dr-prod
spec:
protectionGroupRef:
name: web-frontend-pg
namespace: dr-prod
cleanupPolicy: Automatic
retentionTime: 1h
batchBootTimeoutSeconds: 300
oc --kubeconfig $KUBECONFIG_QUORUM apply -f test-failover.yaml
# Watch progress through phases
oc --kubeconfig $KUBECONFIG_QUORUM get testfailover test-web-frontend -n dr-prod -w
Expected phase progression:
NAME PHASE AGE
test-web-frontend CreatingSnapshots 0m10s
test-web-frontend CreatingVolumes 0m45s
test-web-frontend CreatingVMs 1m20s
test-web-frontend VerifyingData 2m05s
test-web-frontend Succeeded 2m50s
test-web-frontend CleaningUp 3m10s
With cleanupPolicy: Automatic, all test resources are removed after the retentionTime expires. Switch to cleanupPolicy: Manual if you want to inspect the test VMs before cleanup.
Example 4 — Checking replication health with kubectl
# Aggregated health for all Protection Groups
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod
# Detailed view including per-volume sync state and last-sync timestamp
oc --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o yaml
# Look for any RPO violation events
oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod
Example healthy output:
NAME HEALTH LAST-SYNC
web-frontend-pg-status Healthy 2024-01-15T10:32:01Z
Example RPO event (warning):
NAME TYPE SEVERITY AGE
web-frontend-rpo-evt-001 LagSpike warning 5m
An RPO event at warning severity indicates a transient lag spike. A critical severity event requires immediate investigation of the DRBD replication link.
Use the following format for each issue: Symptom → Likely cause → Fix.
ProtectionRequest stays in Pending and does not advance
Symptom: oc get protectionrequest -n dr-prod shows STATUS: Pending for more than two minutes after creation.
Likely cause: The protection-controller on the quorum cluster cannot reach the primary cluster's API server, or the ProtectionZone CR is misconfigured or not yet reconciled.
Fix:
- Check the quorum control plane logs for reconciliation errors:
oc --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \ -l app.kubernetes.io/component=site-recovery-quorum-control-plane --tail=100 - Verify the ProtectionZone status and confirm both clusters are reachable:
oc --kubeconfig $KUBECONFIG_QUORUM get protectionzone -n dr-prod -o yaml - Confirm the kubeconfig Secrets referenced in the ProtectionZone exist in the
dr-prodnamespace:oc --kubeconfig $KUBECONFIG_QUORUM get secrets -n dr-prod
ProtectionRequest reaches Provisioning and then transitions to Failed
Symptom: The ProtectionRequest status shows Failed with an error message referencing DRBDResource creation.
Likely cause: The DRBD node agent DaemonSet on the primary or DR cluster is not running on all nodes, or TCP ports 7000–7999 are blocked between the clusters.
Fix:
- Verify the drbd-node-agent DaemonSet is fully deployed on both clusters:
The
oc --kubeconfig $KUBECONFIG_PRIMARY get daemonset -n trilio-site-recovery-system oc --kubeconfig $KUBECONFIG_DR get daemonset -n trilio-site-recovery-systemDESIREDandREADYcounts must match. If they do not, check individual pod logs:oc --kubeconfig $KUBECONFIG_PRIMARY logs -n trilio-site-recovery-system \ -l app.kubernetes.io/component=drbd-node-agent --tail=50 - From a worker node on the primary cluster, test TCP connectivity to a DR worker node on a DRBD port:
If the connection is refused or times out, open ports 7000–7999 at your network firewall.
nc -zv <dr-worker-ip> 7000
ReplicationGroupStatus shows Degraded or Critical
Symptom: oc get replicationgroupstatus -n dr-prod shows HEALTH: Degraded or HEALTH: Critical, and RPOEvent CRs are present.
Likely cause: The DRBD replication link between primary and DR clusters is experiencing packet loss, bandwidth saturation, or has gone down entirely. For Protocol C (synchronous), even a brief link interruption will degrade replication state.
Fix:
- Inspect the RPOEvent CRs for details on which volumes are affected and the observed lag:
oc --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod -o yaml - Check individual DRBDResource status on both clusters:
oc --kubeconfig $KUBECONFIG_PRIMARY get drbdresource -A -o wide oc --kubeconfig $KUBECONFIG_DR get drbdresource -A -o wide - Review drbd-node-agent logs on affected worker nodes for connection state messages.
- Once the network issue is resolved, DRBD will automatically begin resynchronizing. Monitor
ReplicationGroupStatusuntil it returns toHealthy.
Helm install of the quorum control plane fails with a webhook admission error
Symptom: helm install site-recovery-quorum exits with an error such as failed calling webhook: protectionzone.siterecovery.trilio.io.
Likely cause: The site-recovery-protectionzone-controller chart was not installed first, or its webhook pod is not yet ready.
Fix:
- Confirm the ProtectionZone controller is installed and its pod is in
Runningstate:oc --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod - If the pod is present but not yet ready, wait for it to become ready and retry:
oc --kubeconfig $KUBECONFIG_QUORUM wait --for=condition=Ready pod \ -l app.kubernetes.io/component=site-recovery-protectionzone-controller \ -n dr-prod --timeout=120s - Then re-run the quorum control plane install.
TestFailover stays in CreatingSnapshots for a long time
Symptom: A TestFailover CR has been in CreatingSnapshots phase for more than five minutes.
Likely cause: The VolumeSnapshot class is not configured on the DR cluster, or CDI is not installed.
Fix:
- Confirm CDI is installed on the DR cluster:
oc --kubeconfig $KUBECONFIG_DR get pods -n cdi - Confirm that a VolumeSnapshotClass is available on the DR cluster:
oc --kubeconfig $KUBECONFIG_DR get volumesnapshotclass - Check the test-failover-controller logs on the quorum cluster for snapshot errors:
oc --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod \ -l app.kubernetes.io/component=site-recovery-quorum-control-plane --tail=100 \ | grep -i snapshot
Collecting a diagnostic bundle for support
If you cannot resolve an issue with the steps above, collect a full diagnostic bundle using the tsr-gather must-gather tool. Run it against the quorum cluster; it automatically collects logs and CRD state from all clusters in the DR estate.
oc adm must-gather \
--image=registry.trilio.io/site-recovery-must-gather:<tag> \
-- /usr/bin/gather --since 2h
The bundle never includes secrets, kubeconfigs, cloud-init payloads, or DRBD shared secrets. Attach the resulting archive when opening a support case with TrilioData.