--- title: Getting Started product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: tutorial version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/getting-started --- # Getting Started _Installing the operator and creating your first site recovery custom resource_ ## Overview This page walks you through installing Site Recovery and creating your first protected VM. You will deploy the quorum control plane, install the workload control plane on your primary and DR clusters, and submit your first Custom Resource to bring a VM under DR protection. By the end, you will have a working protection configuration that you can validate with a test failover. ## Prerequisites Before you begin, ensure the following are in place: **Clusters** - Three Kubernetes clusters (quorum, primary, DR) for centralized storage deployments, or two clusters (primary, DR) for DRBD Operator deployments - Kubernetes ≥ 1.28 on all clusters, or OpenShift ≥ 4.14 - KubeVirt ≥ 1.0 installed on the primary and DR clusters **Storage** - DRBD kernel module ≥ 9.0 on all worker nodes - DRBD Operator installed on the primary and DR clusters (DRBD Operator deployment model only) - LVM thin-provisioned storage pools on worker nodes - VolumeSnapshot support (CSI snapshotter) on the primary and DR clusters if you plan to run test failovers **Tooling** - Helm ≥ 3.0 - `kubectl` configured for all three clusters - Ansible (required for Ansible-based deployment playbooks) **Network** - TCP 7000–7999 open between primary and DR worker nodes (DRBD replication) - TCP 6443 open from the quorum cluster to both the primary and DR Kubernetes API servers - Round-trip latency < 10 ms between primary and DR clusters if you intend to use Protocol C (synchronous replication); Protocol A (asynchronous) has no latency requirement **Access** - Kubeconfig files with administrative access for each cluster - The following environment variables set in your shell: ```bash export KUBECONFIG_QUORUM=~/.kube/config-quorum export KUBECONFIG_CLUSTER1=~/.kube/config-cluster1 export KUBECONFIG_CLUSTER2=~/.kube/config-cluster2 ``` ## Quick start The steps below follow the **DRBD Operator deployment model** — the fastest path to a working setup. If you are using a centralized storage controller, the sequence is the same but you will use a `ProtectionGroup` instead of a `ProtectionRequest` to protect VMs. 1. Deploy the workload control plane on the primary cluster: ```bash helm install dr-deployment ./helm/site-recovery-workload-control-plane \ --kubeconfig $KUBECONFIG_CLUSTER1 \ -n trilio-site-recovery-system --create-namespace ``` 2. Deploy the workload control plane on the DR cluster: ```bash helm install dr-deployment ./helm/site-recovery-workload-control-plane \ --kubeconfig $KUBECONFIG_CLUSTER2 \ -n trilio-site-recovery-system --create-namespace ``` 3. Deploy the quorum control plane — install the ProtectionZone webhook and CRDs first, then the unified manager: ```bash helm install dr-deployment ./helm/site-recovery-protectionzone-controller \ --kubeconfig $KUBECONFIG_QUORUM \ -n dr-deployment --create-namespace helm install dr-deployment ./helm/site-recovery-quorum-control-plane \ --kubeconfig $KUBECONFIG_QUORUM \ -n dr-deployment --create-namespace ``` 4. Create a `DRBDReplicationPolicy` that maps storage classes between your clusters and sets the replication protocol: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f drbd-replication-policy.yaml ``` 5. Submit a `ProtectionRequest` to bring a VM under DR protection: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f protect-my-vm.yaml ``` 6. Verify that protection has been established: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-my-vm -n dr-deployment -o yaml ``` When the `ProtectionRequest` status shows the VM is fully protected and its volumes are synchronised, your first VM is under DR protection. ## Steps ### Step 1 — Deploy the workload control plane on the primary cluster The workload control plane installs the node agent and admission webhooks that the quorum controllers depend on. Install it in the `trilio-site-recovery-system` namespace, which is reserved for Site Recovery components on workload clusters. ```bash helm install dr-deployment ./helm/site-recovery-workload-control-plane \ --kubeconfig $KUBECONFIG_CLUSTER1 \ -n trilio-site-recovery-system --create-namespace ``` **Success:** `helm list -n trilio-site-recovery-system --kubeconfig $KUBECONFIG_CLUSTER1` shows `dr-deployment` with status `deployed`. --- ### Step 2 — Deploy the workload control plane on the DR cluster Repeat the same installation on the DR cluster. Both clusters must run the workload control plane before the quorum controllers can manage them. ```bash helm install dr-deployment ./helm/site-recovery-workload-control-plane \ --kubeconfig $KUBECONFIG_CLUSTER2 \ -n trilio-site-recovery-system --create-namespace ``` **Success:** The same check on `$KUBECONFIG_CLUSTER2` shows `deployed`. --- ### Step 3 — Deploy the quorum control plane The quorum cluster hosts the `protection-controller`, `failover-controller`, `pg-sync-controller`, and the Site Manager UI. The ProtectionZone webhook and CRDs must be installed first because the unified manager references them at startup. ```bash # Install the ProtectionZone webhook and CRDs helm install dr-deployment ./helm/site-recovery-protectionzone-controller \ --kubeconfig $KUBECONFIG_QUORUM \ -n dr-deployment --create-namespace # Install the unified quorum manager helm install dr-deployment ./helm/site-recovery-quorum-control-plane \ --kubeconfig $KUBECONFIG_QUORUM \ -n dr-deployment --create-namespace ``` The namespace `dr-deployment` is the DR deployment namespace. Site Recovery isolates every deployment in its own `dr-` namespace; all quorum-side resources for this deployment will live here. **Success:** `helm list -n dr-deployment --kubeconfig $KUBECONFIG_QUORUM` shows both releases as `deployed`. --- ### Step 4 — Define a DRBDReplicationPolicy A `DRBDReplicationPolicy` tells the DRBD Operator how to replicate volumes between your clusters. It specifies the replication protocol, the replication endpoints on each cluster, and which storage class on the primary cluster maps to which storage class on the DR cluster. Create a file named `drbd-replication-policy.yaml`: ```yaml apiVersion: siterecovery.trilio.io/v1 kind: DRBDReplicationPolicy metadata: name: cross-cluster-policy namespace: dr-deployment spec: drbdProtocol: C # Protocol C = synchronous, RPO=0; use A for async over high-latency links storageClassMappings: - primaryStorageClass: ocs-storagecluster-ceph-rbd drStorageClass: ocs-storagecluster-ceph-rbd primaryCluster: name: cluster1 replicationEndpoint: "10.0.0.1:7000" drCluster: name: cluster2 replicationEndpoint: "10.0.0.2:7000" ``` Replace the storage class names and IP addresses with values from your environment. ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f drbd-replication-policy.yaml ``` **Success:** `kubectl --kubeconfig $KUBECONFIG_QUORUM get drbdreplicationpolicy -n dr-deployment` shows `cross-cluster-policy`. --- ### Step 5 — Protect a VM with a ProtectionRequest A `ProtectionRequest` is the declaration that you want a specific VM to be DR-protected. When the `protection-controller` on the quorum cluster picks it up, it validates the VM, creates a `DRBDVolume` for each of the VM's PVCs, waits for initial synchronisation to complete, and then switches the VM to DRBD-backed frontend PVCs. Create a file named `protect-my-vm.yaml`: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-my-vm namespace: dr-deployment spec: vmName: my-vm vmNamespace: default sourceCluster: cluster1 ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f protect-my-vm.yaml ``` The controller progresses through the following lifecycle: validation → `DRBDVolume` creation → synchronisation → frontend PVC switchover → protected. **Success:** Watch the status until it reports the VM is fully protected: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-my-vm \ -n dr-deployment -w ``` --- ### Step 6 — Verify replication health The `replication-monitor` agent is deployed automatically as part of the standard Ansible playbooks and writes `ReplicationGroupStatus` resources that give you an aggregated health view. ```bash # Check aggregated replication health kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-deployment # Inspect the DRBDVolume created for your VM to see per-volume sync progress kubectl --kubeconfig $KUBECONFIG_QUORUM get drbdvolume -n dr-deployment ``` When the `ReplicationGroupStatus` shows `Healthy` and your `DRBDVolume` shows sync progress at 100%, the VM is fully protected and ready for failover. --- ### Step 7 — (Optional) Run a test failover to validate DR readiness Test failover is supported only in DRBD Operator deployment models. It snapshots your production volumes, boots test VMs on the DR cluster, runs verification checks, and then cleans up — without affecting production workloads. ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: validate-my-vm namespace: dr-deployment spec: protectionGroupRef: name: my-first-pg namespace: default cleanupPolicy: Manual retentionTime: 2h EOF # Watch progress kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover validate-my-vm -n dr-deployment -w # When done, clean up test resources kubectl --kubeconfig $KUBECONFIG_QUORUM delete testfailover validate-my-vm -n dr-deployment ``` ## Examples ### Example 1 — DRBDReplicationPolicy with asynchronous replication Use Protocol A when your primary and DR clusters are separated by more than 50 ms round-trip latency. Writes are acknowledged after committing to local disk; data is shipped to the DR cluster in the background. This accepts a small RPO window in exchange for reduced write latency. ```yaml apiVersion: siterecovery.trilio.io/v1 kind: DRBDReplicationPolicy metadata: name: async-policy namespace: dr-deployment spec: drbdProtocol: A storageClassMappings: - primaryStorageClass: local-storage drStorageClass: local-storage-dr primaryCluster: name: cluster1 replicationEndpoint: "10.0.0.1:7000" drCluster: name: cluster2 replicationEndpoint: "10.0.0.2:7000" ``` --- ### Example 2 — ProtectionRequest for a VM with multiple disks The `protection-controller` creates one `DRBDVolume` per PVC that the VM owns. You only need to reference the VM — the controller discovers its volumes automatically. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-db-vm namespace: dr-deployment spec: vmName: database-primary vmNamespace: production sourceCluster: cluster1 ``` After applying, check the `DRBDVolume` resources to see one entry per disk: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get drbdvolume -n dr-deployment ``` Expected output (one row per PVC): ``` NAME SYNC PROGRESS FRONTEND PVC AGE protect-db-vm-data-disk 100% database-primary-data-drbd 5m protect-db-vm-log-disk 100% database-primary-log-drbd 5m ``` --- ### Example 3 — FailoverRequest for a planned failover A planned failover gracefully stops VMs on the primary cluster before promoting volumes on the DR cluster, guaranteeing zero data loss. Submit the `FailoverRequest` on the quorum cluster; the `failover-controller` orchestrates the entire sequence. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: planned-failover-pg1 namespace: dr-deployment spec: protectionGroupRef: name: my-first-pg targetCluster: cluster2 failoverType: planned ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f planned-failover.yaml # Watch the operation progress kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest planned-failover-pg1 \ -n dr-deployment -w ``` --- ### Example 4 — Listing all protected resources across the deployment Use these commands to get a quick health snapshot of your deployment: ```bash # ProtectionRequests on the quorum cluster kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest -n dr-deployment # DRBDVolumes showing per-volume sync state kubectl --kubeconfig $KUBECONFIG_QUORUM get drbdvolume -n dr-deployment # ReplicationGroupStatus for aggregated health kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-deployment # Any RPO violations recorded by the replication-monitor kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-deployment ``` ## Troubleshooting ### ProtectionRequest stays in `Validating` phase **Symptom:** The `ProtectionRequest` does not advance beyond `Validating` after several minutes. **Likely causes:** - The VM named in `spec.vmName` does not exist in `spec.vmNamespace` on the source cluster. - The `protection-controller` on the quorum cluster cannot reach the source cluster's Kubernetes API (TCP 6443). - The workload control plane is not installed on the source cluster. **Fix:** 1. Confirm the VM exists: `kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get vm -n ` 2. Confirm the quorum can reach the source API server and that TCP 6443 is open. 3. Confirm the workload control plane is deployed: `helm list -n trilio-site-recovery-system --kubeconfig $KUBECONFIG_CLUSTER1` 4. Check the `protection-controller` logs on the quorum cluster: `kubectl --kubeconfig $KUBECONFIG_QUORUM logs -l app=protection-controller -n dr-deployment` --- ### DRBDVolume sync progress is stuck below 100% **Symptom:** `kubectl get drbdvolume -n dr-deployment` shows sync progress stalled at a value below 100% for an extended period. **Likely causes:** - TCP 7000–7999 is blocked between primary and DR worker nodes. - The DRBD kernel module is not loaded on one or more worker nodes. - Network latency exceeds 50 ms and Protocol C is configured — the volume pair may be experiencing repeated resync. **Fix:** 1. Verify port connectivity: from a DR worker node, run `nc -zv 7000`. 2. On each worker node, confirm the DRBD module is loaded: `lsmod | grep drbd`. 3. If latency is high, consider switching the `DRBDReplicationPolicy` to `drbdProtocol: A`. 4. Check `RPOEvent` resources for recorded lag violations: `kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-deployment` --- ### Helm install fails for the quorum control plane **Symptom:** `helm install dr-deployment ./helm/site-recovery-quorum-control-plane` fails with a webhook or CRD not found error. **Likely cause:** The ProtectionZone webhook and CRDs were not installed before the unified manager. The manager references CRDs that do not yet exist in the cluster. **Fix:** Install releases in the correct order: ```bash # Step 1 — webhook and CRDs helm install dr-deployment ./helm/site-recovery-protectionzone-controller \ --kubeconfig $KUBECONFIG_QUORUM -n dr-deployment --create-namespace # Step 2 — unified manager (only after step 1 completes successfully) helm install dr-deployment ./helm/site-recovery-quorum-control-plane \ --kubeconfig $KUBECONFIG_QUORUM -n dr-deployment --create-namespace ``` --- ### ReplicationGroupStatus shows `Degraded` or `Critical` **Symptom:** `kubectl get replicationgroupstatus -n dr-deployment` reports a health status of `Degraded` or `Critical`. **Likely cause:** One or more volumes in the Protection Group have fallen behind or lost their replication connection. The `replication-monitor` records details in `RPOEvent` resources. **Fix:** 1. Inspect the `ReplicationGroupStatus` for per-volume detail: `kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-deployment -o yaml` 2. Check `RPOEvent` resources to see which volumes violated their RPO threshold and the recorded severity. 3. If network connectivity between clusters has been interrupted and both sides resumed as primary, you may have a split-brain condition. Do not trigger failover until the split-brain is resolved manually — promoting a diverged volume will cause data loss. 4. After resolving the underlying cause, verify that sync progress returns to 100% on all `DRBDVolume` resources before proceeding with any failover operations. --- ### Test failover fails at `CreatingSnapshots` phase **Symptom:** A `TestFailover` resource stalls or moves to `Failed` during the `CreatingSnapshots` phase. **Likely cause:** VolumeSnapshot support (CSI snapshotter) is not installed on the cluster, or the storage class used by the VM's PVCs does not support snapshots. **Fix:** 1. Confirm the CSI snapshotter is installed on the primary cluster: `kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get crd volumesnapshots.snapshot.storage.k8s.io` 2. Confirm that the storage class used by the VM's PVCs has a matching `VolumeSnapshotClass`. 3. Check the `test-failover-controller` logs on the quorum cluster for the specific error: `kubectl --kubeconfig $KUBECONFIG_QUORUM logs -l app=test-failover-controller -n dr-deployment` > **Note:** Test failover is only supported in DRBD Operator deployment models.