--- title: Creating Your First Resource product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/first-resource --- # Creating Your First Resource _Step-by-step walkthrough of defining and triggering a site recovery workflow_ ## Overview This guide walks you through defining and triggering a complete site recovery workflow from scratch. You will create the core custom resources that Site Recovery uses — a Protection Group or ProtectionRequest to declare what to protect, and a FailoverRequest to trigger recovery — and then observe each step as the controllers reconcile your intent into action. By the end, you will have a working DR configuration that you can validate with a test failover and extend to production use. ## Prerequisites Before you begin, confirm you have the following in place: **Clusters** - Three Kubernetes clusters (≥ 1.28) or OpenShift clusters (≥ 4.14): one quorum cluster, one primary cluster, and one DR cluster - For the DRBD Operator deployment model, a two-cluster setup (primary + DR) is also supported, with the quorum cluster optional but recommended **Software on each cluster** - KubeVirt ≥ 1.0 installed on the primary and DR clusters - DRBD kernel module ≥ 9.0 on all worker nodes that will run replicated volumes - DRBD Operator installed on primary and DR clusters (DRBD Operator deployment model only) - Helm ≥ 3.0 and `kubectl` available on your workstation **Storage** - LVM thin-provisioned storage pools on worker nodes - VolumeSnapshot support (CSI snapshotter) on both clusters if you plan to run test failovers **Networking** - TCP 7000–7999 open between primary and DR worker nodes (DRBD replication traffic) - TCP 6443 open from the quorum cluster to the primary and DR cluster API servers - Round-trip latency < 10 ms between primary and DR nodes for Protocol C (synchronous replication); Protocol A (asynchronous) has no latency requirement **Credentials** - Kubeconfig files with administrative access for each cluster, referenced as: - `$KUBECONFIG_QUORUM` — quorum cluster - `$KUBECONFIG_CLUSTER1` — primary cluster - `$KUBECONFIG_CLUSTER2` — DR cluster **Site Recovery control plane** - Site Recovery Helm charts deployed on the quorum, primary, and DR clusters (see your deployment guide for Ansible playbook or Helm instructions) - The `pgctl` CLI available on your workstation ## Installation This page assumes the Site Recovery control plane is already deployed via the standard Ansible playbooks or Helm charts. If you have not yet deployed the control plane, complete that step first. The instructions below focus on creating your first protection resource. **Step 1: Export kubeconfig paths** Set environment variables so every subsequent command targets the correct cluster without ambiguity: ```bash export KUBECONFIG_QUORUM=~/.kube/config-quorum export KUBECONFIG_CLUSTER1=~/.kube/config-cluster1 export KUBECONFIG_CLUSTER2=~/.kube/config-cluster2 ``` **Step 2: Verify the controllers are running** Confirm the quorum-side controllers are healthy before creating any resources: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get pods -n dr-prod ``` You should see pods for the `failover-controller`, `protection-controller`, `pg-sync-controller`, and the Site Manager API and UI. All pods should be in `Running` state. Also confirm the workload-side controllers are running on both the primary and DR clusters: ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get pods -n trilio-site-recovery-system kubectl --kubeconfig $KUBECONFIG_CLUSTER2 get pods -n trilio-site-recovery-system ``` You should see `protection-group-controller` and `test-failover-controller` pods in `Running` state on each cluster. **Step 3: Choose a deployment model** Site Recovery supports two deployment models. Choose the one that matches your infrastructure: | Model | Primary CRD | When to use | |---|---|---| | Centralized storage | `ProtectionGroup` | Shared storage controller, three-cluster topology | | DRBD Operator | `ProtectionRequest` + `DRBDReplicationPolicy` | Distributed DRBD Operator, two- or three-cluster topology | Continue with the section that matches your model. ## Configuration The key configuration decisions you make when creating your first resource are: which VMs to protect, which replication protocol to use, and how storage classes map between clusters. **Replication protocol** | Setting | Protocol C | Protocol A | |---|---|---| | `drbdProtocol` value | `C` | `A` | | Replication mode | Synchronous — write acknowledged only after both sites commit | Asynchronous — write acknowledged after local commit; DR side receives data in the background | | RPO | Zero (RPO=0) | Near-zero (seconds) | | Latency requirement | < 10 ms RTT between primary and DR nodes | No strict requirement | | Best for | Same-datacenter or low-latency WAN links | Long-distance or high-latency links | Choose Protocol C unless your network latency between clusters exceeds 10 ms. **Storage class mappings (DRBD Operator model)** In the DRBD Operator model, the `DRBDReplicationPolicy` contains a `storageClassMappings` list that pairs each primary cluster storage class with its DR cluster equivalent. This allows Site Recovery to provision matching volumes on both sides automatically. If your storage class names are identical on both clusters, the mapping still needs to be stated explicitly. ```yaml storageClassMappings: - primaryStorageClass: ocs-storagecluster-ceph-rbd drStorageClass: ocs-storagecluster-ceph-rbd - primaryStorageClass: local-storage drStorageClass: local-storage-dr ``` **Replication endpoints** The `replicationEndpoint` field in `DRBDReplicationPolicy` specifies the IP address and port on which DRBD listens for cross-cluster replication traffic. Use the node IP (or a load-balanced VIP) on DRBD's port range (TCP 7000–7999). **Failover type** When you create a `FailoverRequest`, the `failoverType` field controls whether a graceful shutdown is attempted: | Value | Behavior | Use when | |---|---|---| | `planned` | Shuts down VMs on the primary cluster gracefully before promoting volumes on the DR cluster. Zero data loss. | Primary cluster is healthy and accessible | | `unplanned` | Promotes DR volumes and starts VMs immediately, without waiting for primary cluster coordination. May incur minimal data loss if Protocol A was in use. | Primary cluster is unavailable or unreachable | **ProtectionGroup `desiredState`** The `spec.desiredState` field on a `ProtectionGroup` tells the controller what the target running state of the group should be. Set it to `running` for normal operation. The controller uses this to determine which cluster the VMs should be active on. ## Usage The workflow below covers the most common path: protect a VM, verify replication is healthy, run a test failover to validate DR readiness, and then trigger a planned failover. --- ### Centralized storage model: protect VMs with a ProtectionGroup Create a `ProtectionGroup` on the primary cluster, grouping the VMs you want to fail over together as a unit: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: web-tier-pg namespace: default spec: desiredState: running virtualMachines: - name: vm-web-server - name: vm-database replicationPolicy: type: synchronous remoteCluster: cluster2 ``` Apply it to the primary cluster: ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER1 apply -f web-tier-pg.yaml ``` Watch the `ProtectionGroup` status as the `protection-group-controller` sets up replication and tracks per-VM sync progress: ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get protectiongroup web-tier-pg -w ``` Wait until the status shows `Consistent` before proceeding. A `Syncing` state means initial replication is in progress; `Degraded` indicates a problem (see Troubleshooting). --- ### DRBD Operator model: create a replication policy and protect a VM **Step 1: Create the DRBDReplicationPolicy on the quorum cluster** The `DRBDReplicationPolicy` defines how volumes replicate between your two clusters: ```yaml apiVersion: siterecovery.trilio.io/v1 kind: DRBDReplicationPolicy metadata: name: prod-replication-policy namespace: dr-prod spec: drbdProtocol: C storageClassMappings: - primaryStorageClass: ocs-storagecluster-ceph-rbd drStorageClass: ocs-storagecluster-ceph-rbd primaryCluster: name: cluster1 replicationEndpoint: "10.0.0.1:7000" drCluster: name: cluster2 replicationEndpoint: "10.0.0.2:7000" ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f prod-replication-policy.yaml ``` **Step 2: Create a ProtectionRequest for each VM** A `ProtectionRequest` asks the `protection-controller` on the quorum cluster to protect a single VM. The controller validates the VM and its PVCs, creates `DRBDVolume` resources, waits for replication to sync, and then switches the VM to use DRBD-backed frontend PVCs: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-vm-web-server namespace: dr-prod spec: vmName: vm-web-server vmNamespace: default sourceCluster: cluster1 ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f protect-vm-web-server.yaml ``` Track the protection lifecycle: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-vm-web-server -n dr-prod -w ``` The `protection-controller` moves the `ProtectionRequest` through a defined lifecycle: validation → DRBDVolume creation → sync wait → frontend PVC switch → fully protected. **Step 3: Verify replication health** Inspect the `DRBDVolume` resources created for the VM to confirm sync progress: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get drbdvolume -n dr-prod ``` Check the `ReplicationGroupStatus` for an aggregated health view: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o yaml ``` The `status.health` field reports `Healthy`, `Degraded`, or `Critical`. Wait for `Healthy` before running a test failover or triggering a real failover. --- ### Trigger a planned failover Once the Protection Group or ProtectionRequest is in a healthy replication state, you can trigger a planned failover by creating a `FailoverRequest` on the quorum cluster. The `failover-controller` watches for this resource and orchestrates the full sequence: VM shutdown on primary, volume promotion on DR, VM startup on DR. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: web-tier-planned-failover namespace: dr-prod spec: protectionGroupRef: name: web-tier-pg targetCluster: cluster2 failoverType: planned ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f web-tier-failover.yaml # Watch progress kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest web-tier-planned-failover -n dr-prod -w ``` Expect the failover to complete within 3–8 minutes. The `FailoverRequest` status tracks the operation through to `Completed` or `Failed`. ## Examples ### Example 1: Minimal ProtectionGroup (centralized storage model) This is the simplest possible protection configuration for two VMs that should fail over together: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: my-first-pg namespace: default spec: desiredState: running virtualMachines: - name: vm-web-server - name: vm-database replicationPolicy: type: synchronous remoteCluster: cluster2 ``` ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER1 apply -f my-first-pg.yaml ``` Expected output after applying: ``` protectiongroup.siterecovery.trilio.io/my-first-pg created ``` Expected status after replication syncs (query with `-o yaml` for full detail): ``` NAME STATUS VMS AGE my-first-pg Consistent 2 4m12s ``` --- ### Example 2: DRBDReplicationPolicy with Protocol C (DRBD Operator model) A replication policy specifying synchronous replication with identical storage class names on both clusters: ```yaml apiVersion: siterecovery.trilio.io/v1 kind: DRBDReplicationPolicy metadata: name: prod-sync-policy namespace: dr-prod spec: drbdProtocol: C storageClassMappings: - primaryStorageClass: ocs-storagecluster-ceph-rbd drStorageClass: ocs-storagecluster-ceph-rbd primaryCluster: name: cluster1 replicationEndpoint: "10.0.0.1:7000" drCluster: name: cluster2 replicationEndpoint: "10.0.0.2:7000" ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f prod-sync-policy.yaml ``` Expected output: ``` drbdreplicationpolicy.siterecovery.trilio.io/prod-sync-policy created ``` --- ### Example 3: ProtectionRequest for a single VM (DRBD Operator model) ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-vm-database namespace: dr-prod spec: vmName: vm-database vmNamespace: default sourceCluster: cluster1 ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f protect-vm-database.yaml kubectl --kubeconfig $KUBECONFIG_QUORUM get protectionrequest protect-vm-database -n dr-prod -w ``` Expected progression of the `STATUS` column: ``` NAME STATUS protect-vm-database Validating protect-vm-database CreatingDRBDVolumes protect-vm-database WaitingForSync protect-vm-database SwitchingFrontendPVC protect-vm-database Protected ``` --- ### Example 4: Non-disruptive test failover (DRBD Operator model only) A test failover validates your DR capability without affecting production. It snapshots live volumes, boots test VMs on the DR cluster, runs verification, and cleans up: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: validate-web-tier namespace: dr-prod spec: protectionGroupRef: name: my-first-pg namespace: default cleanupPolicy: Manual retentionTime: 2h ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f validate-web-tier.yaml kubectl --kubeconfig $KUBECONFIG_QUORUM get testfailover validate-web-tier -n dr-prod -w ``` Expected phase progression: ``` NAME PHASE AGE validate-web-tier Pending 0s validate-web-tier CreatingSnapshots 12s validate-web-tier CreatingVolumes 45s validate-web-tier CreatingVMs 1m10s validate-web-tier VerifyingData 2m5s validate-web-tier Succeeded 3m22s ``` When you are done inspecting the test VMs, clean up by deleting the resource: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM delete testfailover validate-web-tier -n dr-prod ``` --- ### Example 5: Planned failover FailoverRequest ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: web-tier-planned-fo namespace: dr-prod spec: protectionGroupRef: name: my-first-pg targetCluster: cluster2 failoverType: planned ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f web-tier-planned-fo.yaml kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest web-tier-planned-fo -n dr-prod -w ``` Expected final status: ``` NAME STATUS TYPE TARGET AGE web-tier-planned-fo Completed planned cluster2 6m48s ``` --- ### Example 6: Unplanned failover when the primary cluster is down When the primary cluster is unreachable, set `failoverType: unplanned`. The `failover-controller` promotes DR volumes and starts VMs without waiting for primary cluster coordination: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: web-tier-emergency-fo namespace: dr-prod spec: protectionGroupRef: name: my-first-pg targetCluster: cluster2 failoverType: unplanned ``` ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM apply -f web-tier-emergency-fo.yaml kubectl --kubeconfig $KUBECONFIG_QUORUM get failoverrequest web-tier-emergency-fo -n dr-prod -w ``` ## Troubleshooting Use the following patterns to diagnose common failures when creating your first resources. --- **Symptom:** `ProtectionGroup` status stays `Syncing` for more than 15 minutes *Likely cause:* Initial DRBD sync of large volumes is still in progress, or network throughput between clusters is limited. *Fix:* 1. Check the `ReplicationGroupStatus` for per-volume sync progress: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get replicationgroupstatus -n dr-prod -o yaml ``` 2. Verify DRBD traffic is flowing on TCP 7000–7999 between worker nodes. A firewall rule blocking this port range will stall sync indefinitely. 3. If volumes are large, allow additional time. Initial sync rate depends on storage throughput and network bandwidth. --- **Symptom:** `ProtectionGroup` status is `Degraded` *Likely cause:* One or more DRBD volumes have lost their replication connection, or replication lag has exceeded the configured threshold. *Fix:* 1. Look for `RPOEvent` resources, which record lag violations: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM get rpoevent -n dr-prod -o yaml ``` 2. Check `replication-monitor` logs on the quorum cluster for connection errors: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod -l app=replication-monitor ``` 3. Verify the DRBD replication endpoints in your `DRBDReplicationPolicy` are correct and reachable. 4. If a split-brain is suspected (both sides report primary role), do not trigger a failover until the split-brain is manually resolved. Check DRBD resource state on both clusters before proceeding. --- **Symptom:** `ProtectionRequest` is stuck in `Validating` state *Likely cause:* The VM named in `spec.vmName` does not exist in `spec.vmNamespace` on `spec.sourceCluster`, or the `protection-controller` cannot reach the primary cluster API server. *Fix:* 1. Confirm the VM exists on the primary cluster: ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get vm vm-web-server -n default ``` 2. Check `protection-controller` logs on the quorum cluster: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod -l app=protection-controller ``` 3. Ensure TCP 6443 is open from the quorum cluster to the primary cluster API server. --- **Symptom:** `FailoverRequest` status shows `Failed` *Likely cause:* For a planned failover, the `failover-controller` could not gracefully stop VMs on the primary cluster (for example, the primary cluster became unavailable mid-operation). For an unplanned failover, a volume promotion error on the DR side. *Fix:* 1. Describe the `FailoverRequest` for detailed status conditions: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM describe failoverrequest -n dr-prod ``` 2. Check `failover-controller` logs: ```bash kubectl --kubeconfig $KUBECONFIG_QUORUM logs -n dr-prod -l app=failover-controller ``` 3. If the primary cluster is truly unavailable and you have a failed planned failover, re-create the `FailoverRequest` with `failoverType: unplanned` to proceed without primary cluster coordination. 4. Do not create a second `FailoverRequest` for the same Protection Group until the first one has reached a terminal state (`Completed` or `Failed`) to avoid concurrent failover conflicts. --- **Symptom:** `TestFailover` stays in `CreatingSnapshots` phase and does not advance *Likely cause:* VolumeSnapshot support (CSI snapshotter) is not installed on the target cluster, or the VolumeSnapshotClass is not configured. *Fix:* 1. Verify CSI snapshotter is deployed on the DR cluster: ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER2 get crd volumesnapshots.snapshot.storage.k8s.io ``` 2. Check `test-failover-controller` logs on the primary or DR cluster: ```bash kubectl --kubeconfig $KUBECONFIG_CLUSTER2 logs -n trilio-site-recovery-system -l app=test-failover-controller ``` 3. Note: test failover is only supported in the DRBD Operator deployment model. If you are using the centralized storage model, this feature is not available. --- **Symptom:** Controllers are running but custom resources are not being reconciled *Likely cause:* The resource was created in the wrong namespace, or the controller's RBAC does not cover that namespace. *Fix:* 1. Resources managed by quorum-side controllers (`FailoverRequest`, `ProtectionRequest`, `DRBDReplicationPolicy`) must be created in the deployment namespace on the quorum cluster (for example, `dr-prod`). 2. `ProtectionGroup` and `TestFailover` resources are managed by workload-side controllers and should be created on the appropriate workload cluster in the correct namespace. 3. Use `pgctl` to verify your deployment context is pointing at the correct namespace and cluster: ```bash pgctl context list ```