Creating Your First Resource

This guide walks you through defining your first disaster recovery plan with Trilio Site Recovery and validating it with a non-disruptive test recovery. You will create a ProtectionGroup that groups one or more KubeVirt VirtualMachines into a single replication unit, submit a ProtectionRequest to bring those VMs under DRBD block-level replication, and then run a TestFailover to confirm that your DR copy can boot and pass verification checks — all without touching production workloads. By the end of this guide you will have a working, validated recovery plan expressed entirely as Kubernetes custom resources that can be committed to Git and applied with standard tooling.

Before starting, confirm that the following conditions are met:

• Three OpenShift clusters at version 4.17 or later: one quorum cluster (management plane) and two workload clusters (primary and DR). The quorum cluster is required and not optional.
• Trilio Site Recovery operator installed on the quorum cluster via OLM (CatalogSource + OperatorGroup + Subscription in the trilio-site-recovery-system namespace). See the installation guide if the operator is not yet present.
• A ProtectionZone CR already created and in Available phase. The ProtectionZone controller must have deployed the workload control planes (site-recovery-workload-control-plane) to both workload clusters before you can protect VMs.
• KubeVirt or OpenShift Virtualization installed on both workload clusters.
• DRBD kernel module ≥ 9.x loaded on every worker node of both workload clusters.
• LVM thin pool configured on each worker node, backed by a dedicated block device, for DRBD backing storage.
• Network connectivity between primary and DR worker nodes:
  • TCP ports 7000–8000 inclusive open (one port per replicated VM volume; port 8000 is mandatory — a rule stopping at 7999 silently prevents replication on one volume).
  • Round-trip latency below 50 ms (required for Protocol C synchronous replication).
• oc CLI available and configured. oc is required for OpenShift-specific operations; kubectl-only operation is not a supported mode.
• Kubeconfig files for all three clusters, stored as Kubernetes Secrets in the zone namespace (dr-<zone-name>) on the quorum cluster. These are referenced by the ProtectionZone CR and must already exist.
• The target VMs must be running on the primary cluster and backed by PVCs that can be replicated. VMs using WaitForFirstConsumer storage classes for DRBD metadata are not supported (those classes never bind for DRBD-managed PVCs).

Note: Upstream Kubernetes workload clusters are supported on versions corresponding to OpenShift 4.17 and later. OpenShift 4.14–4.16 are not supported even if the operator bundle can be installed on them.

Trilio Site Recovery installs exclusively through OLM on OpenShift. Customers do not run helm install directly — the Helm charts for the quorum and workload control planes are embedded in the operator binary and deployed automatically by the ProtectionZone controller once the zone reaches Available. This section describes the remaining day-1 steps after the operator is running: creating the kubeconfig Secrets and the ProtectionZone CR that bootstraps the rest of the stack.

Step 1 — Create the zone namespace on the quorum cluster

All quorum-side resources for a DR deployment live in a namespace with the dr- prefix.

oc --context=quorum-cluster create namespace dr-production

Step 2 — Store kubeconfig Secrets for both workload clusters

The ProtectionZone controller reads these Secrets to reach each workload cluster's API server. Create one Secret per cluster in the zone namespace.

# Secret for the primary cluster (cluster1)
oc --context=quorum-cluster create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=/path/to/primary-kubeconfig \
  -n dr-production

# Secret for the DR cluster (cluster2)
oc --context=quorum-cluster create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=/path/to/dr-kubeconfig \
  -n dr-production

Step 3 — Create the webhook serving certificate Secret

Webhook TLS certificates are not managed by the operator. Create the Secret that the admission webhook expects before creating the ProtectionZone.

# Generate a self-signed certificate or use your PKI; the Secret name must
# match webhook.certSecretName (default: webhook-server-cert).
oc --context=quorum-cluster create secret tls webhook-server-cert \
  --cert=/path/to/tls.crt \
  --key=/path/to/tls.key \
  -n dr-production

Step 4 — Create the ProtectionZone CR

The ProtectionZone CR is the single entry point that deploys the quorum control plane, the workload control planes on both workload clusters, and the DRBD node agent DaemonSets. Apply it to the quorum cluster.

# protection-zone.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production
  namespace: dr-production
spec:
  displayName: "Production DR Zone"
  description: "Primary to DR replication for production VMs"
  storageBackendMode: DRBD
  clusters:
    - name: primary-cluster
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr-cluster
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

oc --context=quorum-cluster apply -f protection-zone.yaml

Step 5 — Wait for the zone to reach Available

oc --context=quorum-cluster get pz -n dr-production -w

Expected output when ready:

NAME         DISPLAY NAME          BACKEND   CLUSTERS                         STATUS      AGE
production   Production DR Zone    DRBD      primary-cluster,dr-cluster       Available   3m

While the zone provisions, the controller automatically:

• Deploys the site-recovery-quorum-control-plane into dr-production on the quorum cluster.
• Deploys the site-recovery-workload-control-plane (DRBD node agent DaemonSet, webhooks, RBAC) into trilio-site-recovery-system on both workload clusters.
• Creates the default DRBDReplicationPolicy named production-default with isDefault: true and Protocol C. You do not need to create or modify this object.

Important: Quorum-side components and all quorum CRs live in dr-production. Workload-side components live in trilio-site-recovery-system on each workload cluster. These are different namespaces — do not query quorum pods in trilio-site-recovery-system or workload pods in dr-production.

Most configuration for your first recovery plan is expressed directly in the custom resources described in the Usage section. The key options that affect behavior are summarized here.

ProtectionGroup

ProtectionRequest

TestFailover

Replication policy (managed automatically)

The DRBDReplicationPolicy named <zoneName>-default is created and owned by the ProtectionZone controller. You do not author or modify it. Key behavior:

• Protocol C only — synchronous replication, RPO=0. Protocol A is not supported.
• Port allocation — ports are allocated from the fixed range 7000–8000 (one port per protected VM).
• Storage class mappings — defined in the policy's clusters[].storageClassMappings; if a source class is absent from the map it is used as-is on the DR side.

Node pinning

A protected VM is pinned to one node per cluster for the lifetime of its protection. spec.nodeName is immutable once set. Live migration and cold migration of a protected VM are denied by the admission webhook. oc adm drain of a node hosting a running protected VM will not complete while the VM remains protected.

This section walks through the three core operations: grouping VMs into a protection group, requesting block-level replication for each VM, and running a test failover to validate your DR capability.

All CRs in this section are applied to the quorum cluster unless stated otherwise. The quorum cluster hosts the ProtectionGroup, ProtectionRequest, FailoverRequest, and TestFailover kinds. The DRBDResource kind lives on each workload cluster and is created automatically by the protection controller — you do not create it directly.

1. Create a ProtectionGroup

A ProtectionGroup defines the set of VMs that must fail over together as a coordinated unit. All members replicate together and fail over atomically.

# protection-group.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: web-tier
  namespace: dr-production
spec:
  sourceCluster: cluster1
  desiredState: running
  virtualMachines:
    - name: vm-web-01
      namespace: production
      bootOrder: 1
    - name: vm-web-02
      namespace: production
      bootOrder: 1
    - name: vm-database
      namespace: production
      bootOrder: 2

oc --context=quorum-cluster apply -f protection-group.yaml

bootOrder controls failover sequencing: vm-web-01 and vm-web-02 share order 1 and boot in parallel as one batch; vm-database has order 2 and boots after that batch completes.

2. Submit a ProtectionRequest for each VM

A ProtectionRequest asks the protection controller to bring a single VM under DRBD replication. Submit one per VM in the group.

# protect-vm-web-01.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-web-01
  namespace: dr-production
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm-web-01
    namespace: production
  replicationConfig:
    replicationProtocol: C
    drbdMetadataConfig:
      storageClassName: ocs-storagecluster-ceph-rbd
      size: 128Mi

oc --context=quorum-cluster apply -f protect-vm-web-01.yaml
# Repeat for each VM in the group

The protection controller works through the following phases automatically:

Watch progress:

oc --context=quorum-cluster get pr -n dr-production -w

Example output during sync:

NAME                  VM            NAMESPACE    PHASE     SYNC   AGE
protect-vm-web-01     vm-web-01     production   Syncing   42     4m

Wait for all ProtectionRequests to reach Protected before proceeding.

3. Verify the ProtectionGroup state

Once all VMs are protected, the ProtectionGroup should report Protected:

oc --context=quorum-cluster get pg web-tier -n dr-production

NAME       VMS                              STATE       ACTIVE     AGE
web-tier   vm-web-01,vm-web-02,vm-database  Protected   cluster1   12m

For per-VM replication detail:

oc --context=quorum-cluster get pg web-tier -n dr-production -o yaml

Check status.protectedVMs[].replicationStatus for each VM. A value of Protected means that VM's volumes are fully in sync.

4. Check replication health

The replication monitor automatically creates a ReplicationGroupStatus object named web-tier-replication in dr-production. Use it to see aggregate sync state:

oc --context=quorum-cluster get rgs web-tier-replication -n dr-production

NAME                   PROTECTION-GROUP   RPO-STATUS   OUT-OF-SYNC   SYNCED   CONNECTION   LAST-SYNC   AGE
web-tier-replication   web-tier           Met           0             3        Connected    <time>      10m

RPO-STATUS: Met with OUT-OF-SYNC: 0 confirms Protocol C synchronous replication is healthy and RPO=0.

To filter RPO events by protection group:

oc --context=quorum-cluster get rpoe -n dr-production \
  -l siterecovery.trilio.io/protection-group=web-tier

5. Inspect protected VMs on the workload cluster

Protected VMs are labeled siterecovery.trilio.io/protected-vm=true on the workload cluster. This label is on the VirtualMachine object only — not on DRBDResource or ProtectionRequest objects.

oc --context=primary-cluster get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

To view the DRBDResource for a specific VM on the primary cluster (resources live in the VM's namespace):

oc --context=primary-cluster get dres -n production

Example 1 — Minimal ProtectionGroup and ProtectionRequests

Protect a single VM named vm-app running in the app-team namespace on cluster1.

# pg-app-team.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: app-team-pg
  namespace: dr-production
spec:
  sourceCluster: cluster1
  desiredState: running
  virtualMachines:
    - name: vm-app
      namespace: app-team

# protect-vm-app.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-app
  namespace: dr-production
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm-app
    namespace: app-team
  replicationConfig:
    replicationProtocol: C
    drbdMetadataConfig:
      storageClassName: ocs-storagecluster-ceph-rbd

oc --context=quorum-cluster apply -f pg-app-team.yaml
oc --context=quorum-cluster apply -f protect-vm-app.yaml

# Wait for Protected phase
oc --context=quorum-cluster get pr protect-vm-app -n dr-production -w

Expected final output:

NAME               VM       NAMESPACE   PHASE       SYNC   AGE
protect-vm-app     vm-app   app-team    Protected   100    8m

Example 2 — Test failover with automatic cleanup

Run a non-disruptive DR validation. The test controller creates volume snapshots, provisions test PVCs, starts test VMs in an isolated namespace, runs verification checks, and cleans up — without affecting the production VMs.

# test-failover.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: web-tier-test
  namespace: dr-production
spec:
  protectionGroupRef:
    name: web-tier
    namespace: dr-production
  cleanupPolicy: Automatic
  retentionTime: 2h

oc --context=quorum-cluster apply -f test-failover.yaml

# Watch progress
oc --context=quorum-cluster get tfo web-tier-test -n dr-production -w

The test controller selects the target cluster automatically based on which cluster the VMs are NOT currently running on. You can see which cluster was selected and why in status.targetCluster and status.targetClusterReason.

Expected phase progression:

NAME              PROTECTION GROUP   PHASE                  PASSED   AGE
web-tier-test     web-tier           Queued                          0s
web-tier-test     web-tier           PreparingEnvironment            5s
web-tier-test     web-tier           CreatingSnapshots               12s
web-tier-test     web-tier           CreatingVolumes                 35s
web-tier-test     web-tier           CreatingVMs                     58s
web-tier-test     web-tier           StartingVMs                     70s
web-tier-test     web-tier           Verifying                       95s
web-tier-test     web-tier           Succeeded              true     2m10s

To see the namespace where test resources were created:

oc --context=quorum-cluster get tfo web-tier-test -n dr-production \
  -o jsonpath='{.status.testNamespace}'

The test namespace is generated by the controller (for example site-recovery-web-tier-a3f9c1) and is reported in status.testNamespace. Do not attempt to construct or guess this value.

To inspect verification check results:

oc --context=quorum-cluster get tfo web-tier-test -n dr-production \
  -o jsonpath='{.status.verificationResults}'

With cleanupPolicy: Automatic and retentionTime: 2h, the test resources are removed 2 hours after the test completes. To clean up immediately, delete the CR:

oc --context=quorum-cluster delete tfo web-tier-test -n dr-production

Example 3 — Test failover with network reachability probes

Add optional HTTP probes to verify that test VMs are reachable over the network after boot.

# test-failover-with-probes.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: web-tier-test-probes
  namespace: dr-production
spec:
  protectionGroupRef:
    name: web-tier
    namespace: dr-production
  cleanupPolicy: Manual
  retentionTime: 4h
  verification:
    reachability:
      - type: tcp
        port: 22
        timeout: 30s
      - type: http
        vmName: vm-web-01
        port: 8080
        endpoint: /healthz
        expectedStatus: 200
        timeout: 30s

The vmName field restricts a probe to a specific VM by its source name as listed in the ProtectionGroup. When vmName is omitted, the probe runs against every test VM.

Automatic boot, guest-agent, and filesystem checks always run first. The reachability probes run after those pass.

Example 4 — Check replication status via the TSR web console

On OpenShift, the ProtectionZone controller creates a Route and console link for the TSR web console automatically. Access it from the OpenShift console or directly:

# Get the console Route URL
oc --context=quorum-cluster get route -n dr-production

Navigate to your protection zone in the console. The dashboard shows ReplicationGroupStatus metrics, per-VM sync progress, and RPO compliance for each ProtectionGroup. Zone-scoped pages derive the dr-<zoneId> namespace — a zone named production maps to the dr-production namespace.

Note: Browser access to the TSR web console is OpenShift-only. The console pods are deployed on upstream Kubernetes but no ingress or authentication path is provided, and browser access is not a supported use case there.

ProtectionRequest stuck in Validating

Symptom: kubectl get pr -n dr-production shows PHASE: Validating for more than a few minutes.

Likely causes:

• The VM does not exist on the cluster identified by spec.sourceCluster.
• The VM's PVCs use a WaitForFirstConsumer StorageClass for DRBD metadata (rejected during validation).
• The ProtectionZone is not in Available phase (workload control plane not yet deployed).

Fix:

1. Verify the VM exists on the correct workload cluster:
   Copy

   oc --context=primary-cluster get vm -n <vm-namespace> <vm-name>
   

2. Confirm the ProtectionZone is Available:
   Copy

   oc --context=quorum-cluster get pz -n dr-production
   

3. Inspect conditions on the ProtectionRequest:
   Copy

   oc --context=quorum-cluster get pr <name> -n dr-production -o yaml
   

   Look for Ready=False with reason RoleConflict — this means another ProtectionRequest already holds the DRBD Primary role for this VM's resource.
4. Check quorum control plane logs:
   Copy

   oc --context=quorum-cluster logs -n dr-production \
     -l app.kubernetes.io/component=quorum-control-plane \
     -c manager
   

ProtectionRequest stuck in Syncing

Symptom: SYNC percentage in kubectl get pr is not advancing, or advances very slowly.

Likely causes:

• DRBD ports not open between primary and DR worker nodes (TCP 7000–8000 inclusive).
• Round-trip latency between clusters exceeds 50 ms.
• DRBD node agent not running on the node where the VM is scheduled.

Fix:

1. Verify the DRBD node agent DaemonSet is healthy on both workload clusters:
   Copy

   oc --context=primary-cluster get daemonset -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=drbd-node-agent
   

2. Find the agent pod on the specific VM's node:
   Copy

   # Get the pod name for the node
   oc --context=primary-cluster get pods -n trilio-site-recovery-system \
     --field-selector spec.nodeName=<node-name> \
     -l app.kubernetes.io/component=drbd-node-agent -o name
   # Then get logs
   oc --context=primary-cluster logs -n trilio-site-recovery-system <pod-name>
   

3. Check the DRBDResource on the primary cluster for status.connectionState:
   Copy

   oc --context=primary-cluster get dres -n <vm-namespace>
   

   A connectionState of Disconnected or StandAlone indicates the DRBD peers cannot reach each other. Verify firewall rules allow TCP 7000–8000 between worker node IPs.
4. Check the status.phase of the DRBDResource. A Degraded phase with connectionState: Disconnected confirms a network path problem.

ProtectionRequest reaches Failed

Symptom: PHASE: Failed in kubectl get pr.

Fix:

1. Inspect conditions on the ProtectionRequest:
   Copy

   oc --context=quorum-cluster get pr <name> -n dr-production -o yaml
   

   The conditions array contains the reason for failure.
2. Collect a diagnostic bundle with tsr-gather:
   Copy

   oc --context=quorum-cluster adm must-gather \
     --image=quay.io/triliovault/site-recovery-must-gather:<tag>
   

   Review SUMMARY.md in the bundle first for a summary of what is wrong.
3. After resolving the underlying cause, delete the failed ProtectionRequest and resubmit it. Re-protection always requires a full unprotect and a full initial resync — there is no partial retry.

TestFailover stuck in PreparingEnvironment

Symptom: kubectl get tfo -n dr-production shows PHASE: PreparingEnvironment for an extended time.

Likely causes:

• The target workload cluster cannot be reached from the quorum cluster.
• A NetworkPolicy or namespace quota is preventing the test namespace from being created.

Fix:

1. Check the ProtectionZone cluster reachability:
   Copy

   oc --context=quorum-cluster get pz -n dr-production -o yaml
   

   Look at status.clusters[].status — Unreachable for either cluster explains the stall.
2. Inspect the TestFailover conditions:
   Copy

   oc --context=quorum-cluster get tfo <name> -n dr-production -o yaml
   

3. Check quorum control plane logs for test-failover reconciler output:
   Copy

   oc --context=quorum-cluster logs -n dr-production \
     -l app.kubernetes.io/component=quorum-control-plane \
     -c manager | grep testfailover
   

TestFailover Failed — verification checks did not pass

Symptom: PHASE: Failed, PASSED: false in kubectl get tfo.

Fix:

1. Read the verification check details:
   Copy

   oc --context=quorum-cluster get tfo <name> -n dr-production \
     -o jsonpath='{.status.verificationResults.checks}'
   

   Each check has a type, vm, passed, and message field.
2. Find the test namespace from the status and inspect test VM events on the DR cluster:
   Copy

   TEST_NS=$(oc --context=quorum-cluster get tfo <name> -n dr-production \
     -o jsonpath='{.status.testNamespace}')
   oc --context=dr-cluster get events -n "$TEST_NS" --sort-by='.lastTimestamp'
   

3. Test VMs are isolated by NetworkPolicy in the test namespace. If a reachability probe fails, confirm the probe target port is accessible within the pod network of the test namespace.
4. With cleanupPolicy: Manual, test resources remain after failure so you can inspect them. Do not attempt to manually patch finalizers to force deletion — delete the TestFailover CR to trigger the CleaningUp → Cleaned lifecycle.

ProtectionZone stuck in Provisioning or Degraded

Symptom: kubectl get pz -n dr-production shows a phase other than Available.

Fix:

1. Check cluster reachability:
   Copy

   oc --context=quorum-cluster get pz -n dr-production -o yaml
   

   Review status.clusters[].status and status.clusters[].message for each cluster.
2. Verify the kubeconfig Secrets are correct and the keys match what the ProtectionZone references:
   Copy

   oc --context=quorum-cluster get secret cluster1-kubeconfig -n dr-production \
     -o jsonpath='{.data.kubeconfig}' | base64 -d | head -5
   

3. Confirm the trilio-site-recovery-system namespace exists on each workload cluster (the zone controller probes this namespace by name to confirm the workload chart is installed):
   Copy

   oc --context=primary-cluster get namespace trilio-site-recovery-system
   oc --context=dr-cluster get namespace trilio-site-recovery-system
   

4. Review the protection-zone-controller logs on the quorum cluster:
   Copy

   oc --context=quorum-cluster logs -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=protectionzone-controller