Creating Your First Resource

This page walks you through defining your first disaster recovery plan and validating it with a non-disruptive test recovery. You will create a ProtectionGroup that declares which virtual machines must fail over together, submit a ProtectionRequest for each VM to establish block-level replication, verify that replication reaches a healthy state, and then trigger a TestFailover to confirm your DR capability without touching production workloads. By the end you will have a complete, tested recovery plan expressed entirely as Kubernetes custom resources.

Before starting this walkthrough, ensure the following are in place:

• Three OpenShift clusters (OpenShift 4.17 or later on all three): one TSR-Manager cluster and two workload clusters (primary and DR). The TSR-Manager cluster is required — it is not optional.
• Trilio Site Recovery operator installed on the TSR-Manager cluster via OLM (CatalogSource, OperatorGroup, and Subscription referencing package site-recovery, channel tech-preview, catalog quay.io/triliovault/site-recovery-catalog) in the trilio-site-recovery-system namespace.
• A ProtectionZone CR exists and is Available in a dr-<name> namespace on the TSR-Manager cluster. The ProtectionZone controller must have already deployed the workload control planes to both workload clusters and the zone's status phase must be Available. If the zone is still Provisioning, wait before proceeding.
• KubeVirt or OpenShift Virtualization installed on both workload clusters.
• DRBD kernel module ≥ 9.x loaded on every worker node of both workload clusters.
• LVM thin pool configured on each worker node of both workload clusters, backed by a dedicated block device, for DRBD backing storage.
• Network connectivity: TCP ports 7000–8000 inclusive open between primary and DR worker nodes (one port per replicated VM volume; port 8000 is mandatory — omitting it causes a silent replication failure on one volume). TCP port 6443 open from the TSR-Manager cluster to both workload cluster API servers.
• Network latency below 50 ms RTT between primary and DR clusters (required for Protocol C synchronous replication).
• oc CLI available and kubeconfig files for all three clusters. oc is required for OpenShift-specific paths.
• Target VMs exist and are running on the primary workload cluster, with PVCs bound to storage classes that are mapped in the zone's replication policy.
• Webhook serving certificate Secret named webhook-server-cert created in the zone namespace (dr-<name>) on the TSR-Manager cluster before the ProtectionZone controller deploys the quorum control plane.

The operator and control planes are already installed when the ProtectionZone reaches Available. There is nothing to install at this stage — all DR operations are performed by creating custom resources. The steps below establish the kubeconfig references and zone namespace that subsequent steps depend on.

Step 1 — Confirm your zone is Available

Run the following against the TSR-Manager cluster. Replace <zone-name> with the value of your ProtectionZone's metadata.name.

oc --context tsr-manager get pz <zone-name> -n dr-<zone-name>

The STATUS column must show Available before you proceed. If it shows Provisioning, the workload control planes are still deploying. If it shows Unavailable or Degraded, check cluster reachability:

oc --context tsr-manager get pz <zone-name> -n dr-<zone-name> -o jsonpath='{.status.clusters}' | jq .

Each entry's status field must be Reachable.

Step 2 — Confirm the workload control plane is running on both workload clusters

The DRBD node agent DaemonSet is deployed automatically to trilio-site-recovery-system on each workload cluster.

# Primary cluster
oc --context primary \
  get daemonset \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

# DR cluster
oc --context dr \
  get daemonset \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

All pods in the DaemonSet should be in Running state before you protect any VMs.

Step 3 — Confirm the TSR-Manager control plane is running

The unified controller manager runs in the zone namespace on the TSR-Manager cluster.

oc --context tsr-manager \
  get pods \
  -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane

You should see a single pod with 1/1 ready. This pod hosts the failover, protection, pg-sync, replication-monitor, and test-failover reconcilers.

All configuration relevant to your recovery plan is expressed in custom resource fields. The key objects and their most important fields are described below.

ProtectionGroup

A ProtectionGroup declares which VMs must fail over together as an atomic unit. Key fields:

ProtectionRequest

A ProtectionRequest enables block-level DRBD replication for a single VM. One ProtectionRequest is required per VM. Key fields:

The ProtectionZone controller automatically creates a DRBDReplicationPolicy named <zone-name>-default with isDefault: true in your zone namespace. You do not create or modify this policy — the protection controller consults it automatically when processing each ProtectionRequest.

TestFailover

A TestFailover triggers a non-disruptive DR validation. Key fields:

The controller assigns the test namespace automatically and stores it in status.testNamespace. You cannot specify this namespace — do not attempt to construct or set it.

The workflow has three phases: protect, verify, and test.

Phase 1 — Protect your VMs

1a. Create a ProtectionGroup on the TSR-Manager cluster

The ProtectionGroup declares the set of VMs that must fail over together. Create it in your zone namespace (dr-<zone-name>).

oc --context tsr-manager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: my-app-group
  namespace: dr-<zone-name>
spec:
  sourceCluster: cluster1
  desiredState: running
  virtualMachines:
    - name: vm-web
      namespace: production
      bootOrder: 1
    - name: vm-db
      namespace: production
      bootOrder: 2
  sla:
    rtoTargetSeconds: 300
    rpoTargetSeconds: 0
EOF

Verify the ProtectionGroup was accepted:

oc --context tsr-manager get pg my-app-group -n dr-<zone-name>

1b. Create a ProtectionRequest for each VM

A ProtectionRequest is a per-VM object. Create one for vm-web and one for vm-db. Both are created on the TSR-Manager cluster in the zone namespace.

oc --context tsr-manager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-web
  namespace: dr-<zone-name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm-web
    namespace: production
  replicationConfig:
    replicationProtocol: C
    drbdMetadataConfig:
      storageClassName: ocs-storagecluster-ceph-rbd
      size: 128Mi
EOF

oc --context tsr-manager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-db
  namespace: dr-<zone-name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm-db
    namespace: production
  replicationConfig:
    replicationProtocol: C
    drbdMetadataConfig:
      storageClassName: ocs-storagecluster-ceph-rbd
      size: 128Mi
EOF

1c. Watch protection progress

The protection controller moves each ProtectionRequest through the following phases: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

oc --context tsr-manager get pr -n dr-<zone-name> -w

The initial sync (Syncing phase) copies all VM disk data to the DR cluster. Duration depends on disk size and network throughput. Once both requests reach Protected, replication is established and running in synchronous Protocol C mode.

Important: A protected VM is pinned to one node per cluster for the lifetime of the protection. Live migration and cold migration of a protected VM are denied. Draining a node that hosts a running protected VM will not complete. To move a VM you must unprotect it, migrate it, and re-protect it — re-protection always requires a full initial resync.

Phase 2 — Verify replication health

The replication monitor auto-creates a ReplicationGroupStatus object named <protection-group-name>-replication in the zone namespace:

oc --context tsr-manager get rgs my-app-group-replication -n dr-<zone-name>

The RPO-Status column should show Met and Connection should show Connected before you run a test failover. If RPO-Status shows Degraded or Violated, investigate before proceeding.

For per-VM detail:

oc --context tsr-manager get rgs my-app-group-replication \
  -n dr-<zone-name> \
  -o jsonpath='{.status.volumeReplicationStatuses}' | jq .

Phase 3 — Run a test failover

A TestFailover creates volume snapshots on the DR cluster, provisions test PVCs and test VMs in an isolated namespace, runs verification checks (boot, guest agent, optional reachability probes), and then cleans up. Production VMs are not affected.

Create the TestFailover on the TSR-Manager cluster:

oc --context tsr-manager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: my-app-group-test
  namespace: dr-<zone-name>
spec:
  protectionGroupRef:
    name: my-app-group
  cleanupPolicy: Automatic
  retentionTime: 2h
EOF

Watch progress:

oc --context tsr-manager get tfo my-app-group-test -n dr-<zone-name> -w

The controller moves the TestFailover through: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded (or Failed).

Once the test reaches a terminal phase, the controller records a summary on the ProtectionGroup:

oc --context tsr-manager get pg my-app-group \
  -n dr-<zone-name> \
  -o jsonpath='{.status.lastTestFailover}' | jq .

To retrieve the test namespace the controller assigned (for log inspection while resources still exist):

oc --context tsr-manager get tfo my-app-group-test \
  -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}'

After the retentionTime elapses, cleanup runs automatically. To trigger it immediately, set cleanupRequested: true:

oc --context tsr-manager patch tfo my-app-group-test \
  -n dr-<zone-name> \
  --type merge \
  -p '{"spec":{"cleanupRequested":true}}'

Example 1 — ProtectionGroup with boot ordering

This group protects a two-tier application: the database boots first, then the web server. During a failover the controller waits for vm-db to reach Running before starting vm-web.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: two-tier-app
  namespace: dr-production-zone
spec:
  sourceCluster: cluster1
  desiredState: running
  virtualMachines:
    - name: vm-db
      namespace: production
      bootOrder: 1
    - name: vm-web
      namespace: production
      bootOrder: 2
  sla:
    rtoTargetSeconds: 300
    rpoTargetSeconds: 0

Example 2 — ProtectionRequest with a dedicated metadata storage class

When your cluster uses a fast local storage class for DRBD metadata, specify it explicitly. The class must bind immediately — WaitForFirstConsumer classes are rejected during validation.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-db
  namespace: dr-production-zone
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm-db
    namespace: production
  replicationConfig:
    replicationProtocol: C
    drbdMetadataConfig:
      storageClassName: local-block-immediate
      size: 128Mi

Example 3 — Checking ProtectionRequest phases

# Show all protection requests and their current phase
oc --context tsr-manager get pr -n dr-production-zone

Example 4 — TestFailover with an HTTP reachability probe

This test additionally verifies that the web server is reachable on port 80 after the automatic boot checks pass. The probe runs only against vm-web.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: two-tier-app-test
  namespace: dr-production-zone
spec:
  protectionGroupRef:
    name: two-tier-app
  cleanupPolicy: Automatic
  retentionTime: 2h
  verification:
    reachability:
      - type: http
        vmName: vm-web
        port: 80
        endpoint: /healthz
        expectedStatus: 200
        timeout: 30s

Example 5 — Inspecting per-VM replication health

# Show the ReplicationGroupStatus for the protection group
oc --context tsr-manager get rgs two-tier-app-replication \
  -n dr-production-zone

# Show per-VM replication state detail
oc --context tsr-manager get rgs two-tier-app-replication \
  -n dr-production-zone \
  -o jsonpath='{.status.volumeReplicationStatuses}' | jq .

Example 6 — Filtering RPO events for your group

RPOEvent objects carry the label siterecovery.trilio.io/protection-group set to the group name. Use this label to filter them, because field selectors on custom resource spec or status fields are not supported.

oc --context tsr-manager get rpoe \
  -n dr-production-zone \
  -l siterecovery.trilio.io/protection-group=two-tier-app

Example 7 — Verifying the DRBDResource on a workload cluster

DRBDResource objects live on the workload clusters in the VM's namespace, not on the TSR-Manager cluster.

# On the primary cluster, in the VM's namespace
oc --context primary get dres -n production

# On the DR cluster
oc --context dr get dres -n production

Each resource name follows the pattern drbd-<namespace>-<vmName>-<hash8>. The Phase column should show Consistent once initial sync is complete.

ProtectionRequest is stuck in Validating

Symptom: oc get pr -n dr-<zone-name> shows the request in Validating phase for more than a few minutes.

Likely causes:

• The VM does not exist on the cluster identified by spec.sourceCluster, or its PVCs are not bound.
• The target workload cluster is unreachable from the TSR-Manager cluster (check zone cluster status).
• The webhook-server-cert Secret is missing or malformed in the zone namespace.

Fix:

1. Confirm the VM exists and its PVCs are bound on the primary cluster: oc --context primary get vm <name> -n <namespace>
2. Check zone cluster reachability: oc --context tsr-manager get pz <zone-name> -n dr-<zone-name> -o jsonpath='{.status.clusters}'
3. Check the protection controller logs: oc --context tsr-manager logs -n dr-<zone-name> -l app.kubernetes.io/component=manager-control-plane

ProtectionRequest is stuck in Syncing

Symptom: The request enters Syncing but status.syncProgress does not advance, or advances very slowly.

Likely causes:

• TCP ports 7000–8000 are not fully open between primary and DR worker nodes. A firewall rule that ends at 7999 silently prevents replication on one volume.
• The DRBD node agent is not running on the node the VM is pinned to.
• Network latency between clusters exceeds 50 ms RTT, causing Protocol C write pressure.

Fix:

1. Verify DRBD node agents are running on all worker nodes of both clusters: oc --context primary get pods -n trilio-site-recovery-system -l app.kubernetes.io/component=drbd-node-agent
2. Confirm port 8000 is open: test TCP connectivity from a primary worker to a DR worker on port 8000.
3. Check the DRBDResource on the primary cluster for connection state: oc --context primary get dres -n <vm-namespace> — the State column should show Connected.
4. Check node agent logs on the node the VM is pinned to:
   Copy

   # Find the agent pod on the relevant node
   oc --context primary get pods \
     -n trilio-site-recovery-system \
     --field-selector spec.nodeName=<node-name> \
     -o jsonpath='{.items[0].metadata.name}'
   # Then fetch its logs
   oc --context primary logs <pod-name> -n trilio-site-recovery-system
   

ProtectionRequest shows Ready=False with reason RoleConflict

Symptom: oc get pr <name> -n dr-<zone-name> -o jsonpath='{.status.conditions}' shows a condition with reason: RoleConflict.

Likely cause: The DRBD replication pair has inconsistent role state — both copies are attempting to hold the Primary role simultaneously, which can occur after an unclean recovery.

Fix: Contact Trilio support with a diagnostic bundle collected via oc adm must-gather from the TSR-Manager cluster.

TestFailover is stuck in CreatingSnapshots

Symptom: The TestFailover does not advance past CreatingSnapshots after several minutes.

Likely causes:

• The VolumeSnapshot API or a VolumeSnapshotClass is not available on the DR cluster.
• The CSI driver on the DR cluster does not support snapshots for the storage class used by the DRBD frontend PVCs.

Fix:

1. Check that a VolumeSnapshotClass is installed on the DR cluster.
2. Check the test-failover controller logs on the TSR-Manager cluster: oc --context tsr-manager logs -n dr-<zone-name> -l app.kubernetes.io/component=manager-control-plane

TestFailover result is Failed at Verifying phase

Symptom: The TestFailover reaches Verifying but then moves to Failed. status.verificationResults.checks shows which check failed.

Likely causes:

• The test VM did not boot successfully (guest agent not responding within the settle window).
• A user-configured reachability probe could not reach the test VM on the specified port.
• A test-verification hook returned a non-zero exit code.

Fix:

1. Read the verification results: oc --context tsr-manager get tfo <name> -n dr-<zone-name> -o jsonpath='{.status.verificationResults}' | jq .
2. If using reachability probes, confirm the port and endpoint are correct and that the test VM's network policy allows inbound traffic on that port from within the test namespace.
3. Find the test namespace: oc --context tsr-manager get tfo <name> -n dr-<zone-name> -o jsonpath='{.status.testNamespace}' then inspect pods in that namespace on the DR cluster.

oc get dres returns server doesn't have a resource type on the TSR-Manager cluster

Symptom: Running oc --context tsr-manager get dres fails with a resource-not-found error.

Cause: DRBDResource CRs live only on the workload clusters, not on the TSR-Manager cluster. The CRD is not registered there by design.

Fix: Run oc get dres against the primary or DR cluster context, in the VM's namespace:

oc --context primary get dres -n <vm-namespace>

Node agent pod is not running — node has a NoSchedule taint

Symptom: A worker node shows the taint siterecovery.trilio.io/not-ready:NoSchedule, and new pods will not schedule on it.

Cause: The DRBD node agent applies this taint to its own node until the DRBD volumes it manages have converged. This is expected during initial sync or after a node restart.

Fix: Monitor the DRBDResource status on the affected cluster. The taint is removed automatically once DRBD converges. If the taint persists after sync completes, check the node agent logs on that node.