Custom Resource Reference
Complete API reference for all site recovery custom resources
This page is the complete API reference for all Trilio Site Recovery Custom Resource Definitions (CRDs). Every DR operation—protecting VMs, triggering failover, running non-disruptive DR tests, and monitoring replication health—is expressed as a Kubernetes custom resource under the siterecovery.trilio.io/v1alpha1 API group. Understanding these CRDs lets you build GitOps workflows, write automation with kubectl, and reason precisely about the state of your DR estate. Each entry below describes a CRD's purpose, its full spec field reference, status fields reported by the relevant controller, and annotated examples.
Before working with Site Recovery custom resources, ensure the following are in place:
- OpenShift 4.14 or later on all clusters (quorum, primary, and DR)
- OpenShift Virtualization (CNV) 1.0 or later — VMs are
VirtualMachineresources managed by KubeVirt - DRBD kernel module 9.x or later on every worker node
- Helm 3.0 or later — used to install the control plane charts before CRDs become active
kubectlorocCLI configured with kubeconfig files for every cluster in your DR estate- Site Recovery control plane deployed — the
site-recovery-protectionzone-controllerchart must be installed on the quorum cluster before any other chart, and thesite-recovery-quorum-control-planeandsite-recovery-workload-control-planecharts must be deployed before custom resources are applied - Network connectivity — TCP 7000–7999 open between primary and DR worker nodes (DRBD replication); TCP 6443 open from the quorum cluster to primary and DR API servers
- DRBD Operator installed on the primary and DR clusters (required for
ProtectionRequest,DRBDResource, andDRBDReplicationPolicyworkflows)
CRD manifests are installed automatically by the Helm charts. You do not apply CRD YAML files independently under normal circumstances. The installation order matters because the quorum control plane depends on the ProtectionZone webhook.
Step 1 — Install the ProtectionZone controller on the quorum cluster
This chart registers the ProtectionZone CRD and its admission webhook. It must exist before the quorum control plane chart is deployed.
helm upgrade --install site-recovery-protectionzone-controller \
oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
--namespace dr-system \
--create-namespace \
--kubeconfig ~/.kube/config-quorum
Step 2 — Install the quorum control plane on the quorum cluster
This chart deploys the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers as a single controller manager, along with all quorum-side CRDs (FailoverRequest, TestFailover, RPOEvent, ReplicationGroupStatus).
helm upgrade --install site-recovery-quorum-control-plane \
oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
--namespace dr-system \
--kubeconfig ~/.kube/config-quorum \
--set primaryCluster.kubeconfig="$(base64 -w0 ~/.kube/config-primary)" \
--set drCluster.kubeconfig="$(base64 -w0 ~/.kube/config-dr)"
Step 3 — Install the workload control plane on the primary cluster
This chart deploys the drbd-node-agent DaemonSet, admission webhooks, and RBAC on the primary cluster. It also registers the ProtectionGroup, ProtectionRequest, DRBDResource, and DRBDReplicationPolicy CRDs on the workload side.
helm upgrade --install site-recovery-workload-control-plane \
oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
--namespace dr-system \
--create-namespace \
--kubeconfig ~/.kube/config-primary
Step 4 — Install the workload control plane on the DR cluster
helm upgrade --install site-recovery-workload-control-plane \
oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
--namespace dr-system \
--create-namespace \
--kubeconfig ~/.kube/config-dr
Step 5 — Verify CRD registration
Run this against each cluster to confirm all expected CRDs are present:
kubectl get crds | grep siterecovery.trilio.io
Expected output includes:
drbdreplicationpolicies.siterecovery.trilio.io
drbdresources.siterecovery.trilio.io
failoverrequests.siterecovery.trilio.io
protectiongroups.siterecovery.trilio.io
protectionrequests.siterecovery.trilio.io
protectionzones.siterecovery.trilio.io
replicationgroupstatuses.siterecovery.trilio.io
rpoevents.siterecovery.trilio.io
testfailovers.siterecovery.trilio.io
All configuration for Site Recovery is expressed through custom resource spec fields. The sections below document every field for each CRD, grouped by resource kind. Required fields are marked (required).
ProtectionZone
API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Quorum cluster
Managed by: site-recovery-protectionzone-controller
A ProtectionZone defines the top-level DR deployment scope: which clusters participate, which storage backend model is in use, and optional operational settings. Create one ProtectionZone per DR deployment namespace (dr-<name>) on the quorum cluster.
| Field | Type | Required | Description |
|---|---|---|---|
spec.displayName | string | Yes | Human-readable name for this DR deployment, shown in the TSR web console. |
spec.storageBackendMode | string | Yes | Storage orchestration model. Accepted values: drbd-operator. Determines which protection workflow (ProtectionRequest vs. ProtectionGroup) is active. |
spec.clusters | array | Yes | List of cluster references (name, kubeconfig secret reference) that belong to this protection zone. Minimum two entries (primary and DR). |
spec.description | string | No | Free-text description of the protection zone for operator reference. |
spec.logging | object | No | Controls log verbosity and output format for controllers operating within this zone. |
ProtectionGroup
API group/version: siterecovery.trilio.io/v1alpha1
Short name: pg
Where applied: Primary and DR clusters
Managed by: protection-group-controller (workload clusters), pg-sync-controller (quorum cluster)
A ProtectionGroup groups one or more VMs that must fail over together as a coordinated unit. The group's status is the authoritative source for replication health and per-VM protection state. All VMs in a group must reside in the same namespace.
| Field | Type | Required | Description |
|---|---|---|---|
spec.virtualMachines | array | Yes | Ordered list of VM references (name string) to include in this group. All VMs must exist in the same namespace as the ProtectionGroup. |
spec.desiredState | string | No | Operator-driven lifecycle directive. Values: running, stopped. The protection-group-controller reconciles each VM's spec.running field to match. Omit to leave VM lifecycle unmanaged by the controller. |
spec.sourceCluster | string | No | Name of the cluster where VMs are currently active. Informational; used by the failover controller to determine source and target during orchestration. |
spec.resourceGroupName | string | No | Maps this group to a named resource group in the storage backend. |
spec.sla | object | No | Service-level agreement parameters (RPO targets, alerting thresholds) applied to this group. |
Status fields (set by controllers, read-only):
| Field | Description |
|---|---|
status.state | Overall group state: Active, Degraded, Failed. |
status.currentState | Actual running state of VMs: running, stopped, mixed. |
status.replicationHealth | Aggregated replication health: Healthy, Degraded, Critical. |
status.protectedVMs | Per-VM list with replicationStatus, volume details, and sync state. |
ProtectionRequest
API group/version: siterecovery.trilio.io/v1alpha1
Short names: pr, protect
Where applied: Primary cluster (DRBD Operator deployments)
Managed by: protection-controller (quorum cluster)
A ProtectionRequest requests block-level replication protection for a single VM. The protection-controller validates the VM, creates a DRBDResource pair covering all VM disks, and switches the VM to DRBD-backed frontend PVCs.
| Field | Type | Required | Description |
|---|---|---|---|
spec.virtualMachine | object | Yes | Reference to the VM to protect: name (string) and namespace (string). |
spec.sourceCluster | string | Yes | Name of the cluster where the VM currently runs. |
spec.replicationConfig | object | No | Inline replication parameters overriding the default DRBDReplicationPolicy for this VM. Accepts the same sub-fields as DRBDReplicationPolicy.spec. |
Status fields:
| Field | Description |
|---|---|
status.phase | Protection lifecycle phase: Pending, ValidatingVM, ProvisioningDRBDResource, SwitchingPVCs, Protected, Failed. |
status.drbdResourceRef | Name of the DRBDResource created for this VM. |
status.message | Human-readable status message or error detail. |
DRBDReplicationPolicy
API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Primary and DR clusters (DRBD Operator deployments)
A DRBDReplicationPolicy defines how volumes replicate between clusters: storage class mappings, replication endpoints, protocol, and resync behavior. Policies can be marked as default (isDefault: true) so that ProtectionRequest resources without an inline replicationConfig inherit them automatically.
| Field | Type | Required | Description |
|---|---|---|---|
spec.clusters | array | Yes | List of cluster-side definitions, each containing the cluster name and storage class mapping. Minimum two entries. |
spec.replicationMode | string | No | Replication direction mode. Accepted values: active-passive (default). |
spec.drbdProtocol | string | No | DRBD wire protocol. A = asynchronous (low-latency tolerance, small RPO window in seconds). C = synchronous (RPO=0, requires <50ms RTT). Defaults to C. |
spec.isDefault | boolean | No | When true, this policy is applied to any ProtectionRequest that does not specify an inline replicationConfig. At most one policy per namespace should be marked default. |
spec.diskConfig | object | No | Block device and filesystem parameters applied when provisioning DRBD-backed PVCs on the DR side. |
spec.networkConfig | object | No | DRBD replication endpoint addressing, port ranges, and transport options applied to all DRBDResource objects governed by this policy. |
spec.resyncConfig | object | No | Controls initial sync and resync rate limits to avoid saturating the replication link. |
spec.rpo | object | No | RPO threshold configuration: warning and critical lag thresholds in seconds used by the replication monitor to generate RPOEvent records. |
DRBDResource
API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Primary and DR clusters
Managed by: protection-controller (creates); drbd-node-agent (reports state)
A DRBDResource represents all disks of a single VM as one replicated DRBD resource. One CR exists per VM per cluster side (primary side and DR side). Typically created automatically by the protection-controller when processing a ProtectionRequest; you may also create them manually for advanced scenarios.
| Field | Type | Required | Description |
|---|---|---|---|
spec.role | string | Yes | DRBD role for this side. Values: Primary, Secondary. |
spec.side | string | Yes | Logical cluster side. Values: source, target. |
spec.port | integer | Yes | TCP port (in the 7000–7999 range) on which this DRBD resource listens for replication traffic. |
spec.volumes | array | Yes | List of volume definitions: each entry maps a PVC name to its DRBD minor number and backing device path. |
spec.endpoints | array | No | Override list of peer endpoint addresses for cross-cluster replication. If omitted, derived from the governing DRBDReplicationPolicy.spec.networkConfig. |
spec.nodeName | string | No | Worker node on which this resource's DRBD device is active. Informational. |
spec.paused | boolean | No | When true, the drbd-node-agent suspends reconciliation for this resource without deleting it. Useful during maintenance. Defaults to false. |
spec.replication | object | No | Per-resource replication tuning that overrides the parent DRBDReplicationPolicy. |
spec.replicationIntent | string | No | Describes the intended replication relationship (for example, the name of the paired resource on the peer cluster). |
Status fields:
| Field | Description |
|---|---|
status.connectionState | DRBD connection state with the peer: Connected, Connecting, Disconnected, StandAlone. |
status.replicationState | Per-volume sync state: UpToDate, Inconsistent, Outdated, SyncSource, SyncTarget. |
status.syncProgress | Percentage of initial sync or resync completed. |
FailoverRequest
API group/version: siterecovery.trilio.io/v1alpha1
Short name: fr
Where applied: Quorum cluster
Managed by: failover-controller
Creating a FailoverRequest triggers either a planned or unplanned failover for the referenced ProtectionGroup. The failover-controller watches this CR and drives the full orchestration: stopping VMs on the source cluster (planned only), promoting DRBD volumes on the target cluster, and starting VMs there.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the target ProtectionGroup: name and namespace. |
spec.targetCluster | string | Yes | Name of the cluster to which VMs should be moved. |
spec.failoverType | string | No | planned (default) — gracefully stops VMs before promoting volumes, achieving zero data loss. unplanned — force-promotes volumes on the DR cluster without waiting for the primary, for use when the primary is unavailable. |
spec.drainTimeoutSeconds | integer | No | Maximum time in seconds to wait for VMs to stop on the source cluster before aborting (planned failover only). Defaults to 300. |
spec.batchBootTimeoutSeconds | integer | No | Maximum time in seconds to wait for each batch of VMs to reach Running state on the target cluster. Defaults to 600. |
Status fields:
| Field | Description |
|---|---|
status.phase | Failover phase: Pending, StoppingOnSource, PromotingVolumes, StartingOnTarget, Completed, Failed. |
status.message | Human-readable status or error message. |
status.startTime | RFC3339 timestamp when the failover began. |
status.completionTime | RFC3339 timestamp when the failover reached a terminal phase. |
TestFailover
API group/version: siterecovery.trilio.io/v1alpha1
Short name: tf
Where applied: Quorum cluster
Managed by: test-failover-controller
A TestFailover triggers a non-disruptive DR validation. The test-failover-controller creates volume snapshots of protected PVCs, provisions test PVCs from those snapshots, starts cloned VMs in an isolated namespace on the DR cluster, runs optional verification checks, and then cleans up — without affecting production workloads.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the ProtectionGroup to test: name and namespace. |
spec.cleanupPolicy | string | No | Controls what happens after the test ends. Automatic (default) — resources are cleaned up immediately when the test reaches a terminal phase. Retain — resources are kept until the retentionTime elapses, allowing manual inspection. |
spec.retentionTime | string | No | Duration string (e.g., 2h, 30m) specifying how long to retain test resources when cleanupPolicy: Retain. |
spec.timeout | string | No | Maximum total duration for the test failover before the controller marks it Failed and begins cleanup. Example: 1h. |
spec.batchBootTimeoutSeconds | integer | No | Maximum time in seconds to wait for each batch of test VMs to reach Running state. |
spec.verification | object | No | Optional verification configuration: scripts or probes to run against test VMs before the test is marked Succeeded. |
Status fields:
| Field | Description |
|---|---|
status.phase | Test phase: CreatingSnapshots, ProvisioningPVCs, StartingVMs, VerifyingData, Succeeded, Failed, CleaningUp. |
status.testNamespace | Isolated namespace on the DR cluster where test VMs run. |
status.message | Human-readable status or error detail. |
ReplicationGroupStatus
API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Quorum cluster
Managed by: Replication-monitor reconciler (part of site-recovery-quorum-control-plane)
A ReplicationGroupStatus provides an aggregated health view for a ProtectionGroup. The replication monitor creates and updates one of these per group. You read this resource to determine overall health; you do not typically create it manually.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the monitored ProtectionGroup. |
spec.replicationProtocol | string | No | The DRBD protocol in use for this group (A or C). Informational; derived from the governing DRBDReplicationPolicy. |
spec.rpoObjectiveSeconds | integer | No | The RPO objective in seconds against which replication lag is evaluated. Violations produce RPOEvent records. |
spec.pollingIntervalSeconds | integer | No | How frequently the replication monitor polls DRBD volume state. Defaults to 30. Lower values increase API server load. |
Status fields:
| Field | Description |
|---|---|
status.health | Aggregated health: Healthy, Degraded, Critical. |
status.perVolumeDetail | Per-PVC list of sync state, current lag in seconds, and last-sync timestamp. |
status.lastSyncTime | RFC3339 timestamp of the last successful synchronization across all volumes in the group. |
RPOEvent
API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Quorum cluster
Managed by: Replication-monitor reconciler
An RPOEvent is created automatically by the replication monitor when replication lag exceeds the configured RPO threshold for a ProtectionGroup. These records form an immutable audit trail for compliance reporting and can be used to drive alerts. You do not create RPOEvent records manually.
| Field | Type | Required | Description |
|---|---|---|---|
spec.protectionGroupRef | object | Yes | Reference to the ProtectionGroup affected by this violation. |
spec.eventType | string | Yes | Classification of the event. Values: RPOViolation, RPORecovered. |
spec.timestamp | string | Yes | RFC3339 timestamp when the violation was detected. |
spec.rpoAtEvent | integer | No | Observed replication lag in seconds at the time of the event. |
spec.affectedVolumes | array | No | List of PVC names whose replication was lagging at event time. |
spec.violationReason | string | No | Human-readable description of why the RPO was violated (e.g., network congestion, peer disconnected). |
spec.outOfSyncBytesAtEvent | integer | No | Approximate bytes out of sync at event time (derived from DRBD sync progress). |
spec.connectionDetails | object | No | DRBD connection state snapshot captured at event time. |
spec.previousRPOStatus | string | No | Health status immediately before this event (Healthy, Degraded, Critical). |
spec.newRPOStatus | string | No | Health status after this event. |
spec.recoveryEstimate | object | No | Estimated time to full resync if lag is ongoing. |
spec.secondsSinceLastEvent | integer | No | Time elapsed since the previous RPOEvent for this group. |
spec.message | string | No | Additional human-readable context. |
The following workflows represent the most common day-to-day operations using Site Recovery custom resources. All commands use kubectl; substitute oc if you prefer the OpenShift CLI.
Establish a Protection Zone
Before protecting any VMs, define a ProtectionZone on the quorum cluster. This is the top-level container for a DR deployment.
kubectl apply -f protectionzone.yaml --kubeconfig ~/.kube/config-quorum -n dr-production
Protect a VM (DRBD Operator model)
Create a ProtectionRequest on the primary cluster. The protection-controller picks it up and drives the full protection workflow.
kubectl apply -f protectionrequest.yaml --kubeconfig ~/.kube/config-primary -n production
Watch the protection phase progress:
kubectl get protectionrequest my-vm-pr -n production \
--kubeconfig ~/.kube/config-primary \
-w -o jsonpath='{.status.phase}'
When status.phase reaches Protected, the VM's disks are fully replicated.
Check replication health
Read the ReplicationGroupStatus for a group to get an aggregated view:
kubectl get replicationgroupstatus -n dr-production \
--kubeconfig ~/.kube/config-quorum
For per-volume detail:
kubectl get replicationgroupstatus my-group-status -n dr-production \
--kubeconfig ~/.kube/config-quorum -o yaml
Review any recent RPO violations:
kubectl get rpoevents -n dr-production \
--kubeconfig ~/.kube/config-quorum \
--sort-by='.spec.timestamp'
Run a non-disruptive test failover
Create a TestFailover on the quorum cluster. Production workloads continue unaffected.
kubectl apply -f testfailover.yaml --kubeconfig ~/.kube/config-quorum -n dr-production
Watch progress:
kubectl get testfailover my-dr-test -n dr-production \
--kubeconfig ~/.kube/config-quorum \
-w -o jsonpath='{.status.phase}'
Phases progress: CreatingSnapshots → ProvisioningPVCs → StartingVMs → VerifyingData → Succeeded → CleaningUp.
Execute a planned failover
Create a FailoverRequest on the quorum cluster. The failover-controller stops VMs gracefully on the source cluster, promotes DRBD volumes on the target, and starts VMs there.
kubectl apply -f failoverrequest.yaml --kubeconfig ~/.kube/config-quorum -n dr-production
Monitor progress:
kubectl get failoverrequest my-failover -n dr-production \
--kubeconfig ~/.kube/config-quorum \
-w -o jsonpath='{.status.phase}'
The failover is complete when status.phase is Completed.
Execute an unplanned failover
When the primary cluster is unavailable, set spec.failoverType: unplanned. DRBD volumes are force-promoted on the DR cluster without waiting for the primary. Apply the FailoverRequest to the quorum cluster.
kubectl apply -f unplanned-failoverrequest.yaml \
--kubeconfig ~/.kube/config-quorum -n dr-production
Pause replication for maintenance
To suspend DRBD reconciliation for a single VM's DRBDResource without deleting it:
kubectl patch drbdresource my-vm-primary -n production \
--kubeconfig ~/.kube/config-primary \
--type merge -p '{"spec":{"paused":true}}'
Set paused: false to resume.
List all protection state across a namespace
# ProtectionRequests
kubectl get pr -n production --kubeconfig ~/.kube/config-primary
# DRBDResources
kubectl get drbdresource -n production --kubeconfig ~/.kube/config-primary
# FailoverRequests
kubectl get fr -n dr-production --kubeconfig ~/.kube/config-quorum
# TestFailovers
kubectl get tf -n dr-production --kubeconfig ~/.kube/config-quorum
Each example below is self-contained. Copy the YAML, adjust names and cluster-specific values, and apply with kubectl apply -f <file>.
Example 1 — ProtectionZone: Define a DR deployment
Apply to the quorum cluster in namespace dr-production.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: production-zone
namespace: dr-production
spec:
displayName: "Production DR Zone"
storageBackendMode: drbd-operator
description: "Primary-to-DR replication for production VMs"
clusters:
- name: primary-cluster
kubeconfigSecretRef:
name: primary-cluster-kubeconfig
key: kubeconfig
- name: dr-cluster
kubeconfigSecretRef:
name: dr-cluster-kubeconfig
key: kubeconfig
Expected result: The ProtectionZone admission webhook validates the resource and the zone becomes active. Verify:
kubectl get protectionzone production-zone -n dr-production \
--kubeconfig ~/.kube/config-quorum
Example 2 — DRBDReplicationPolicy: Define replication parameters
Apply to the primary cluster. This policy uses synchronous replication (Protocol C, RPO=0) and is marked as the namespace default.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
name: sync-replication-policy
namespace: production
spec:
isDefault: true
drbdProtocol: "C"
replicationMode: active-passive
clusters:
- clusterName: primary-cluster
storageClass: fast-ssd
- clusterName: dr-cluster
storageClass: fast-ssd-dr
networkConfig:
portRange:
start: 7000
end: 7999
rpo:
warningThresholdSeconds: 5
criticalThresholdSeconds: 30
Expected result: Any ProtectionRequest in the production namespace without an inline replicationConfig uses this policy.
Example 3 — ProtectionRequest: Protect a single VM
Apply to the primary cluster. The protection-controller on the quorum cluster validates the VM, creates a DRBDResource pair, and switches the VM to DRBD-backed frontend PVCs.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: web-server-vm-pr
namespace: production
spec:
sourceCluster: primary-cluster
virtualMachine:
name: web-server-vm
namespace: production
Watch the protection lifecycle:
kubectl get pr web-server-vm-pr -n production \
--kubeconfig ~/.kube/config-primary \
-w -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message'
Expected output progression:
NAME PHASE MESSAGE
web-server-vm-pr Pending Awaiting reconciliation
web-server-vm-pr ValidatingVM Checking VM and PVC configuration
web-server-vm-pr ProvisioningDRBDResource Creating DRBD resource pair
web-server-vm-pr SwitchingPVCs Attaching DRBD-backed frontend PVCs
web-server-vm-pr Protected VM successfully protected
Example 4 — DRBDResource: Inspect a protected VM's replication state
The protection-controller creates DRBDResource objects automatically. To inspect the replication state of one VM:
kubectl get drbdresource -n production \
--kubeconfig ~/.kube/config-primary \
-o custom-columns='NAME:.metadata.name,ROLE:.spec.role,CONNECTION:.status.connectionState,SYNC:.status.replicationState,PROGRESS:.status.syncProgress'
Expected output (during initial sync):
NAME ROLE CONNECTION SYNC PROGRESS
web-server-vm-primary Primary Connected SyncSource 100%
Expected output (steady state):
NAME ROLE CONNECTION SYNC PROGRESS
web-server-vm-primary Primary Connected UpToDate 100%
To pause replication during a maintenance window:
apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDResource
metadata:
name: web-server-vm-primary
namespace: production
spec:
role: Primary
side: source
port: 7100
paused: true
volumes:
- pvcName: web-server-vm-disk-0
minorNumber: 1
Example 5 — TestFailover: Non-disruptive DR validation
Apply to the quorum cluster. Uses automatic cleanup after the test completes.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
name: quarterly-dr-test
namespace: dr-production
spec:
protectionGroupRef:
name: production-protection-group
namespace: production
cleanupPolicy: Automatic
timeout: 1h
batchBootTimeoutSeconds: 300
verification:
httpProbe:
path: /healthz
port: 8080
Monitor progress:
kubectl get tf quarterly-dr-test -n dr-production \
--kubeconfig ~/.kube/config-quorum \
-w -o jsonpath='{.status.phase}'
Expected phase progression:
CreatingSnapshots → ProvisioningPVCs → StartingVMs → VerifyingData → Succeeded → CleaningUp
Example 6 — FailoverRequest: Planned failover
Apply to the quorum cluster. VMs on the primary cluster are gracefully stopped before volumes are promoted on the DR cluster.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
name: planned-failover-20260101
namespace: dr-production
spec:
protectionGroupRef:
name: production-protection-group
namespace: production
targetCluster: dr-cluster
failoverType: planned
drainTimeoutSeconds: 300
batchBootTimeoutSeconds: 600
Monitor progress:
kubectl get fr planned-failover-20260101 -n dr-production \
--kubeconfig ~/.kube/config-quorum \
-w -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message'
Expected phase progression:
NAME PHASE MESSAGE
planned-failover-20260101 Pending Analyzing source and target clusters
planned-failover-20260101 StoppingOnSource Stopping VMs on primary-cluster
planned-failover-20260101 PromotingVolumes Promoting DRBD volumes on dr-cluster
planned-failover-20260101 StartingOnTarget Starting VMs on dr-cluster
planned-failover-20260101 Completed All VMs running on dr-cluster
Example 7 — FailoverRequest: Unplanned failover
Use when the primary cluster is unreachable. Set failoverType: unplanned; the controller force-promotes DRBD volumes without waiting for primary shutdown.
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
name: emergency-failover-20260101
namespace: dr-production
spec:
protectionGroupRef:
name: production-protection-group
namespace: production
targetCluster: dr-cluster
failoverType: unplanned
batchBootTimeoutSeconds: 600
Example 8 — ReplicationGroupStatus and RPOEvents: Health audit
List all groups with their aggregated health:
kubectl get replicationgroupstatus -n dr-production \
--kubeconfig ~/.kube/config-quorum \
-o custom-columns='NAME:.metadata.name,HEALTH:.status.health,LAST-SYNC:.status.lastSyncTime'
Expected output:
NAME HEALTH LAST-SYNC
production-protection-group-rgs Healthy 2026-01-01T10:00:00Z
List RPO violations in the last DR namespace (newest first):
kubectl get rpoevents -n dr-production \
--kubeconfig ~/.kube/config-quorum \
--sort-by='.spec.timestamp' \
-o custom-columns='TIME:.spec.timestamp,TYPE:.spec.eventType,LAG:.spec.rpoAtEvent,REASON:.spec.violationReason'
Use a consistent approach for each issue: check the relevant CR's status.phase and status.message first, then inspect controller logs using kubectl logs on the relevant controller pod in the quorum or workload cluster's dr-system namespace.
For a full diagnostic bundle covering all clusters, run:
oc adm must-gather --image=registry.trilio.io/tsr-gather:latest \
--dest-dir=./tsr-bundle \
--kubeconfig ~/.kube/config-quorum
ProtectionRequest stuck in ValidatingVM
Symptom: status.phase stays at ValidatingVM for more than a few minutes; status.message references a PVC or VM validation failure.
Likely causes:
- The referenced VM does not exist in the specified
spec.virtualMachine.namespace. - The VM's PVCs use a storage class that is not mapped in the governing
DRBDReplicationPolicy. - The
protection-controllerpod on the quorum cluster is not running or cannot reach the primary cluster API server.
Fix:
- Confirm the VM exists:
kubectl get vm <name> -n <namespace> --kubeconfig ~/.kube/config-primary - Confirm the PVC storage class is listed in
DRBDReplicationPolicy.spec.clusters[*].storageClassfor the primary cluster entry. - Check
protection-controllerlogs:kubectl logs -l app=site-recovery-quorum-control-plane -n dr-system --kubeconfig ~/.kube/config-quorum | grep protection-controller - Verify network connectivity from the quorum cluster to the primary API server on TCP 6443.
ProtectionRequest stuck in ProvisioningDRBDResource
Symptom: Phase advances past ValidatingVM but does not reach SwitchingPVCs.
Likely causes:
- The
drbd-node-agentDaemonSet on the primary cluster is not ready on all worker nodes. - TCP port range 7000–7999 is not open between primary and DR worker nodes.
- The
DRBDResourceCRD was created but thedrbd-node-agentreports a connection error.
Fix:
- Check the DaemonSet:
kubectl get daemonset -n dr-system --kubeconfig ~/.kube/config-primary - Inspect the created
DRBDResource:kubectl get drbdresource -n <namespace> --kubeconfig ~/.kube/config-primary -o yaml - Look at
status.connectionState; if it showsDisconnectedorStandAlone, firewall rules are the most common cause. - Verify DRBD ports: from a primary worker node,
nc -zv <dr-worker-node-ip> 7000.
FailoverRequest stuck in StoppingOnSource
Symptom: status.phase stays at StoppingOnSource; VMs on the primary cluster remain in Running state.
Likely causes:
- The
failover-controllercannot reach the primary cluster API server. - The
ProtectionGroupon the primary cluster hasstatus.currentState: mixedbecause one or more VMs failed to stop withinspec.drainTimeoutSeconds.
Fix:
- Verify quorum-to-primary connectivity:
kubectl get nodes --kubeconfig ~/.kube/config-primaryfrom the quorum cluster's network context. - Check the
ProtectionGroupstatus on the primary cluster:kubectl get pg <name> -n <namespace> --kubeconfig ~/.kube/config-primary -o yaml - Look at
status.protectedVMsfor any VM withreplicationStatusother thanProtected. - If a single VM is blocking the drain, check KubeVirt VMI status:
kubectl get vmi -n <namespace> --kubeconfig ~/.kube/config-primary - If the primary is genuinely unreachable, switch
spec.failoverTypetounplannedby deleting the existingFailoverRequestand applying a new one withfailoverType: unplanned.
FailoverRequest stuck in PromotingVolumes
Symptom: Source VMs stopped but DR-side VMs never start; phase remains PromotingVolumes.
Likely causes:
- DRBD volumes on the DR cluster cannot be promoted because the
drbd-node-agentreports them asInconsistentorOutdated. - The replication link was interrupted before all data was flushed (Protocol A asynchronous mode).
Fix:
- Check
DRBDResourcestatus on the DR cluster:kubectl get drbdresource -n <namespace> --kubeconfig ~/.kube/config-dr -o yaml - Review
status.replicationStateandstatus.syncProgressfor all volumes. - If volumes are
Inconsistent, assess data loss tolerance. For Protocol C (synchronous), this should not occur; investigate network issues. For Protocol A (asynchronous), some data loss may have occurred at the point of disconnection. - Check
failover-controllerlogs for specific DRBD promotion error messages.
TestFailover stuck in VerifyingData or failing at Succeeded
Symptom: Test VMs start but the phase never advances past VerifyingData.
Likely causes:
- The
spec.verificationprobe (HTTP or script) is failing because the test VM application did not start cleanly from the snapshot. - The
batchBootTimeoutSecondswas too short and the controller timed out waiting for VMs. - The test namespace on the DR cluster is missing required network policies or service accounts.
Fix:
- Find the test namespace:
kubectl get tf <name> -n dr-production --kubeconfig ~/.kube/config-quorum -o jsonpath='{.status.testNamespace}' - Check test VM status in that namespace:
kubectl get vm,vmi -n <testNamespace> --kubeconfig ~/.kube/config-dr - Review test VM logs or console output for application startup errors.
- If the verification probe is the issue, increase
spec.batchBootTimeoutSecondsor relax the probe configuration inspec.verification. - To clean up a stuck TestFailover manually:
kubectl delete tf <name> -n dr-production --kubeconfig ~/.kube/config-quorum; thetest-failover-controllerwill run cleanup.
RPOEvents appearing frequently / ReplicationGroupStatus shows Degraded
Symptom: kubectl get rpoevents shows repeated RPOViolation entries; ReplicationGroupStatus.status.health is Degraded or Critical.
Likely causes:
- Network bandwidth between primary and DR is insufficient for the write rate of protected VMs (most common with Protocol A).
- Intermittent packet loss or elevated latency on the replication link.
- A
DRBDResourceis reportingDisconnected.
Fix:
- Check connection state for all resources:
kubectl get drbdresource -n <namespace> --kubeconfig ~/.kube/config-primary -o jsonpath='{range .items[*]}{.metadata.name}: {.status.connectionState}\n{end}' - Review
spec.rpoAtEventin recentRPOEventrecords to understand lag magnitude. - If latency is the issue and you are using Protocol C, verify RTT is below 50ms:
ping <dr-worker-node-ip>from a primary worker node. - If using Protocol A and lag is acceptable for your RPO SLA, raise
spec.rpo.warningThresholdSecondsin theDRBDReplicationPolicyto reduce alert noise. - Check
drbd-node-agentlogs on affected nodes:kubectl logs -l app=drbd-node-agent -n dr-system --kubeconfig ~/.kube/config-primary
ProtectionGroup validating webhook rejects an update
Symptom: kubectl apply or kubectl patch returns an admission webhook error referencing namespace or VM membership rules.
Likely cause: The update would result in VMs from more than one namespace belonging to the same ProtectionGroup. This is a current tech preview restriction (all member VMs must share one namespace).
Fix:
- Ensure all VMs in the
spec.virtualMachineslist reside in the same namespace as theProtectionGroup. - If your existing group spans multiple namespaces from before this restriction was enforced, delete the group and recreate it as separate single-namespace groups.
- Do not attempt to edit a multi-namespace group; the webhook will reject any spec change on it.