Trilio Site Recovery for Kubernetes
Guide

Custom Resource Reference

Complete API reference for all site recovery custom resources


Overview

This page is the complete API reference for all Trilio Site Recovery Custom Resource Definitions (CRDs). Every DR operation—protecting VMs, triggering failover, running non-disruptive DR tests, and monitoring replication health—is expressed as a Kubernetes custom resource under the siterecovery.trilio.io/v1alpha1 API group. Understanding these CRDs lets you build GitOps workflows, write automation with kubectl, and reason precisely about the state of your DR estate. Each entry below describes a CRD's purpose, its full spec field reference, status fields reported by the relevant controller, and annotated examples.


Prerequisites

Before working with Site Recovery custom resources, ensure the following are in place:

  • OpenShift 4.14 or later on all clusters (quorum, primary, and DR)
  • OpenShift Virtualization (CNV) 1.0 or later — VMs are VirtualMachine resources managed by KubeVirt
  • DRBD kernel module 9.x or later on every worker node
  • Helm 3.0 or later — used to install the control plane charts before CRDs become active
  • kubectl or oc CLI configured with kubeconfig files for every cluster in your DR estate
  • Site Recovery control plane deployed — the site-recovery-protectionzone-controller chart must be installed on the quorum cluster before any other chart, and the site-recovery-quorum-control-plane and site-recovery-workload-control-plane charts must be deployed before custom resources are applied
  • Network connectivity — TCP 7000–7999 open between primary and DR worker nodes (DRBD replication); TCP 6443 open from the quorum cluster to primary and DR API servers
  • DRBD Operator installed on the primary and DR clusters (required for ProtectionRequest, DRBDResource, and DRBDReplicationPolicy workflows)

Installation

CRD manifests are installed automatically by the Helm charts. You do not apply CRD YAML files independently under normal circumstances. The installation order matters because the quorum control plane depends on the ProtectionZone webhook.

Step 1 — Install the ProtectionZone controller on the quorum cluster

This chart registers the ProtectionZone CRD and its admission webhook. It must exist before the quorum control plane chart is deployed.

helm upgrade --install site-recovery-protectionzone-controller \
  oci://registry.trilio.io/charts/site-recovery-protectionzone-controller \
  --namespace dr-system \
  --create-namespace \
  --kubeconfig ~/.kube/config-quorum

Step 2 — Install the quorum control plane on the quorum cluster

This chart deploys the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers as a single controller manager, along with all quorum-side CRDs (FailoverRequest, TestFailover, RPOEvent, ReplicationGroupStatus).

helm upgrade --install site-recovery-quorum-control-plane \
  oci://registry.trilio.io/charts/site-recovery-quorum-control-plane \
  --namespace dr-system \
  --kubeconfig ~/.kube/config-quorum \
  --set primaryCluster.kubeconfig="$(base64 -w0 ~/.kube/config-primary)" \
  --set drCluster.kubeconfig="$(base64 -w0 ~/.kube/config-dr)"

Step 3 — Install the workload control plane on the primary cluster

This chart deploys the drbd-node-agent DaemonSet, admission webhooks, and RBAC on the primary cluster. It also registers the ProtectionGroup, ProtectionRequest, DRBDResource, and DRBDReplicationPolicy CRDs on the workload side.

helm upgrade --install site-recovery-workload-control-plane \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --namespace dr-system \
  --create-namespace \
  --kubeconfig ~/.kube/config-primary

Step 4 — Install the workload control plane on the DR cluster

helm upgrade --install site-recovery-workload-control-plane \
  oci://registry.trilio.io/charts/site-recovery-workload-control-plane \
  --namespace dr-system \
  --create-namespace \
  --kubeconfig ~/.kube/config-dr

Step 5 — Verify CRD registration

Run this against each cluster to confirm all expected CRDs are present:

kubectl get crds | grep siterecovery.trilio.io

Expected output includes:

drbdreplicationpolicies.siterecovery.trilio.io
drbdresources.siterecovery.trilio.io
failoverrequests.siterecovery.trilio.io
protectiongroups.siterecovery.trilio.io
protectionrequests.siterecovery.trilio.io
protectionzones.siterecovery.trilio.io
replicationgroupstatuses.siterecovery.trilio.io
rpoevents.siterecovery.trilio.io
testfailovers.siterecovery.trilio.io

Configuration

All configuration for Site Recovery is expressed through custom resource spec fields. The sections below document every field for each CRD, grouped by resource kind. Required fields are marked (required).


ProtectionZone

API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Quorum cluster
Managed by: site-recovery-protectionzone-controller

A ProtectionZone defines the top-level DR deployment scope: which clusters participate, which storage backend model is in use, and optional operational settings. Create one ProtectionZone per DR deployment namespace (dr-<name>) on the quorum cluster.

FieldTypeRequiredDescription
spec.displayNamestringYesHuman-readable name for this DR deployment, shown in the TSR web console.
spec.storageBackendModestringYesStorage orchestration model. Accepted values: drbd-operator. Determines which protection workflow (ProtectionRequest vs. ProtectionGroup) is active.
spec.clustersarrayYesList of cluster references (name, kubeconfig secret reference) that belong to this protection zone. Minimum two entries (primary and DR).
spec.descriptionstringNoFree-text description of the protection zone for operator reference.
spec.loggingobjectNoControls log verbosity and output format for controllers operating within this zone.

ProtectionGroup

API group/version: siterecovery.trilio.io/v1alpha1
Short name: pg
Where applied: Primary and DR clusters
Managed by: protection-group-controller (workload clusters), pg-sync-controller (quorum cluster)

A ProtectionGroup groups one or more VMs that must fail over together as a coordinated unit. The group's status is the authoritative source for replication health and per-VM protection state. All VMs in a group must reside in the same namespace.

FieldTypeRequiredDescription
spec.virtualMachinesarrayYesOrdered list of VM references (name string) to include in this group. All VMs must exist in the same namespace as the ProtectionGroup.
spec.desiredStatestringNoOperator-driven lifecycle directive. Values: running, stopped. The protection-group-controller reconciles each VM's spec.running field to match. Omit to leave VM lifecycle unmanaged by the controller.
spec.sourceClusterstringNoName of the cluster where VMs are currently active. Informational; used by the failover controller to determine source and target during orchestration.
spec.resourceGroupNamestringNoMaps this group to a named resource group in the storage backend.
spec.slaobjectNoService-level agreement parameters (RPO targets, alerting thresholds) applied to this group.

Status fields (set by controllers, read-only):

FieldDescription
status.stateOverall group state: Active, Degraded, Failed.
status.currentStateActual running state of VMs: running, stopped, mixed.
status.replicationHealthAggregated replication health: Healthy, Degraded, Critical.
status.protectedVMsPer-VM list with replicationStatus, volume details, and sync state.

ProtectionRequest

API group/version: siterecovery.trilio.io/v1alpha1
Short names: pr, protect
Where applied: Primary cluster (DRBD Operator deployments)
Managed by: protection-controller (quorum cluster)

A ProtectionRequest requests block-level replication protection for a single VM. The protection-controller validates the VM, creates a DRBDResource pair covering all VM disks, and switches the VM to DRBD-backed frontend PVCs.

FieldTypeRequiredDescription
spec.virtualMachineobjectYesReference to the VM to protect: name (string) and namespace (string).
spec.sourceClusterstringYesName of the cluster where the VM currently runs.
spec.replicationConfigobjectNoInline replication parameters overriding the default DRBDReplicationPolicy for this VM. Accepts the same sub-fields as DRBDReplicationPolicy.spec.

Status fields:

FieldDescription
status.phaseProtection lifecycle phase: Pending, ValidatingVM, ProvisioningDRBDResource, SwitchingPVCs, Protected, Failed.
status.drbdResourceRefName of the DRBDResource created for this VM.
status.messageHuman-readable status message or error detail.

DRBDReplicationPolicy

API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Primary and DR clusters (DRBD Operator deployments)

A DRBDReplicationPolicy defines how volumes replicate between clusters: storage class mappings, replication endpoints, protocol, and resync behavior. Policies can be marked as default (isDefault: true) so that ProtectionRequest resources without an inline replicationConfig inherit them automatically.

FieldTypeRequiredDescription
spec.clustersarrayYesList of cluster-side definitions, each containing the cluster name and storage class mapping. Minimum two entries.
spec.replicationModestringNoReplication direction mode. Accepted values: active-passive (default).
spec.drbdProtocolstringNoDRBD wire protocol. A = asynchronous (low-latency tolerance, small RPO window in seconds). C = synchronous (RPO=0, requires <50ms RTT). Defaults to C.
spec.isDefaultbooleanNoWhen true, this policy is applied to any ProtectionRequest that does not specify an inline replicationConfig. At most one policy per namespace should be marked default.
spec.diskConfigobjectNoBlock device and filesystem parameters applied when provisioning DRBD-backed PVCs on the DR side.
spec.networkConfigobjectNoDRBD replication endpoint addressing, port ranges, and transport options applied to all DRBDResource objects governed by this policy.
spec.resyncConfigobjectNoControls initial sync and resync rate limits to avoid saturating the replication link.
spec.rpoobjectNoRPO threshold configuration: warning and critical lag thresholds in seconds used by the replication monitor to generate RPOEvent records.

DRBDResource

API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Primary and DR clusters
Managed by: protection-controller (creates); drbd-node-agent (reports state)

A DRBDResource represents all disks of a single VM as one replicated DRBD resource. One CR exists per VM per cluster side (primary side and DR side). Typically created automatically by the protection-controller when processing a ProtectionRequest; you may also create them manually for advanced scenarios.

FieldTypeRequiredDescription
spec.rolestringYesDRBD role for this side. Values: Primary, Secondary.
spec.sidestringYesLogical cluster side. Values: source, target.
spec.portintegerYesTCP port (in the 7000–7999 range) on which this DRBD resource listens for replication traffic.
spec.volumesarrayYesList of volume definitions: each entry maps a PVC name to its DRBD minor number and backing device path.
spec.endpointsarrayNoOverride list of peer endpoint addresses for cross-cluster replication. If omitted, derived from the governing DRBDReplicationPolicy.spec.networkConfig.
spec.nodeNamestringNoWorker node on which this resource's DRBD device is active. Informational.
spec.pausedbooleanNoWhen true, the drbd-node-agent suspends reconciliation for this resource without deleting it. Useful during maintenance. Defaults to false.
spec.replicationobjectNoPer-resource replication tuning that overrides the parent DRBDReplicationPolicy.
spec.replicationIntentstringNoDescribes the intended replication relationship (for example, the name of the paired resource on the peer cluster).

Status fields:

FieldDescription
status.connectionStateDRBD connection state with the peer: Connected, Connecting, Disconnected, StandAlone.
status.replicationStatePer-volume sync state: UpToDate, Inconsistent, Outdated, SyncSource, SyncTarget.
status.syncProgressPercentage of initial sync or resync completed.

FailoverRequest

API group/version: siterecovery.trilio.io/v1alpha1
Short name: fr
Where applied: Quorum cluster
Managed by: failover-controller

Creating a FailoverRequest triggers either a planned or unplanned failover for the referenced ProtectionGroup. The failover-controller watches this CR and drives the full orchestration: stopping VMs on the source cluster (planned only), promoting DRBD volumes on the target cluster, and starting VMs there.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the target ProtectionGroup: name and namespace.
spec.targetClusterstringYesName of the cluster to which VMs should be moved.
spec.failoverTypestringNoplanned (default) — gracefully stops VMs before promoting volumes, achieving zero data loss. unplanned — force-promotes volumes on the DR cluster without waiting for the primary, for use when the primary is unavailable.
spec.drainTimeoutSecondsintegerNoMaximum time in seconds to wait for VMs to stop on the source cluster before aborting (planned failover only). Defaults to 300.
spec.batchBootTimeoutSecondsintegerNoMaximum time in seconds to wait for each batch of VMs to reach Running state on the target cluster. Defaults to 600.

Status fields:

FieldDescription
status.phaseFailover phase: Pending, StoppingOnSource, PromotingVolumes, StartingOnTarget, Completed, Failed.
status.messageHuman-readable status or error message.
status.startTimeRFC3339 timestamp when the failover began.
status.completionTimeRFC3339 timestamp when the failover reached a terminal phase.

TestFailover

API group/version: siterecovery.trilio.io/v1alpha1
Short name: tf
Where applied: Quorum cluster
Managed by: test-failover-controller

A TestFailover triggers a non-disruptive DR validation. The test-failover-controller creates volume snapshots of protected PVCs, provisions test PVCs from those snapshots, starts cloned VMs in an isolated namespace on the DR cluster, runs optional verification checks, and then cleans up — without affecting production workloads.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the ProtectionGroup to test: name and namespace.
spec.cleanupPolicystringNoControls what happens after the test ends. Automatic (default) — resources are cleaned up immediately when the test reaches a terminal phase. Retain — resources are kept until the retentionTime elapses, allowing manual inspection.
spec.retentionTimestringNoDuration string (e.g., 2h, 30m) specifying how long to retain test resources when cleanupPolicy: Retain.
spec.timeoutstringNoMaximum total duration for the test failover before the controller marks it Failed and begins cleanup. Example: 1h.
spec.batchBootTimeoutSecondsintegerNoMaximum time in seconds to wait for each batch of test VMs to reach Running state.
spec.verificationobjectNoOptional verification configuration: scripts or probes to run against test VMs before the test is marked Succeeded.

Status fields:

FieldDescription
status.phaseTest phase: CreatingSnapshots, ProvisioningPVCs, StartingVMs, VerifyingData, Succeeded, Failed, CleaningUp.
status.testNamespaceIsolated namespace on the DR cluster where test VMs run.
status.messageHuman-readable status or error detail.

ReplicationGroupStatus

API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Quorum cluster
Managed by: Replication-monitor reconciler (part of site-recovery-quorum-control-plane)

A ReplicationGroupStatus provides an aggregated health view for a ProtectionGroup. The replication monitor creates and updates one of these per group. You read this resource to determine overall health; you do not typically create it manually.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the monitored ProtectionGroup.
spec.replicationProtocolstringNoThe DRBD protocol in use for this group (A or C). Informational; derived from the governing DRBDReplicationPolicy.
spec.rpoObjectiveSecondsintegerNoThe RPO objective in seconds against which replication lag is evaluated. Violations produce RPOEvent records.
spec.pollingIntervalSecondsintegerNoHow frequently the replication monitor polls DRBD volume state. Defaults to 30. Lower values increase API server load.

Status fields:

FieldDescription
status.healthAggregated health: Healthy, Degraded, Critical.
status.perVolumeDetailPer-PVC list of sync state, current lag in seconds, and last-sync timestamp.
status.lastSyncTimeRFC3339 timestamp of the last successful synchronization across all volumes in the group.

RPOEvent

API group/version: siterecovery.trilio.io/v1alpha1
Where applied: Quorum cluster
Managed by: Replication-monitor reconciler

An RPOEvent is created automatically by the replication monitor when replication lag exceeds the configured RPO threshold for a ProtectionGroup. These records form an immutable audit trail for compliance reporting and can be used to drive alerts. You do not create RPOEvent records manually.

FieldTypeRequiredDescription
spec.protectionGroupRefobjectYesReference to the ProtectionGroup affected by this violation.
spec.eventTypestringYesClassification of the event. Values: RPOViolation, RPORecovered.
spec.timestampstringYesRFC3339 timestamp when the violation was detected.
spec.rpoAtEventintegerNoObserved replication lag in seconds at the time of the event.
spec.affectedVolumesarrayNoList of PVC names whose replication was lagging at event time.
spec.violationReasonstringNoHuman-readable description of why the RPO was violated (e.g., network congestion, peer disconnected).
spec.outOfSyncBytesAtEventintegerNoApproximate bytes out of sync at event time (derived from DRBD sync progress).
spec.connectionDetailsobjectNoDRBD connection state snapshot captured at event time.
spec.previousRPOStatusstringNoHealth status immediately before this event (Healthy, Degraded, Critical).
spec.newRPOStatusstringNoHealth status after this event.
spec.recoveryEstimateobjectNoEstimated time to full resync if lag is ongoing.
spec.secondsSinceLastEventintegerNoTime elapsed since the previous RPOEvent for this group.
spec.messagestringNoAdditional human-readable context.

Usage

The following workflows represent the most common day-to-day operations using Site Recovery custom resources. All commands use kubectl; substitute oc if you prefer the OpenShift CLI.


Establish a Protection Zone

Before protecting any VMs, define a ProtectionZone on the quorum cluster. This is the top-level container for a DR deployment.

kubectl apply -f protectionzone.yaml --kubeconfig ~/.kube/config-quorum -n dr-production

Protect a VM (DRBD Operator model)

Create a ProtectionRequest on the primary cluster. The protection-controller picks it up and drives the full protection workflow.

kubectl apply -f protectionrequest.yaml --kubeconfig ~/.kube/config-primary -n production

Watch the protection phase progress:

kubectl get protectionrequest my-vm-pr -n production \
  --kubeconfig ~/.kube/config-primary \
  -w -o jsonpath='{.status.phase}'

When status.phase reaches Protected, the VM's disks are fully replicated.


Check replication health

Read the ReplicationGroupStatus for a group to get an aggregated view:

kubectl get replicationgroupstatus -n dr-production \
  --kubeconfig ~/.kube/config-quorum

For per-volume detail:

kubectl get replicationgroupstatus my-group-status -n dr-production \
  --kubeconfig ~/.kube/config-quorum -o yaml

Review any recent RPO violations:

kubectl get rpoevents -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  --sort-by='.spec.timestamp'

Run a non-disruptive test failover

Create a TestFailover on the quorum cluster. Production workloads continue unaffected.

kubectl apply -f testfailover.yaml --kubeconfig ~/.kube/config-quorum -n dr-production

Watch progress:

kubectl get testfailover my-dr-test -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  -w -o jsonpath='{.status.phase}'

Phases progress: CreatingSnapshotsProvisioningPVCsStartingVMsVerifyingDataSucceededCleaningUp.


Execute a planned failover

Create a FailoverRequest on the quorum cluster. The failover-controller stops VMs gracefully on the source cluster, promotes DRBD volumes on the target, and starts VMs there.

kubectl apply -f failoverrequest.yaml --kubeconfig ~/.kube/config-quorum -n dr-production

Monitor progress:

kubectl get failoverrequest my-failover -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  -w -o jsonpath='{.status.phase}'

The failover is complete when status.phase is Completed.


Execute an unplanned failover

When the primary cluster is unavailable, set spec.failoverType: unplanned. DRBD volumes are force-promoted on the DR cluster without waiting for the primary. Apply the FailoverRequest to the quorum cluster.

kubectl apply -f unplanned-failoverrequest.yaml \
  --kubeconfig ~/.kube/config-quorum -n dr-production

Pause replication for maintenance

To suspend DRBD reconciliation for a single VM's DRBDResource without deleting it:

kubectl patch drbdresource my-vm-primary -n production \
  --kubeconfig ~/.kube/config-primary \
  --type merge -p '{"spec":{"paused":true}}'

Set paused: false to resume.


List all protection state across a namespace

# ProtectionRequests
kubectl get pr -n production --kubeconfig ~/.kube/config-primary

# DRBDResources
kubectl get drbdresource -n production --kubeconfig ~/.kube/config-primary

# FailoverRequests
kubectl get fr -n dr-production --kubeconfig ~/.kube/config-quorum

# TestFailovers
kubectl get tf -n dr-production --kubeconfig ~/.kube/config-quorum

Examples

Each example below is self-contained. Copy the YAML, adjust names and cluster-specific values, and apply with kubectl apply -f <file>.


Example 1 — ProtectionZone: Define a DR deployment

Apply to the quorum cluster in namespace dr-production.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production-zone
  namespace: dr-production
spec:
  displayName: "Production DR Zone"
  storageBackendMode: drbd-operator
  description: "Primary-to-DR replication for production VMs"
  clusters:
    - name: primary-cluster
      kubeconfigSecretRef:
        name: primary-cluster-kubeconfig
        key: kubeconfig
    - name: dr-cluster
      kubeconfigSecretRef:
        name: dr-cluster-kubeconfig
        key: kubeconfig

Expected result: The ProtectionZone admission webhook validates the resource and the zone becomes active. Verify:

kubectl get protectionzone production-zone -n dr-production \
  --kubeconfig ~/.kube/config-quorum

Example 2 — DRBDReplicationPolicy: Define replication parameters

Apply to the primary cluster. This policy uses synchronous replication (Protocol C, RPO=0) and is marked as the namespace default.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-replication-policy
  namespace: production
spec:
  isDefault: true
  drbdProtocol: "C"
  replicationMode: active-passive
  clusters:
    - clusterName: primary-cluster
      storageClass: fast-ssd
    - clusterName: dr-cluster
      storageClass: fast-ssd-dr
  networkConfig:
    portRange:
      start: 7000
      end: 7999
  rpo:
    warningThresholdSeconds: 5
    criticalThresholdSeconds: 30

Expected result: Any ProtectionRequest in the production namespace without an inline replicationConfig uses this policy.


Example 3 — ProtectionRequest: Protect a single VM

Apply to the primary cluster. The protection-controller on the quorum cluster validates the VM, creates a DRBDResource pair, and switches the VM to DRBD-backed frontend PVCs.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: web-server-vm-pr
  namespace: production
spec:
  sourceCluster: primary-cluster
  virtualMachine:
    name: web-server-vm
    namespace: production

Watch the protection lifecycle:

kubectl get pr web-server-vm-pr -n production \
  --kubeconfig ~/.kube/config-primary \
  -w -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message'

Expected output progression:

NAME               PHASE                  MESSAGE
web-server-vm-pr   Pending                Awaiting reconciliation
web-server-vm-pr   ValidatingVM           Checking VM and PVC configuration
web-server-vm-pr   ProvisioningDRBDResource  Creating DRBD resource pair
web-server-vm-pr   SwitchingPVCs          Attaching DRBD-backed frontend PVCs
web-server-vm-pr   Protected              VM successfully protected

Example 4 — DRBDResource: Inspect a protected VM's replication state

The protection-controller creates DRBDResource objects automatically. To inspect the replication state of one VM:

kubectl get drbdresource -n production \
  --kubeconfig ~/.kube/config-primary \
  -o custom-columns='NAME:.metadata.name,ROLE:.spec.role,CONNECTION:.status.connectionState,SYNC:.status.replicationState,PROGRESS:.status.syncProgress'

Expected output (during initial sync):

NAME                        ROLE      CONNECTION   SYNC         PROGRESS
web-server-vm-primary       Primary   Connected    SyncSource   100%

Expected output (steady state):

NAME                        ROLE      CONNECTION   SYNC       PROGRESS
web-server-vm-primary       Primary   Connected    UpToDate   100%

To pause replication during a maintenance window:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: DRBDResource
metadata:
  name: web-server-vm-primary
  namespace: production
spec:
  role: Primary
  side: source
  port: 7100
  paused: true
  volumes:
    - pvcName: web-server-vm-disk-0
      minorNumber: 1

Example 5 — TestFailover: Non-disruptive DR validation

Apply to the quorum cluster. Uses automatic cleanup after the test completes.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: quarterly-dr-test
  namespace: dr-production
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: production
  cleanupPolicy: Automatic
  timeout: 1h
  batchBootTimeoutSeconds: 300
  verification:
    httpProbe:
      path: /healthz
      port: 8080

Monitor progress:

kubectl get tf quarterly-dr-test -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  -w -o jsonpath='{.status.phase}'

Expected phase progression:

CreatingSnapshots → ProvisioningPVCs → StartingVMs → VerifyingData → Succeeded → CleaningUp

Example 6 — FailoverRequest: Planned failover

Apply to the quorum cluster. VMs on the primary cluster are gracefully stopped before volumes are promoted on the DR cluster.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-20260101
  namespace: dr-production
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: production
  targetCluster: dr-cluster
  failoverType: planned
  drainTimeoutSeconds: 300
  batchBootTimeoutSeconds: 600

Monitor progress:

kubectl get fr planned-failover-20260101 -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  -w -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message'

Expected phase progression:

NAME                         PHASE              MESSAGE
planned-failover-20260101    Pending            Analyzing source and target clusters
planned-failover-20260101    StoppingOnSource   Stopping VMs on primary-cluster
planned-failover-20260101    PromotingVolumes   Promoting DRBD volumes on dr-cluster
planned-failover-20260101    StartingOnTarget   Starting VMs on dr-cluster
planned-failover-20260101    Completed          All VMs running on dr-cluster

Example 7 — FailoverRequest: Unplanned failover

Use when the primary cluster is unreachable. Set failoverType: unplanned; the controller force-promotes DRBD volumes without waiting for primary shutdown.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover-20260101
  namespace: dr-production
spec:
  protectionGroupRef:
    name: production-protection-group
    namespace: production
  targetCluster: dr-cluster
  failoverType: unplanned
  batchBootTimeoutSeconds: 600

Example 8 — ReplicationGroupStatus and RPOEvents: Health audit

List all groups with their aggregated health:

kubectl get replicationgroupstatus -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  -o custom-columns='NAME:.metadata.name,HEALTH:.status.health,LAST-SYNC:.status.lastSyncTime'

Expected output:

NAME                            HEALTH    LAST-SYNC
production-protection-group-rgs Healthy   2026-01-01T10:00:00Z

List RPO violations in the last DR namespace (newest first):

kubectl get rpoevents -n dr-production \
  --kubeconfig ~/.kube/config-quorum \
  --sort-by='.spec.timestamp' \
  -o custom-columns='TIME:.spec.timestamp,TYPE:.spec.eventType,LAG:.spec.rpoAtEvent,REASON:.spec.violationReason'

Troubleshooting

Use a consistent approach for each issue: check the relevant CR's status.phase and status.message first, then inspect controller logs using kubectl logs on the relevant controller pod in the quorum or workload cluster's dr-system namespace.

For a full diagnostic bundle covering all clusters, run:

oc adm must-gather --image=registry.trilio.io/tsr-gather:latest \
  --dest-dir=./tsr-bundle \
  --kubeconfig ~/.kube/config-quorum

ProtectionRequest stuck in ValidatingVM

Symptom: status.phase stays at ValidatingVM for more than a few minutes; status.message references a PVC or VM validation failure.

Likely causes:

  • The referenced VM does not exist in the specified spec.virtualMachine.namespace.
  • The VM's PVCs use a storage class that is not mapped in the governing DRBDReplicationPolicy.
  • The protection-controller pod on the quorum cluster is not running or cannot reach the primary cluster API server.

Fix:

  1. Confirm the VM exists: kubectl get vm <name> -n <namespace> --kubeconfig ~/.kube/config-primary
  2. Confirm the PVC storage class is listed in DRBDReplicationPolicy.spec.clusters[*].storageClass for the primary cluster entry.
  3. Check protection-controller logs: kubectl logs -l app=site-recovery-quorum-control-plane -n dr-system --kubeconfig ~/.kube/config-quorum | grep protection-controller
  4. Verify network connectivity from the quorum cluster to the primary API server on TCP 6443.

ProtectionRequest stuck in ProvisioningDRBDResource

Symptom: Phase advances past ValidatingVM but does not reach SwitchingPVCs.

Likely causes:

  • The drbd-node-agent DaemonSet on the primary cluster is not ready on all worker nodes.
  • TCP port range 7000–7999 is not open between primary and DR worker nodes.
  • The DRBDResource CRD was created but the drbd-node-agent reports a connection error.

Fix:

  1. Check the DaemonSet: kubectl get daemonset -n dr-system --kubeconfig ~/.kube/config-primary
  2. Inspect the created DRBDResource: kubectl get drbdresource -n <namespace> --kubeconfig ~/.kube/config-primary -o yaml
  3. Look at status.connectionState; if it shows Disconnected or StandAlone, firewall rules are the most common cause.
  4. Verify DRBD ports: from a primary worker node, nc -zv <dr-worker-node-ip> 7000.

FailoverRequest stuck in StoppingOnSource

Symptom: status.phase stays at StoppingOnSource; VMs on the primary cluster remain in Running state.

Likely causes:

  • The failover-controller cannot reach the primary cluster API server.
  • The ProtectionGroup on the primary cluster has status.currentState: mixed because one or more VMs failed to stop within spec.drainTimeoutSeconds.

Fix:

  1. Verify quorum-to-primary connectivity: kubectl get nodes --kubeconfig ~/.kube/config-primary from the quorum cluster's network context.
  2. Check the ProtectionGroup status on the primary cluster: kubectl get pg <name> -n <namespace> --kubeconfig ~/.kube/config-primary -o yaml
  3. Look at status.protectedVMs for any VM with replicationStatus other than Protected.
  4. If a single VM is blocking the drain, check KubeVirt VMI status: kubectl get vmi -n <namespace> --kubeconfig ~/.kube/config-primary
  5. If the primary is genuinely unreachable, switch spec.failoverType to unplanned by deleting the existing FailoverRequest and applying a new one with failoverType: unplanned.

FailoverRequest stuck in PromotingVolumes

Symptom: Source VMs stopped but DR-side VMs never start; phase remains PromotingVolumes.

Likely causes:

  • DRBD volumes on the DR cluster cannot be promoted because the drbd-node-agent reports them as Inconsistent or Outdated.
  • The replication link was interrupted before all data was flushed (Protocol A asynchronous mode).

Fix:

  1. Check DRBDResource status on the DR cluster: kubectl get drbdresource -n <namespace> --kubeconfig ~/.kube/config-dr -o yaml
  2. Review status.replicationState and status.syncProgress for all volumes.
  3. If volumes are Inconsistent, assess data loss tolerance. For Protocol C (synchronous), this should not occur; investigate network issues. For Protocol A (asynchronous), some data loss may have occurred at the point of disconnection.
  4. Check failover-controller logs for specific DRBD promotion error messages.

TestFailover stuck in VerifyingData or failing at Succeeded

Symptom: Test VMs start but the phase never advances past VerifyingData.

Likely causes:

  • The spec.verification probe (HTTP or script) is failing because the test VM application did not start cleanly from the snapshot.
  • The batchBootTimeoutSeconds was too short and the controller timed out waiting for VMs.
  • The test namespace on the DR cluster is missing required network policies or service accounts.

Fix:

  1. Find the test namespace: kubectl get tf <name> -n dr-production --kubeconfig ~/.kube/config-quorum -o jsonpath='{.status.testNamespace}'
  2. Check test VM status in that namespace: kubectl get vm,vmi -n <testNamespace> --kubeconfig ~/.kube/config-dr
  3. Review test VM logs or console output for application startup errors.
  4. If the verification probe is the issue, increase spec.batchBootTimeoutSeconds or relax the probe configuration in spec.verification.
  5. To clean up a stuck TestFailover manually: kubectl delete tf <name> -n dr-production --kubeconfig ~/.kube/config-quorum; the test-failover-controller will run cleanup.

RPOEvents appearing frequently / ReplicationGroupStatus shows Degraded

Symptom: kubectl get rpoevents shows repeated RPOViolation entries; ReplicationGroupStatus.status.health is Degraded or Critical.

Likely causes:

  • Network bandwidth between primary and DR is insufficient for the write rate of protected VMs (most common with Protocol A).
  • Intermittent packet loss or elevated latency on the replication link.
  • A DRBDResource is reporting Disconnected.

Fix:

  1. Check connection state for all resources: kubectl get drbdresource -n <namespace> --kubeconfig ~/.kube/config-primary -o jsonpath='{range .items[*]}{.metadata.name}: {.status.connectionState}\n{end}'
  2. Review spec.rpoAtEvent in recent RPOEvent records to understand lag magnitude.
  3. If latency is the issue and you are using Protocol C, verify RTT is below 50ms: ping <dr-worker-node-ip> from a primary worker node.
  4. If using Protocol A and lag is acceptable for your RPO SLA, raise spec.rpo.warningThresholdSeconds in the DRBDReplicationPolicy to reduce alert noise.
  5. Check drbd-node-agent logs on affected nodes: kubectl logs -l app=drbd-node-agent -n dr-system --kubeconfig ~/.kube/config-primary

ProtectionGroup validating webhook rejects an update

Symptom: kubectl apply or kubectl patch returns an admission webhook error referencing namespace or VM membership rules.

Likely cause: The update would result in VMs from more than one namespace belonging to the same ProtectionGroup. This is a current tech preview restriction (all member VMs must share one namespace).

Fix:

  1. Ensure all VMs in the spec.virtualMachines list reside in the same namespace as the ProtectionGroup.
  2. If your existing group spans multiple namespaces from before this restriction was enforced, delete the group and recreate it as separate single-namespace groups.
  3. Do not attempt to edit a multi-namespace group; the webhook will reject any spec change on it.