Key Concepts

This page defines the core Kubernetes and Trilio Site Recovery (TSR)-specific concepts you need to understand before planning, deploying, or operating a disaster recovery environment. TSR is a Kubernetes operator (Tech Preview) that automates synchronous block-level replication of KubeVirt virtual machines across two workload clusters, with a third quorum cluster hosting the management plane. Every DR operation—protecting VMs, triggering failover, running non-disruptive tests, and monitoring replication health—is driven by Custom Resource Definitions in the siterecovery.trilio.io/v1alpha1 API group. Understanding the concepts on this page will help you map your infrastructure to TSR's object model and interpret the status fields and events the operator produces.

Cluster topology

A TSR deployment always involves exactly three clusters:

The two workload clusters occupy the fixed deployment slots cluster1 and cluster2. These identifiers are used throughout the API (for example, ProtectionRequest.spec.sourceCluster and FailoverRequest.spec.targetCluster). The slot assignment is fixed at zone creation time and cannot be changed without re-creating the zone.

Single-Node OpenShift (SNO) is supported for both workload clusters and the quorum cluster. On a single-node workload cluster, a node failure means the entire cluster side is lost—plan accordingly.

ProtectionZone

A ProtectionZone (pz) is the top-level configuration object for one TSR deployment. It lives on the quorum cluster in a namespace named dr-<zone-name> and declares:

• Which two workload clusters participate (spec.clusters, exactly two entries, one per slot)
• The kubeconfig Secrets the quorum controllers use to reach each cluster (kubeconfigSecretRef)
• The storage backend (storageBackendMode: DRBD—the only supported value in Tech Preview)
• The OS family of each cluster's nodes (distributionName: RHEL-CoreOS-9 or RHEL-CoreOS-10), which selects the DRBD kernel-module loader image
• Optional runtime log-level tuning (spec.logging)

When a ProtectionZone reaches the Available phase, the ProtectionZone controller automatically deploys:

1. The site-recovery-quorum-control-plane Helm chart to the quorum cluster (into the dr-<name> namespace)
2. The site-recovery-workload-control-plane Helm chart to each workload cluster (into trilio-site-recovery-system)

Customers do not run helm install directly. The charts are embedded in the operator binary and deployed automatically.

The ProtectionZone controller also creates the TSR web console (site-recovery-backend and site-recovery-frontend) and, on OpenShift, an oauth-proxy Route and console link that provide browser access. On upstream Kubernetes, the console pods are deployed but no ingress or authentication path is created—browser access is not a supported use case on non-OpenShift clusters.

Zone lifecycle phases: Provisioning → Available → Degraded or Unavailable → Deleting

Namespace placement rule: Every quorum-side CR (ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent, DRBDReplicationPolicy) must live in the dr-<name> namespace. Workload-side CRs (DRBDResource) live in the protected VM's own namespace on each workload cluster, in trilio-site-recovery-system. DRBD CRs submitted outside a dr--prefixed namespace are rejected by the admission webhook.

DRBDReplicationPolicy

A DRBDReplicationPolicy (drp, drbdpolicy) defines how volumes replicate between the two clusters: storage class mappings, replication endpoints, and DRBD protocol settings.

You do not create or modify this object. When a ProtectionZone is created, the ProtectionZone controller automatically creates one default policy named <zoneName>-default with isDefault: true in the zone's dr-<name> namespace. Only the isDefault policy in a ProtectionRequest's namespace is ever consulted.

Key characteristics of the default policy:

• drbdProtocol: C — Protocol C synchronous replication only (see below)
• replicationMode: Synchronous
• isDefault: true

The networkConfig.portRange field is accepted by the API but has no effect on port allocation in Tech Preview. Ports are always allocated from the fixed range 7000–8000 (one port per replicated VM).

Protocol C (synchronous replication)

Protocol C is DRBD's synchronous replication mode. A write is acknowledged to the application only after it has been committed to disk on both the primary and DR clusters. This guarantees zero data loss (RPO = 0).

Protocol C is the only supported replication protocol in Tech Preview. The admission webhook rejects replicationProtocol: A with the error ReplicationProtocolA is not supported yet. Protocol B is not supported either.

Because replication is always synchronous:

• RPO is always 0 and is not tunable. Setting a non-zero rpo with Protocol C is rejected: rpo must be 0 when spec.replicationConfig.replicationProtocol is C.
• Synchronous replication requires round-trip latency below approximately 50 ms between primary and DR worker nodes. Higher latency increases write latency for VM workloads proportionally.
• TCP ports 7000–8000 inclusive (one port per replicated VM volume) must be open between primary and DR worker nodes. Port 8000 is mandatory—a firewall rule stopping at 7999 silently prevents replication on one volume.

ProtectionGroup

A ProtectionGroup (pg) is a logical bundle of KubeVirt VMs that replicate and fail over together as an atomic unit. It lives on the quorum cluster. All members of a ProtectionGroup share the same failover outcome: you cannot fail over individual VMs from within a group.

Key fields:

• spec.virtualMachines — the list of VMs to protect (at least one required); each entry can carry a bootOrder to control startup sequencing during failover
• spec.sourceCluster — the deployment-time source cluster slot (cluster1 or cluster2); updated by the failover controller when the active cluster changes
• spec.desiredState — whether VMs should be running or stopped on the active cluster

Status fields:

• status.state — one of Protected, Syncing, Degraded, Failed
• status.activeClusterID — which cluster slot (cluster1 or cluster2) currently holds the running VMs
• status.protectedVMs — per-VM replication status (Protected, Syncing, Degraded, NotProtected)

Important: spec.sla.rpoTargetSeconds and spec.sla.rtoTargetSeconds are accepted by the API for compliance reporting but have no operational effect on replication or failover behavior in Tech Preview.

ProtectionRequest

A ProtectionRequest (pr, protect) requests block-level DRBD replication protection for a single KubeVirt VM. It lives on the quorum cluster in the zone's dr-<name> namespace.

What the protection-controller does when you create a ProtectionRequest:

1. Validates the VM and its PVCs on the source cluster
2. Creates a DRBDResource pair—one CR per cluster side, in the VM's namespace on each workload cluster—covering all of the VM's disks as a single DRBD resource
3. Syncs data from the source to the DR side (initial resynchronization)
4. Switches the VM to DRBD-backed frontend PVCs once the initial sync completes

Protection phases (exact, in order): Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected

Additional phases: Unprotecting (teardown in progress), Failed.

Node pinning: A protected VM is pinned to one node per cluster for the lifetime of the protection. The source-side node is the node the VM was running on at protect time (the VM must be running to be protected). The DR-side node is chosen by the scheduler and then latched. spec.nodeName is immutable once set. There is no rebind operation—moving a protected VM to a different node requires a full unprotect and re-protect cycle.

Label on protected VMs: The label siterecovery.trilio.io/protected-vm=true is stamped on the KubeVirt VirtualMachine object on the workload cluster. Use oc get vm -A -l siterecovery.trilio.io/protected-vm=true to list all protected VMs.

Constraints on protected VMs:

• Live migration is denied by admission webhook
• Cold migration (stop, move, start) is not supported for the lifetime of the protection—the node binding is immutable
• oc adm drain of a node hosting a running protected VM will not complete
• Disk addition, removal, resize, and online volume expansion are not supported

Re-protection: Re-protecting a VM always requires a full unprotect and a full initial resynchronization. There is no incremental re-protect path.

DRBDResource

A DRBDResource (dres) represents all disks of a single protected VM as one replicated DRBD resource. One CR exists per VM per cluster side, in the VM's namespace on that workload cluster—never on the quorum cluster.

DRBDResource naming follows the pattern drbd-<namespace>-<vmName>-<hash8>. One CR is created on the primary cluster (with spec.side: Source) and one on the DR cluster (with spec.side: Target). Both share the same metadata.name, the same replication port, and the same volume numbers—they differ in side, role, node binding, and local PVC names.

A VM with up to 15 replicated disks is supported (the API enforces MaxItems: 15 on spec.volumes). Each disk is a volume within the single DRBD resource, numbered from 0 (boot disk).

Status phases: Pending → Provisioning → ReadyToActivate → Syncing → Consistent → Degraded → Failed → Deleting

Each DRBDResource also creates one external-metadata PVC per volume, named drbd-md-<resourceName>-v<volumeNumber>.

The DRBD node agent applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its own node at startup, before touching DRBD, and removes it once every DRBD resource on that node has converged. Workloads that do not tolerate this taint will not schedule on a freshly booted node until DRBD has converged.

FailoverRequest

A FailoverRequest (fr) triggers a failover of a ProtectionGroup from its currently active cluster to a designated target cluster. It lives on the quorum cluster. You create one FailoverRequest per failover event; the object is not reused.

Two failover types:

Planned failover

spec.failoverType: planned (the default). Used when the primary cluster is available and you want a graceful, zero-data-loss switchover. The failover controller:

1. Stops VMs on the source cluster in reverse boot-order
2. Waits for DRBD to confirm the final sync
3. Promotes the DR DRBD volumes to Primary
4. Starts VMs on the target cluster in boot-order sequence

Because Protocol C guarantees all writes were committed to both sides before acknowledgment, a successful planned failover results in zero data loss.

Unplanned failover

spec.failoverType: unplanned. Used when the primary cluster is unavailable (a disaster scenario). DRBD volumes are force-promoted on the DR cluster without waiting for the primary to shut down gracefully. If replication was Healthy and Consistent at the moment of failure, zero data loss occurs. If replication was Degraded, minimal data loss may occur.

Failover phases (granular): Initializing → Validating → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → Completed or Failed

Overall state: Pending → InProgress → Completed or Failed or Cancelled

Per-VM status: The status.vmStatuses array reports each VM's individual phase (Pending, Stopping, Stopped, Starting, Running, Failed). There is no failedVMs field—use status.vmStatuses to find failed VMs.

Wall-clock budget: The failover controller operates within a 5-minute wall-clock budget.

Note on drainTimeoutSeconds: This field is accepted and range-checked by the admission webhook but is not consumed by the failover controller in Tech Preview. It has no effect on drain or failover behavior and should not be relied upon.

TestFailover

A TestFailover (tfo) triggers a non-disruptive DR validation. It does not affect production VMs. The test-failover controller:

1. Prepares an isolated test namespace (status.testNamespace, generated as site-recovery-<protection-group>-<random>) with a NetworkPolicy fence
2. Creates volume snapshots of the DR-side DRBD volumes
3. Provisions test PVCs from those snapshots
4. Starts test VMs in the isolated namespace
5. Runs verification checks (automatic boot/agent/filesystem checks, plus optional user-configured network reachability probes)
6. Cleans up all test resources

The test namespace is generated by the controller and reported on status.testNamespace. You do not specify, construct, or guess it.

TestFailover phases (exact, in order): Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded or Failed → CleaningUp → Cleaned

Target cluster selection: The controller automatically selects the cluster that is not currently running the production VMs. The selection signal is stamped on status.targetClusterReason (one of vm_runstrategy, pg_status, or pg_spec_sourceCluster).

Cleanup policy: spec.cleanupPolicy: Automatic (default) tears down test resources automatically after spec.retentionTime (default 2h). cleanupPolicy: Manual leaves test resources until you delete the TestFailover CR.

Admission gates: A TestFailover is rejected at creation if a FailoverRequest in Pending or InProgress state already exists for the same ProtectionGroup. A Completed FailoverRequest does not block.

ReplicationGroupStatus

A ReplicationGroupStatus (rgs) aggregates replication health for a ProtectionGroup into a single object. The replication-monitor reconciler auto-creates exactly one per ProtectionGroup, named <protection-group-name>-replication. You do not create or delete this object—it is owned by the replication monitor.

Key status fields:

• status.rpoStatus — Met, Degraded, Violated, or Unknown
• status.aggregated.connectionState — worst-case connection state across all VMs: Connected, Degraded, or Disconnected
• status.aggregated.syncedVolumeCount / totalVolumeCount — how many VMs (not disks) are fully in sync
• status.volumeReplicationStatuses — one entry per VM (one per DRBD resource, since all of a VM's disks are volumes of a single resource since TSR-90)
• status.lastFullySyncedAt — when every VM in the group was last fully in sync

To find the ReplicationGroupStatus for a ProtectionGroup, use the label siterecovery.trilio.io/protection-group=<pg-name>. Field selectors on spec.* or status.* do not work on custom resources.

RPOEvent

An RPOEvent (rpoe) is an immutable audit record of a single replication event for a ProtectionGroup. The replication monitor writes these automatically; you cannot create or modify them. Each RPOEvent captures the event type, RPO at the time of the event, affected volumes, and suggested remediation actions.

Event types: SyncCompleted, RPODegraded, RPOViolated, ConnectionLost, ConnectionRestored, ResyncStarted, ResyncCompleted, VolumeAdded, VolumeRemoved, MetadataUndersized, ResizePending, ResizeComplete

RPOEvents provide an audit trail for compliance reporting. Because they are immutable, any update attempt is rejected.

To find RPOEvents for a specific ProtectionGroup, filter by the label siterecovery.trilio.io/protection-group=<pg-name>.

DRBD node agent

The drbd-node-agent runs as a DaemonSet on every worker node of the primary and DR clusters. It manages DRBD kernel-level block replication for individual VM volumes and reports volume and connection state back to the quorum controllers via DRBDResource status.

The DaemonSet object name follows the pattern <release>-site-recovery-workload-control-plane-agent. The component label is app.kubernetes.io/component=drbd-node-agent.

The agent requires privileged access (privileged: true, hostNetwork: true, hostPID: true) and is bound to the stock privileged SCC on OpenShift automatically.

TSR web console

The TSR web console consists of two components installed by the operator with no feature flag and no opt-out:

• site-recovery-frontend — a Next.js/React UI
• site-recovery-backend — a Python Flask API served under gunicorn

On OpenShift, the ProtectionZone controller creates an oauth-proxy Route and console link, providing browser access automatically. On upstream Kubernetes, the pods are deployed but no ingress or authentication path is provided—browser access is not a supported use case.

Zone-scoped console pages derive the dr-<zoneId> namespace from the zone ID. A zone in a differently named namespace will appear in listings but its detail pages will not work.

tsr-gather (support bundle)

tsr-gather is an oc adm must-gather-compatible tool packaged as the site-recovery-must-gather image. Run it against the quorum cluster to collect logs, CRD state, and configuration from all clusters in the DR estate into a single diagnostic bundle.

The binary inside the image is /usr/bin/gather. No -- <command> override is needed when invoking oc adm must-gather—it runs the ENTRYPOINT automatically.

Scale limits and constraints summary

Namespace reference

Examining a ProtectionZone

After installing TSR and creating a ProtectionZone, check its phase and per-cluster reachability:

# On the quorum cluster — short name 'pz' works
oc get pz -n dr-prod

Expected output (columns from the CRD's additionalPrinterColumns):

NAME   DISPLAY NAME      BACKEND   CLUSTERS           STATUS      AGE
prod   Production Zone   DRBD      cluster1,cluster2   Available   12m

Inspect cluster reachability:

oc get pz prod -n dr-prod -o jsonpath='{.status.clusters}' | python3 -m json.tool

[
  {"name": "cluster1", "clusterID": "cluster1", "status": "Reachable"},
  {"name": "cluster2", "clusterID": "cluster2", "status": "Reachable"}
]

Checking a ProtectionRequest's phase and sync progress

# Short names: pr or protect
oc get pr -n dr-prod

Expected output:

NAME          VM        NAMESPACE   PHASE       SYNC   AGE
protect-vm1   vm1       prod-vms    Syncing     47     8m

To watch initial sync progress on a specific ProtectionRequest:

oc get pr protect-vm1 -n dr-prod -w \
  -o custom-columns='PHASE:.status.phase,SYNC:.status.syncProgress,SLOWEST:.status.slowestVolumeSyncProgress.percentage'

Listing all protected VMs

The label siterecovery.trilio.io/protected-vm=true is stamped on the KubeVirt VirtualMachine object on the workload cluster:

# Run against the primary workload cluster
oc get vm -A -l siterecovery.trilio.io/protected-vm=true

NAMESPACE   NAME   AGE   STATUS    READY
prod-vms    vm1    10m   Running   True
prod-vms    vm2    10m   Running   True

Viewing replication health for a ProtectionGroup

The ReplicationGroupStatus is auto-created as <pg-name>-replication:

oc get rgs my-pg-replication -n dr-prod

NAME                PROTECTION-GROUP   RPO-STATUS   OUT-OF-SYNC   SYNCED   CONNECTION   LAST-SYNC
my-pg-replication   my-pg              Met           0             3        Connected    2m ago

For per-VM detail:

oc get rgs my-pg-replication -n dr-prod \
  -o jsonpath='{.status.volumeReplicationStatuses}' | python3 -m json.tool

Checking DRBD node agent status on a specific node

The DaemonSet object name follows the pattern <release>-site-recovery-workload-control-plane-agent. To find the agent pod on a specific node (two steps, because kubectl logs has no --field-selector):

# Step 1: find the pod name
POD=$(oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=worker-1 \
  -o jsonpath='{.items[0].metadata.name}')

# Step 2: tail its logs
oc logs -n trilio-site-recovery-system "$POD" --tail=50

Monitoring a planned failover

oc get fr -n dr-prod -w

NAME             PROTECTION GROUP   TARGET     STATE        PHASE             AGE
pg-failover-1    my-pg              cluster2   InProgress   StoppingSource    45s
pg-failover-1    my-pg              cluster2   InProgress   RemovingTaints    90s
pg-failover-1    my-pg              cluster2   InProgress   StartingTarget    2m
pg-failover-1    my-pg              cluster2   Completed    Completed         4m

To see per-VM status during the failover:

oc get fr pg-failover-1 -n dr-prod \
  -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool

Monitoring a TestFailover

oc get tfo my-test -n dr-prod -w

NAME      PROTECTION GROUP   PHASE          PASSED   AGE
my-test   my-pg              Verifying             3m
my-test   my-pg              Succeeded      true     6m

To find the isolated test namespace the controller generated:

oc get tfo my-test -n dr-prod -o jsonpath='{.status.testNamespace}'

site-recovery-my-pg-a3f7c2

Collecting a diagnostic bundle

Run tsr-gather from the quorum cluster. The oc adm must-gather command runs the /usr/bin/gather binary inside the image automatically:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag>

To restrict collection to a shorter log window or a single zone:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --zone prod

• Plan and verify infrastructure — prerequisite checklist for network ports, DRBD kernel module, LVM thin pools, and cluster connectivity before deploying TSR
• Deploy the workload control plane — how the ProtectionZone controller installs the DRBD node agent DaemonSet and workload admission webhooks automatically
• Deploy the quorum control plane — OLM installation of the operator bundle and ProtectionZone CR creation
• Configure cross-cluster replication — understanding storage class mappings and the auto-created DRBDReplicationPolicy
• Protect virtual machines — creating ProtectionRequests and interpreting protection phases
• Verify replication health — reading ReplicationGroupStatus and RPOEvent objects
• Run a test failover — creating a TestFailover CR and interpreting verification results
• Execute a planned failover — creating a FailoverRequest with failoverType: planned
• Execute an unplanned failover — creating a FailoverRequest with failoverType: unplanned when the primary cluster is unavailable
• Perform failback — returning VMs to the original primary after a failover
• Diagnose a degraded or failed node or replication link — runbooks for DRBDResource node loss and replication health issues
• Upgrade Site Recovery components — operator upgrade path via OLM