Trilio Site Recovery for Kubernetes
Docs/site-recovery/Getting Started
Tutorial

Getting Started

Installing the operator and creating your first site recovery resource

Overview

This page walks you through installing Trilio Site Recovery on your OpenShift quorum cluster and creating your first protection zone — the foundational resource that wires together your primary and DR clusters and unlocks every DR workflow. By the end you will have a running operator, a ProtectionZone in the Available phase, and a protected virtual machine replicating synchronously (Protocol C, RPO=0) between two OpenShift clusters. Trilio Site Recovery is currently in Tech Preview; no component is separately GA.

Prerequisites

Before you begin, confirm the following are in place:

Clusters

  • Three OpenShift clusters: one quorum cluster (management plane) and two workload clusters (primary and DR). The quorum cluster is required — it is not optional.
  • All three clusters must run OpenShift 4.17 or later. The OLM bundle annotations (com.redhat.openshift.versions: v4.14-v4.19, from deploy/olm-catalog/site-recovery/metadata/annotations.yaml) describe the installable range; the validated floor per docs/tech-preview-supported-configurations.md is OpenShift 4.17. Versions 4.14–4.16 are not supported even if the bundle can be installed on them.
  • Single-Node OpenShift is supported for both workload and quorum roles.

Virtualization

  • OpenShift Virtualization (CNV) installed on both workload clusters.

Storage

  • DRBD kernel module ≥ 9.x available on every worker node of both workload clusters (the operator's DRBD module-loader init container handles this on RHEL CoreOS 9 nodes automatically).
  • An LVM thin pool configured on each worker node of the primary and DR clusters, backed by a dedicated block device, for DRBD backing storage.

Network

  • TCP ports 7000–8000 inclusive open between primary and DR worker nodes (one port per replicated VM volume; port 8000 is mandatory — a firewall that stops at 7999 will silently prevent replication on the last volume).
  • TCP port 6443 open from the quorum cluster to both workload cluster API servers.
  • Round-trip latency below 50 ms between primary and DR clusters (required for Protocol C synchronous replication).

CLI tools

  • oc CLI — required for all OpenShift-specific paths (oc adm must-gather, Routes, SCC binding). kubectl-only operation is not a supported mode.
  • Kubeconfig files for all three clusters.

Registry access

  • Outbound HTTPS access to quay.io/triliovault from all three clusters.

Optional components (enable additional capabilities)

  • Containerized Data Importer (CDI) — for DataVolume-based VMs.
  • Multus CNI — for multi-network VMs.
Quick start

The minimal path from zero to a protected virtual machine:

  1. Install the operator on the quorum cluster via OperatorHub by creating a CatalogSource, OperatorGroup, and Subscription in the trilio-site-recovery-system namespace.

  2. Create kubeconfig Secrets for both workload clusters in the zone namespace on the quorum cluster.

  3. Create a ProtectionZone CR in a dr-<name> namespace on the quorum cluster — the controller automatically deploys the quorum and workload control planes.

  4. Wait for the zone to reach Available (oc get pz -n dr-<name>).

  5. Protect a VM by creating a ProtectionRequest CR on the quorum cluster and waiting for Protected phase.

Detailed steps for each of these actions follow in the Steps section below.

Steps

Step 1 — Install the operator on the quorum cluster

All operator resources install into the trilio-site-recovery-system namespace. The OLM bundle deploys three components: the protection-zone-controller, the site-recovery-backend Flask API, and the site-recovery-frontend Next.js UI with its OpenShift oauth-proxy sidecar. There is no feature flag and no opt-out for the console.

Create a CatalogSource pointing at the Trilio catalog image:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: trilio-site-recovery-catalog
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/triliovault/site-recovery-catalog:latest
  displayName: Trilio Site Recovery
  publisher: Trilio Data, Inc.

oc --kubeconfig $KUBECONFIG_QUORUM apply -f catalogsource.yaml

Create an OperatorGroup scoped to trilio-site-recovery-system:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trilio-site-recovery-og
  namespace: trilio-site-recovery-system
spec:
  targetNamespaces:
    - trilio-site-recovery-system

oc --kubeconfig $KUBECONFIG_QUORUM apply -f operatorgroup.yaml

Create a Subscription for the site-recovery package on the tech-preview channel:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: tech-preview
  name: site-recovery
  source: trilio-site-recovery-catalog
  sourceNamespace: openshift-marketplace

oc --kubeconfig $KUBECONFIG_QUORUM apply -f subscription.yaml

Verify the operator pod reaches Running:

oc --kubeconfig $KUBECONFIG_QUORUM get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery

Expected output: the protection-zone-controller, site-recovery-backend, and site-recovery-frontend pods are all Running.

Step 2 — Create kubeconfig Secrets for both workload clusters

The ProtectionZone controller reaches each workload cluster through a kubeconfig stored as a Kubernetes Secret in the zone namespace. Create the zone namespace first (it must follow the dr-<name> pattern — the DRBD admission webhook rejects CRs outside a dr--prefixed namespace):

oc --kubeconfig $KUBECONFIG_QUORUM create namespace dr-prod

Create a Secret for the primary cluster (cluster1):

oc --kubeconfig $KUBECONFIG_QUORUM create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=$KUBECONFIG_CLUSTER1 \
  -n dr-prod

Create a Secret for the DR cluster (cluster2):

oc --kubeconfig $KUBECONFIG_QUORUM create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=$KUBECONFIG_CLUSTER2 \
  -n dr-prod

Why these Secrets matter: the quorum controller projects them into the site-recovery-quorum-control-plane manager pod at /etc/kubeconfig/cluster1 and /etc/kubeconfig/cluster2. The manager exits at startup if either file is missing, so both Secrets must exist before the zone reaches Available.

Step 3 — Create the ProtectionZone CR

The ProtectionZone CR is the root configuration object for a DR deployment. Creating it triggers the protection-zone-controller to:

  1. Deploy the site-recovery-quorum-control-plane Helm chart into the dr-prod namespace on the quorum cluster.
  2. Deploy the site-recovery-workload-control-plane Helm chart (DRBD node agent DaemonSet, webhooks, RBAC) into trilio-site-recovery-system on both workload clusters.
  3. Create a default DRBDReplicationPolicy named dr-prod-default with isDefault: true and Protocol C (you do not author or modify this object).
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: dr-prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  description: "Primary to DR site synchronous replication."
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionzone.yaml

Key fields explained:

  • storageBackendMode: DRBD — the only supported backend; the admission webhook rejects Linstor.
  • clusterID — must be cluster1 or cluster2; these stable slot identifiers are what all other CRs (ProtectionRequest, FailoverRequest) reference.
  • distributionName — selects the DRBD kernel-module loader image. Use RHEL-CoreOS-9 for OpenShift 4.17–4.19 on RHEL CoreOS 9 nodes. If your cluster has moved to a RHEL CoreOS 10 base, use RHEL-CoreOS-10 (not validated in Tech Preview per docs/tech-preview-supported-configurations.md).
  • kubeconfigSecretRef — resolved in the ProtectionZone's own namespace (dr-prod).

Step 4 — Verify the zone reaches Available

Watch zone status on the quorum cluster. Note that quorum-side components and CRs live in dr-prod; workload-side components live in trilio-site-recovery-system on each workload cluster — these two namespaces are always distinct.

# Watch zone phase (quorum cluster)
oc --kubeconfig $KUBECONFIG_QUORUM get pz -n dr-prod -w

# Inspect per-cluster reachability
oc --kubeconfig $KUBECONFIG_QUORUM get pz dr-prod -n dr-prod -o yaml

Successful output from get pz:

NAME      DISPLAY NAME          BACKEND   CLUSTERS        STATUS      AGE
dr-prod   Production DR Zone    DRBD      primary,dr      Available   2m

Also verify the control planes came up on both workload clusters:

# DRBD node agent DaemonSet on the primary cluster
oc --kubeconfig $KUBECONFIG_CLUSTER1 get daemonset \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

# Quorum manager on the quorum cluster
oc --kubeconfig $KUBECONFIG_QUORUM get deploy \
  -n dr-prod \
  -l app.kubernetes.io/component=quorum-control-plane

The quorum manager pod runs a single container named manager and shows 1/1 Running. The node-agent DaemonSet should show one pod per worker node on each workload cluster.

If the zone stays in Provisioning or moves to Degraded, check the status conditions:

oc --kubeconfig $KUBECONFIG_QUORUM get pz dr-prod -n dr-prod \
  -o jsonpath='{.status.conditions}' | jq .

A Unreachable cluster entry means the controller cannot reach the workload API server — verify the kubeconfig Secret content and TCP port 6443 connectivity from the quorum cluster.

Step 5 — Create a webhook serving certificate

The admission webhooks on both the quorum and workload control planes require a TLS serving certificate. The charts do not provision certificates automatically — you must create the Secret out of band before the managers can serve webhooks. The default secret name on the quorum side is webhook-server-cert (in the dr-prod namespace); on the workload side it is webhook (in trilio-site-recovery-system).

Generate a self-signed certificate and create the Secrets (adjust SANs for your cluster's DNS):

# Quorum webhook cert (dr-prod namespace)
oc --kubeconfig $KUBECONFIG_QUORUM create secret tls webhook-server-cert \
  --cert=tls.crt --key=tls.key \
  -n dr-prod

# Workload webhook cert — primary cluster
oc --kubeconfig $KUBECONFIG_CLUSTER1 create secret tls webhook \
  --cert=tls.crt --key=tls.key \
  -n trilio-site-recovery-system

# Workload webhook cert — DR cluster
oc --kubeconfig $KUBECONFIG_CLUSTER2 create secret tls webhook \
  --cert=tls.crt --key=tls.key \
  -n trilio-site-recovery-system

The cert SANs must cover the webhook Service's in-cluster DNS names, for example:

  • <release>-site-recovery-quorum-control-plane-webhook.dr-prod.svc
  • <release>-site-recovery-quorum-control-plane-webhook.dr-prod.svc.cluster.local

Step 6 — Protect a virtual machine

With the zone Available, you can protect your first VM. A ProtectionRequest tells the protection controller to:

  1. Validate the VM and its PVCs on the source cluster.
  2. Provision a DRBDResource pair (one CR per cluster side, named drbd-<namespace>-<vm>-<hash8>, in the VM's namespace on each workload cluster).
  3. Switch the VM to DRBD-backed frontend PVCs.

Create the ProtectionRequest on the quorum cluster in the zone namespace:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-my-vm
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: my-vm
    namespace: default

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectionrequest.yaml

Watch the protection phases:

oc --kubeconfig $KUBECONFIG_QUORUM get pr protect-my-vm -n dr-prod -w

The phase column progresses through: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

Once Protected, the VM on the primary cluster carries the label siterecovery.trilio.io/protected-vm=true. Verify it:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

Important operational constraints after protection:

  • The VM is pinned to one node per cluster; spec.nodeName is immutable for the lifetime of the protection.
  • Live migration and cold migration of protected VMs are denied by admission webhook.
  • oc adm drain on a node hosting a running protected VM will not complete.
  • Re-protecting a VM always requires a full unprotect followed by a full initial resync.
Examples

Example 1 — Minimal ProtectionZone for a production DR deployment

This is the smallest valid ProtectionZone for a two-cluster DRBD deployment. The controller creates the default replication policy (dr-prod-default) automatically.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: dr-prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Expected oc get pz output after the controller reconciles:

NAME      DISPLAY NAME          BACKEND   CLUSTERS     STATUS      AGE
dr-prod   Production DR Zone    DRBD      primary,dr   Available   3m

Example 2 — ProtectionRequest for a single VM

Protect a VM named web-server running in the production namespace on cluster1.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-server
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-server
    namespace: production

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protect-web-server.yaml

Watch until Protected:

oc --kubeconfig $KUBECONFIG_QUORUM get pr protect-web-server -n dr-prod -w

Expected output once complete:

NAME                  VM           NAMESPACE    PHASE       SYNC   AGE
protect-web-server    web-server   production   Protected   100    4m

Check the DRBD resource on the primary workload cluster (the CR lives in the VM's namespace, not the zone namespace):

oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n production

Expected output:

NAME                          SIDE     ROLE      NODE         STATE      PHASE       AGE
drbd-production-web-server-a1b2c3d4   Source   Primary   worker-0   Connected   Syncing     2m

Example 3 — ProtectionGroup for coordinated failover of multiple VMs

A ProtectionGroup groups VMs that must fail over together. Create it on the quorum cluster after all member VMs have been individually protected via ProtectionRequest.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: web-tier
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachines:
    - name: web-server
      namespace: production
    - name: db-server
      namespace: production

oc --kubeconfig $KUBECONFIG_QUORUM apply -f protectiongroup.yaml

Check group replication health:

oc --kubeconfig $KUBECONFIG_QUORUM get pg web-tier -n dr-prod
oc --kubeconfig $KUBECONFIG_QUORUM get rgs web-tier-replication -n dr-prod

Expected rgs output when healthy:

NAME                   PROTECTION-GROUP   RPO-STATUS   OUT-OF-SYNC   SYNCED   CONNECTION   LAST-SYNC
web-tier-replication   web-tier           Met          0             2        Connected    30s

Example 4 — Accessing the TSR web console (OpenShift only)

The ProtectionZone controller creates an OpenShift Route and console link for the TSR web console automatically when the zone reaches Available. Browser access is supported on OpenShift only — no ingress is created on upstream Kubernetes.

Find the console Route on the quorum cluster:

oc --kubeconfig $KUBECONFIG_QUORUM get route -n trilio-site-recovery-system

Navigate to the URL shown in the HOST/PORT column. Authentication is handled by the OpenShift oauth-proxy sidecar; your OpenShift credentials are used directly. Zone-scoped pages derive the dr-<zoneId> namespace — ensure your zone name matches the namespace (a zone in a differently named namespace will be listed but its detail pages will not function).

Example 5 — Collecting a diagnostic bundle

Run tsr-gather (an oc adm must-gather-compatible tool) against the quorum cluster. It collects logs, CRD state, and configuration from all clusters in the DR estate into a single bundle.

oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:latest

To limit the log window or scope to a single zone:

# Last 2 hours of logs only
oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:latest \
  -- /usr/bin/gather --since 2h

# Single zone only
oc --kubeconfig $KUBECONFIG_QUORUM adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:latest \
  -- /usr/bin/gather --zone dr-prod

Read SUMMARY.md in the bundle first for a summary of detected issues, then manifest.yaml for the full collection manifest. Secret data, kubeconfigs, and DRBD shared secrets are never written to the bundle.

Troubleshooting

Use a consistent pattern for each issue: Symptom → Likely cause → Fix.

ProtectionZone stuck in Provisioning or moves to Degraded

Symptom: oc get pz -n dr-prod shows STATUS: Provisioning or Degraded after several minutes.

Likely cause: The controller cannot reach one or both workload clusters, the kubeconfig Secret is missing or malformed, or TCP port 6443 is blocked from the quorum cluster to the workload API servers.

Fix:

# Check per-cluster reachability in the zone status
oc --kubeconfig $KUBECONFIG_QUORUM get pz dr-prod -n dr-prod \
  -o jsonpath='{.status.clusters}' | jq .

# Check protection-zone-controller logs
oc --kubeconfig $KUBECONFIG_QUORUM logs -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller \
  --tail=100

If status.clusters[].status is Unreachable, verify the Secret key name matches kubeconfigSecretRef.key, confirm the embedded kubeconfig points to the correct API server URL, and test connectivity: oc --kubeconfig $KUBECONFIG_CLUSTER1 get namespace trilio-site-recovery-system.

DRBD node agent pods stuck in Init phase

Symptom: On a workload cluster, agent pods show Init:CrashLoopBackOff or Init:Error.

Likely cause: The DRBD module-loader init container (drbd-module-loader) cannot find a precompiled kernel module matching the running kernel. This happens when the node's kernel version is not covered by the mirrored LINBIT release.

Fix:

# Find the agent DaemonSet name (release-prefixed)
oc --kubeconfig $KUBECONFIG_CLUSTER1 get daemonset -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

# Get logs from the init container on a failing node
AGENT_POD=$(oc --kubeconfig $KUBECONFIG_CLUSTER1 get pods \
  --field-selector spec.nodeName=<node-name> \
  -n trilio-site-recovery-system -o jsonpath='{.items[0].metadata.name}')

oc --kubeconfig $KUBECONFIG_CLUSTER1 logs $AGENT_POD \
  -n trilio-site-recovery-system \
  -c drbd-module-loader

If the kmod loader reports no matching kernel, either update the node to a supported RHCOS kernel version or pre-load the DRBD module out of band and set agent.moduleLoader.enabled=false in the workload chart values (the chart values are managed by the ProtectionZone controller — contact Trilio support to override this setting).

ProtectionRequest stuck in Syncing with Sync percentage not advancing

Symptom: oc get pr -n dr-prod shows PHASE: Syncing and SYNC is not increasing over time.

Likely cause: DRBD replication ports (7000–8000) are not fully open between primary and DR worker nodes, the round-trip latency exceeds 50 ms, or the LVM thin pool on a worker node is exhausted.

Fix:

# Check the DRBDResource on the primary workload cluster
oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n <vm-namespace>
oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres <drbd-resource-name> -n <vm-namespace> -o yaml

Look at status.connectionState and status.volumes[].replicationState. A Disconnected or StandAlone connection state indicates a network problem. Verify that all ports 7000–8000 inclusive are open — port 8000 is mandatory. Check RTT: ping -c 10 <dr-worker-ip> from a primary worker node and confirm it is below 50 ms.

ProtectionRequest in Failed phase with RoleConflict reason

Symptom: oc get pr -n dr-prod shows PHASE: Failed. Inspecting conditions shows Ready=False with reason: RoleConflict.

Likely cause: Both DRBD copies have promoted to Primary simultaneously (split-brain scenario), or a previous failover left the DR-side copy in Primary role.

Fix:

# Check DRBD resource role on both clusters
oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -n <vm-namespace> -o wide
oc --kubeconfig $KUBECONFIG_CLUSTER2 get dres -n <vm-namespace> -o wide

The ROLE column shows Primary or Secondary for each side. Only one side may be Primary at any time. To recover: unprotect the VM (oc delete pr <name> -n dr-prod) and re-protect it. Re-protect always requires a full initial resync.

oc get dres returns server doesn't have a resource type

Symptom: Running oc --kubeconfig $KUBECONFIG_QUORUM get dres returns an error about an unknown resource type.

Likely cause: DRBDResource CRDs are installed only on workload clusters, not on the quorum cluster. The command is being run against the wrong cluster context.

Fix: Always query dres against a workload cluster:

oc --kubeconfig $KUBECONFIG_CLUSTER1 get dres -A
oc --kubeconfig $KUBECONFIG_CLUSTER2 get dres -A

TSR web console pages for a zone are blank or return errors

Symptom: The console loads but zone-scoped pages show errors or empty data.

Likely cause: Zone-scoped console pages derive the dr-<zoneId> namespace. If the ProtectionZone metadata name does not match its namespace name (for example, the zone is named prod but lives in namespace dr-production), the page cannot resolve the correct namespace and all data calls fail.

Fix: Ensure the ProtectionZone metadata.name matches the dr- prefixed namespace it lives in — for example, a zone named prod must live in namespace dr-prod. If the zone was created in the wrong namespace, delete it and recreate it in the correctly named namespace.