Installation
Installing CRDs and deploying the site-recovery operator into your cluster
This page walks you through installing Site Recovery into your Kubernetes environment — from applying Custom Resource Definitions (CRDs) to deploying the control-plane components on each of your clusters. Site Recovery uses three Helm charts that distribute controllers across the quorum cluster and your primary and DR clusters, so understanding which chart goes where before you begin will save you from having to redo steps. Complete this installation before configuring deployments, protecting VMs, or testing failover.
Before you begin, ensure the following are in place:
Clusters
- Three Kubernetes clusters (≥ 1.28) or OpenShift clusters (≥ 4.14) for the centralized storage model: one quorum cluster and one each for primary and DR
- Two clusters minimum for the DRBD Operator model (quorum cluster is still recommended)
- KubeVirt ≥ 1.0 installed on the primary and DR clusters
Storage
- DRBD kernel module ≥ 9.0 loaded on all worker nodes of the primary and DR clusters
- DRBD Operator installed on the primary and DR clusters (DRBD Operator deployment model only)
- LVM thin-provisioned storage pools configured on worker nodes
- VolumeSnapshot support (CSI snapshotter) on primary and DR clusters if you plan to use test failover (DRBD Operator model only)
Tooling
- Helm ≥ 3.0
kubectlconfigured with access to all clusters- Ansible (used by the deployment playbooks that wire up credentials and run health checks)
- A kubeconfig file with cluster-admin access for each cluster
Networking
- TCP 7000–7999 open between primary and DR worker nodes (DRBD replication)
- TCP 6443 open from the quorum cluster to both the primary and DR Kubernetes API servers
- Round-trip latency < 10 ms between primary and DR clusters if you intend to use Protocol C (synchronous replication); Protocol A (asynchronous) works at any latency
Environment variables used throughout this page
export KUBECONFIG_QUORUM=~/.kube/config-quorum
export KUBECONFIG_CLUSTER1=~/.kube/config-cluster1
export KUBECONFIG_CLUSTER2=~/.kube/config-cluster2
Site Recovery is delivered as three Helm charts. Install them in the order shown below; the ProtectionZone webhook must be up before the quorum control plane, and the workload control plane must be deployed on both the primary and DR clusters before you create any DR deployments.
Step 1 — Deploy the ProtectionZone controller (quorum cluster)
The site-recovery-protectionzone-controller chart installs the webhook and CRDs that the quorum control plane depends on. Install it first, in a namespace that matches your intended deployment name (conventionally dr-<name>).
helm install dr-prod ./helm/site-recovery-protectionzone-controller \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace
Wait for the controller pod to reach Running before continuing:
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod \
wait --for=condition=available deployment --all --timeout=120s
Step 2 — Deploy the quorum control plane (quorum cluster)
The site-recovery-quorum-control-plane chart deploys the failover-controller, protection-controller, and pg-sync-controller into the same namespace. These controllers watch FailoverRequest, ProtectionRequest, and ProtectionGroup resources and orchestrate all cross-cluster operations.
helm install dr-prod ./helm/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace
Verify all quorum-side controllers are running:
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod get pods
Expect to see pods for the quorum manager (which contains the failover-controller, protection-controller, and pg-sync-controller) and the Site Manager UI (frontend and backend) all in Running state.
Step 3 — Deploy the workload control plane (primary and DR clusters)
The site-recovery-workload-control-plane chart installs the protection-group-controller, test-failover-controller, and the DRBD node agent on a workload cluster. Deploy it on both the primary and DR clusters. The chart always installs into the trilio-site-recovery-system namespace.
# Primary cluster
helm install dr-prod ./helm/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_CLUSTER1 \
-n trilio-site-recovery-system --create-namespace
# DR cluster
helm install dr-prod ./helm/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_CLUSTER2 \
-n trilio-site-recovery-system --create-namespace
Confirm both workload control planes are healthy:
kubectl --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system get pods
kubectl --kubeconfig $KUBECONFIG_CLUSTER2 -n trilio-site-recovery-system get pods
Step 4 — Verify CRD installation
All CRDs are bundled in the Helm charts and applied automatically during helm install. Confirm they are present on the quorum cluster:
kubectl --kubeconfig $KUBECONFIG_QUORUM get crds | grep siterecovery.trilio.io
You should see entries for:
drbdreplicationpolicies.siterecovery.trilio.io
drbdvolumes.siterecovery.trilio.io
failoverrequests.siterecovery.trilio.io
protectiongroups.siterecovery.trilio.io
protectionrequests.siterecovery.trilio.io
replicationgroupstatuses.siterecovery.trilio.io
rpoevents.siterecovery.trilio.io
testfailovers.siterecovery.trilio.io
On the workload clusters, confirm the workload-side CRDs are present:
kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get crds | grep siterecovery.trilio.io
Step 5 (DRBD Operator model only) — Confirm the replication-monitor is running
The replication-monitor agent is deployed automatically as part of the standard Ansible playbooks for DRBD Operator deployments. It runs on the primary and DR clusters and writes RPOEvent and ReplicationGroupStatus resources. Verify it is present:
kubectl --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system \
get pods -l app=replication-monitor
If the replication-monitor pod is absent, re-run the Ansible provisioning playbook for the affected cluster before proceeding.
Step 6 — Access the Site Manager UI
The Site Manager UI is deployed on the quorum cluster as part of the quorum control plane chart. Retrieve the service details to find the access URL:
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod get svc
Expose the UI using a route, ingress, or port-forward depending on your cluster's networking configuration. The UI provides dashboards for cluster health, Protection Group management, failover triggering, and operation monitoring.
Each Helm chart accepts values that control resource allocation, image references, and behavioral tuning. The most important options are described below. Pass any of these with --set key=value or a -f values.yaml file during helm install or helm upgrade.
site-recovery-quorum-control-plane
| Key | Default | Effect |
|---|---|---|
image.repository | quay.io/triliodata/site-recovery-control-plane | The container image used for the quorum manager (failover-controller, protection-controller, pg-sync-controller). Change this if pulling from a private registry. |
image.tag | Release version | Pin to a specific release tag. If unset, defaults to the chart's bundled version. |
replicaCount | 1 | Number of quorum manager replicas. The manager uses leader election, so increasing this value adds HA without split-brain risk. |
site-recovery-workload-control-plane
| Key | Default | Effect |
|---|---|---|
image.repository | quay.io/triliodata/site-recovery-control-plane | Container image for the workload manager (protection-group-controller, test-failover-controller). |
nodeAgent.image.repository | quay.io/triliodata/site-recovery-drbd-node-agent | Separate image for the DRBD node agent that runs as a DaemonSet. Change when using a private registry. |
replicaCount | 1 | Replicas for the workload manager deployment. |
site-recovery-protectionzone-controller
| Key | Default | Effect |
|---|---|---|
image.repository | quay.io/triliodata/site-recovery-protection-zone-controller | Image for the ProtectionZone webhook controller. |
DRBDReplicationPolicy (DRBD Operator model)
After the charts are installed, you configure cross-cluster replication by creating a DRBDReplicationPolicy resource on the quorum cluster. The key fields are:
| Field | Valid values | Effect |
|---|---|---|
spec.drbdProtocol | A or C | C = synchronous (RPO=0, requires < 10 ms RTT); A = asynchronous (near-zero RPO, tolerates any latency) |
spec.storageClassMappings | List of primaryStorageClass / drStorageClass pairs | Maps storage class names between clusters so the system can provision matching volumes on both sides |
spec.primaryCluster.replicationEndpoint | "<IP>:<port>" | The IP and port (in the 7000–7999 range) on the primary cluster that DRBD peers connect to |
spec.drCluster.replicationEndpoint | "<IP>:<port>" | The equivalent endpoint on the DR cluster |
See the Configure a DR Deployment page for a complete DRBDReplicationPolicy example.
Once installation is complete, your day-to-day interactions with Site Recovery happen through kubectl and Kubernetes custom resources. The Site Manager UI on the quorum cluster provides a visual alternative for dashboards and one-click operations.
Check installed chart versions at any time:
helm list -n dr-prod --kubeconfig $KUBECONFIG_QUORUM
Upgrade a chart after a new release:
helm upgrade dr-prod ./helm/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM -n dr-prod
Inspect the running controllers on the quorum cluster:
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod get pods
List all Site Recovery CRDs currently known to the quorum cluster:
kubectl --kubeconfig $KUBECONFIG_QUORUM get crds | grep siterecovery.trilio.io
Use quorum-deployments.sh for interactive operational tasks:
The quorum-deployments.sh interactive shell script provides a menu-driven interface for more than 20 tasks including deploying the quorum, triggering failovers, backing up and restoring state, and running health checks. Run it directly on a host with access to your quorum cluster's kubeconfig:
bash quorum-deployments.sh
After installation, your next step is to configure a DR deployment — see Configure a DR Deployment — which wires up the primary and DR cluster credentials and creates the per-deployment namespace and controller instances.
Example 1 — Full installation for a DRBD Operator deployment named dr-prod
This sequence installs all three charts and confirms the control plane is healthy.
# 1. Install the ProtectionZone webhook on the quorum cluster
helm install dr-prod ./helm/site-recovery-protectionzone-controller \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace
# 2. Install the quorum control plane
helm install dr-prod ./helm/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace
# 3. Install the workload control plane on the primary cluster
helm install dr-prod ./helm/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_CLUSTER1 \
-n trilio-site-recovery-system --create-namespace
# 4. Install the workload control plane on the DR cluster
helm install dr-prod ./helm/site-recovery-workload-control-plane \
--kubeconfig $KUBECONFIG_CLUSTER2 \
-n trilio-site-recovery-system --create-namespace
# 5. Confirm quorum pods are running
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod get pods
Expected output (quorum pods):
NAME READY STATUS RESTARTS AGE
dr-prod-quorum-manager-7d9f6b8c4-xk9lp 2/2 Running 0 90s
dr-prod-site-manager-backend-6b5dc7f8-mnpq2 1/1 Running 0 90s
dr-prod-site-manager-frontend-8c4fd9b-zrwt1 1/1 Running 0 90s
Example 2 — Verify CRDs are installed correctly
kubectl --kubeconfig $KUBECONFIG_QUORUM get crds | grep siterecovery.trilio.io
Expected output:
drbdreplicationpolicies.siterecovery.trilio.io 2024-01-15T10:23:01Z
drbdvolumes.siterecovery.trilio.io 2024-01-15T10:23:01Z
failoverrequests.siterecovery.trilio.io 2024-01-15T10:23:01Z
protectiongroups.siterecovery.trilio.io 2024-01-15T10:23:01Z
protectionrequests.siterecovery.trilio.io 2024-01-15T10:23:01Z
replicationgroupstatuses.siterecovery.trilio.io 2024-01-15T10:23:01Z
rpoevents.siterecovery.trilio.io 2024-01-15T10:23:01Z
testfailovers.siterecovery.trilio.io 2024-01-15T10:23:01Z
Example 3 — Install using a private registry override
If your clusters pull images from a private registry, override the image repository at install time:
helm install dr-prod ./helm/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod --create-namespace \
--set image.repository=registry.example.com/site-recovery-control-plane \
--set image.tag=1.2.0
Example 4 — Upgrade the quorum control plane to a new version
helm upgrade dr-prod ./helm/site-recovery-quorum-control-plane \
--kubeconfig $KUBECONFIG_QUORUM \
-n dr-prod
Expected output:
Release "dr-prod" has been upgraded. Happy Helming!
NAME: dr-prod
LAST DEPLOYED: Mon Jan 15 11:05:22 2024
NAMESPACE: dr-prod
STATUS: deployed
REVISION: 2
Issue: Helm install fails with "webhook not found" or admission error
Symptom: The site-recovery-quorum-control-plane install fails with an admission webhook error.
Cause: The site-recovery-protectionzone-controller chart (Step 1) was not installed first, or its pod has not yet reached Running.
Fix:
# Check the ProtectionZone controller is running
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod get pods
# If not running, check events
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod describe pod -l app=site-recovery-protection-zone-controller
Wait until the ProtectionZone controller pod is Running, then retry the quorum control plane install.
Issue: CRDs are missing after helm install
Symptom: kubectl get crds | grep siterecovery.trilio.io returns no results or fewer CRDs than expected.
Cause: The Helm chart's crds/ directory may be out of sync with the source, or the install targeted the wrong cluster.
Fix: Confirm you used the correct --kubeconfig flag and the correct namespace. Then verify the chart contains the expected CRD files:
ls ./helm/site-recovery-quorum-control-plane/crds/
If CRD files are missing from the chart directory, the repository may need make manifests to regenerate them. Contact your distribution maintainer if using a release artifact.
Issue: Quorum manager pod is in CrashLoopBackOff
Symptom: The quorum manager pod repeatedly restarts.
Cause: The manager cannot reach the primary or DR cluster API servers, or kubeconfig secrets for the deployment have not yet been created.
Fix:
# Inspect logs
kubectl --kubeconfig $KUBECONFIG_QUORUM -n dr-prod logs -l app=site-recovery-quorum-manager --previous
# Verify TCP 6443 is reachable from the quorum cluster
curl -k https://<primary-api-server>:6443/healthz
curl -k https://<dr-api-server>:6443/healthz
Ensure the kubeconfig secrets for your deployment have been created in the dr-prod namespace before the manager starts.
Issue: Workload control plane pods not appearing on primary or DR cluster
Symptom: After running helm install for site-recovery-workload-control-plane, no pods appear in trilio-site-recovery-system.
Cause: The install may have targeted the wrong cluster's kubeconfig, or image pull is failing.
Fix:
# Confirm the namespace exists on the intended cluster
kubectl --kubeconfig $KUBECONFIG_CLUSTER1 get ns trilio-site-recovery-system
# Check for image pull errors
kubectl --kubeconfig $KUBECONFIG_CLUSTER1 -n trilio-site-recovery-system describe pods
If you see ImagePullBackOff, verify that your worker nodes can reach the image registry, or provide a private registry override with --set image.repository=....
Issue: replication-monitor pod missing on workload cluster (DRBD Operator model)
Symptom: No pod matching app=replication-monitor exists in trilio-site-recovery-system after deploying the workload control plane.
Cause: The replication-monitor is deployed by the Ansible provisioning playbooks, not directly by the Helm chart. If the playbooks were skipped or failed, the agent is absent.
Fix: Re-run the Ansible provisioning playbook for the affected cluster. Consult the playbook output for any failed tasks before retrying.
Issue: DRBD ports blocked — replication does not start
Symptom: DRBDVolume resources show a Pending or Disconnected sync state after protecting a VM.
Cause: TCP ports 7000–7999 between primary and DR worker nodes are not open.
Fix: From a primary worker node, test connectivity to a DR worker node:
nc -zv <dr-worker-node-ip> 7000
If the connection is refused or times out, update your firewall or security group rules to permit TCP 7000–7999 between all primary and DR worker nodes, then re-check the DRBDVolume status:
kubectl --kubeconfig $KUBECONFIG_QUORUM get drbdvolumes -A