Deployment Guide

This guide covers production deployment considerations for Site Recovery, including RBAC configuration, resource limits for controllers, leader election settings, and multi-cluster topology planning. Site Recovery spans three Kubernetes clusters — a quorum cluster hosting the management plane, a primary cluster running production VMs, and a DR cluster receiving replicated storage — and each cluster requires specific role bindings and resource allocations to operate correctly at scale. Following these guidelines helps ensure that controllers remain highly available, replication monitoring stays responsive, and multi-tenant DR deployments remain isolated from one another. Read this guide before deploying into a production or pre-production environment.

Before deploying Site Recovery in a production environment, ensure the following requirements are met:

Cluster infrastructure:

• Three Kubernetes clusters (≥ 1.28) for the centralized storage model: one quorum cluster, one primary cluster, one DR cluster
  • Minimum two clusters for the DRBD Operator deployment model (primary + DR)
• OpenShift ≥ 4.14 if deploying on OpenShift (optional)
• KubeVirt ≥ 1.0 installed on primary and DR clusters

Storage:

• DRBD kernel module ≥ 9.0 on all worker nodes that will host replicated VMs
• LVM thin-provisioned storage pools on worker nodes
• DRBD Operator installed on primary and DR clusters (required for DRBD Operator deployment model)
• VolumeSnapshot support (CSI snapshotter) on primary and DR clusters if you plan to use test failover (DRBD Operator model only)

Tooling:

• Helm ≥ 3.0
• kubectl configured for each cluster
• Ansible (for running the provided playbooks)
• pgctl installed and available in your PATH

Network:

• TCP 7000–7999 open between primary and DR worker nodes (DRBD replication traffic)
• TCP 6443 open from the quorum cluster to the API servers of the primary and DR clusters
• RTT < 10 ms between primary and DR clusters for Protocol C (synchronous) replication; any latency is acceptable for Protocol A (asynchronous)
• RTT < 50 ms is the maximum supported for Protocol C

Access:

• Kubeconfig files with cluster-admin or equivalent access for each of the three clusters
• Sufficient permissions to create namespaces, CustomResourceDefinitions, ClusterRoles, and ClusterRoleBindings

Site Recovery is deployed using a combination of Ansible playbooks (for infrastructure bootstrapping and quorum setup) and the quorum-deployments.sh interactive script (for day-2 operational tasks). Follow the steps below in order.

Step 1 — Verify cluster connectivity from the quorum node

Before running any playbooks, confirm the quorum cluster can reach both the primary and DR cluster API servers:

# Replace with your actual kubeconfig paths
export KUBECONFIG=/path/to/quorum-kubeconfig.yaml
kubectl get nodes

# Verify connectivity to primary cluster
kubectl --kubeconfig=/path/to/primary-kubeconfig.yaml get nodes

# Verify connectivity to DR cluster
kubectl --kubeconfig=/path/to/dr-kubeconfig.yaml get nodes

All three commands must return node lists before proceeding.

Step 2 — Deploy DRBD kernel modules and storage components

Run the Ansible playbook that installs the DRBD kernel module (≥ 9.0) and provisions LVM thin pools on worker nodes in both the primary and DR clusters:

ansible-playbook -i inventory/production.ini playbooks/deploy-drbd-storage.yml \
  --extra-vars "primary_kubeconfig=/path/to/primary-kubeconfig.yaml" \
  --extra-vars "dr_kubeconfig=/path/to/dr-kubeconfig.yaml"

Verify module load on a representative worker node:

ssh worker-node-1 "lsmod | grep drbd"
# Expected: drbd   <size>   0

Step 3 — Install the DRBD Operator on primary and DR clusters (DRBD Operator model)

If you are using the DRBD Operator deployment model, install the operator on both the primary and DR clusters using Helm:

# Primary cluster
helm install drbd-operator charts/drbd-operator \
  --namespace drbd-operator \
  --create-namespace \
  --kubeconfig /path/to/primary-kubeconfig.yaml

# DR cluster
helm install drbd-operator charts/drbd-operator \
  --namespace drbd-operator \
  --create-namespace \
  --kubeconfig /path/to/dr-kubeconfig.yaml

Wait for the operator pods to become ready:

kubectl --kubeconfig /path/to/primary-kubeconfig.yaml \
  -n drbd-operator wait pod --all --for=condition=Ready --timeout=120s

Step 4 — Bootstrap the quorum cluster

Run the quorum bootstrap playbook. This deploys the management plane components — failover-controller, protection-controller, pg-sync-controller, the Site Manager UI, and the Site Manager API — onto the quorum cluster:

ansible-playbook -i inventory/production.ini playbooks/deploy-quorum.yml \
  --extra-vars "quorum_kubeconfig=/path/to/quorum-kubeconfig.yaml"

The replication-monitor agent is deployed automatically as part of this playbook run on each cluster where it is required.

Step 5 — Install per-cluster controllers on primary and DR clusters

Deploy the protection-group-controller and test-failover-controller on both the primary and DR clusters:

ansible-playbook -i inventory/production.ini playbooks/deploy-cluster-controllers.yml \
  --extra-vars "primary_kubeconfig=/path/to/primary-kubeconfig.yaml" \
  --extra-vars "dr_kubeconfig=/path/to/dr-kubeconfig.yaml"

Step 6 — Configure your first DR deployment

Use quorum-deployments.sh to create the initial DR deployment (deployment context). The script is interactive and menu-driven:

bash quorum-deployments.sh

From the menu, select the option to create a new deployment. You will be prompted for:

• A deployment name (used as the namespace suffix: dr-<name>)
• The path to the primary cluster kubeconfig
• The path to the DR cluster kubeconfig

Alternatively, use pgctl to register the deployment context after the namespace and secrets have been created:

pgctl context create \
  --name production \
  --primary-kubeconfig /path/to/primary-kubeconfig.yaml \
  --dr-kubeconfig /path/to/dr-kubeconfig.yaml

Step 7 — Verify the deployment

Confirm that all controllers are running and the deployment context is reachable:

# Check quorum-cluster controllers
kubectl -n dr-production get pods

# Validate the deployment context with pgctl
pgctl context validate --name production

All pods should be in the Running state and the context validation should report no errors before you proceed to protect VMs.

The following sections describe the key configuration dimensions for a production Site Recovery deployment. Configuration is expressed through Kubernetes manifests and Helm values; there are no application-level config files to edit directly.

RBAC

Each controller operates under a dedicated ServiceAccount with the minimum RBAC permissions needed for its role. The Ansible playbooks create these automatically, but in environments with restricted RBAC provisioning you may need to apply them manually.

Quorum cluster — controllers that require cluster-wide read access:

• failover-controller: needs get, list, watch, update, and patch on FailoverRequest, ProtectionGroup, and related CRDs; needs get and list on Secrets (to load kubeconfigs for remote clusters).
• protection-controller: needs get, list, watch, create, update, and patch on ProtectionRequest and DRBDVolume.
• pg-sync-controller: needs get, list, watch, and update on ProtectionGroup resources across the namespaces it manages.

Primary and DR clusters — per-cluster controllers:

• protection-group-controller: needs get, list, watch, update, and patch on ProtectionGroup and ReplicationGroupStatus.
• test-failover-controller: needs get, list, watch, create, update, patch, and delete on TestFailover resources and VolumeSnapshot objects.
• replication-monitor: needs get, list, watch, create, and update on RPOEvent and ReplicationGroupStatus.

Important: In multi-tenant deployments each DR deployment lives in its own dr-<name> namespace. Namespace-scoped RoleBindings are preferred over ClusterRoleBindings wherever the controller does not require cluster-wide access, to preserve tenant isolation.

Resource limits

Set explicit resource requests and limits on all controller Deployments to prevent resource contention on the quorum cluster, especially when managing many DR deployments simultaneously.

Recommended baseline values for the quorum cluster controllers:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

For the replication-monitor agent running on primary and DR clusters, which polls DRBD replication state continuously, slightly higher memory headroom is advisable in large clusters:

resources:
  requests:
    cpu: "50m"
    memory: "64Mi"
  limits:
    cpu: "200m"
    memory: "256Mi"

Adjust these values upward if you observe OOMKilled events or CPU throttling during large-scale replication sync operations.

Leader election

All quorum-cluster controllers use Kubernetes leader election (via coordination.k8s.io/leases) to ensure only one replica is active at a time. In production, run each controller with two replicas so that a second instance can immediately assume leadership if the active pod fails.

Key leader election parameters (set via controller Deployment environment variables or Helm values):

These defaults are appropriate for most deployments. Increase LEADER_ELECT_LEASE_DURATION only if the quorum cluster's API server experiences high latency (> 500 ms) and you observe unnecessary leader re-elections.

DRBDReplicationPolicy settings

The DRBDReplicationPolicy CRD is the primary configuration surface for DRBD Operator deployments. The most operationally significant fields are:

Namespace isolation for multi-tenant deployments

Each DR deployment creates a dedicated namespace named dr-<deployment-name>. All controllers, secrets, and CRD instances for that deployment are scoped to this namespace. Do not share credentials or controller instances across namespaces, as this breaks tenant isolation and can cause cross-deployment failover interference.

Site Manager UI and API

The Site Manager UI and Site Manager API are deployed on the quorum cluster. The API is a Python Flask application. Ensure the Service exposing the API is accessible to the quorum cluster's ingress or load balancer, and that authentication is configured before exposing it externally. The UI relies on the API backend; both must be running for the dashboard to function correctly.

Once Site Recovery is deployed, your primary day-to-day interfaces are pgctl, kubectl (for CRD inspection), quorum-deployments.sh (for bulk operational tasks), and the Site Manager UI.

Managing deployment contexts with pgctl

List all registered deployment contexts:

pgctl context list

Switch to a specific deployment context before running further commands:

pgctl context use --name production

Validate that the active context's clusters are reachable and controllers are healthy:

pgctl context validate

Inspecting Protection Groups

List all Protection Groups in the active deployment:

pgctl pg list

Inspect the replication state of a specific Protection Group:

pgctl pg status --name my-protection-group

You can also inspect the underlying CRDs directly with kubectl:

# On the primary or DR cluster
kubectl get protectiongroups -A
kubectl describe protectiongroup my-protection-group -n dr-production

Monitoring replication health

The replication-monitor agent writes ReplicationGroupStatus and RPOEvent resources continuously. Use these to monitor replication health:

# Overall replication health per Protection Group
kubectl get replicationgroupstatuses -n dr-production

# Recent RPO violations
kubectl get rpoevents -n dr-production --sort-by=.metadata.creationTimestamp

The Site Manager UI provides the same information in dashboard form, with per-VM sync state and last-successful-sync timestamps.

Using quorum-deployments.sh for operational tasks

The quorum-deployments.sh script provides a menu-driven interface for more than 20 tasks. Launch it and navigate using the numbered menu:

bash quorum-deployments.sh

Common tasks available through the menu include:

• Deploying or updating the quorum management plane
• Creating or removing DR deployments
• Triggering planned or unplanned failovers
• Backing up and restoring quorum cluster state
• Running health checks across all managed clusters

Triggering a failover

For production failovers, prefer using pgctl or the Site Manager UI so that intent is recorded as a FailoverRequest CRD that the failover-controller orchestrates:

# Planned failover
pgctl failover trigger \
  --protection-group my-protection-group \
  --type planned

# Monitor progress
pgctl failover status --protection-group my-protection-group

The failover-controller on the quorum cluster watches for the FailoverRequest resource and drives volume promotion, VM shutdown on the primary, and VM startup on the DR cluster automatically.

Example 1 — Create a DRBDReplicationPolicy for synchronous replication

This policy configures Protocol C (synchronous, RPO=0) replication between two storage classes. Use this when your primary and DR clusters have < 10 ms RTT.

apiVersion: drbd.site-recovery.io/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-replication-policy
  namespace: dr-production
spec:
  protocol: C
  storageClassMappings:
    - primary: fast-ssd
      dr: fast-ssd-dr
  replicationEndpoints:
    - primary: 10.0.1.10:7000
      dr: 10.0.2.10:7000

Apply it to the quorum cluster:

kubectl apply -f sync-replication-policy.yaml

Expected: The DRBDReplicationPolicy is created and the DRBD Operator on each cluster begins configuring replication resources according to the policy.

Example 2 — Request DR protection for a single VM (DRBD Operator model)

Create a ProtectionRequest to protect a VM named web-server-01:

apiVersion: site-recovery.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-server-01
  namespace: dr-production
spec:
  vmName: web-server-01
  vmNamespace: production-vms
  replicationPolicy: sync-replication-policy

kubectl apply -f protect-web-server-01.yaml

Monitor the lifecycle:

kubectl get protectionrequest protect-web-server-01 -n dr-production -w

Expected output (progressing through lifecycle phases):

NAME                     STATUS       AGE
protect-web-server-01    Validating   5s
protect-web-server-01    Creating     15s
protect-web-server-01    Syncing      45s
protect-web-server-01    Protected    3m10s

Once the status is Protected, the VM is running on a frontend PVC backed by DRBD replication and the DR cluster has a synchronized replica.

Example 3 — Create a Protection Group

Group two VMs so they fail over together:

apiVersion: site-recovery.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: web-tier
  namespace: dr-production
spec:
  vms:
    - name: web-server-01
      namespace: production-vms
    - name: web-server-02
      namespace: production-vms

kubectl apply -f web-tier-pg.yaml

Check the Protection Group state:

kubectl get protectiongroup web-tier -n dr-production

Expected output:

NAME       STATE        AGE
web-tier   Consistent   8m

Consistent indicates all VMs in the group are replicated and ready for failover.

Example 4 — Trigger a planned failover

apiVersion: site-recovery.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-web-tier-planned
  namespace: dr-production
spec:
  protectionGroup: web-tier
  type: Planned

kubectl apply -f failover-web-tier-planned.yaml

# Watch progress
kubectl get failoverrequest failover-web-tier-planned -n dr-production -w

Expected progression:

NAME                          STATUS          AGE
failover-web-tier-planned     ShuttingDown    10s
failover-web-tier-planned     Promoting       35s
failover-web-tier-planned     Starting        55s
failover-web-tier-planned     Completed       2m15s

Example 5 — Inspect ReplicationGroupStatus for a Protection Group

kubectl describe replicationgroupstatus web-tier -n dr-production

Expected output (abridged):

Name:         web-tier
Namespace:    dr-production
Status:
  Health:              Healthy
  Last Successful Sync: 2024-01-15T14:32:01Z
  Volume States:
    web-server-01-disk:  Synced
    web-server-02-disk:  Synced

Example 6 — Multi-tenant deployment: add a second DR deployment

Each deployment gets its own namespace. Use quorum-deployments.sh to add it interactively:

bash quorum-deployments.sh
# Select: Add new deployment
# Enter deployment name: staging
# Enter primary kubeconfig path: /path/to/staging-primary-kubeconfig.yaml
# Enter DR kubeconfig path: /path/to/staging-dr-kubeconfig.yaml

Or register the context with pgctl after manually creating the dr-staging namespace and uploading credentials:

pgctl context create \
  --name staging \
  --primary-kubeconfig /path/to/staging-primary-kubeconfig.yaml \
  --dr-kubeconfig /path/to/staging-dr-kubeconfig.yaml

pgctl context validate --name staging

The new deployment runs in the dr-staging namespace, fully isolated from the dr-production deployment.

Use the following patterns to diagnose and resolve the most common deployment and operational issues.

Issue: Controller pod is in CrashLoopBackOff on the quorum cluster

Symptom: One or more of failover-controller, protection-controller, or pg-sync-controller pods repeatedly crash on the quorum cluster.

Likely causes:

• Missing or malformed kubeconfig Secrets for the primary or DR clusters
• RBAC permissions insufficient for the controller's ServiceAccount
• Leader election lease cannot be created (missing permissions on coordination.k8s.io/leases)

Fix:

# Check controller logs
kubectl -n dr-production logs deploy/failover-controller --previous

# Verify the Secret containing cluster credentials exists
kubectl -n dr-production get secrets

# Check RBAC bindings for the controller ServiceAccount
kubectl get rolebindings,clusterrolebindings -A \
  -o wide | grep failover-controller

If the error mentions leases, add get, create, update, and patch verbs on coordination.k8s.io/leases to the controller's ClusterRole.

Issue: ProtectionRequest is stuck in Syncing state

Symptom: A ProtectionRequest has been in Syncing status for longer than expected (> 30 minutes for small volumes).

Likely causes:

• DRBD replication traffic is blocked on TCP 7000–7999 between worker nodes
• The DRBD Operator on the primary or DR cluster is not running
• LVM thin pool is full or not properly provisioned

Fix:

# Check DRBDVolume sync progress
kubectl get drbdvolumes -n dr-production

# Check the DRBD Operator pods on primary cluster
kubectl --kubeconfig /path/to/primary-kubeconfig.yaml \
  -n drbd-operator get pods

# Verify network connectivity on a worker node
ssh worker-node-1 "nc -zv <dr-worker-ip> 7000"

# Check replication-monitor for RPO events indicating lag
kubectl get rpoevents -n dr-production

If TCP 7000–7999 is blocked, update your firewall or NetworkPolicy rules to permit this range between primary and DR worker nodes.

Issue: FailoverRequest is stuck in Promoting state

Symptom: A FailoverRequest has not progressed past Promoting for several minutes.

Likely causes:

• The primary cluster is unreachable and the failover type is Planned (which requires graceful VM shutdown)
• ReplicationGroupStatus for the Protection Group shows Degraded or Critical, indicating volumes are not fully synced
• The failover-controller on the quorum cluster is not running

Fix:

# Check failover-controller logs
kubectl -n dr-production logs deploy/failover-controller

# Check Protection Group state
kubectl get protectiongroup <pg-name> -n dr-production

# Check replication health
kubectl get replicationgroupstatus <pg-name> -n dr-production

If the primary cluster is genuinely unreachable and you need an emergency failover, delete the existing FailoverRequest and create a new one with type: Unplanned. Be aware that if Protocol A was in use, a small amount of data loss may occur.

Issue: Site Manager UI shows clusters as unreachable

Symptom: The Site Manager UI dashboard reports one or both clusters as unreachable.

Likely causes:

• The kubeconfig Secret for the affected cluster is expired or has an invalid token
• The quorum cluster cannot reach the cluster's API server on TCP 6443
• The cluster genuinely has no Ready nodes

Fix:

# Test API server reachability from the quorum cluster
kubectl --kubeconfig /path/to/affected-kubeconfig.yaml get nodes

# If the token is expired, rotate the kubeconfig and update the Secret
kubectl -n dr-production create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=/path/to/renewed-kubeconfig.yaml \
  --dry-run=client -o yaml | kubectl apply -f -

After updating the Secret, the Site Manager API will pick up the new credentials on its next reconciliation cycle.

Issue: RPOEvent resources are accumulating — replication lag is high

Symptom: kubectl get rpoevents -n dr-production shows a growing list of lag violation events for one or more Protection Groups.

Likely causes:

• Network bandwidth between primary and DR clusters is saturated
• Protocol C (synchronous) is configured but RTT has increased above 50 ms
• A worker node hosting a DRBD volume is under heavy I/O load

Fix:

# Identify which Protection Group and volume is affected
kubectl describe rpoevent <event-name> -n dr-production

# Check current replication health
kubectl get replicationgroupstatus -n dr-production

# Check round-trip latency to the DR cluster
ping -c 20 <dr-worker-node-ip>

If RTT consistently exceeds 50 ms, consider switching to Protocol A (asynchronous) by updating the DRBDReplicationPolicy protocol field. If bandwidth is the bottleneck, work with your network team to increase capacity or implement QoS policies that prioritize DRBD traffic on TCP 7000–7999.

Issue: Split-brain detected on a DRBD volume

Symptom: The ReplicationGroupStatus for a Protection Group shows Critical and the DRBD Operator logs indicate a split-brain condition on one or more volumes.

Likely causes:

• Both the primary and DR DRBD volumes simultaneously assumed the primary role, typically after a network partition followed by recovery.

Fix:

Split-brain resolution requires manual intervention to choose which side's data to keep. Do not attempt automated recovery without identifying which replica has the most recent data.

# Identify the affected DRBDVolume
kubectl get drbdvolumes -n dr-production
kubectl describe drbdvolume <volume-name> -n dr-production

Follow your organization's data integrity runbook to designate one side as the authoritative replica, then force the other side to resynchronize from it. After resolution, verify the ReplicationGroupStatus returns to Healthy before resuming normal operations or attempting a failover.