--- title: Deployment Guide product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/deployment-guide --- # Deployment Guide _Production deployment considerations including RBAC setup, resource limits, leader election, and multi-cluster scenarios_ ## Overview This guide covers production deployment considerations for Site Recovery, including RBAC configuration, resource limits for controllers, leader election settings, and multi-cluster topology planning. Site Recovery spans three Kubernetes clusters — a quorum cluster hosting the management plane, a primary cluster running production VMs, and a DR cluster receiving replicated storage — and each cluster requires specific role bindings and resource allocations to operate correctly at scale. Following these guidelines helps ensure that controllers remain highly available, replication monitoring stays responsive, and multi-tenant DR deployments remain isolated from one another. Read this guide before deploying into a production or pre-production environment. ## Prerequisites Before deploying Site Recovery in a production environment, ensure the following requirements are met: **Cluster infrastructure:** - Three Kubernetes clusters (≥ 1.28) for the centralized storage model: one quorum cluster, one primary cluster, one DR cluster - Minimum two clusters for the DRBD Operator deployment model (primary + DR) - OpenShift ≥ 4.14 if deploying on OpenShift (optional) - KubeVirt ≥ 1.0 installed on primary and DR clusters **Storage:** - DRBD kernel module ≥ 9.0 on all worker nodes that will host replicated VMs - LVM thin-provisioned storage pools on worker nodes - DRBD Operator installed on primary and DR clusters (required for DRBD Operator deployment model) - VolumeSnapshot support (CSI snapshotter) on primary and DR clusters if you plan to use test failover (DRBD Operator model only) **Tooling:** - Helm ≥ 3.0 - kubectl configured for each cluster - Ansible (for running the provided playbooks) - `pgctl` installed and available in your PATH **Network:** - TCP 7000–7999 open between primary and DR worker nodes (DRBD replication traffic) - TCP 6443 open from the quorum cluster to the API servers of the primary and DR clusters - RTT < 10 ms between primary and DR clusters for Protocol C (synchronous) replication; any latency is acceptable for Protocol A (asynchronous) - RTT < 50 ms is the maximum supported for Protocol C **Access:** - Kubeconfig files with cluster-admin or equivalent access for each of the three clusters - Sufficient permissions to create namespaces, CustomResourceDefinitions, ClusterRoles, and ClusterRoleBindings ## Installation Site Recovery is deployed using a combination of Ansible playbooks (for infrastructure bootstrapping and quorum setup) and the `quorum-deployments.sh` interactive script (for day-2 operational tasks). Follow the steps below in order. ### Step 1 — Verify cluster connectivity from the quorum node Before running any playbooks, confirm the quorum cluster can reach both the primary and DR cluster API servers: ```bash # Replace with your actual kubeconfig paths export KUBECONFIG=/path/to/quorum-kubeconfig.yaml kubectl get nodes # Verify connectivity to primary cluster kubectl --kubeconfig=/path/to/primary-kubeconfig.yaml get nodes # Verify connectivity to DR cluster kubectl --kubeconfig=/path/to/dr-kubeconfig.yaml get nodes ``` All three commands must return node lists before proceeding. ### Step 2 — Deploy DRBD kernel modules and storage components Run the Ansible playbook that installs the DRBD kernel module (≥ 9.0) and provisions LVM thin pools on worker nodes in both the primary and DR clusters: ```bash ansible-playbook -i inventory/production.ini playbooks/deploy-drbd-storage.yml \ --extra-vars "primary_kubeconfig=/path/to/primary-kubeconfig.yaml" \ --extra-vars "dr_kubeconfig=/path/to/dr-kubeconfig.yaml" ``` Verify module load on a representative worker node: ```bash ssh worker-node-1 "lsmod | grep drbd" # Expected: drbd 0 ``` ### Step 3 — Install the DRBD Operator on primary and DR clusters (DRBD Operator model) If you are using the DRBD Operator deployment model, install the operator on both the primary and DR clusters using Helm: ```bash # Primary cluster helm install drbd-operator charts/drbd-operator \ --namespace drbd-operator \ --create-namespace \ --kubeconfig /path/to/primary-kubeconfig.yaml # DR cluster helm install drbd-operator charts/drbd-operator \ --namespace drbd-operator \ --create-namespace \ --kubeconfig /path/to/dr-kubeconfig.yaml ``` Wait for the operator pods to become ready: ```bash kubectl --kubeconfig /path/to/primary-kubeconfig.yaml \ -n drbd-operator wait pod --all --for=condition=Ready --timeout=120s ``` ### Step 4 — Bootstrap the quorum cluster Run the quorum bootstrap playbook. This deploys the management plane components — `failover-controller`, `protection-controller`, `pg-sync-controller`, the Site Manager UI, and the Site Manager API — onto the quorum cluster: ```bash ansible-playbook -i inventory/production.ini playbooks/deploy-quorum.yml \ --extra-vars "quorum_kubeconfig=/path/to/quorum-kubeconfig.yaml" ``` The `replication-monitor` agent is deployed automatically as part of this playbook run on each cluster where it is required. ### Step 5 — Install per-cluster controllers on primary and DR clusters Deploy the `protection-group-controller` and `test-failover-controller` on both the primary and DR clusters: ```bash ansible-playbook -i inventory/production.ini playbooks/deploy-cluster-controllers.yml \ --extra-vars "primary_kubeconfig=/path/to/primary-kubeconfig.yaml" \ --extra-vars "dr_kubeconfig=/path/to/dr-kubeconfig.yaml" ``` ### Step 6 — Configure your first DR deployment Use `quorum-deployments.sh` to create the initial DR deployment (deployment context). The script is interactive and menu-driven: ```bash bash quorum-deployments.sh ``` From the menu, select the option to create a new deployment. You will be prompted for: - A deployment name (used as the namespace suffix: `dr-`) - The path to the primary cluster kubeconfig - The path to the DR cluster kubeconfig Alternatively, use `pgctl` to register the deployment context after the namespace and secrets have been created: ```bash pgctl context create \ --name production \ --primary-kubeconfig /path/to/primary-kubeconfig.yaml \ --dr-kubeconfig /path/to/dr-kubeconfig.yaml ``` ### Step 7 — Verify the deployment Confirm that all controllers are running and the deployment context is reachable: ```bash # Check quorum-cluster controllers kubectl -n dr-production get pods # Validate the deployment context with pgctl pgctl context validate --name production ``` All pods should be in the `Running` state and the context validation should report no errors before you proceed to protect VMs. ## Configuration The following sections describe the key configuration dimensions for a production Site Recovery deployment. Configuration is expressed through Kubernetes manifests and Helm values; there are no application-level config files to edit directly. ### RBAC Each controller operates under a dedicated ServiceAccount with the minimum RBAC permissions needed for its role. The Ansible playbooks create these automatically, but in environments with restricted RBAC provisioning you may need to apply them manually. **Quorum cluster — controllers that require cluster-wide read access:** - `failover-controller`: needs `get`, `list`, `watch`, `update`, and `patch` on `FailoverRequest`, `ProtectionGroup`, and related CRDs; needs `get` and `list` on Secrets (to load kubeconfigs for remote clusters). - `protection-controller`: needs `get`, `list`, `watch`, `create`, `update`, and `patch` on `ProtectionRequest` and `DRBDVolume`. - `pg-sync-controller`: needs `get`, `list`, `watch`, and `update` on `ProtectionGroup` resources across the namespaces it manages. **Primary and DR clusters — per-cluster controllers:** - `protection-group-controller`: needs `get`, `list`, `watch`, `update`, and `patch` on `ProtectionGroup` and `ReplicationGroupStatus`. - `test-failover-controller`: needs `get`, `list`, `watch`, `create`, `update`, `patch`, and `delete` on `TestFailover` resources and VolumeSnapshot objects. - `replication-monitor`: needs `get`, `list`, `watch`, `create`, and `update` on `RPOEvent` and `ReplicationGroupStatus`. > **Important:** In multi-tenant deployments each DR deployment lives in its own `dr-` namespace. Namespace-scoped RoleBindings are preferred over ClusterRoleBindings wherever the controller does not require cluster-wide access, to preserve tenant isolation. ### Resource limits Set explicit resource requests and limits on all controller Deployments to prevent resource contention on the quorum cluster, especially when managing many DR deployments simultaneously. Recommended baseline values for the quorum cluster controllers: ```yaml resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "512Mi" ``` For the `replication-monitor` agent running on primary and DR clusters, which polls DRBD replication state continuously, slightly higher memory headroom is advisable in large clusters: ```yaml resources: requests: cpu: "50m" memory: "64Mi" limits: cpu: "200m" memory: "256Mi" ``` Adjust these values upward if you observe OOMKilled events or CPU throttling during large-scale replication sync operations. ### Leader election All quorum-cluster controllers use Kubernetes leader election (via `coordination.k8s.io/leases`) to ensure only one replica is active at a time. In production, run each controller with two replicas so that a second instance can immediately assume leadership if the active pod fails. Key leader election parameters (set via controller Deployment environment variables or Helm values): | Parameter | Default | Effect | |---|---|---| | `LEADER_ELECT` | `true` | Enables leader election. Do not disable in production. | | `LEADER_ELECT_LEASE_DURATION` | `15s` | How long a lease is held before it can be acquired by another replica. | | `LEADER_ELECT_RENEW_DEADLINE` | `10s` | How long the active leader retries refreshing before giving up. | | `LEADER_ELECT_RETRY_PERIOD` | `2s` | How frequently non-leader replicas attempt to acquire the lease. | These defaults are appropriate for most deployments. Increase `LEADER_ELECT_LEASE_DURATION` only if the quorum cluster's API server experiences high latency (> 500 ms) and you observe unnecessary leader re-elections. ### DRBDReplicationPolicy settings The `DRBDReplicationPolicy` CRD is the primary configuration surface for DRBD Operator deployments. The most operationally significant fields are: | Field | Values | Effect | |---|---|---| | `protocol` | `C` (synchronous) or `A` (asynchronous) | Protocol C guarantees RPO=0 but requires < 50 ms RTT. Protocol A supports any latency with a small RPO window. | | `storageClassMappings` | List of `{primary: "", dr: ""}` pairs | Maps primary storage classes to equivalent DR storage classes for automatic PVC provisioning on the DR side. | | `replicationEndpoints` | List of IP:port pairs | The DRBD replication addresses (TCP 7000–7999) used for block-level data transfer between clusters. | ### Namespace isolation for multi-tenant deployments Each DR deployment creates a dedicated namespace named `dr-`. All controllers, secrets, and CRD instances for that deployment are scoped to this namespace. Do not share credentials or controller instances across namespaces, as this breaks tenant isolation and can cause cross-deployment failover interference. ### Site Manager UI and API The Site Manager UI and Site Manager API are deployed on the quorum cluster. The API is a Python Flask application. Ensure the Service exposing the API is accessible to the quorum cluster's ingress or load balancer, and that authentication is configured before exposing it externally. The UI relies on the API backend; both must be running for the dashboard to function correctly. ## Usage Once Site Recovery is deployed, your primary day-to-day interfaces are `pgctl`, `kubectl` (for CRD inspection), `quorum-deployments.sh` (for bulk operational tasks), and the Site Manager UI. ### Managing deployment contexts with pgctl List all registered deployment contexts: ```bash pgctl context list ``` Switch to a specific deployment context before running further commands: ```bash pgctl context use --name production ``` Validate that the active context's clusters are reachable and controllers are healthy: ```bash pgctl context validate ``` ### Inspecting Protection Groups List all Protection Groups in the active deployment: ```bash pgctl pg list ``` Inspect the replication state of a specific Protection Group: ```bash pgctl pg status --name my-protection-group ``` You can also inspect the underlying CRDs directly with kubectl: ```bash # On the primary or DR cluster kubectl get protectiongroups -A kubectl describe protectiongroup my-protection-group -n dr-production ``` ### Monitoring replication health The `replication-monitor` agent writes `ReplicationGroupStatus` and `RPOEvent` resources continuously. Use these to monitor replication health: ```bash # Overall replication health per Protection Group kubectl get replicationgroupstatuses -n dr-production # Recent RPO violations kubectl get rpoevents -n dr-production --sort-by=.metadata.creationTimestamp ``` The Site Manager UI provides the same information in dashboard form, with per-VM sync state and last-successful-sync timestamps. ### Using quorum-deployments.sh for operational tasks The `quorum-deployments.sh` script provides a menu-driven interface for more than 20 tasks. Launch it and navigate using the numbered menu: ```bash bash quorum-deployments.sh ``` Common tasks available through the menu include: - Deploying or updating the quorum management plane - Creating or removing DR deployments - Triggering planned or unplanned failovers - Backing up and restoring quorum cluster state - Running health checks across all managed clusters ### Triggering a failover For production failovers, prefer using `pgctl` or the Site Manager UI so that intent is recorded as a `FailoverRequest` CRD that the `failover-controller` orchestrates: ```bash # Planned failover pgctl failover trigger \ --protection-group my-protection-group \ --type planned # Monitor progress pgctl failover status --protection-group my-protection-group ``` The `failover-controller` on the quorum cluster watches for the `FailoverRequest` resource and drives volume promotion, VM shutdown on the primary, and VM startup on the DR cluster automatically. ## Examples ### Example 1 — Create a DRBDReplicationPolicy for synchronous replication This policy configures Protocol C (synchronous, RPO=0) replication between two storage classes. Use this when your primary and DR clusters have < 10 ms RTT. ```yaml apiVersion: drbd.site-recovery.io/v1alpha1 kind: DRBDReplicationPolicy metadata: name: sync-replication-policy namespace: dr-production spec: protocol: C storageClassMappings: - primary: fast-ssd dr: fast-ssd-dr replicationEndpoints: - primary: 10.0.1.10:7000 dr: 10.0.2.10:7000 ``` Apply it to the quorum cluster: ```bash kubectl apply -f sync-replication-policy.yaml ``` Expected: The `DRBDReplicationPolicy` is created and the DRBD Operator on each cluster begins configuring replication resources according to the policy. --- ### Example 2 — Request DR protection for a single VM (DRBD Operator model) Create a `ProtectionRequest` to protect a VM named `web-server-01`: ```yaml apiVersion: site-recovery.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-web-server-01 namespace: dr-production spec: vmName: web-server-01 vmNamespace: production-vms replicationPolicy: sync-replication-policy ``` ```bash kubectl apply -f protect-web-server-01.yaml ``` Monitor the lifecycle: ```bash kubectl get protectionrequest protect-web-server-01 -n dr-production -w ``` Expected output (progressing through lifecycle phases): ``` NAME STATUS AGE protect-web-server-01 Validating 5s protect-web-server-01 Creating 15s protect-web-server-01 Syncing 45s protect-web-server-01 Protected 3m10s ``` Once the status is `Protected`, the VM is running on a frontend PVC backed by DRBD replication and the DR cluster has a synchronized replica. --- ### Example 3 — Create a Protection Group Group two VMs so they fail over together: ```yaml apiVersion: site-recovery.io/v1alpha1 kind: ProtectionGroup metadata: name: web-tier namespace: dr-production spec: vms: - name: web-server-01 namespace: production-vms - name: web-server-02 namespace: production-vms ``` ```bash kubectl apply -f web-tier-pg.yaml ``` Check the Protection Group state: ```bash kubectl get protectiongroup web-tier -n dr-production ``` Expected output: ``` NAME STATE AGE web-tier Consistent 8m ``` `Consistent` indicates all VMs in the group are replicated and ready for failover. --- ### Example 4 — Trigger a planned failover ```yaml apiVersion: site-recovery.io/v1alpha1 kind: FailoverRequest metadata: name: failover-web-tier-planned namespace: dr-production spec: protectionGroup: web-tier type: Planned ``` ```bash kubectl apply -f failover-web-tier-planned.yaml # Watch progress kubectl get failoverrequest failover-web-tier-planned -n dr-production -w ``` Expected progression: ``` NAME STATUS AGE failover-web-tier-planned ShuttingDown 10s failover-web-tier-planned Promoting 35s failover-web-tier-planned Starting 55s failover-web-tier-planned Completed 2m15s ``` --- ### Example 5 — Inspect ReplicationGroupStatus for a Protection Group ```bash kubectl describe replicationgroupstatus web-tier -n dr-production ``` Expected output (abridged): ``` Name: web-tier Namespace: dr-production Status: Health: Healthy Last Successful Sync: 2024-01-15T14:32:01Z Volume States: web-server-01-disk: Synced web-server-02-disk: Synced ``` --- ### Example 6 — Multi-tenant deployment: add a second DR deployment Each deployment gets its own namespace. Use `quorum-deployments.sh` to add it interactively: ```bash bash quorum-deployments.sh # Select: Add new deployment # Enter deployment name: staging # Enter primary kubeconfig path: /path/to/staging-primary-kubeconfig.yaml # Enter DR kubeconfig path: /path/to/staging-dr-kubeconfig.yaml ``` Or register the context with `pgctl` after manually creating the `dr-staging` namespace and uploading credentials: ```bash pgctl context create \ --name staging \ --primary-kubeconfig /path/to/staging-primary-kubeconfig.yaml \ --dr-kubeconfig /path/to/staging-dr-kubeconfig.yaml pgctl context validate --name staging ``` The new deployment runs in the `dr-staging` namespace, fully isolated from the `dr-production` deployment. ## Troubleshooting Use the following patterns to diagnose and resolve the most common deployment and operational issues. --- ### Issue: Controller pod is in CrashLoopBackOff on the quorum cluster **Symptom:** One or more of `failover-controller`, `protection-controller`, or `pg-sync-controller` pods repeatedly crash on the quorum cluster. **Likely causes:** - Missing or malformed kubeconfig Secrets for the primary or DR clusters - RBAC permissions insufficient for the controller's ServiceAccount - Leader election lease cannot be created (missing permissions on `coordination.k8s.io/leases`) **Fix:** ```bash # Check controller logs kubectl -n dr-production logs deploy/failover-controller --previous # Verify the Secret containing cluster credentials exists kubectl -n dr-production get secrets # Check RBAC bindings for the controller ServiceAccount kubectl get rolebindings,clusterrolebindings -A \ -o wide | grep failover-controller ``` If the error mentions `leases`, add `get`, `create`, `update`, and `patch` verbs on `coordination.k8s.io/leases` to the controller's ClusterRole. --- ### Issue: ProtectionRequest is stuck in `Syncing` state **Symptom:** A `ProtectionRequest` has been in `Syncing` status for longer than expected (> 30 minutes for small volumes). **Likely causes:** - DRBD replication traffic is blocked on TCP 7000–7999 between worker nodes - The DRBD Operator on the primary or DR cluster is not running - LVM thin pool is full or not properly provisioned **Fix:** ```bash # Check DRBDVolume sync progress kubectl get drbdvolumes -n dr-production # Check the DRBD Operator pods on primary cluster kubectl --kubeconfig /path/to/primary-kubeconfig.yaml \ -n drbd-operator get pods # Verify network connectivity on a worker node ssh worker-node-1 "nc -zv 7000" # Check replication-monitor for RPO events indicating lag kubectl get rpoevents -n dr-production ``` If TCP 7000–7999 is blocked, update your firewall or NetworkPolicy rules to permit this range between primary and DR worker nodes. --- ### Issue: FailoverRequest is stuck in `Promoting` state **Symptom:** A `FailoverRequest` has not progressed past `Promoting` for several minutes. **Likely causes:** - The primary cluster is unreachable and the failover type is `Planned` (which requires graceful VM shutdown) - `ReplicationGroupStatus` for the Protection Group shows `Degraded` or `Critical`, indicating volumes are not fully synced - The `failover-controller` on the quorum cluster is not running **Fix:** ```bash # Check failover-controller logs kubectl -n dr-production logs deploy/failover-controller # Check Protection Group state kubectl get protectiongroup -n dr-production # Check replication health kubectl get replicationgroupstatus -n dr-production ``` If the primary cluster is genuinely unreachable and you need an emergency failover, delete the existing `FailoverRequest` and create a new one with `type: Unplanned`. Be aware that if Protocol A was in use, a small amount of data loss may occur. --- ### Issue: Site Manager UI shows clusters as `unreachable` **Symptom:** The Site Manager UI dashboard reports one or both clusters as `unreachable`. **Likely causes:** - The kubeconfig Secret for the affected cluster is expired or has an invalid token - The quorum cluster cannot reach the cluster's API server on TCP 6443 - The cluster genuinely has no Ready nodes **Fix:** ```bash # Test API server reachability from the quorum cluster kubectl --kubeconfig /path/to/affected-kubeconfig.yaml get nodes # If the token is expired, rotate the kubeconfig and update the Secret kubectl -n dr-production create secret generic cluster1-kubeconfig \ --from-file=kubeconfig=/path/to/renewed-kubeconfig.yaml \ --dry-run=client -o yaml | kubectl apply -f - ``` After updating the Secret, the Site Manager API will pick up the new credentials on its next reconciliation cycle. --- ### Issue: RPOEvent resources are accumulating — replication lag is high **Symptom:** `kubectl get rpoevents -n dr-production` shows a growing list of lag violation events for one or more Protection Groups. **Likely causes:** - Network bandwidth between primary and DR clusters is saturated - Protocol C (synchronous) is configured but RTT has increased above 50 ms - A worker node hosting a DRBD volume is under heavy I/O load **Fix:** ```bash # Identify which Protection Group and volume is affected kubectl describe rpoevent -n dr-production # Check current replication health kubectl get replicationgroupstatus -n dr-production # Check round-trip latency to the DR cluster ping -c 20 ``` If RTT consistently exceeds 50 ms, consider switching to Protocol A (asynchronous) by updating the `DRBDReplicationPolicy` `protocol` field. If bandwidth is the bottleneck, work with your network team to increase capacity or implement QoS policies that prioritize DRBD traffic on TCP 7000–7999. --- ### Issue: Split-brain detected on a DRBD volume **Symptom:** The `ReplicationGroupStatus` for a Protection Group shows `Critical` and the DRBD Operator logs indicate a split-brain condition on one or more volumes. **Likely causes:** - Both the primary and DR DRBD volumes simultaneously assumed the primary role, typically after a network partition followed by recovery. **Fix:** Split-brain resolution requires manual intervention to choose which side's data to keep. Do not attempt automated recovery without identifying which replica has the most recent data. ```bash # Identify the affected DRBDVolume kubectl get drbdvolumes -n dr-production kubectl describe drbdvolume -n dr-production ``` Follow your organization's data integrity runbook to designate one side as the authoritative replica, then force the other side to resynchronize from it. After resolution, verify the `ReplicationGroupStatus` returns to `Healthy` before resuming normal operations or attempting a failover.