Src

The src module provides the core programmatic interface for interacting with Site Recovery's DR orchestration layer from within the quorum control plane. It exposes controller-level functions—such as controller.add_vm_protection, controller.remove_vm_protection, and controller.get_quorum_cluster_client—that the site-recovery-quorum-control-plane operator uses internally to reconcile ProtectionRequests, manage ProtectionGroups, and coordinate cross-cluster failover operations. Understanding this subsystem is useful if you are extending the operator, writing integration tests, or diagnosing reconciliation loops that touch VM protection lifecycle.

Before working with the src module, ensure the following are in place:

• Kubernetes ≥ 1.28 on all clusters (quorum, primary, and DR)
• KubeVirt ≥ 1.0 on primary and DR clusters (VMs must be KubeVirt VirtualMachine objects)
• Helm ≥ 3.0 (required to deploy the quorum and workload control plane charts)
• DRBD kernel module ≥ 9.0 on all worker nodes in the primary and DR clusters
• DRBD Operator deployed on both primary and DR clusters (DRBD Operator model)
• site-recovery-protectionzone-controller deployed on the quorum cluster before the quorum control plane chart
• site-recovery-quorum-control-plane deployed and healthy on the quorum cluster
• site-recovery-workload-control-plane deployed (Deployment + DaemonSet) on both primary and DR clusters
• Kubeconfig Secrets for the primary and DR clusters stored on the quorum cluster
• TCP port 6443 open from the quorum cluster to both primary and DR API servers
• TCP ports 7000–7999 open between primary and DR worker nodes (DRBD replication traffic)
• A DRBDReplicationPolicy CR created in the relevant quorum namespace before any protection is requested

The src module is shipped as part of the site-recovery-quorum-control-plane operator image and is not installed independently. The steps below deploy the full quorum control plane, which activates all src reconcilers.

Step 1 — Add the Helm repository

helm repo add site-recovery https://charts.example.com/site-recovery
helm repo update

Step 2 — Deploy the ProtectionZone controller (prerequisite)

The site-recovery-protectionzone-controller must be deployed before the quorum control plane chart because it installs the CRDs that the quorum control plane depends on.

helm upgrade --install site-recovery-protectionzone-controller \
  site-recovery/site-recovery-protectionzone-controller \
  --namespace site-recovery-system \
  --create-namespace

Wait for the controller to become ready:

kubectl rollout status deployment/site-recovery-protectionzone-controller \
  -n site-recovery-system

Step 3 — Create kubeconfig Secrets for the primary and DR clusters

The quorum control plane uses these Secrets to call controller.get_quorum_cluster_client and authenticate against each workload cluster's API server.

kubectl create secret generic primary-cluster-kubeconfig \
  --from-file=kubeconfig=/path/to/primary-kubeconfig \
  --namespace dr-<deployment-name>

kubectl create secret generic dr-cluster-kubeconfig \
  --from-file=kubeconfig=/path/to/dr-kubeconfig \
  --namespace dr-<deployment-name>

Replace <deployment-name> with the name of your DR Deployment (the quorum namespace convention is dr-<name>).

Step 4 — Deploy the quorum control plane

helm upgrade --install site-recovery-quorum-control-plane \
  site-recovery/site-recovery-quorum-control-plane \
  --namespace site-recovery-system \
  --set deployment.name=<deployment-name> \
  --set primaryCluster.kubeconfigSecret=primary-cluster-kubeconfig \
  --set drCluster.kubeconfigSecret=dr-cluster-kubeconfig

Step 5 — Deploy the workload control plane on primary and DR clusters

Run this command once targeting the primary cluster and once targeting the DR cluster, adjusting the --kubeconfig flag accordingly.

helm upgrade --install site-recovery-workload-control-plane \
  site-recovery/site-recovery-workload-control-plane \
  --namespace site-recovery-system \
  --create-namespace \
  --kubeconfig /path/to/primary-kubeconfig

helm upgrade --install site-recovery-workload-control-plane \
  site-recovery/site-recovery-workload-control-plane \
  --namespace site-recovery-system \
  --create-namespace \
  --kubeconfig /path/to/dr-kubeconfig

Step 6 — Verify all components are running

# Quorum cluster
kubectl get pods -n site-recovery-system

# Primary cluster
kubectl get pods -n site-recovery-system --kubeconfig /path/to/primary-kubeconfig

# DR cluster
kubectl get pods -n site-recovery-system --kubeconfig /path/to/dr-kubeconfig

Expect to see the quorum control plane pod, the workload control plane Deployment pod, and drbd-node-agent DaemonSet pods on each worker node of the primary and DR clusters.

The src module's behavior is governed by the following key configuration surfaces. All values are specified either as Helm values (chart-level) or as fields in Custom Resource manifests.

controller.get_quorum_cluster_client

This function constructs an authenticated Kubernetes client for each workload cluster. Its behavior is driven by the kubeconfig Secrets you created in the dr-<deployment-name> namespace.

If the Secret is missing or the key is absent, the reconciler enters an error state and emits an event on the relevant ProtectionRequest or ProtectionGroup.

controller.add_vm_protection

This function is invoked when the protection controller reconciles a ProtectionRequest. Its behavior is shaped by the DRBDReplicationPolicy that must exist in the same namespace.

When add_vm_protection completes successfully, the ProtectionRequest status phase transitions to Protected and a DRBDResource is created on each cluster side representing the VM's disks.

controller.remove_vm_protection

This function is invoked when a ProtectionRequest is deleted or when protection is explicitly revoked. It tears down the DRBD replication link and optionally restores the VM's original PVC.

| Helm Value | Default | Effect | |---|---|---|---| | protection.retainFrontendPVC | false | If true, the DRBD-backed frontend PVC is preserved after protection removal; if false, it is deleted and the original PVC binding is restored | | protection.gracefulDrainTimeout | 120s | Maximum time to wait for in-flight DRBD writes to flush before severing the replication link |

Warning: Setting protection.retainFrontendPVC: false on a VM that has no snapshot or backup will result in data loss if the original PVC was deleted during protection enrollment. Verify backup posture before removing protection.

The src module functions are not called directly by operators—they are triggered by creating, updating, or deleting Custom Resources on the quorum cluster. The following workflows cover the most common interactions.

Enrolling a VM in DR protection (controller.add_vm_protection)

Create a DRBDReplicationPolicy to define replication parameters, then create a ProtectionRequest to enroll the VM.

# drbd-replication-policy.yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: policy-protocol-c
  namespace: dr-production
spec:
  protocol: C
  storageClassMappings:
    - primaryStorageClass: fast-nvme
      drStorageClass: fast-nvme-dr
  primaryEndpoints:
    - 192.168.10.11:7000
    - 192.168.10.12:7000
  drEndpoints:
    - 10.20.30.11:7000
    - 10.20.30.12:7000

kubectl apply -f drbd-replication-policy.yaml

# protection-request.yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-database
  namespace: dr-production
spec:
  vmName: vm-database
  vmNamespace: production
  replicationPolicyRef:
    name: policy-protocol-c

kubectl apply -f protection-request.yaml

Monitor progress by watching the ProtectionRequest status:

kubectl get protectionrequest protect-vm-database -n dr-production -w

The status phase progresses: Pending → Validating → ProvisioningVolumes → Synchronizing → Protected.

Inspecting replication health after enrollment

Once the VM is protected, inspect the DRBDResource objects created on each cluster side to verify sync state:

# On the quorum cluster, list all DRBDResources for the deployment
kubectl get drbdresource -n dr-production

Check the ReplicationGroupStatus for an aggregated health view:

kubectl get replicationgroupstatus -n dr-production -o wide

Removing VM protection (controller.remove_vm_protection)

Delete the ProtectionRequest to invoke controller.remove_vm_protection. The controller drains in-flight writes, tears down the DRBD link, and transitions the VM back to its original storage configuration.

kubectl delete protectionrequest protect-vm-database -n dr-production

Watch the ProtectionRequest until it is fully removed from the API server:

kubectl get protectionrequest protect-vm-database -n dr-production -w

Obtaining the quorum cluster client context (controller.get_quorum_cluster_client)

This function is used internally by every reconciler to build authenticated clients for the primary and DR clusters. If you are writing an integration test or a custom admission webhook that calls quorum control plane logic, you can simulate this behavior by mounting the same kubeconfig Secrets into your test pod and constructing a rest.Config from them:

# Verify that the kubeconfig Secrets are present and well-formed
kubectl get secret primary-cluster-kubeconfig -n dr-production -o jsonpath='{.data.kubeconfig}' | base64 -d | kubectl --kubeconfig /dev/stdin cluster-info

kubectl get secret dr-cluster-kubeconfig -n dr-production -o jsonpath='{.data.kubeconfig}' | base64 -d | kubectl --kubeconfig /dev/stdin cluster-info

If either command fails, the quorum control plane will be unable to reconcile any CRD in the dr-production namespace.

Example 1 — Protect a database VM with synchronous replication (RPO=0)

This example enrolls a KubeVirt VM named vm-postgres in the production namespace with Protocol C replication.

# Step 1: Create the replication policy
apiVersion: site-recovery.example.com/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: sync-policy
  namespace: dr-production
spec:
  protocol: C
  storageClassMappings:
    - primaryStorageClass: premium-ssd
      drStorageClass: premium-ssd-dr
  primaryEndpoints:
    - 192.168.1.10:7000
    - 192.168.1.11:7000
  drEndpoints:
    - 10.0.1.10:7000
    - 10.0.1.11:7000

kubectl apply -f sync-policy.yaml

# Step 2: Create the ProtectionRequest
apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-postgres
  namespace: dr-production
spec:
  vmName: vm-postgres
  vmNamespace: production
  replicationPolicyRef:
    name: sync-policy

kubectl apply -f protection-request-postgres.yaml

Expected output after the controller processes the request:

NAME                   PHASE       AGE
protect-vm-postgres    Protected   4m32s

Example 2 — Protect a VM with asynchronous replication (long-distance DR)

Use Protocol A when the primary and DR clusters are separated by more than ~50ms RTT. This introduces a small RPO window (seconds) but removes the network latency constraint.

apiVersion: site-recovery.example.com/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: async-policy-wan
  namespace: dr-remote
spec:
  protocol: A
  storageClassMappings:
    - primaryStorageClass: standard
      drStorageClass: standard-dr
  primaryEndpoints:
    - 203.0.113.10:7000
  drEndpoints:
    - 198.51.100.10:7000

kubectl apply -f async-policy-wan.yaml

apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-appserver
  namespace: dr-remote
spec:
  vmName: vm-appserver
  vmNamespace: apps
  replicationPolicyRef:
    name: async-policy-wan

kubectl apply -f protection-request-appserver.yaml

Expected output:

NAME                    PHASE       AGE
protect-vm-appserver    Protected   6m11s

Example 3 — Verify quorum cluster client connectivity

Before enrolling VMs, verify that the quorum control plane can reach both workload clusters by checking the conditions on an existing ProtectionRequest:

kubectl describe protectionrequest protect-vm-postgres -n dr-production

Expected output (healthy connectivity):

Conditions:
  Type                  Status   Reason
  ----                  ------   ------
  PrimaryClusterReady   True     Connected
  DRClusterReady        True     Connected
  ReplicationActive     True     Synchronizing

Example 4 — Remove VM protection

kubectl delete protectionrequest protect-vm-postgres -n dr-production

Expected output (watch until deletion):

NAME                   PHASE          AGE
protect-vm-postgres    Terminating    8m14s
# (resource disappears from list once teardown completes)

Example 5 — Inspect RPO violations after protection enrollment

If the replication monitor detects that a Protection Group has exceeded its RPO threshold, it creates an RPOEvent:

kubectl get rpoevent -n dr-production

Expected output:

NAME                                SEVERITY   LAG       AGE
rpoevent-protect-vm-postgres-001    Warning    12s       2m04s

See the troubleshooting section for how to resolve replication lag.

Use the following structured guide to diagnose common failures in the src module's controller functions.

Issue 1 — ProtectionRequest stuck in Pending phase

Symptom: The ProtectionRequest has been created but its status phase does not advance beyond Pending after several minutes.

Likely causes:

• The referenced DRBDReplicationPolicy does not exist in the same namespace.
• The kubeconfig Secrets for the primary or DR cluster are missing or contain stale credentials.
• The site-recovery-quorum-control-plane pod is not running.

Fix:

# Check that the DRBDReplicationPolicy exists
kubectl get drbdreplicationpolicy -n dr-<deployment-name>

# Check the quorum control plane pod
kubectl get pods -n site-recovery-system

# View controller logs for reconciliation errors
kubectl logs -n site-recovery-system -l app=site-recovery-quorum-control-plane --tail=100

# Verify kubeconfig Secrets
kubectl get secret primary-cluster-kubeconfig dr-cluster-kubeconfig -n dr-<deployment-name>

Issue 2 — ProtectionRequest stuck in ProvisioningVolumes phase

Symptom: The phase reached ProvisioningVolumes but has not progressed to Synchronizing after 10+ minutes.

Likely causes:

• The drbd-node-agent DaemonSet pods on the primary or DR cluster are not running.
• DRBD kernel module is not loaded on the target worker node.
• TCP ports 7000–7999 are blocked between primary and DR worker nodes.

Fix:

# Check drbd-node-agent DaemonSet on primary
kubectl get daemonset -n site-recovery-system --kubeconfig /path/to/primary-kubeconfig
kubectl get pods -n site-recovery-system -l app=drbd-node-agent --kubeconfig /path/to/primary-kubeconfig

# Check drbd-node-agent logs on a specific node
kubectl logs -n site-recovery-system <drbd-node-agent-pod> --kubeconfig /path/to/primary-kubeconfig

# Verify DRBD module on a worker node
kubectl debug node/<worker-node-name> -it --image=busybox --kubeconfig /path/to/primary-kubeconfig -- lsmod | grep drbd

Issue 3 — controller.get_quorum_cluster_client fails (cluster unreachable)

Symptom: The quorum control plane logs show errors such as failed to build client for primary cluster or connection refused to port 6443.

Likely causes:

• The kubeconfig Secret contains an expired token or certificate.
• TCP port 6443 is blocked from the quorum cluster to the workload cluster API server.
• The kubeconfig references an internal cluster hostname not resolvable from the quorum cluster.

Fix:

# Test reachability from the quorum cluster
kubectl run connectivity-test --image=curlimages/curl --restart=Never --rm -it \
  -- curl -k https://<primary-api-server>:6443/healthz

# Rotate the kubeconfig Secret if credentials are stale
kubectl create secret generic primary-cluster-kubeconfig \
  --from-file=kubeconfig=/path/to/refreshed-primary-kubeconfig \
  --namespace dr-<deployment-name> \
  --dry-run=client -o yaml | kubectl apply -f -

Issue 4 — RPOEvent created; replication lag exceeds threshold

Symptom: kubectl get rpoevent -n dr-<deployment-name> shows one or more events with Warning or Critical severity.

Likely causes:

• Network congestion or increased latency between primary and DR worker nodes.
• A DR cluster worker node is down, causing the DRBD connection to drop.
• Protocol A buffer is filling faster than it can drain to the DR cluster.

Fix:

# Check ReplicationGroupStatus for aggregate health
kubectl get replicationgroupstatus -n dr-<deployment-name> -o wide

# Check DRBDResource status for per-volume sync state
kubectl describe drbdresource -n dr-<deployment-name>

# Check drbd-node-agent logs on the node with lag
kubectl logs -n site-recovery-system <drbd-node-agent-pod> --kubeconfig /path/to/dr-kubeconfig

# Collect a full support bundle for escalation
oc adm must-gather --image=<tsr-gather-image> -- /usr/bin/tsr-gather

Issue 5 — controller.remove_vm_protection does not complete; ProtectionRequest stuck in Terminating

Symptom: After running kubectl delete protectionrequest <name>, the resource remains in Terminating state for more than 5 minutes.

Likely causes:

• A finalizer on the ProtectionRequest is blocked because the quorum control plane cannot reach one of the workload clusters.
• In-flight DRBD writes have not flushed within the configured gracefulDrainTimeout.

Fix:

# Check quorum control plane logs for finalizer errors
kubectl logs -n site-recovery-system -l app=site-recovery-quorum-control-plane --tail=200 | grep -i finalizer

# If the cluster is permanently unreachable and you must force-remove the resource,
# remove the finalizer manually (use with caution — data loss risk):
kubectl patch protectionrequest <name> -n dr-<deployment-name> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

Warning: Manually removing finalizers bypasses the graceful teardown of DRBD replication. Only do this if the target cluster is confirmed permanently unreachable and you accept potential data inconsistency on the DR side.