Architecture

This page describes the internal architecture of Site Recovery: how its controllers are structured, how they communicate through Kubernetes Custom Resources, and how control flows across the quorum, primary, and DR clusters during protection, failover, and failback operations. Understanding this architecture helps you reason about reconciliation behavior, diagnose failures, and plan operational decisions such as concurrent failover safety and multi-tenant deployments. Site Recovery follows the Kubernetes operator pattern throughout — controllers continuously reconcile declared state in CRDs against actual cluster state, and all user interactions happen through kubectl and custom resource manifests rather than direct API calls.

Site Recovery distributes its control plane across three cluster roles. The following describes each component, where it runs, and what it does from an operational perspective.

Quorum cluster components

Primary and DR cluster components

CLI tools

Custom Resources (CRDs)

The following traces control flow from a user's intent to protected, running VMs across two clusters — then through a planned failover. This covers both the protection and failover workflows end-to-end.

Phase 1: Protecting a VM (DRBD Operator model)

1. You declare protection intent. Using pgctl or kubectl, you apply a ProtectionRequest resource on the quorum cluster specifying the VM to protect and the DRBDReplicationPolicy to use.

2. protection-controller validates and provisions. The protection-controller on the quorum cluster picks up the ProtectionRequest, validates that the target VM exists and is in a compatible state, and creates a DRBDVolume resource. The DRBD Operator on the primary cluster provisions a DRBD-replicated PVC and begins synchronizing data to the DR cluster over TCP ports 7000–7999.

3. Frontend PVC swap. Once the DRBDVolume reports full sync, the protection-controller switches the VM to a DRBD-backed frontend PVC. From this point forward, all VM disk I/O flows through the DRBD replication layer.

4. ProtectionGroup enrollment. You apply or update a ProtectionGroup resource on the primary cluster that references this VM alongside others that should fail over as a unit. The protection-group-controller begins tracking per-VM replication state within the group.

5. pg-sync-controller mirrors state. The pg-sync-controller on the quorum cluster continuously synchronizes ProtectionGroup metadata to the DR cluster, keeping standby VM specs accurate.

6. replication-monitor watches health. On both clusters, replication-monitor tracks DRBD sync progress. If replication lag exceeds configured thresholds, it writes an RPOEvent and updates the ReplicationGroupStatus for the affected Protection Group. You can observe these with kubectl get rpoevent or via the Site Manager UI.

Phase 2: Planned Failover

1. You trigger a failover. You create a FailoverRequest resource (via pgctl, the Site Manager UI, or kubectl apply) specifying the Protection Group and failover type (planned).

2. failover-controller acquires a lock. The failover-controller on the quorum cluster picks up the FailoverRequest. It acquires a Kubernetes Lease (failover-lock-{protection-group-name}) to prevent concurrent failover operations on the same Protection Group.

3. VMs stopped on source via Protection Group. The failover-controller patches ProtectionGroup.spec.desiredState: stopped on the primary cluster. It does not touch VMs directly. The protection-group-controller on the primary cluster observes the desiredState change and reconciles each VM in the group by patching spec.running: false. It then updates status.currentState: stopped.

4. Failover-controller waits for source confirmation. The failover-controller polls the primary cluster's ProtectionGroup.status.currentState until it reaches stopped. The failover state machine advances through StoppingOnSource.

5. DRBD volume promotion. The failover-controller removes quorum taints on the DR cluster (after verifying no other Protection Groups with running VMs would be affected) and waits for DRBD to promote the DR-side volumes to primary role.

6. VMs started on target via Protection Group. The failover-controller patches ProtectionGroup.spec.desiredState: running on the DR cluster. The protection-group-controller on the DR cluster reconciles VM state, starting each VM. It updates status.currentState: running. The failover state machine advances through StartingOnTarget.

7. FailoverRequest marked complete. Once the DR cluster's ProtectionGroup.status.currentState is running, the failover-controller marks the FailoverRequest status as Completed. You can verify this with pgctl or kubectl get failoverrequest.

Control flow summary

You (pgctl / kubectl)
  → FailoverRequest CRD (quorum cluster)
    → failover-controller
      → patches ProtectionGroup.spec.desiredState (primary cluster)
        → protection-group-controller (primary)
          → stops VMs → updates status.currentState
      → removes DRBD quorum taints (DR cluster)
      → patches ProtectionGroup.spec.desiredState (DR cluster)
        → protection-group-controller (DR)
          → starts VMs → updates status.currentState
      → updates FailoverRequest.status = Completed

At no point does the failover-controller directly patch VM resources. All VM lifecycle operations are delegated to the protection-group-controller on the respective cluster through the ProtectionGroup CRD.

Decision 1: Hybrid CRD-based controller architecture

Site Recovery uses a layered architecture where each controller operates only on its local cluster and communicates intent to peer controllers exclusively through CRDs. The failover-controller never directly patches VM resources — it patches ProtectionGroup.spec.desiredState, and the protection-group-controller on the target cluster is responsible for reconciling VMs to match that state.

Rationale: This enforces separation of concerns: the protection-group-controller owns VM lifecycle on its cluster; the failover-controller owns cross-cluster workflow orchestration. Keeping VM reconciliation logic in a single controller makes it easier to add pre/post hooks, health checks, or rollback logic without duplicating code across controllers. It also makes the system idempotent — the protection-group-controller always reconciles to the declared desiredState regardless of how many times it is triggered.

Decision 2: Quorum cluster as dedicated management plane

The failover-controller, protection-controller, and pg-sync-controller all run on a dedicated quorum cluster rather than on the primary or DR clusters. The quorum cluster does not run application workloads and does not relay storage replication traffic.

Rationale: Separating the management plane from the data plane means a failure on the primary cluster does not affect the ability to trigger failover. The quorum cluster can reach both primary and DR cluster APIs during a site failure, which is necessary for unplanned failover orchestration. It also provides a neutral vantage point for multi-tenant management — multiple independent DR deployments, each in their own dr-<name> namespace, share a single management cluster without cross-tenant interference.

Decision 3: Kubernetes Leases for per-Protection Group failover locking

Before executing a failover, the failover-controller acquires a Kubernetes Lease resource named failover-lock-{protection-group-name} in the Protection Group's namespace. The lease has a 5-minute expiry to handle cases where a controller crashes mid-operation.

Rationale: DRBD quorum taints are applied at the node level, not per-Protection Group or per-PVC. Removing taints for one Protection Group affects scheduling for all workloads on those nodes. Without a locking mechanism, two concurrent failover operations targeting different Protection Groups on the same cluster could remove taints prematurely and disrupt each other's VMs. The lease approach is Kubernetes-native, requires no external coordination service, and produces a clear audit trail of who held the lock and when.

Decision 4: Safety checks before quorum taint removal

Before removing DRBD quorum taints on a target cluster, the failover-controller checks whether any other Protection Groups in the same namespace have a status.currentState of running on that cluster. If conflicts exist, taint removal is aborted unless a force flag is explicitly set on the FailoverRequest.

Rationale: Removing node-level taints while other Protection Groups have running VMs on those nodes can cause unexpected VM rescheduling or disruption. The safety check prevents accidental cross-PG interference in environments where multiple Protection Groups coexist. The force override exists for emergency scenarios — such as an unplanned failover during a maintenance window — where the operator has explicitly accepted the risk.

Decision 5: ProtectionGroup spec.desiredState as the VM lifecycle contract

The ProtectionGroup CRD exposes a spec.desiredState field (running or stopped) that the failover-controller sets and the protection-group-controller reconciles. The actual state of VMs is reflected in status.currentState (running, stopped, mixed, or unknown).

Rationale: This declarative contract means the failover-controller does not need to know which specific VMs are in the group or track their individual states. It simply declares intent and waits for the protection-group-controller to confirm convergence. The mixed state allows the protection-group-controller to signal partial progress, giving the failover-controller a stable signal to wait on rather than polling individual VM statuses.

Known limitations

DRBD taints are node-scoped, not PVC-scoped.
Quorum taints applied by DRBD are node-level constructs. Site Recovery cannot selectively remove taints for a specific Protection Group without affecting all workloads scheduled on those nodes. The safety check and force flag mitigate this, but they do not eliminate the underlying constraint — that constraint is inherent to how DRBD and Kubernetes node taints interact.

Safety checks are namespace-scoped.
The concurrent failover safety check inspects Protection Groups within the same namespace as the failing-over group. Protection Groups in other namespaces on the same cluster are not checked. In multi-tenant deployments where multiple dr-<name> namespaces coexist on overlapping worker nodes, operators should coordinate failover timing manually or use the Site Manager UI to confirm cluster-wide readiness.

No global failover queue.
Site Recovery does not implement a cluster-wide failover scheduler or queue. If two operators independently trigger failovers for different Protection Groups toward the same target cluster simultaneously, the second failover will be blocked by the safety check (not queued). The operator must retry after the first failover completes. A future failover coordinator is a planned enhancement.

Test failover is only supported in DRBD Operator deployment models.
The TestFailover resource and the test-failover-controller are not available in centralized storage deployments. If your deployment uses a shared storage controller rather than the DRBD Operator, you cannot perform non-disruptive DR validation using the TestFailover workflow.

RTO depends on VM count and storage sync time.
Site Recovery targets 3–8 minutes for automated failover. This range assumes a reasonably sized Protection Group. Very large groups with many VMs or volumes, or environments where DRBD resync is required before volume promotion, may exceed this target.

Alternatives not chosen

Direct VM manipulation from the failover-controller.
An earlier design had the failover-controller patch individual VM resources directly across clusters. This was rejected because it duplicated VM lifecycle logic, bypassed the protection-group-controller's idempotent reconciliation, and made it harder to add pre/post hooks to the VM stop/start sequence. All VM operations now go through ProtectionGroup.spec.desiredState.

Script-driven failover without CRDs.
Shell scripts that directly stop and start VMs without creating a FailoverRequest remain functional for development and testing purposes but are not recommended for production. They produce no audit trail, cannot be queried for status after the fact, and bypass the locking and safety mechanisms built into the failover-controller.

Running management controllers on the primary cluster.
Collocating the failover-controller on the primary cluster would make it unavailable during a primary site failure — exactly the scenario where it is most needed. The quorum cluster's isolation from application workloads is a prerequisite for reliable unplanned failover.

When to choose a different approach

• If your environment cannot support a third dedicated quorum cluster, the DRBD Operator deployment model with two clusters (primary + DR) provides a reduced-footprint alternative, though cross-cluster orchestration capabilities are constrained.
• If you require synchronous replication with zero RPO, ensure your primary-to-DR network round-trip latency is under 50ms. Protocol A (asynchronous) is the appropriate choice for links above that threshold, with the understanding that a small RPO window (measured in seconds) applies.
• If you need to protect workloads that are not KubeVirt or OpenShift Virtualization VMs, Site Recovery is not the right tool — its protection model is built specifically around VM block storage replication via DRBD.