---
title: Architecture
product: trilio-site-recovery-for-kubernetes-openshift-virtualization
doc_type: architecture
version: feature-tsr-24
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/architecture
---

# Architecture

_Controller architecture, reconciliation loops, and component interaction for site recovery workflows_

## Overview

This page describes the internal architecture of Site Recovery: how its controllers are structured, how they communicate through Kubernetes Custom Resources, and how control flows across the quorum, primary, and DR clusters during protection, failover, and failback operations. Understanding this architecture helps you reason about reconciliation behavior, diagnose failures, and plan operational decisions such as concurrent failover safety and multi-tenant deployments. Site Recovery follows the Kubernetes operator pattern throughout — controllers continuously reconcile declared state in CRDs against actual cluster state, and all user interactions happen through kubectl and custom resource manifests rather than direct API calls.

## Architecture diagram

```mermaid
flowchart TB
    subgraph Quorum["Quorum Cluster (Management Plane)"]
        SM["Site Manager UI"]
        SMAPI["Site Manager API"]
        FC["failover-controller"]
        PC["protection-controller"]
        PSC["pg-sync-controller"]
        SM --> SMAPI
        SMAPI --> FC
        SMAPI --> PC
    end

    subgraph Primary["Primary Cluster (Active Site)"]
        PGC_P["protection-group-controller"]
        TFC_P["test-failover-controller"]
        RM_P["replication-monitor"]
        DO_P["DRBD Operator"]
        VM_P["Production VMs"]
        PVC_P["Frontend PVCs (DRBD-backed)"]
        PGC_P --> VM_P
        VM_P --> PVC_P
        DO_P --> PVC_P
        RM_P -->|"writes"| RPO_P["RPOEvent / ReplicationGroupStatus"]
    end

    subgraph DR["DR Cluster (Standby Site)"]
        PGC_D["protection-group-controller"]
        TFC_D["test-failover-controller"]
        RM_D["replication-monitor"]
        DO_D["DRBD Operator"]
        VM_D["Standby VMs (stopped)"]
        PVC_D["Replicated PVCs"]
        PGC_D --> VM_D
        DO_D --> PVC_D
        RM_D -->|"writes"| RPO_D["RPOEvent / ReplicationGroupStatus"]
    end

    DRBD_REP[/"DRBD Block Replication\nTCP 7000-7999"/]
    PVC_P -->|"sync"| DRBD_REP
    DRBD_REP -->|"replicated"| PVC_D

    FC -->|"watches FailoverRequest"| FC
    FC -->|"patches ProtectionGroup\ndesiredState"| PGC_P
    FC -->|"patches ProtectionGroup\ndesiredState"| PGC_D
    PC -->|"creates DRBDVolume\nswaps Frontend PVC"| PVC_P
    PSC -->|"syncs PG metadata"| PGC_D

    pgctl["pgctl (CLI)"] -->|"kubectl apply CRDs"| Quorum
    pgctl -->|"kubectl apply CRDs"| Primary
    pgctl -->|"kubectl apply CRDs"| DR
```

## Components

Site Recovery distributes its control plane across three cluster roles. The following describes each component, where it runs, and what it does from an operational perspective.

**Quorum cluster components**

| Component | Type | Role |
|---|---|---|
| **failover-controller** | Operator | Watches `FailoverRequest` resources and orchestrates the full failover sequence: stopping VMs on the source cluster by patching `ProtectionGroup.spec.desiredState`, waiting for volume promotion, then starting VMs on the target cluster. It never manipulates VMs directly. |
| **protection-controller** | Operator | Handles `ProtectionRequest` resources in DRBD Operator deployments. Validates VMs, creates `DRBDVolume` resources, waits for initial sync, and switches VMs to DRBD-backed frontend PVCs. |
| **pg-sync-controller** | Operator | Keeps `ProtectionGroup` metadata synchronized between primary and DR clusters so that standby VM specs remain accurate and consistent with production. |
| **Site Manager UI** | Service | A web-based management interface providing dashboards for cluster health, VM protection status, Protection Group management, failover triggering, and operation monitoring. Deployed on the quorum cluster. |
| **Site Manager API** | API | The Python Flask REST API backend for the Site Manager UI. Exposes endpoints for deployments, VMs, Protection Groups, operations, health, and replication status. Can also be used for custom integrations. |

**Primary and DR cluster components**

| Component | Type | Role |
|---|---|---|
| **protection-group-controller** | Operator | Runs on both primary and DR clusters. Manages `ProtectionGroup` resources, tracks per-VM replication state, and reconciles VM lifecycle (start/stop) in response to `spec.desiredState` changes set by the failover-controller. Updates `status.currentState` to reflect actual VM state. |
| **test-failover-controller** | Operator | Runs on both primary and DR clusters. Manages `TestFailover` resources, orchestrating snapshot creation, test VM provisioning, verification checks, and cleanup without affecting production workloads. Test failover is supported only in DRBD Operator deployment models. |
| **replication-monitor** | Agent | Monitors DRBD replication health on each cluster. Creates `RPOEvent` resources when lag thresholds are exceeded and maintains `ReplicationGroupStatus` resources for operator visibility. Deployed automatically as part of the standard Ansible playbooks. |
| **DRBD Operator** | Operator | Installed on primary and DR clusters in the DRBD Operator deployment model. Manages the lifecycle of DRBD-replicated volumes using `DRBDReplicationPolicy` and `DRBDVolume` resources. |

**CLI tools**

| Component | Type | Role |
|---|---|---|
| **pgctl** | CLI | The primary command-line tool for managing deployment contexts, creating and inspecting Protection Groups, validating configuration, and triggering or monitoring failover operations. |
| **quorum-deployments.sh** | CLI | An interactive shell script providing a menu-driven interface for more than 20 operational tasks including deploying the quorum, triggering failovers, backing up and restoring state, and running health checks. |

**Custom Resources (CRDs)**

| CRD | Purpose |
|---|---|
| `ProtectionGroup` | Groups VMs that must fail over together; tracks collective replication state (Syncing, Consistent, or Degraded). |
| `ProtectionRequest` | Requests DR protection for a single VM in DRBD Operator deployments. |
| `FailoverRequest` | Triggers planned or unplanned failover for a Protection Group; tracks progress to completion or failure. |
| `TestFailover` | Initiates a non-disruptive DR validation run using volume snapshots. |
| `DRBDReplicationPolicy` | Defines cross-cluster replication settings for DRBD Operator deployments. |
| `DRBDVolume` | Represents a single DRBD-replicated PVC; exposes sync progress and frontend PVC references. |
| `RPOEvent` | Records replication lag violations for auditing and alerting. |
| `ReplicationGroupStatus` | Provides an aggregated replication health summary for a Protection Group. |

## Data flow

The following traces control flow from a user's intent to protected, running VMs across two clusters — then through a planned failover. This covers both the protection and failover workflows end-to-end.

### Phase 1: Protecting a VM (DRBD Operator model)

1. **You declare protection intent.** Using `pgctl` or `kubectl`, you apply a `ProtectionRequest` resource on the quorum cluster specifying the VM to protect and the `DRBDReplicationPolicy` to use.

2. **protection-controller validates and provisions.** The protection-controller on the quorum cluster picks up the `ProtectionRequest`, validates that the target VM exists and is in a compatible state, and creates a `DRBDVolume` resource. The DRBD Operator on the primary cluster provisions a DRBD-replicated PVC and begins synchronizing data to the DR cluster over TCP ports 7000–7999.

3. **Frontend PVC swap.** Once the `DRBDVolume` reports full sync, the protection-controller switches the VM to a DRBD-backed frontend PVC. From this point forward, all VM disk I/O flows through the DRBD replication layer.

4. **ProtectionGroup enrollment.** You apply or update a `ProtectionGroup` resource on the primary cluster that references this VM alongside others that should fail over as a unit. The protection-group-controller begins tracking per-VM replication state within the group.

5. **pg-sync-controller mirrors state.** The pg-sync-controller on the quorum cluster continuously synchronizes `ProtectionGroup` metadata to the DR cluster, keeping standby VM specs accurate.

6. **replication-monitor watches health.** On both clusters, replication-monitor tracks DRBD sync progress. If replication lag exceeds configured thresholds, it writes an `RPOEvent` and updates the `ReplicationGroupStatus` for the affected Protection Group. You can observe these with `kubectl get rpoevent` or via the Site Manager UI.

### Phase 2: Planned Failover

1. **You trigger a failover.** You create a `FailoverRequest` resource (via `pgctl`, the Site Manager UI, or `kubectl apply`) specifying the Protection Group and failover type (`planned`).

2. **failover-controller acquires a lock.** The failover-controller on the quorum cluster picks up the `FailoverRequest`. It acquires a Kubernetes Lease (`failover-lock-{protection-group-name}`) to prevent concurrent failover operations on the same Protection Group.

3. **VMs stopped on source via Protection Group.** The failover-controller patches `ProtectionGroup.spec.desiredState: stopped` on the primary cluster. It does not touch VMs directly. The protection-group-controller on the primary cluster observes the `desiredState` change and reconciles each VM in the group by patching `spec.running: false`. It then updates `status.currentState: stopped`.

4. **Failover-controller waits for source confirmation.** The failover-controller polls the primary cluster's `ProtectionGroup.status.currentState` until it reaches `stopped`. The failover state machine advances through `StoppingOnSource`.

5. **DRBD volume promotion.** The failover-controller removes quorum taints on the DR cluster (after verifying no other Protection Groups with running VMs would be affected) and waits for DRBD to promote the DR-side volumes to primary role.

6. **VMs started on target via Protection Group.** The failover-controller patches `ProtectionGroup.spec.desiredState: running` on the DR cluster. The protection-group-controller on the DR cluster reconciles VM state, starting each VM. It updates `status.currentState: running`. The failover state machine advances through `StartingOnTarget`.

7. **FailoverRequest marked complete.** Once the DR cluster's `ProtectionGroup.status.currentState` is `running`, the failover-controller marks the `FailoverRequest` status as `Completed`. You can verify this with `pgctl` or `kubectl get failoverrequest`.

### Control flow summary

```
You (pgctl / kubectl)
  → FailoverRequest CRD (quorum cluster)
    → failover-controller
      → patches ProtectionGroup.spec.desiredState (primary cluster)
        → protection-group-controller (primary)
          → stops VMs → updates status.currentState
      → removes DRBD quorum taints (DR cluster)
      → patches ProtectionGroup.spec.desiredState (DR cluster)
        → protection-group-controller (DR)
          → starts VMs → updates status.currentState
      → updates FailoverRequest.status = Completed
```

At no point does the failover-controller directly patch VM resources. All VM lifecycle operations are delegated to the protection-group-controller on the respective cluster through the `ProtectionGroup` CRD.

## Design decisions

### Decision 1: Hybrid CRD-based controller architecture

Site Recovery uses a layered architecture where each controller operates only on its local cluster and communicates intent to peer controllers exclusively through CRDs. The failover-controller never directly patches VM resources — it patches `ProtectionGroup.spec.desiredState`, and the protection-group-controller on the target cluster is responsible for reconciling VMs to match that state.

**Rationale:** This enforces separation of concerns: the protection-group-controller owns VM lifecycle on its cluster; the failover-controller owns cross-cluster workflow orchestration. Keeping VM reconciliation logic in a single controller makes it easier to add pre/post hooks, health checks, or rollback logic without duplicating code across controllers. It also makes the system idempotent — the protection-group-controller always reconciles to the declared `desiredState` regardless of how many times it is triggered.

### Decision 2: Quorum cluster as dedicated management plane

The failover-controller, protection-controller, and pg-sync-controller all run on a dedicated quorum cluster rather than on the primary or DR clusters. The quorum cluster does not run application workloads and does not relay storage replication traffic.

**Rationale:** Separating the management plane from the data plane means a failure on the primary cluster does not affect the ability to trigger failover. The quorum cluster can reach both primary and DR cluster APIs during a site failure, which is necessary for unplanned failover orchestration. It also provides a neutral vantage point for multi-tenant management — multiple independent DR deployments, each in their own `dr-<name>` namespace, share a single management cluster without cross-tenant interference.

### Decision 3: Kubernetes Leases for per-Protection Group failover locking

Before executing a failover, the failover-controller acquires a Kubernetes Lease resource named `failover-lock-{protection-group-name}` in the Protection Group's namespace. The lease has a 5-minute expiry to handle cases where a controller crashes mid-operation.

**Rationale:** DRBD quorum taints are applied at the node level, not per-Protection Group or per-PVC. Removing taints for one Protection Group affects scheduling for all workloads on those nodes. Without a locking mechanism, two concurrent failover operations targeting different Protection Groups on the same cluster could remove taints prematurely and disrupt each other's VMs. The lease approach is Kubernetes-native, requires no external coordination service, and produces a clear audit trail of who held the lock and when.

### Decision 4: Safety checks before quorum taint removal

Before removing DRBD quorum taints on a target cluster, the failover-controller checks whether any other Protection Groups in the same namespace have a `status.currentState` of `running` on that cluster. If conflicts exist, taint removal is aborted unless a `force` flag is explicitly set on the `FailoverRequest`.

**Rationale:** Removing node-level taints while other Protection Groups have running VMs on those nodes can cause unexpected VM rescheduling or disruption. The safety check prevents accidental cross-PG interference in environments where multiple Protection Groups coexist. The `force` override exists for emergency scenarios — such as an unplanned failover during a maintenance window — where the operator has explicitly accepted the risk.

### Decision 5: ProtectionGroup `spec.desiredState` as the VM lifecycle contract

The `ProtectionGroup` CRD exposes a `spec.desiredState` field (`running` or `stopped`) that the failover-controller sets and the protection-group-controller reconciles. The actual state of VMs is reflected in `status.currentState` (`running`, `stopped`, `mixed`, or `unknown`).

**Rationale:** This declarative contract means the failover-controller does not need to know which specific VMs are in the group or track their individual states. It simply declares intent and waits for the protection-group-controller to confirm convergence. The `mixed` state allows the protection-group-controller to signal partial progress, giving the failover-controller a stable signal to wait on rather than polling individual VM statuses.

## Trade-offs

### Known limitations

**DRBD taints are node-scoped, not PVC-scoped.**  
Quorum taints applied by DRBD are node-level constructs. Site Recovery cannot selectively remove taints for a specific Protection Group without affecting all workloads scheduled on those nodes. The safety check and `force` flag mitigate this, but they do not eliminate the underlying constraint — that constraint is inherent to how DRBD and Kubernetes node taints interact.

**Safety checks are namespace-scoped.**  
The concurrent failover safety check inspects Protection Groups within the same namespace as the failing-over group. Protection Groups in other namespaces on the same cluster are not checked. In multi-tenant deployments where multiple `dr-<name>` namespaces coexist on overlapping worker nodes, operators should coordinate failover timing manually or use the Site Manager UI to confirm cluster-wide readiness.

**No global failover queue.**  
Site Recovery does not implement a cluster-wide failover scheduler or queue. If two operators independently trigger failovers for different Protection Groups toward the same target cluster simultaneously, the second failover will be blocked by the safety check (not queued). The operator must retry after the first failover completes. A future failover coordinator is a planned enhancement.

**Test failover is only supported in DRBD Operator deployment models.**  
The `TestFailover` resource and the test-failover-controller are not available in centralized storage deployments. If your deployment uses a shared storage controller rather than the DRBD Operator, you cannot perform non-disruptive DR validation using the `TestFailover` workflow.

**RTO depends on VM count and storage sync time.**  
Site Recovery targets 3–8 minutes for automated failover. This range assumes a reasonably sized Protection Group. Very large groups with many VMs or volumes, or environments where DRBD resync is required before volume promotion, may exceed this target.

### Alternatives not chosen

**Direct VM manipulation from the failover-controller.**  
An earlier design had the failover-controller patch individual VM resources directly across clusters. This was rejected because it duplicated VM lifecycle logic, bypassed the protection-group-controller's idempotent reconciliation, and made it harder to add pre/post hooks to the VM stop/start sequence. All VM operations now go through `ProtectionGroup.spec.desiredState`.

**Script-driven failover without CRDs.**  
Shell scripts that directly stop and start VMs without creating a `FailoverRequest` remain functional for development and testing purposes but are not recommended for production. They produce no audit trail, cannot be queried for status after the fact, and bypass the locking and safety mechanisms built into the failover-controller.

**Running management controllers on the primary cluster.**  
Collocating the failover-controller on the primary cluster would make it unavailable during a primary site failure — exactly the scenario where it is most needed. The quorum cluster's isolation from application workloads is a prerequisite for reliable unplanned failover.

### When to choose a different approach

- If your environment cannot support a third dedicated quorum cluster, the DRBD Operator deployment model with two clusters (primary + DR) provides a reduced-footprint alternative, though cross-cluster orchestration capabilities are constrained.
- If you require synchronous replication with zero RPO, ensure your primary-to-DR network round-trip latency is under 50ms. Protocol A (asynchronous) is the appropriate choice for links above that threshold, with the understanding that a small RPO window (measured in seconds) applies.
- If you need to protect workloads that are not KubeVirt or OpenShift Virtualization VMs, Site Recovery is not the right tool — its protection model is built specifically around VM block storage replication via DRBD.