---
title: Custom Resource Schemas
product: trilio-site-recovery-for-kubernetes-openshift-virtualization
doc_type: guide
version: feature-tsr-24
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/crd-schema
---

# Custom Resource Schemas

_CRD structure, field validation rules, and status conditions for site recovery resources_

## Overview

This page documents the structure, field validation rules, and status conditions for all Custom Resource Definitions (CRDs) used by Site Recovery. Understanding these schemas lets you author valid manifests, interpret controller-reported status, and build automation around the lifecycle of protection, failover, and replication health. Each CRD is described with its spec fields, accepted values and constraints, and the status subresource conditions that controllers write back as they reconcile your desired state.

## Prerequisites

Before working with Site Recovery CRDs, ensure the following are in place:

- **Kubernetes ≥ 1.28** on all clusters (quorum, primary, and DR)
- **KubeVirt ≥ 1.0** installed on primary and DR clusters
- **DRBD kernel module ≥ 9.0** on worker nodes that host replicated volumes
- **kubectl** configured with access to the target cluster
- Site Recovery controllers deployed on the quorum cluster (failover-controller, protection-controller, pg-sync-controller)
- protection-group-controller and test-failover-controller deployed on primary and DR clusters
- replication-monitor deployed on primary and DR clusters (deployed automatically by the standard Ansible playbooks)
- For DRBD Operator deployments: DRBD Operator installed on both primary and DR clusters
- VolumeSnapshot support (CSI snapshotter) on primary and DR clusters if you intend to use TestFailover resources
- Familiarity with Kubernetes custom resources and `kubectl apply` workflows

## Installation

The CRDs are installed automatically as part of the Site Recovery deployment playbooks. You do not apply them manually. The steps below show how to verify that all CRDs are present after deployment and how to inspect their schemas.

**Step 1 — Confirm CRDs are registered on the quorum cluster**

```bash
kubectl get crds | grep site-recovery
```

Expected output lists the following CRDs (group names may vary by your installation):

```
protectiongroups.<group>
protectionrequests.<group>
failoverrequests.<group>
testfailovers.<group>
drbdreplicationpolicies.<group>
drbdvolumes.<group>
rpoevents.<group>
replicationgroupstatuses.<group>
```

**Step 2 — Inspect the OpenAPI schema for a specific CRD**

```bash
kubectl explain protectiongroup --recursive
kubectl explain protectionrequest.spec --recursive
```

**Step 3 — Confirm CRDs are registered on primary and DR clusters**

ProtectionGroup, TestFailover, DRBDVolume, DRBDReplicationPolicy, and ReplicationGroupStatus resources are also watched on primary and DR clusters. Verify:

```bash
# Run on primary cluster
kubectl --kubeconfig /path/to/primary-kubeconfig.yaml get crds | grep site-recovery

# Run on DR cluster
kubectl --kubeconfig /path/to/dr-kubeconfig.yaml get crds | grep site-recovery
```

**Step 4 — Validate a manifest before applying**

Use `pgctl` to validate a resource manifest against the registered schema:

```bash
pgctl validate -f my-protection-group.yaml
```

## Configuration

The following sections describe every CRD spec field, its type, validation constraints, default value (where applicable), and behavioral effect. Status fields written by controllers are listed separately under each CRD.

---

### ProtectionGroup

A ProtectionGroup declares which VMs should fail over together as a coordinated unit and tracks their collective replication state. It is used in centralized storage deployments. The protection-group-controller on the primary and DR clusters reconciles this resource; the pg-sync-controller on the quorum cluster keeps both sides synchronized.

**Spec fields**

| Field | Type | Required | Validation | Description |
|---|---|---|---|---|
| `vmNames` | `[]string` | Yes | Non-empty list | Names of VMs in the same namespace that belong to this group. All VMs must exist before protection can be established. |
| `namespace` | `string` | Yes | Valid DNS label | Namespace on the primary cluster where the listed VMs reside. |
| `drNamespace` | `string` | No | Valid DNS label | Namespace on the DR cluster where standby VM specs are maintained. Defaults to the same value as `namespace`. |
| `replicationPolicyRef` | `string` | No | Name of a DRBDReplicationPolicy | For DRBD Operator deployments, the name of the DRBDReplicationPolicy to apply to all volumes in this group. |
| `consistencyMode` | `string` | No | `Strict` \| `BestEffort` | Controls whether failover is blocked when any VM in the group is not fully synced. `Strict` blocks failover; `BestEffort` proceeds with a warning. Defaults to `Strict`. |

**Status fields**

| Field | Type | Description |
|---|---|---|
| `phase` | `string` | Aggregate replication state: `Syncing`, `Consistent`, or `Degraded`. |
| `conditions` | `[]Condition` | Standard Kubernetes condition list. Condition types include `Ready`, `AllVMsProtected`, and `ReplicationHealthy`. |
| `vmStatuses` | `[]VMStatus` | Per-VM replication state within the group. Each entry includes `vmName`, `phase`, and `lastSyncTime`. |
| `lastTransitionTime` | `string` | RFC 3339 timestamp of the last phase transition. |

---

### ProtectionRequest

A ProtectionRequest requests DR protection for a single VM in DRBD Operator deployments. The protection-controller on the quorum cluster drives the resource through a defined lifecycle: validation → DRBDVolume creation → sync wait → frontend PVC switchover → protected.

**Spec fields**

| Field | Type | Required | Validation | Description |
|---|---|---|---|---|
| `vmName` | `string` | Yes | Non-empty | Name of the VM to protect. The VM must be running on the primary cluster. |
| `vmNamespace` | `string` | Yes | Valid DNS label | Namespace of the VM on the primary cluster. |
| `replicationPolicyRef` | `string` | Yes | Name of a DRBDReplicationPolicy | The replication policy that defines how the VM's volumes are replicated. |
| `protectionGroupRef` | `string` | No | Name of a ProtectionGroup | If set, associates this VM with a Protection Group after protection is established. |

**Status fields**

| Field | Type | Description |
|---|---|---|
| `phase` | `string` | Current lifecycle phase: `Validating`, `CreatingVolumes`, `WaitingForSync`, `SwitchingFrontend`, `Protected`, or `Failed`. |
| `conditions` | `[]Condition` | Condition types include `VMValidated`, `VolumesCreated`, `ReplicationEstablished`, `FrontendSwitched`. |
| `drbdVolumeRefs` | `[]string` | Names of DRBDVolume resources created for this VM's PVCs. |
| `frontendPVCRefs` | `[]string` | Names of the DRBD-backed frontend PVCs that replaced the VM's original PVCs. |
| `message` | `string` | Human-readable description of the current state or failure reason. |

---

### FailoverRequest

A FailoverRequest triggers a planned or unplanned failover for a Protection Group. The failover-controller on the quorum cluster watches for this resource and orchestrates the full failover sequence: VM shutdown (planned only), volume promotion on the DR cluster, and VM startup.

**Spec fields**

| Field | Type | Required | Validation | Description |
|---|---|---|---|---|
| `protectionGroupRef` | `string` | Yes | Name of an existing ProtectionGroup | The Protection Group to fail over. |
| `type` | `string` | Yes | `Planned` \| `Unplanned` | `Planned` performs a graceful VM shutdown before promoting DR volumes, guaranteeing zero data loss. `Unplanned` promotes volumes immediately without a graceful shutdown, for use when the primary cluster is unavailable. |
| `targetNamespace` | `string` | No | Valid DNS label | Namespace on the DR cluster where VMs should start. Defaults to the namespace declared in the ProtectionGroup. |
| `force` | `boolean` | No | — | When `true` on an `Unplanned` failover, skips reachability checks for the primary cluster. Use only when the primary is confirmed unreachable. Defaults to `false`. |

**Status fields**

| Field | Type | Description |
|---|---|---|
| `phase` | `string` | Operation phase: `Pending`, `ShuttingDownVMs`, `PromotingVolumes`, `StartingVMs`, `Completed`, or `Failed`. |
| `conditions` | `[]Condition` | Condition types include `VMsShutdown`, `VolumesPromoted`, `VMsStarted`. |
| `startTime` | `string` | RFC 3339 timestamp when the failover-controller began processing. |
| `completionTime` | `string` | RFC 3339 timestamp when the operation reached `Completed` or `Failed`. |
| `message` | `string` | Human-readable status or failure reason. |

---

### TestFailover

A TestFailover initiates a non-disruptive DR validation run. The test-failover-controller on the primary and DR clusters orchestrates snapshot creation, test VM provisioning on the DR cluster, verification checks, and cleanup — without affecting production workloads. TestFailover is supported only for DRBD Operator deployment models.

**Spec fields**

| Field | Type | Required | Validation | Description |
|---|---|---|---|---|
| `protectionGroupRef` | `string` | Yes | Name of an existing ProtectionGroup | The Protection Group whose volumes are snapshotted for the test. |
| `testNamespace` | `string` | No | Valid DNS label | Namespace on the DR cluster where test VMs are created. Defaults to `<protectionGroupRef>-test`. |
| `snapshotClass` | `string` | No | Name of a VolumeSnapshotClass | The CSI snapshot class to use when snapshotting volumes. If omitted, the cluster default is used. |
| `autoCleanup` | `boolean` | No | — | When `true`, the controller removes test VMs and snapshots after verification completes. Defaults to `true`. |
| `verificationTimeoutSeconds` | `integer` | No | ≥ 60 | How long the controller waits for test VMs to reach Running state before marking the test as failed. Defaults to `300`. |

**Status fields**

| Field | Type | Description |
|---|---|---|
| `phase` | `string` | Lifecycle phase: `CreatingSnapshots`, `ProvisioningTestVMs`, `Verifying`, `CleaningUp`, `Succeeded`, or `Failed`. |
| `conditions` | `[]Condition` | Condition types include `SnapshotsCreated`, `TestVMsRunning`, `VerificationPassed`, `CleanupComplete`. |
| `snapshotRefs` | `[]string` | Names of VolumeSnapshot resources created during the test. |
| `testVMRefs` | `[]string` | Names of test VM resources created on the DR cluster. |
| `startTime` | `string` | RFC 3339 timestamp when the test began. |
| `completionTime` | `string` | RFC 3339 timestamp when the test reached a terminal phase. |
| `message` | `string` | Human-readable result or failure reason. |

---

### DRBDReplicationPolicy

A DRBDReplicationPolicy defines cross-cluster replication settings for DRBD Operator deployments. The DRBD Operator on primary and DR clusters reads this resource to configure DRBD protocol, storage class mappings, and replication endpoints.

**Spec fields**

| Field | Type | Required | Validation | Description |
|---|---|---|---|---|
| `protocol` | `string` | Yes | `A` \| `C` | DRBD replication protocol. `C` (synchronous) acknowledges writes only after both sides commit; requires < 50 ms RTT and achieves RPO=0. `A` (asynchronous) acknowledges after local commit; suitable for high-latency links; achieves near-zero RPO. |
| `storageClassMappings` | `[]StorageClassMapping` | Yes | Non-empty list | Pairs a primary-cluster storage class name with the corresponding DR-cluster storage class name. |
| `storageClassMappings[].primaryStorageClass` | `string` | Yes | Non-empty | Storage class name on the primary cluster. |
| `storageClassMappings[].drStorageClass` | `string` | Yes | Non-empty | Storage class name on the DR cluster. |
| `replicationEndpoints` | `[]ReplicationEndpoint` | Yes | Non-empty list | IP addresses or hostnames of worker nodes participating in DRBD replication. Must be reachable over TCP ports 7000–7999. |
| `replicationEndpoints[].address` | `string` | Yes | Valid IP or hostname | Replication endpoint address. |
| `replicationEndpoints[].port` | `integer` | No | 7000–7999 | DRBD replication port. Defaults to `7789`. |
| `autoRepair` | `boolean` | No | — | When `true`, the DRBD Operator attempts automatic resync after a split-brain is detected. Defaults to `false`; manual intervention is recommended for split-brain resolution. |

**Status fields**

| Field | Type | Description |
|---|---|---|
| `phase` | `string` | Policy status: `Active`, `Degraded`, or `Invalid`. |
| `conditions` | `[]Condition` | Condition types include `EndpointsReachable`, `StorageClassesValid`. |
| `message` | `string` | Human-readable status or validation error. |

---

### DRBDVolume

A DRBDVolume represents a single DRBD-replicated PVC. It is created automatically by the protection-controller when a ProtectionRequest is processed. You inspect DRBDVolume resources to verify per-volume replication status; you do not typically create them by hand.

**Spec fields**

| Field | Type | Required | Validation | Description |
|---|---|---|---|---|
| `sourcePVCRef` | `string` | Yes | Name of an existing PVC | The original PVC on the primary cluster that is being replicated. |
| `sourceNamespace` | `string` | Yes | Valid DNS label | Namespace of the source PVC. |
| `replicationPolicyRef` | `string` | Yes | Name of a DRBDReplicationPolicy | The policy governing this volume's replication behavior. |

**Status fields**

| Field | Type | Description |
|---|---|---|
| `phase` | `string` | Volume replication state: `Provisioning`, `Syncing`, `UpToDate`, `Degraded`, or `Failed`. |
| `syncProgress` | `integer` | Percentage of data synchronized to the DR cluster (0–100). |
| `frontendPVCRef` | `string` | Name of the DRBD-backed frontend PVC that replaced the VM's original PVC on the primary cluster. |
| `drPVCRef` | `string` | Name of the standby PVC on the DR cluster that receives replicated data. |
| `lastSyncTime` | `string` | RFC 3339 timestamp of the most recent successful sync. |
| `conditions` | `[]Condition` | Condition types include `ReplicationEstablished`, `UpToDate`. |

---

### RPOEvent

An RPOEvent records a replication lag violation — an instance where the observed lag between primary and DR exceeded the acceptable threshold. The replication-monitor agent creates these resources; you read them for auditing and alerting.

**Spec fields**

RPOEvent resources are created exclusively by the replication-monitor. The spec fields are set at creation time and are immutable.

| Field | Type | Description |
|---|---|---|
| `protectionGroupRef` | `string` | Name of the Protection Group affected by the lag violation. |
| `observedLagSeconds` | `integer` | The replication lag observed at the time of the event, in seconds. |
| `threshold` | `integer` | The lag threshold (in seconds) that was exceeded, as configured in the replication-monitor. |
| `severity` | `string` | Event severity: `Warning` or `Critical`. |
| `timestamp` | `string` | RFC 3339 timestamp when the violation was detected. |
| `affectedVolumes` | `[]string` | Names of DRBDVolume resources that were lagging at the time of the event. |

**Status fields**

RPOEvent has no controller-managed status subresource. The full record is contained in the spec.

---

### ReplicationGroupStatus

A ReplicationGroupStatus provides an aggregated replication health summary for a Protection Group. The replication-monitor agent creates and updates these resources so that operators and the failover-controller can assess group-level readiness without inspecting individual DRBDVolume resources.

**Spec fields**

ReplicationGroupStatus resources are maintained by the replication-monitor. The fields below are written by the agent and are not user-editable.

| Field | Type | Description |
|---|---|---|
| `protectionGroupRef` | `string` | Name of the Protection Group this status describes. |
| `overallHealth` | `string` | Aggregated health: `Healthy`, `Degraded`, or `Critical`. |
| `perVolumeStatus` | `[]VolumeStatus` | Per-volume sync state. Each entry includes `volumeName`, `phase`, and `syncProgress`. |
| `lastSuccessfulSyncTime` | `string` | RFC 3339 timestamp of the most recent point at which all volumes in the group were fully synchronized. |
| `message` | `string` | Human-readable summary of the current health state. |

## Usage

The following patterns represent the most common ways you will interact with Site Recovery CRDs during day-to-day operations.

**Protecting a VM with a ProtectionRequest (DRBD Operator model)**

Create a ProtectionRequest to request DRBD-backed protection for a single VM. The protection-controller picks up the resource and drives it through validation, volume creation, sync, and frontend PVC switchover.

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-web01
  namespace: dr-production
spec:
  vmName: web01
  vmNamespace: production
  replicationPolicyRef: policy-production
  protectionGroupRef: pg-production-web
```

Watch the status phase progress:

```bash
kubectl get protectionrequest protect-vm-web01 -n dr-production -w
```

**Grouping VMs into a Protection Group**

Declare a ProtectionGroup to ensure that a set of VMs always fails over together. The protection-group-controller and pg-sync-controller keep this resource reconciled on both clusters.

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionGroup
metadata:
  name: pg-production-web
  namespace: dr-production
spec:
  vmNames:
    - web01
    - web02
    - db01
  namespace: production
  replicationPolicyRef: policy-production
  consistencyMode: Strict
```

**Inspecting replication health**

Read the ReplicationGroupStatus for a group to get an aggregated view without querying each DRBDVolume individually:

```bash
kubectl get replicationgroupstatus -n dr-production -o yaml
```

Check per-volume sync progress by listing DRBDVolume resources:

```bash
kubectl get drbdvolume -n dr-production
```

**Triggering a planned failover**

Create a FailoverRequest with `type: Planned`. The failover-controller gracefully stops VMs on the primary cluster before promoting volumes on the DR cluster.

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production-planned
  namespace: dr-production
spec:
  protectionGroupRef: pg-production-web
  type: Planned
```

Alternatively, use `pgctl` to trigger the same operation:

```bash
pgctl failover --group pg-production-web --type planned --deployment production
```

**Triggering an unplanned failover**

When the primary cluster is unavailable, set `type: Unplanned` and optionally `force: true` to bypass primary reachability checks:

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production-emergency
  namespace: dr-production
spec:
  protectionGroupRef: pg-production-web
  type: Unplanned
  force: true
```

**Running a non-disruptive test failover**

Create a TestFailover resource to validate DR readiness without affecting production. This requires VolumeSnapshot support on both clusters and is only available for DRBD Operator deployments.

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: TestFailover
metadata:
  name: test-production-web
  namespace: dr-production
spec:
  protectionGroupRef: pg-production-web
  autoCleanup: true
  verificationTimeoutSeconds: 300
```

**Auditing RPO violations**

List all RPOEvent resources to review recent replication lag violations:

```bash
kubectl get rpoevent -n dr-production --sort-by=.spec.timestamp
```

## Examples

### Example 1 — Define a DRBDReplicationPolicy for synchronous replication

This policy configures Protocol C (synchronous) replication between two clusters, mapping their respective storage classes and declaring the replication endpoints.

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: DRBDReplicationPolicy
metadata:
  name: policy-production
  namespace: dr-production
spec:
  protocol: C
  storageClassMappings:
    - primaryStorageClass: fast-ssd
      drStorageClass: fast-ssd-dr
  replicationEndpoints:
    - address: 10.0.1.10
      port: 7789
    - address: 10.0.1.11
      port: 7789
  autoRepair: false
```

Verify the policy is active:

```bash
kubectl get drbdreplicationpolicy policy-production -n dr-production -o jsonpath='{.status.phase}'
```

Expected output:

```
Active
```

---

### Example 2 — Create a ProtectionRequest and track its lifecycle

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-vm-db01
  namespace: dr-production
spec:
  vmName: db01
  vmNamespace: production
  replicationPolicyRef: policy-production
```

Apply and watch the phase transitions:

```bash
kubectl apply -f protect-vm-db01.yaml
kubectl get protectionrequest protect-vm-db01 -n dr-production -w
```

Expected phase sequence:

```
NAME               PHASE
protect-vm-db01    Validating
protect-vm-db01    CreatingVolumes
protect-vm-db01    WaitingForSync
protect-vm-db01    SwitchingFrontend
protect-vm-db01    Protected
```

Inspect the resulting DRBDVolume references:

```bash
kubectl get protectionrequest protect-vm-db01 -n dr-production \
  -o jsonpath='{.status.drbdVolumeRefs}'
```

---

### Example 3 — ProtectionGroup with Strict consistency and multiple VMs

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: ProtectionGroup
metadata:
  name: pg-production-web
  namespace: dr-production
spec:
  vmNames:
    - web01
    - web02
    - db01
  namespace: production
  replicationPolicyRef: policy-production
  consistencyMode: Strict
```

Check the aggregated group phase:

```bash
kubectl get protectiongroup pg-production-web -n dr-production \
  -o jsonpath='{.status.phase}'
```

Expected output when all VMs are fully synced:

```
Consistent
```

---

### Example 4 — Planned failover and status monitoring

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-planned-20240115
  namespace: dr-production
spec:
  protectionGroupRef: pg-production-web
  type: Planned
```

```bash
kubectl apply -f failover-planned-20240115.yaml
kubectl get failoverrequest failover-planned-20240115 -n dr-production -w
```

Expected phase sequence:

```
NAME                          PHASE
failover-planned-20240115     Pending
failover-planned-20240115     ShuttingDownVMs
failover-planned-20240115     PromotingVolumes
failover-planned-20240115     StartingVMs
failover-planned-20240115     Completed
```

---

### Example 5 — Non-disruptive test failover (DRBD Operator model only)

```yaml
apiVersion: site-recovery.example.com/v1alpha1
kind: TestFailover
metadata:
  name: test-web-20240115
  namespace: dr-production
spec:
  protectionGroupRef: pg-production-web
  testNamespace: pg-production-web-test
  autoCleanup: true
  verificationTimeoutSeconds: 300
```

```bash
kubectl apply -f test-web-20240115.yaml
kubectl get testfailover test-web-20240115 -n dr-production -w
```

Expected output on success:

```
NAME                 PHASE
test-web-20240115    CreatingSnapshots
test-web-20240115    ProvisioningTestVMs
test-web-20240115    Verifying
test-web-20240115    CleaningUp
test-web-20240115    Succeeded
```

---

### Example 6 — Inspect RPO violations

```bash
kubectl get rpoevent -n dr-production -o custom-columns=\
NAME:.metadata.name,\
GROUP:.spec.protectionGroupRef,\
LAG:.spec.observedLagSeconds,\
SEVERITY:.spec.severity,\
TIME:.spec.timestamp
```

Example output:

```
NAME                GROUP                  LAG   SEVERITY   TIME
rpo-evt-20240115a   pg-production-web      47    Warning    2024-01-15T02:13:00Z
rpo-evt-20240115b   pg-production-web      91    Critical   2024-01-15T02:47:00Z
```

## Troubleshooting

Use the following format for each issue: **Symptom → Likely cause → Fix.**

---

**ProtectionRequest stuck in `Validating` phase**

*Symptom:* A ProtectionRequest remains in `Validating` for more than a few minutes. The `message` field may say the VM cannot be found or is not in a running state.

*Likely cause:* The VM named in `spec.vmName` does not exist in `spec.vmNamespace` on the primary cluster, or the VM is in a stopped or error state.

*Fix:* Confirm the VM exists and is running:
```bash
kubectl get vm <vmName> -n <vmNamespace> --kubeconfig /path/to/primary-kubeconfig.yaml
```
Correct the `vmName` or `vmNamespace` in the ProtectionRequest, or start the VM before re-applying.

---

**ProtectionRequest stuck in `WaitingForSync` phase**

*Symptom:* The ProtectionRequest has been in `WaitingForSync` for an extended period. The corresponding DRBDVolume shows a `syncProgress` value that is not advancing.

*Likely cause:* Network connectivity between primary and DR worker nodes on DRBD replication ports (TCP 7000–7999) is blocked or degraded, or the replication endpoint addresses in the DRBDReplicationPolicy are incorrect.

*Fix:* Verify network connectivity:
```bash
# From a primary worker node
nc -zv <dr-worker-ip> 7789
```
Confirm the `replicationEndpoints` in the DRBDReplicationPolicy match the actual DR worker node addresses. Check DRBD kernel module status on both sides with `dmesg | grep drbd`.

---

**ProtectionGroup phase is `Degraded`**

*Symptom:* `kubectl get protectiongroup` shows `Degraded`. The Site Manager UI shows a red health indicator for the group. RPOEvents may be present.

*Likely cause:* One or more DRBDVolume resources in the group are not `UpToDate`. This can result from a transient network issue, a node restart, or a replication lag threshold violation detected by the replication-monitor.

*Fix:* Identify which volumes are degraded:
```bash
kubectl get drbdvolume -n dr-production
```
Check the ReplicationGroupStatus for the group:
```bash
kubectl get replicationgroupstatus -n dr-production -o yaml
```
If a DRBDVolume is in `Degraded` state, inspect its events and the replication-monitor logs for the root cause. If sync is stalled, refer to the node and replication diagnostics guide.

---

**FailoverRequest stuck in `ShuttingDownVMs` phase (planned failover)**

*Symptom:* A planned FailoverRequest does not progress past `ShuttingDownVMs`. VMs on the primary cluster appear to still be running.

*Likely cause:* One or more VMs in the Protection Group are not responding to graceful shutdown requests, or the failover-controller cannot reach the primary cluster API server.

*Fix:* Verify primary cluster API connectivity from the quorum cluster:
```bash
kubectl get nodes --kubeconfig /path/to/primary-kubeconfig.yaml
```
Check failover-controller logs on the quorum cluster:
```bash
kubectl logs -n dr-production -l app=failover-controller --tail=100
```
If VMs are unresponsive to shutdown, consider deleting the FailoverRequest and re-creating it with `type: Unplanned` if the situation warrants emergency failover.

---

**TestFailover fails in `ProvisioningTestVMs` phase**

*Symptom:* A TestFailover resource enters `Failed` phase during `ProvisioningTestVMs`. The `message` field references snapshot or PVC provisioning errors.

*Likely cause:* VolumeSnapshot support (CSI snapshotter) is not installed or the specified `snapshotClass` does not exist on the DR cluster. TestFailover is only supported for DRBD Operator deployment models.

*Fix:* Confirm VolumeSnapshot CRDs and the CSI snapshotter are present on the DR cluster:
```bash
kubectl get crd volumesnapshots.snapshot.storage.k8s.io \
  --kubeconfig /path/to/dr-kubeconfig.yaml
kubectl get volumesnapshotclass \
  --kubeconfig /path/to/dr-kubeconfig.yaml
```
If the `snapshotClass` field is set in the TestFailover spec, ensure the named class exists. Remove the field to use the cluster default.

---

**DRBDReplicationPolicy shows phase `Invalid`**

*Symptom:* A DRBDReplicationPolicy resource reports `phase: Invalid` in its status. The `message` field describes a validation failure.

*Likely cause:* A referenced storage class name does not exist on one of the clusters, or a replication endpoint is not reachable on the configured port.

*Fix:* Check the status message for specifics:
```bash
kubectl get drbdreplicationpolicy policy-production -n dr-production \
  -o jsonpath='{.status.message}'
```
Verify storage classes exist on both clusters:
```bash
kubectl get sc --kubeconfig /path/to/primary-kubeconfig.yaml
kubectl get sc --kubeconfig /path/to/dr-kubeconfig.yaml
```
Correct the `storageClassMappings` or `replicationEndpoints` and re-apply the policy.

---

**pgctl validate reports schema errors**

*Symptom:* Running `pgctl validate -f <manifest>` outputs field validation errors before you apply a resource.

*Likely cause:* A required field is missing, an enum value is incorrect (for example, `protocol: B` instead of `A` or `C`), or a cross-reference (such as `replicationPolicyRef`) names a resource that does not yet exist.

*Fix:* Review the field tables in the Configuration section above. Use `kubectl explain <kind>.spec.<field>` for inline schema documentation. Ensure referenced resources (ProtectionGroup, DRBDReplicationPolicy) exist in the target namespace before applying dependent resources.