Reconciliation Issues

This page helps you diagnose and resolve situations where Site Recovery custom resources become stuck in pending or degraded states, reconciliation loops report errors, or status conditions go stale. These conditions typically surface in ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, DRBDVolume, and ReplicationGroupStatus resources managed by controllers running on the quorum cluster and on your primary and DR clusters. Understanding the reconciliation model — where controllers continuously compare desired state declared in CRDs against actual cluster state and drive the system toward convergence — is essential for effective troubleshooting.

Before working through reconciliation issues, ensure you have:

• kubectl configured with access to the quorum cluster, primary cluster, and DR cluster
• Administrative kubeconfig files for all three clusters
• pgctl installed and a valid deployment context configured
• Familiarity with the controller architecture: failover-controller and protection-controller run on the quorum cluster; protection-group-controller and test-failover-controller run on the primary and DR clusters; replication-monitor runs on both data clusters
• Sufficient RBAC permissions to read and patch custom resources across all namespaces involved (typically dr-<deployment-name>)
• Kubernetes ≥ 1.28 on all clusters
• DRBD kernel module ≥ 9.0 loaded on worker nodes if diagnosing replication-layer issues

No additional software installation is required to diagnose reconciliation issues. All diagnostic commands use kubectl and pgctl, which should already be present from your initial deployment.

Step 1: Verify controller pods are running on the quorum cluster

kubectl get pods -n dr-<deployment-name> \
  -l app.kubernetes.io/part-of=site-recovery

Expected output shows failover-controller, protection-controller, and pg-sync-controller pods in Running state.

Step 2: Verify controller pods are running on primary and DR clusters

# Primary cluster
kubectl --kubeconfig ~/.kube/config-primary \
  get pods -n dr-<deployment-name> \
  -l app.kubernetes.io/part-of=site-recovery

# DR cluster
kubectl --kubeconfig ~/.kube/config-dr \
  get pods -n dr-<deployment-name> \
  -l app.kubernetes.io/part-of=site-recovery

Expected output shows protection-group-controller and test-failover-controller pods in Running state on each cluster.

Step 3: Confirm the replication-monitor agent is running

The replication-monitor is deployed automatically by the Ansible playbooks. Verify it is present:

kubectl --kubeconfig ~/.kube/config-primary \
  get pods -n dr-<deployment-name> -l app=replication-monitor

Step 4: Confirm CRD registration

kubectl get crds | grep siterecovery

You should see entries for protectiongroups, protectionrequests, failoverrequests, testfailovers, drbdvolumes, drbdreplicationpolicies, rpoevents, and replicationgroupstatuses.

Reconciliation behavior is governed by controller settings embedded in the deployment manifests. The following parameters influence how quickly issues surface and how aggressively controllers retry.

Validation rules enforced at reconciliation time (DRBD Operator model)

The protection-group-controller validates the following conditions whenever a ProtectionGroup is created or updated. Failure in any critical check sets status.state to Failed and leaves the resource unchanged:

ReplicationGroupStatus health values

• Healthy — all volumes in sync, no lag violations
• Degraded — one or more volumes lagging or partially synced
• Critical — replication halted or data loss risk detected

Use the following workflow to systematically locate and resolve a stuck or degraded resource. Work from the highest-level resource down to the underlying controller logs.

Check high-level resource state

Start with the resource that was reported as stuck:

# ProtectionGroup
kubectl --kubeconfig ~/.kube/config-primary \
  get protectiongroup <pg-name> -n <namespace> -o yaml

# ProtectionRequest
kubectl --kubeconfig ~/.kube/quorum \
  get protectionrequest <pr-name> -n dr-<deployment-name> -o yaml

# FailoverRequest
kubectl --kubeconfig ~/.kube/quorum \
  get failoverrequest <fr-name> -n dr-<deployment-name> -o yaml

Pay attention to:

• status.state or status.phase — the current position in the lifecycle
• status.conditions — structured conditions with type, status, reason, and message fields
• status.warnings — non-blocking issues surfaced by validation

Stream controller logs

Once you know which controller owns the resource, stream its logs:

# failover-controller (quorum cluster)
kubectl --kubeconfig ~/.kube/quorum \
  logs -n dr-<deployment-name> \
  -l app=failover-controller -f

# protection-controller (quorum cluster)
kubectl --kubeconfig ~/.kube/quorum \
  logs -n dr-<deployment-name> \
  -l app=protection-controller -f

# protection-group-controller (primary or DR cluster)
kubectl --kubeconfig ~/.kube/config-primary \
  logs -n dr-<deployment-name> \
  -l app=protection-group-controller -f

# replication-monitor (primary or DR cluster)
kubectl --kubeconfig ~/.kube/config-primary \
  logs -n dr-<deployment-name> \
  -l app=replication-monitor -f

Check replication health

# Aggregated group health
kubectl --kubeconfig ~/.kube/config-primary \
  get replicationgroupstatus -n <namespace>

# Per-volume detail
kubectl --kubeconfig ~/.kube/config-primary \
  get drbdvolume -n <namespace>

# Recent RPO violations
kubectl --kubeconfig ~/.kube/config-primary \
  get rpoevent -n <namespace> --sort-by=.metadata.creationTimestamp

Use pgctl for deployment-aware diagnostics

# Validate the deployment context
pgctl validate --deployment <deployment-name>

# Show ProtectionGroup status across clusters
pgctl pg status <pg-name> --deployment <deployment-name>

Use the quorum-deployments.sh script for guided health checks

./quorum-deployments.sh
# Select: Run health checks

Example 1: ProtectionGroup stuck in Pending state after creation

Inspect the resource to find the blocking condition:

kubectl --kubeconfig ~/.kube/config-primary \
  get protectiongroup production-pg -n default -o yaml

Expected output when validation failed:

status:
  state: Failed
  conditions:
    - type: ValidationFailed
      status: "True"
      reason: InsufficientReplication
      message: >-
        PVC prod-vm-2-disk for VM prod-vm-2: storage class local-storage
        does not meet minimum replica requirements for DR
  warnings: []

The protection-group-controller rejected the update because the PVC's storage class does not satisfy replication requirements. Correct the VM's PVC to use a storage class with adequate replica placement, then re-apply the ProtectionGroup spec.

Example 2: FailoverRequest stuck in StoppingOnSource phase

Check the FailoverRequest status on the quorum cluster:

kubectl --kubeconfig ~/.kube/quorum \
  get failoverrequest my-failover -n dr-production -o yaml

status:
  phase: StoppingOnSource
  sourceCluster: cluster1
  targetCluster: cluster2
  conditions:
    - type: VmsStopTimeout
      status: "True"
      message: "ProtectionGroup production-pg on cluster1 has not reached currentState=stopped after 120s"

Check the ProtectionGroup state on the primary cluster:

kubectl --kubeconfig ~/.kube/config-primary \
  get protectiongroup production-pg -n default \
  -o jsonpath='{.status.currentState}'

If it returns mixed, the protection-group-controller is still stopping individual VMs. Check for VMs that are not responding to stop signals:

kubectl --kubeconfig ~/.kube/config-primary \
  get vm -n default -l protectiongroup=production-pg

Example 3: FailoverRequest blocked by a stale failover lock

If a previous failover process crashed, the Kubernetes Lease may still be held. Inspect it:

kubectl --kubeconfig ~/.kube/quorum \
  get lease failover-lock-production-pg \
  -n dr-production -o yaml

spec:
  holderIdentity: failover-controller-48291
  leaseDurationSeconds: 300
  acquireTime: "2024-01-15T09:12:00Z"
  renewTime: "2024-01-15T09:12:00Z"

If renewTime is not advancing and the lease duration has elapsed, the lock expired automatically. If it has not yet elapsed and you are certain no failover is in progress, delete the lease manually to unblock:

kubectl --kubeconfig ~/.kube/quorum \
  delete lease failover-lock-production-pg \
  -n dr-production

Only delete a lease when you have confirmed no failover operation is actively running.

Example 4: DRBDVolume in Degraded state with RPOEvent

List recent RPO violations:

kubectl --kubeconfig ~/.kube/config-primary \
  get rpoevent -n default \
  -o custom-columns=\
NAME:.metadata.name,\
GROUP:.spec.protectionGroup,\
LAG:.spec.observedLag,\
SEVERITY:.spec.severity,\
TIME:.metadata.creationTimestamp

Expected output:

NAME                          GROUP          LAG    SEVERITY   TIME
rpoevent-production-pg-8f2a   production-pg  42s    Warning    2024-01-15T10:05:00Z

Inspect the aggregated group status:

kubectl --kubeconfig ~/.kube/config-primary \
  get replicationgroupstatus production-pg-status \
  -n default -o yaml

status:
  health: Degraded
  lastSuccessfulSync: "2024-01-15T09:58:00Z"
  volumes:
    - pvcName: prod-vm-1-disk
      syncState: Syncing
      syncPercent: 87
    - pvcName: prod-vm-2-disk
      syncState: Consistent
      syncPercent: 100

A Syncing volume is catching up after a temporary network interruption. Monitor until it reaches Consistent. If it does not progress, check DRBD kernel module status on the affected worker node.

Use the following reference for the most common reconciliation failures. Each entry follows the pattern: Symptom → Likely cause → Fix.

ProtectionGroup remains in Failed state after creation or update

Symptom: status.state is Failed; status.conditions contains a ValidationFailed condition with a message referencing a PVC or storage class.

Likely cause: The protection-group-controller rejected the create or update because one or more VMs have PVCs that do not meet replication requirements (insufficient replica count, unsupported storage class, or missing PVC).

Fix:

1. Read the message field in the ValidationFailed condition to identify the specific VM and PVC.
2. Correct the PVC or VM definition on the primary cluster.
3. Re-apply the ProtectionGroup spec. The controller re-validates on every update.

ProtectionRequest stuck in an intermediate lifecycle phase

Symptom: A ProtectionRequest on the quorum cluster remains in a phase such as CreatingDRBDVolume or WaitingForSync for longer than expected.

Likely cause: The protection-controller cannot create the DRBDVolume, or the DRBDVolume was created but replication has not yet completed initial sync.

Fix:

1. Check the DRBDVolume resource referenced in the ProtectionRequest status:
   Copy

   kubectl --kubeconfig ~/.kube/quorum \
     get drbdvolume -n dr-<deployment-name>
   

2. If the DRBDVolume is missing, check protection-controller logs for API errors.
3. If the DRBDVolume exists but syncPercent is not advancing, verify DRBD port reachability (TCP 7000–7999) between primary and DR worker nodes.
4. If the DRBDVolume shows an error condition, check the DRBD Operator logs on the affected cluster.

FailoverRequest does not advance past StoppingOnSource

Symptom: The FailoverRequest phase is StoppingOnSource for an extended period. The ProtectionGroup currentState on the source cluster remains mixed or running.

Likely cause: One or more VMs in the ProtectionGroup are not shutting down cleanly. The protection-group-controller waits for all VMs to stop before the failover-controller advances.

Fix:

1. Identify VMs that are still running:
   Copy

   kubectl --kubeconfig ~/.kube/config-primary \
     get vm -n <namespace>
   

2. Check KubeVirt events for the stuck VM:
   Copy

   kubectl --kubeconfig ~/.kube/config-primary \
     describe vmi <vm-name> -n <namespace>
   

3. For a planned failover, resolve the underlying VM issue and allow the controller to retry. For an unplanned failover where the primary cluster is unreachable, create the FailoverRequest with spec.type: Unplanned so the failover-controller does not wait for a graceful shutdown.

FailoverRequest blocked by a concurrent failover lock

Symptom: The failover-controller log contains Failed to acquire failover lock and names a holder identity from a previous operation.

Likely cause: A previous failover process crashed before releasing the Kubernetes Lease failover-lock-<pg-name>. The lock has a 300-second TTL and will expire automatically.

Fix:

1. Wait for the 300-second lease duration to expire, after which the controller will acquire the lock on its next reconciliation cycle.
2. If the lock has already expired (check renewTime is stale) but the FailoverRequest is still blocked, delete the lease manually:
   Copy

   kubectl --kubeconfig ~/.kube/quorum \
     delete lease failover-lock-<pg-name> \
     -n dr-<deployment-name>
   

3. Do not delete the lease if another failover operation is actively running.

ReplicationGroupStatus shows Degraded or Critical

Symptom: kubectl get replicationgroupstatus returns health: Degraded or health: Critical. Recent RPOEvents exist for the affected Protection Group.

Likely cause: DRBD replication lag has exceeded the configured threshold. Common causes include network saturation, high I/O on the primary, or a temporary network partition.

Fix:

1. Check which volumes are lagging:
   Copy

   kubectl --kubeconfig ~/.kube/config-primary \
     get drbdvolume -n <namespace>
   

2. For Syncing volumes: monitor progress. Transient lag self-resolves once the network recovers.
3. For volumes that are not making sync progress: verify TCP 7000–7999 is open between primary and DR worker nodes and that no firewall rule change has occurred.
4. For Critical health: check whether a split-brain condition exists. DRBD split-brain requires manual resolution — do not attempt automated failover until the split-brain is resolved.

Status conditions are stale and not updating

Symptom: A resource's status.conditions shows a timestamp that is minutes or hours old and does not reflect the current state of the system.

Likely cause: The responsible controller pod has crashed, restarted, or lost connectivity to the cluster it is reconciling against.

Fix:

1. Check controller pod health:
   Copy

   kubectl --kubeconfig ~/.kube/quorum \
     get pods -n dr-<deployment-name>
   

2. If a controller is in CrashLoopBackOff or Error, inspect its logs from the previous execution:
   Copy

   kubectl --kubeconfig ~/.kube/quorum \
     logs -n dr-<deployment-name> \
     -l app=<controller-name> --previous
   

3. If the controller is running but stale: verify the controller has network access to the target cluster's Kubernetes API (TCP 6443). The quorum cluster must reach both primary and DR cluster APIs.
4. If kubeconfig credentials have rotated, update the secret in the dr-<deployment-name> namespace and restart the affected controller pod.

pg-sync-controller fails to keep Protection Group metadata synchronized

Symptom: VMs on the DR cluster have outdated specs compared to the primary, or the pg-sync-controller log shows repeated sync errors.

Likely cause: The pg-sync-controller on the quorum cluster cannot reach the DR cluster API, or there is a namespace or RBAC mismatch.

Fix:

1. Check pg-sync-controller logs:
   Copy

   kubectl --kubeconfig ~/.kube/quorum \
     logs -n dr-<deployment-name> \
     -l app=pg-sync-controller -f
   

2. Confirm the DR cluster kubeconfig secret is valid and the API endpoint is reachable from the quorum cluster.
3. Verify the dr-<deployment-name> namespace exists on the DR cluster and that RBAC is correctly applied.