--- title: Reconciliation Issues product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/troubleshooting-reconciliation --- # Reconciliation Issues _Resources stuck in pending or degraded state, reconciliation loop errors, and stale status conditions_ ## Overview This page helps you diagnose and resolve situations where Site Recovery custom resources become stuck in pending or degraded states, reconciliation loops report errors, or status conditions go stale. These conditions typically surface in ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, DRBDVolume, and ReplicationGroupStatus resources managed by controllers running on the quorum cluster and on your primary and DR clusters. Understanding the reconciliation model — where controllers continuously compare desired state declared in CRDs against actual cluster state and drive the system toward convergence — is essential for effective troubleshooting. ## Prerequisites Before working through reconciliation issues, ensure you have: - `kubectl` configured with access to the quorum cluster, primary cluster, and DR cluster - Administrative kubeconfig files for all three clusters - `pgctl` installed and a valid deployment context configured - Familiarity with the controller architecture: **failover-controller** and **protection-controller** run on the quorum cluster; **protection-group-controller** and **test-failover-controller** run on the primary and DR clusters; **replication-monitor** runs on both data clusters - Sufficient RBAC permissions to read and patch custom resources across all namespaces involved (typically `dr-`) - Kubernetes ≥ 1.28 on all clusters - DRBD kernel module ≥ 9.0 loaded on worker nodes if diagnosing replication-layer issues ## Installation No additional software installation is required to diagnose reconciliation issues. All diagnostic commands use `kubectl` and `pgctl`, which should already be present from your initial deployment. **Step 1: Verify controller pods are running on the quorum cluster** ```bash kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery ``` Expected output shows `failover-controller`, `protection-controller`, and `pg-sync-controller` pods in `Running` state. **Step 2: Verify controller pods are running on primary and DR clusters** ```bash # Primary cluster kubectl --kubeconfig ~/.kube/config-primary \ get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery # DR cluster kubectl --kubeconfig ~/.kube/config-dr \ get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery ``` Expected output shows `protection-group-controller` and `test-failover-controller` pods in `Running` state on each cluster. **Step 3: Confirm the replication-monitor agent is running** The `replication-monitor` is deployed automatically by the Ansible playbooks. Verify it is present: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get pods -n dr- -l app=replication-monitor ``` **Step 4: Confirm CRD registration** ```bash kubectl get crds | grep siterecovery ``` You should see entries for `protectiongroups`, `protectionrequests`, `failoverrequests`, `testfailovers`, `drbdvolumes`, `drbdreplicationpolicies`, `rpoevents`, and `replicationgroupstatuses`. ## Configuration Reconciliation behavior is governed by controller settings embedded in the deployment manifests. The following parameters influence how quickly issues surface and how aggressively controllers retry. | Parameter | Controller | Default | Effect | |-----------|-----------|---------|--------| | Reconciliation interval | failover-controller (timer) | 10 seconds | How often the failover controller re-evaluates a FailoverRequest and advances its state machine | | ProtectionGroup re-validation on update | protection-group-controller | Always | Every spec change triggers full VM and PVC re-validation; failed validation sets state to `Failed` and blocks the update | | RPO lag threshold | replication-monitor | Deployment-specific | Exceeding this threshold causes the monitor to write an RPOEvent and set ReplicationGroupStatus health to `Degraded` or `Critical` | | Failover lock duration | failover-controller | 300 seconds | A Kubernetes Lease named `failover-lock-` expires after this period, releasing a stuck lock from a crashed process | **Validation rules enforced at reconciliation time (DRBD Operator model)** The protection-group-controller validates the following conditions whenever a ProtectionGroup is created or updated. Failure in any critical check sets `status.state` to `Failed` and leaves the resource unchanged: | Check | Severity | Behavior on failure | |-------|----------|--------------------| | VM resource exists in namespace | Critical | Rejects create or update | | PVC resource exists and is bound | Critical | Rejects create or update | | PVC uses a supported storage class | Critical | Rejects create or update | | Storage class minimum replica count met | Critical | Rejects create or update | | Cross-site placement rules present | Warning | Allows create or update, surfaces warning in `status.warnings` | **ReplicationGroupStatus health values** - `Healthy` — all volumes in sync, no lag violations - `Degraded` — one or more volumes lagging or partially synced - `Critical` — replication halted or data loss risk detected ## Usage Use the following workflow to systematically locate and resolve a stuck or degraded resource. Work from the highest-level resource down to the underlying controller logs. **Check high-level resource state** Start with the resource that was reported as stuck: ```bash # ProtectionGroup kubectl --kubeconfig ~/.kube/config-primary \ get protectiongroup -n -o yaml # ProtectionRequest kubectl --kubeconfig ~/.kube/quorum \ get protectionrequest -n dr- -o yaml # FailoverRequest kubectl --kubeconfig ~/.kube/quorum \ get failoverrequest -n dr- -o yaml ``` Pay attention to: - `status.state` or `status.phase` — the current position in the lifecycle - `status.conditions` — structured conditions with `type`, `status`, `reason`, and `message` fields - `status.warnings` — non-blocking issues surfaced by validation **Stream controller logs** Once you know which controller owns the resource, stream its logs: ```bash # failover-controller (quorum cluster) kubectl --kubeconfig ~/.kube/quorum \ logs -n dr- \ -l app=failover-controller -f # protection-controller (quorum cluster) kubectl --kubeconfig ~/.kube/quorum \ logs -n dr- \ -l app=protection-controller -f # protection-group-controller (primary or DR cluster) kubectl --kubeconfig ~/.kube/config-primary \ logs -n dr- \ -l app=protection-group-controller -f # replication-monitor (primary or DR cluster) kubectl --kubeconfig ~/.kube/config-primary \ logs -n dr- \ -l app=replication-monitor -f ``` **Check replication health** ```bash # Aggregated group health kubectl --kubeconfig ~/.kube/config-primary \ get replicationgroupstatus -n # Per-volume detail kubectl --kubeconfig ~/.kube/config-primary \ get drbdvolume -n # Recent RPO violations kubectl --kubeconfig ~/.kube/config-primary \ get rpoevent -n --sort-by=.metadata.creationTimestamp ``` **Use pgctl for deployment-aware diagnostics** ```bash # Validate the deployment context pgctl validate --deployment # Show ProtectionGroup status across clusters pgctl pg status --deployment ``` **Use the quorum-deployments.sh script for guided health checks** ```bash ./quorum-deployments.sh # Select: Run health checks ``` ## Examples **Example 1: ProtectionGroup stuck in `Pending` state after creation** Inspect the resource to find the blocking condition: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get protectiongroup production-pg -n default -o yaml ``` Expected output when validation failed: ```yaml status: state: Failed conditions: - type: ValidationFailed status: "True" reason: InsufficientReplication message: >- PVC prod-vm-2-disk for VM prod-vm-2: storage class local-storage does not meet minimum replica requirements for DR warnings: [] ``` The protection-group-controller rejected the update because the PVC's storage class does not satisfy replication requirements. Correct the VM's PVC to use a storage class with adequate replica placement, then re-apply the ProtectionGroup spec. --- **Example 2: FailoverRequest stuck in `StoppingOnSource` phase** Check the FailoverRequest status on the quorum cluster: ```bash kubectl --kubeconfig ~/.kube/quorum \ get failoverrequest my-failover -n dr-production -o yaml ``` ```yaml status: phase: StoppingOnSource sourceCluster: cluster1 targetCluster: cluster2 conditions: - type: VmsStopTimeout status: "True" message: "ProtectionGroup production-pg on cluster1 has not reached currentState=stopped after 120s" ``` Check the ProtectionGroup state on the primary cluster: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get protectiongroup production-pg -n default \ -o jsonpath='{.status.currentState}' ``` If it returns `mixed`, the protection-group-controller is still stopping individual VMs. Check for VMs that are not responding to stop signals: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get vm -n default -l protectiongroup=production-pg ``` --- **Example 3: FailoverRequest blocked by a stale failover lock** If a previous failover process crashed, the Kubernetes Lease may still be held. Inspect it: ```bash kubectl --kubeconfig ~/.kube/quorum \ get lease failover-lock-production-pg \ -n dr-production -o yaml ``` ```yaml spec: holderIdentity: failover-controller-48291 leaseDurationSeconds: 300 acquireTime: "2024-01-15T09:12:00Z" renewTime: "2024-01-15T09:12:00Z" ``` If `renewTime` is not advancing and the lease duration has elapsed, the lock expired automatically. If it has not yet elapsed and you are certain no failover is in progress, delete the lease manually to unblock: ```bash kubectl --kubeconfig ~/.kube/quorum \ delete lease failover-lock-production-pg \ -n dr-production ``` **Only delete a lease when you have confirmed no failover operation is actively running.** --- **Example 4: DRBDVolume in `Degraded` state with RPOEvent** List recent RPO violations: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get rpoevent -n default \ -o custom-columns=\ NAME:.metadata.name,\ GROUP:.spec.protectionGroup,\ LAG:.spec.observedLag,\ SEVERITY:.spec.severity,\ TIME:.metadata.creationTimestamp ``` Expected output: ``` NAME GROUP LAG SEVERITY TIME rpoevent-production-pg-8f2a production-pg 42s Warning 2024-01-15T10:05:00Z ``` Inspect the aggregated group status: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get replicationgroupstatus production-pg-status \ -n default -o yaml ``` ```yaml status: health: Degraded lastSuccessfulSync: "2024-01-15T09:58:00Z" volumes: - pvcName: prod-vm-1-disk syncState: Syncing syncPercent: 87 - pvcName: prod-vm-2-disk syncState: Consistent syncPercent: 100 ``` A `Syncing` volume is catching up after a temporary network interruption. Monitor until it reaches `Consistent`. If it does not progress, check DRBD kernel module status on the affected worker node. ## Troubleshooting Use the following reference for the most common reconciliation failures. Each entry follows the pattern: **Symptom → Likely cause → Fix**. --- **ProtectionGroup remains in `Failed` state after creation or update** *Symptom:* `status.state` is `Failed`; `status.conditions` contains a `ValidationFailed` condition with a message referencing a PVC or storage class. *Likely cause:* The protection-group-controller rejected the create or update because one or more VMs have PVCs that do not meet replication requirements (insufficient replica count, unsupported storage class, or missing PVC). *Fix:* 1. Read the `message` field in the `ValidationFailed` condition to identify the specific VM and PVC. 2. Correct the PVC or VM definition on the primary cluster. 3. Re-apply the ProtectionGroup spec. The controller re-validates on every update. --- **ProtectionRequest stuck in an intermediate lifecycle phase** *Symptom:* A ProtectionRequest on the quorum cluster remains in a phase such as `CreatingDRBDVolume` or `WaitingForSync` for longer than expected. *Likely cause:* The protection-controller cannot create the DRBDVolume, or the DRBDVolume was created but replication has not yet completed initial sync. *Fix:* 1. Check the DRBDVolume resource referenced in the ProtectionRequest status: ```bash kubectl --kubeconfig ~/.kube/quorum \ get drbdvolume -n dr- ``` 2. If the DRBDVolume is missing, check protection-controller logs for API errors. 3. If the DRBDVolume exists but `syncPercent` is not advancing, verify DRBD port reachability (TCP 7000–7999) between primary and DR worker nodes. 4. If the DRBDVolume shows an error condition, check the DRBD Operator logs on the affected cluster. --- **FailoverRequest does not advance past `StoppingOnSource`** *Symptom:* The FailoverRequest phase is `StoppingOnSource` for an extended period. The ProtectionGroup `currentState` on the source cluster remains `mixed` or `running`. *Likely cause:* One or more VMs in the ProtectionGroup are not shutting down cleanly. The protection-group-controller waits for all VMs to stop before the failover-controller advances. *Fix:* 1. Identify VMs that are still running: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get vm -n ``` 2. Check KubeVirt events for the stuck VM: ```bash kubectl --kubeconfig ~/.kube/config-primary \ describe vmi -n ``` 3. For a planned failover, resolve the underlying VM issue and allow the controller to retry. For an unplanned failover where the primary cluster is unreachable, create the FailoverRequest with `spec.type: Unplanned` so the failover-controller does not wait for a graceful shutdown. --- **FailoverRequest blocked by a concurrent failover lock** *Symptom:* The failover-controller log contains `Failed to acquire failover lock` and names a holder identity from a previous operation. *Likely cause:* A previous failover process crashed before releasing the Kubernetes Lease `failover-lock-`. The lock has a 300-second TTL and will expire automatically. *Fix:* 1. Wait for the 300-second lease duration to expire, after which the controller will acquire the lock on its next reconciliation cycle. 2. If the lock has already expired (check `renewTime` is stale) but the FailoverRequest is still blocked, delete the lease manually: ```bash kubectl --kubeconfig ~/.kube/quorum \ delete lease failover-lock- \ -n dr- ``` 3. Do not delete the lease if another failover operation is actively running. --- **ReplicationGroupStatus shows `Degraded` or `Critical`** *Symptom:* `kubectl get replicationgroupstatus` returns `health: Degraded` or `health: Critical`. Recent RPOEvents exist for the affected Protection Group. *Likely cause:* DRBD replication lag has exceeded the configured threshold. Common causes include network saturation, high I/O on the primary, or a temporary network partition. *Fix:* 1. Check which volumes are lagging: ```bash kubectl --kubeconfig ~/.kube/config-primary \ get drbdvolume -n ``` 2. For `Syncing` volumes: monitor progress. Transient lag self-resolves once the network recovers. 3. For volumes that are not making sync progress: verify TCP 7000–7999 is open between primary and DR worker nodes and that no firewall rule change has occurred. 4. For `Critical` health: check whether a split-brain condition exists. DRBD split-brain requires manual resolution — do not attempt automated failover until the split-brain is resolved. --- **Status conditions are stale and not updating** *Symptom:* A resource's `status.conditions` shows a timestamp that is minutes or hours old and does not reflect the current state of the system. *Likely cause:* The responsible controller pod has crashed, restarted, or lost connectivity to the cluster it is reconciling against. *Fix:* 1. Check controller pod health: ```bash kubectl --kubeconfig ~/.kube/quorum \ get pods -n dr- ``` 2. If a controller is in `CrashLoopBackOff` or `Error`, inspect its logs from the previous execution: ```bash kubectl --kubeconfig ~/.kube/quorum \ logs -n dr- \ -l app= --previous ``` 3. If the controller is running but stale: verify the controller has network access to the target cluster's Kubernetes API (TCP 6443). The quorum cluster must reach both primary and DR cluster APIs. 4. If kubeconfig credentials have rotated, update the secret in the `dr-` namespace and restart the affected controller pod. --- **pg-sync-controller fails to keep Protection Group metadata synchronized** *Symptom:* VMs on the DR cluster have outdated specs compared to the primary, or the `pg-sync-controller` log shows repeated sync errors. *Likely cause:* The pg-sync-controller on the quorum cluster cannot reach the DR cluster API, or there is a namespace or RBAC mismatch. *Fix:* 1. Check pg-sync-controller logs: ```bash kubectl --kubeconfig ~/.kube/quorum \ logs -n dr- \ -l app=pg-sync-controller -f ``` 2. Confirm the DR cluster kubeconfig secret is valid and the API endpoint is reachable from the quorum cluster. 3. Verify the `dr-` namespace exists on the DR cluster and that RBAC is correctly applied.