Trilio Site Recovery for Kubernetes
Docs/site-recovery/Troubleshooting/Reconciliation Issues
Guide

Reconciliation Issues

Resources stuck in pending or in-progress states, reconciliation loop errors

Overview

This page helps you diagnose and resolve situations where Trilio Site Recovery custom resources become stuck in Pending or InProgress states, or where controllers enter reconciliation loop errors. These conditions block protection, failover, and test failover workflows and require prompt attention. Because all Site Recovery operations are driven by Kubernetes custom resources and reconciled continuously by controllers running on the quorum and workload clusters, understanding the reconciliation model is essential to effective troubleshooting.

Prerequisites

Before troubleshooting reconciliation issues, ensure you have:

  • kubectl or oc CLI configured with kubeconfig access to all three clusters (quorum, primary, DR)
  • Permissions to read and patch custom resources in the relevant DR namespace (dr-<name>) on the quorum cluster and in VM namespaces on the workload clusters
  • Permissions to read pod logs from the site-recovery-quorum-control-plane deployment on the quorum cluster and from the site-recovery-workload-control-plane components on primary and DR clusters
  • The tsr-gather must-gather tool available if you need to collect a full diagnostic bundle
  • OpenShift 4.14 or later on all clusters
  • KubeVirt / OpenShift Virtualization 1.0 or later
Installation

No additional installation is required to perform reconciliation troubleshooting. All diagnostic steps use kubectl/oc commands against resources already deployed by the site-recovery-quorum-control-plane and site-recovery-workload-control-plane Helm charts.

If you need to collect a full diagnostic bundle, run tsr-gather against the quorum cluster:

oc adm must-gather --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle

This collects logs, CRD state, and configuration from the quorum cluster and all associated workload clusters without capturing secrets or sensitive credentials.

Configuration

The following custom resource fields directly influence reconciliation behavior and are the most common sources of stuck or looping resources.

ProtectionGroup — spec.desiredState

The desiredState field drives the protection-group-controller reconciliation loop on the workload clusters. Valid values are running and stopped. If desiredState and status.currentState disagree and no progress is made, the controller is likely blocked waiting on a VM or DRBD volume condition.

spec:
  desiredState: running   # or: stopped
  virtualMachines:        # required; at least one entry
    - name: my-vm

FailoverRequest — spec.failoverType and spec.drainTimeoutSeconds

failoverType controls whether the failover-controller waits for a graceful VM shutdown (planned) or force-promotes DRBD volumes immediately (unplanned). An incorrect or missing failoverType can leave the controller waiting indefinitely. drainTimeoutSeconds caps how long the controller waits for source VMs to stop before declaring failure.

spec:
  failoverType: planned         # or: unplanned
  drainTimeoutSeconds: 300      # default if omitted varies; set explicitly
  protectionGroupRef:           # required
    name: my-pg
    namespace: my-namespace
  targetCluster: dr-cluster     # required

TestFailover — spec.cleanupPolicy and spec.timeout

cleanupPolicy determines whether test resources are removed automatically after the test. If set incorrectly, test namespaces and PVCs accumulate and subsequent TestFailover resources may become stuck waiting for available snapshot capacity. timeout bounds the entire test operation.

spec:
  protectionGroupRef:           # required
    name: my-pg
    namespace: my-namespace
  cleanupPolicy: OnSuccess      # or: Always, Never
  timeout: 30m

ProtectionRequest — spec.sourceCluster and spec.virtualMachine

The protection-controller uses sourceCluster to locate the VM and provision the DRBDResource pair. If sourceCluster does not match the cluster where the VM exists, the controller cannot find the VM and the request stays Pending.

spec:
  sourceCluster: primary-cluster   # required; must match actual cluster
  virtualMachine:                  # required
    name: my-vm
    namespace: my-namespace

DRBDReplicationPolicy — spec.drbdProtocol and spec.replicationMode

An invalid or mismatched drbdProtocol value can cause the drbd-node-agent to fail to configure replication, leaving DRBDResource CRDs in a degraded state and blocking ProtectionRequest processing.

spec:
  clusters:                # required array
    - name: primary-cluster
    - name: dr-cluster
  drbdProtocol: C          # C = synchronous (RPO=0); A = asynchronous
  replicationMode: sync    # matches drbdProtocol selection

ReplicationGroupStatus — spec.pollingIntervalSeconds and spec.rpoObjectiveSeconds

If pollingIntervalSeconds is set too high, replication health data is stale and RPOEvent records lag. If rpoObjectiveSeconds is set lower than the actual achievable RPO for your link, the replication-monitor reconciler will continuously emit RPOEvents and log warnings, which can be mistaken for a reconciliation loop.

spec:
  protectionGroupRef:        # required
    name: my-pg
    namespace: my-namespace
  pollingIntervalSeconds: 30
  rpoObjectiveSeconds: 60
Usage

Use the patterns below to locate, diagnose, and resolve stuck or looping resources. Work from the quorum cluster outward to the workload clusters, following the reconciliation chain.

Identify which resource is stuck

Check the status of key CRDs in the DR namespace on the quorum cluster:

# Replace dr-<name> with your deployment namespace
oc get protectionrequests,failoverrequests,testfailovers \
  -n dr-<name> \
  -o wide

Look for resources with PHASE or STATUS values of Pending or InProgress that have not changed recently. Check the AGE column — a resource stuck for more than 10 minutes in InProgress is a strong signal of a blocked reconciliation.

Read the controller logs

The site-recovery-quorum-control-plane deployment on the quorum cluster runs the failover-controller, protection-controller, pg-sync-controller, test-failover-controller, and replication-monitor reconcilers in a single pod. Fetch its logs to find reconciliation errors:

oc logs -n dr-<name> \
  deployment/site-recovery-quorum-control-plane \
  --tail=200

To stream logs and filter for errors:

oc logs -n dr-<name> \
  deployment/site-recovery-quorum-control-plane \
  -f | grep -E 'ERROR|error|failed|Failed|stuck'

For drbd-node-agent issues on the primary or DR cluster, fetch DaemonSet pod logs:

# On the primary cluster
oc logs -n <drbd-namespace> \
  daemonset/drbd-node-agent \
  --all-containers=true \
  --tail=100

Inspect the stuck resource's status conditions

Every CRD exposes a status block that contains the current phase and conditions. Describe the resource for full detail:

oc describe protectionrequest <name> -n dr-<name>
oc describe failoverrequest <name> -n dr-<name>
oc describe testfailover <name> -n dr-<name>

Pay attention to:

  • status.phase — which reconciliation phase the controller is in
  • status.conditions — specific blocking conditions with messages
  • status.message — human-readable explanation of the current state

Check ProtectionGroup replication health

A FailoverRequest or ProtectionRequest that is stuck often reflects an upstream ProtectionGroup that is Degraded. Check it on the workload clusters:

# On the primary cluster
oc get protectiongroup <pg-name> -n <namespace> -o yaml

If status.replicationHealth is Degraded or Critical, resolve the replication issue before retrying the failover or protection operation.

Check RPOEvents for replication lag

oc get rpoevents -n dr-<name> --sort-by='.spec.timestamp'

High-frequency RPOEvents with Critical severity indicate persistent replication lag that may block synchronous-mode operations.

Examples

Example 1: ProtectionRequest stuck in Pending

A ProtectionRequest remains Pending for more than five minutes.

Inspect the resource:

oc describe protectionrequest my-vm-pr -n dr-production

Expected output (truncated):

Name:         my-vm-pr
Namespace:    dr-production
Status:
  Phase:    Pending
  Message:  VirtualMachine my-vm not found on cluster primary-cluster

Identify the mismatch:

oc get protectionrequest my-vm-pr -n dr-production \
  -o jsonpath='{.spec.sourceCluster}'

primary-cluster-old

The spec.sourceCluster value does not match the registered cluster name. Patch the resource:

oc patch protectionrequest my-vm-pr -n dr-production \
  --type merge \
  -p '{"spec":{"sourceCluster":"primary-cluster"}}'

Verify the phase advances:

oc get protectionrequest my-vm-pr -n dr-production -w

NAME        PHASE     AGE
my-vm-pr    Pending   6m
my-vm-pr    Validating  6m30s
my-vm-pr    Protected   7m10s

Example 2: FailoverRequest stuck in InProgress at StoppingOnSource

A FailoverRequest with failoverType: planned is stuck with the protection-group on the source cluster not reaching stopped.

Check the FailoverRequest status:

oc get failoverrequest my-failover -n dr-production -o yaml

status:
  phase: StoppingOnSource
  message: "Waiting for ProtectionGroup my-pg on primary-cluster to reach currentState=stopped"

Check the ProtectionGroup on the primary cluster:

oc get protectiongroup my-pg -n production -o yaml

status:
  currentState: mixed
  replicationHealth: Degraded

Check which VMs have not stopped:

oc get vm -n production -l protectiongroup=my-pg

NAME      AGE   STATUS    READY
vm-web    2d    Running   True
vm-db     2d    Stopping  False

vm-db is hung in Stopping. Check the VMI events:

oc describe vmi vm-db -n production | tail -20

Resolve the stuck VM (for example, force-delete the VMI if the guest OS is unresponsive), then verify the ProtectionGroup reaches stopped. The failover-controller will advance on its next reconciliation cycle (within 10 seconds).

If the primary cluster is genuinely unreachable and an emergency failover is required, delete the current FailoverRequest and create a new one with failoverType: unplanned:

oc delete failoverrequest my-failover -n dr-production

oc apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: my-failover-emergency
  namespace: dr-production
spec:
  failoverType: unplanned
  protectionGroupRef:
    name: my-pg
    namespace: production
  targetCluster: dr-cluster
  drainTimeoutSeconds: 60
EOF

Example 3: TestFailover stuck in CreatingSnapshots

Check the TestFailover status:

oc describe testfailover my-test -n dr-production

Status:
  Phase:    CreatingSnapshots
  Message:  VolumeSnapshot my-pg-vm-web-snap not ready after 8m

Check the VolumeSnapshot on the DR cluster:

oc get volumesnapshot -n test-my-test -o wide

NAME                    READYTOUSE   SOURCEPVC        RESTORESIZE   AGE
my-pg-vm-web-snap       false        vm-web-disk      <unset>       9m

Describe the snapshot for error events:

oc describe volumesnapshot my-pg-vm-web-snap -n test-my-test

If the VolumeSnapshotContent is missing or the CSI driver reports an error, check the CSI driver pods and the drbd-node-agent logs on the DR cluster worker nodes:

oc logs -n <drbd-namespace> daemonset/drbd-node-agent --tail=50

Once the underlying snapshot issue is resolved, the test-failover-controller will detect the snapshot becoming ready and advance through VerifyingData to Succeeded automatically.

To abandon the stuck test and clean up, delete the TestFailover resource. If cleanupPolicy is Always or OnSuccess, the controller will remove test PVCs and the isolated namespace:

oc delete testfailover my-test -n dr-production

Example 4: Replication loop errors from mismatched DRBDReplicationPolicy

The quorum control plane logs show repeated errors for a DRBDResource.

Collect logs:

oc logs -n dr-production deployment/site-recovery-quorum-control-plane \
  --tail=300 | grep 'DRBDResource\|replication\|protocol'

ERROR reconcile DRBDResource vm-db-primary: protocol mismatch: policy specifies C, resource reports A
ERROR reconcile DRBDResource vm-db-primary: retrying in 10s

Inspect the DRBDReplicationPolicy:

oc get drbdreplicationpolicy default-policy -n dr-production -o yaml

spec:
  drbdProtocol: C
  replicationMode: sync

Inspect the DRBDResource on the primary cluster:

oc get drbdresource vm-db-primary -n production -o yaml | grep -A5 replication

If the drbd-node-agent reports the volume is configured in Protocol A (for example, because the link latency exceeds 50ms RTT), either:

  • Fix the network link to meet the sub-50ms requirement for Protocol C, or
  • Update the DRBDReplicationPolicy to use Protocol A and replicationMode: async:
oc patch drbdreplicationpolicy default-policy -n dr-production \
  --type merge \
  -p '{"spec":{"drbdProtocol":"A","replicationMode":"async"}}'

The replication-monitor reconciler will detect the policy change and stop emitting loop errors once the DRBDResource aligns with the policy.

Troubleshooting

Use the following reference to match symptoms to causes and fixes. Each issue follows the pattern: Symptom → Likely Cause → Fix.

ProtectionRequest stays in Pending indefinitely

Symptom: oc get protectionrequest shows Phase: Pending for more than 5 minutes. Controller logs show VirtualMachine not found or cluster not found.

Likely cause: spec.sourceCluster does not match the cluster name registered in the ProtectionZone, or the VM does not exist in the specified namespace.

Fix:

  1. Verify the VM exists on the cluster: oc get vm <name> -n <namespace> using the kubeconfig for the primary cluster.
  2. Verify the cluster name matches: oc get protectionzone -n dr-<name> -o yaml and compare spec.clusters[].name to spec.sourceCluster in the ProtectionRequest.
  3. Patch spec.sourceCluster to the correct value, or recreate the ProtectionRequest with the correct cluster name.

FailoverRequest stuck in InProgress / StoppingOnSource

Symptom: A planned FailoverRequest does not advance past StoppingOnSource. The ProtectionGroup on the source cluster has currentState: mixed.

Likely cause: One or more VMs in the ProtectionGroup cannot stop — the guest OS is unresponsive, or the KubeVirt VMI is stuck in Terminating.

Fix:

  1. Identify the stuck VM: oc get vmi -n <namespace> -l protectiongroup=<pg-name>.
  2. Force-delete the stuck VMI if the guest is confirmed unreachable: oc delete vmi <name> -n <namespace> --grace-period=0 --force.
  3. Wait for the ProtectionGroup currentState to reach stopped.
  4. The failover-controller reconciles every 10 seconds and will automatically advance the FailoverRequest once the source ProtectionGroup is stopped.

If the primary cluster is completely unreachable, delete the current FailoverRequest and submit a new one with spec.failoverType: unplanned and a short spec.drainTimeoutSeconds.

FailoverRequest stuck in InProgress / StartingOnTarget

Symptom: Source VMs have stopped but VMs do not start on the DR cluster. Controller logs mention DRBD quorum taints or lost-quorum events.

Likely cause: Worker nodes on the DR cluster have the drbd.linbit.com/lost-quorum taint applied. The failover-controller's taint-removal safety check may be blocking taint removal because another ProtectionGroup has running VMs on the same target cluster.

Fix:

  1. Check for other ProtectionGroups with running VMs on the target cluster: oc get protectiongroup -n <namespace> -o custom-columns=NAME:.metadata.name,STATE:.status.currentState.
  2. If another ProtectionGroup is running on the target cluster, coordinate the failover sequence — wait for the other operation to complete before retrying, or determine whether an emergency override is appropriate.
  3. Check drbd-node-agent logs on the DR cluster for quorum-loss detail: oc logs -n <drbd-namespace> daemonset/drbd-node-agent --tail=100.
  4. Verify DRBD replication network connectivity (TCP 7000–7999) between primary and DR worker nodes is intact.

TestFailover stuck in CreatingSnapshots

Symptom: TestFailover does not advance past CreatingSnapshots. VolumeSnapshot objects on the DR cluster have READYTOUSE: false for more than a few minutes.

Likely cause: The CSI driver or drbd-node-agent on the DR cluster cannot create the volume snapshot — insufficient storage capacity, a driver error, or the DRBD volume is not in a consistent state.

Fix:

  1. Describe the stuck VolumeSnapshot: oc describe volumesnapshot <name> -n test-<testfailover-name>.
  2. Check drbd-node-agent logs on the DR cluster for DRBD resource state errors.
  3. Check available storage capacity on the DR cluster.
  4. Resolve the underlying snapshot error, then wait for the test-failover-controller to detect the snapshot becoming ready (it polls continuously).
  5. If the test cannot be recovered, delete the TestFailover resource to trigger cleanup according to spec.cleanupPolicy.

TestFailover leaves orphaned resources after failure

Symptom: After a TestFailover enters Failed, test PVCs and an isolated namespace remain on the DR cluster.

Likely cause: spec.cleanupPolicy is set to Never, or the test-failover-controller crashed during the CleaningUp phase.

Fix:

  1. Check spec.cleanupPolicy on the TestFailover: oc get testfailover <name> -n dr-<name> -o jsonpath='{.spec.cleanupPolicy}'.
  2. If set to Never, manually delete the test namespace: oc delete namespace test-<testfailover-name>.
  3. If the controller crashed during cleanup, restart it: oc rollout restart deployment/site-recovery-quorum-control-plane -n dr-<name> and then delete the TestFailover resource to re-trigger the cleanup reconciliation.

Reconciliation loop: same error repeating every 10 seconds

Symptom: The site-recovery-quorum-control-plane logs show the same error for a specific resource on every reconciliation cycle with no state change.

Likely cause: A permanent validation error — for example, a DRBDReplicationPolicy protocol mismatch, a ProtectionGroup referencing a VM in a different namespace (multi-namespace violation), or a missing required field in a CRD spec.

Fix:

  1. Read the full error message from controller logs: oc logs -n dr-<name> deployment/site-recovery-quorum-control-plane --tail=100 | grep ERROR.
  2. Describe the named resource to inspect status conditions: oc describe <kind> <name> -n <namespace>.
  3. Correct the spec field causing the permanent error (for example, fix the drbdProtocol, move all VMs in a ProtectionGroup to the same namespace, or add a missing required field).
  4. Note: all VMs in a ProtectionGroup must be in the same namespace. A ProtectionGroup with members spanning multiple namespaces cannot be edited — delete and recreate it as single-namespace groups.

pg-sync-controller not synchronizing ProtectionGroup metadata

Symptom: The standby ProtectionGroup on the DR cluster has stale or missing VM specs. The pg-sync-controller logs show errors converting DataVolume references.

Likely cause: The pg-sync-controller on the quorum cluster cannot reach the DR cluster API server (TCP 6443), or the VM specs contain dataVolumeTemplates that have not yet been converted to PVC references.

Fix:

  1. Verify connectivity from the quorum cluster to the DR API server: oc exec -n dr-<name> deployment/site-recovery-quorum-control-plane -- curl -k https://<dr-api-server>:6443/healthz.
  2. Check pg-sync-controller logs for conversion errors: oc logs -n dr-<name> deployment/site-recovery-quorum-control-plane --tail=100 | grep pg-sync.
  3. If dataVolumeTemplates conversion is failing, ensure Containerized Data Importer (CDI) is installed and healthy on both workload clusters.

Collecting a full diagnostic bundle

If you cannot isolate the issue from logs and CRD state alone, collect a complete support bundle:

oc adm must-gather --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle

The tsr-gather tool collects logs, CRD state, and configuration from the quorum cluster and all associated workload clusters. Share the resulting bundle with Trilio support.