Guide

Reconciliation Issues

Resources stuck in pending or in-progress states, reconciliation loop errors

Overview

This page helps you diagnose and resolve situations where Trilio Site Recovery custom resources become stuck in pending or in-progress states, or where reconciliation loops emit repeated errors. Because all DR operations are driven by Custom Resource Definitions, a stalled reconciler or a misconfigured resource blocks the entire workflow it owns — from initial VM protection through failover and test failover. Understanding which controller owns each resource, where its logs live, and which conditions signal the root cause lets you restore forward progress without guesswork.

Prerequisites

Before using this guide, ensure you have:

Access to the quorum cluster with oc (required for OpenShift-specific paths such as oc adm must-gather)
Kubeconfig files for the quorum cluster and both workload clusters
Permissions to read pods, logs, events, and custom resources in the dr-<name> namespace on the quorum cluster and in trilio-site-recovery-system on each workload cluster
Familiarity with the two-namespace model: quorum-side components and all quorum-side CRs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent, DRBDReplicationPolicy) live in dr-<name>; workload-side components and DRBDResource CRs live in trilio-site-recovery-system on each workload cluster
OpenShift 4.17 or later on all three clusters (quorum, primary, DR)

Installation

No additional installation is required for diagnostics. The tools and resources described on this page are part of the standard Trilio Site Recovery deployment.

To collect a full diagnostic bundle for a stuck or failed resource, use tsr-gather (the oc adm must-gather compatible tool):

# Run against the quorum cluster; no -- <command> override is needed
oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --dest-dir=./tsr-bundle

This collects logs, CRD state, and configuration from all clusters in the DR estate into a single bundle. Attach the bundle when opening a support case.

Configuration

The following configuration knobs affect reconciliation behaviour and are relevant when diagnosing stuck resources.

Log level

Log verbosity is controlled per namespace at runtime via the site-recovery-logging ConfigMap in the relevant namespace. Valid levels are error, info, debug, and trace (there is no warn level).

Key	Scope	Default
`level`	All components in this namespace	`info`
`level.control-plane`	Quorum controller manager only	inherits `level`
`level.drbd-node-agent`	Node agent DaemonSet only	inherits `level`
`level.protectionzone`	ProtectionZone controller only	inherits `level`

To raise verbosity on the quorum cluster while diagnosing a stuck ProtectionRequest:

# On the quorum cluster, in the zone namespace
oc --context=quorum patch configmap site-recovery-logging \
  -n dr-<name> \
  --type merge \
  -p '{"data":{"level":"debug"}}'

Restore to info after diagnosis to avoid log volume growth.

Metadata provisioning deadline

The zoneConfig.metadataProvisioningFailureDeadline chart value (default 10m) controls how long the protection controller waits for DRBD metadata PVCs to become bound before failing a ProtectionRequest with a ProvisioningDeadlineExceeded warning. If your storage provisioner is slow, this deadline may expire before provisioning completes.

Note: This value is set in the embedded Helm chart deployed automatically by the ProtectionZone controller. Customers do not run helm install or helm upgrade directly; contact support to adjust this value.

Leader election

Leader election is enabled by default (leaderElect: true). If the quorum controller manager pod is restarting, a second pod may be waiting to acquire the leader lease. Check for multiple pods in the zone namespace and confirm only one is Running before concluding the reconciler is stuck.

Webhook certificate

Webhook serving certificates are not managed by the charts. The Secret named webhook-server-cert must exist in the release namespace before admission webhooks can serve. A missing or expired Secret causes all admission requests to be rejected, which surfaces as resources stuck in Pending with admission errors in events.

Usage

Use the patterns below to identify why a specific resource is not progressing.

Locating the controller responsible for a stuck resource

All quorum-side reconcilers (failover, protection, pg-sync, replication-monitor, and test-failover) run inside a single manager container in the quorum controller manager Deployment. The pod output is 1/1.

# On the quorum cluster — list the controller manager pod
oc --context=quorum get pods \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

The DRBD node agent runs as a DaemonSet on every worker node of both workload clusters:

# On a workload cluster — list node agent pods
oc --context=primary get pods \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent

Reading controller logs

# Quorum controller manager logs (all reconcilers)
oc --context=quorum logs \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  --container manager \
  --since=15m

# Node agent log for a specific node (two-step: find pod, then fetch logs)
NODE_POD=$(oc --context=primary get pods \
  -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')

oc --context=primary logs \
  -n trilio-site-recovery-system \
  "$NODE_POD" \
  --since=15m

Inspecting conditions on a stuck resource

Every Site Recovery CR exposes a status.conditions array. Conditions are the primary structured signal for why reconciliation has stalled.

# ProtectionRequest conditions
oc --context=quorum get pr <name> -n dr-<name> -o jsonpath='{.status.conditions}' | jq .

# ProtectionGroup conditions
oc --context=quorum get pg <name> -n dr-<name> -o jsonpath='{.status.conditions}' | jq .

# FailoverRequest conditions
oc --context=quorum get fr <name> -n dr-<name> -o jsonpath='{.status.conditions}' | jq .

# TestFailover conditions
oc --context=quorum get tfo <name> -n dr-<name> -o jsonpath='{.status.conditions}' | jq .

# DRBDResource conditions (on a workload cluster, in the VM's namespace)
oc --context=primary get dres <name> -n <vm-namespace> -o jsonpath='{.status.conditions}' | jq .

Checking ProtectionRequest phase

Valid ProtectionRequest phases are: Pending, Validating, CreatingDRBD, Syncing, ReadyToActivate, Activating, Protected, Unprotecting, Failed.

oc --context=quorum get pr -n dr-<name>

Expected columns: VM, NAMESPACE, PHASE, SYNC, AGE.

ReadyToActivate is a normal operator-action state, not a stall — the protection controller is waiting for you to activate the protected VM. If a resource has been in Syncing for an unusually long time, check sync progress:

oc --context=quorum get pr <name> -n dr-<name> \
  -o jsonpath='{.status.slowestVolumeSyncProgress}'

Checking FailoverRequest phase and per-VM status

Valid FailoverRequest phases are: Initializing, Validating, StoppingSource, WaitingForLocks, RemovingTaints, StartingTarget, Monitoring, Completed, Failed.

FailoverRequest has no message field and no failedVMs field. Per-VM detail is in status.vmStatuses:

oc --context=quorum get fr <name> -n dr-<name> \
  -o jsonpath='{.status.vmStatuses}' | jq .

Checking TestFailover phase

Valid TestFailover phases are: Queued, PreparingEnvironment, CreatingSnapshots, CreatingVolumes, CreatingVMs, StartingVMs, Verifying, Succeeded, Failed, CleaningUp, Cleaned.

The test namespace is controller-generated and exposed in status.testNamespace. Do not attempt to construct or guess this value:

oc --context=quorum get tfo <name> -n dr-<name> \
  -o jsonpath='{.status.testNamespace}'

Filtering ReplicationGroupStatus and RPOEvents by protection group

CRD field selectors on spec.* or status.* are not supported. Use the siterecovery.trilio.io/protection-group label:

# ReplicationGroupStatus for a specific protection group
oc --context=quorum get rgs \
  -n dr-<name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

# RPOEvents for a specific protection group
oc --context=quorum get rpoe \
  -n dr-<name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

The ReplicationGroupStatus for a ProtectionGroup is auto-created by the replication monitor under the name <protection-group-name>-replication.

Checking DRBDResource placement readiness

A DRBDResource that is waiting for its anchor pod to schedule will show ConditionPlacementReady=False. This gates metadata PVC provisioning.

oc --context=primary get dres <name> -n <vm-namespace> \
  -o jsonpath='{.status.conditions[?(@.type=="PlacementReady")]}' | jq .

If the reason is AwaitingAnchorSchedule, at least one candidate node is carrying the siterecovery.trilio.io/not-ready:NoSchedule taint. Check taint status:

oc --context=primary get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'

Checking ProtectionZone reachability

If the ProtectionZone is Unavailable or Degraded, check cluster reachability status:

oc --context=quorum get pz <name> -n dr-<name> \
  -o jsonpath='{.status.clusters}' | jq .

A cluster in Unreachable state means the quorum controller cannot reach that workload cluster's API server. Verify that the kubeconfig Secret in dr-<name> is valid and that TCP port 6443 is open from the quorum cluster to the workload cluster API server.

Examples

Example 1: ProtectionRequest stuck in `CreatingDRBD`

A ProtectionRequest that stays in CreatingDRBD for more than 10 minutes indicates the protection controller cannot provision the DRBDResource pair.

oc --kubeconfig ~/.kube/config-quorum \
  get protectionrequest my-vm-pr -n dr-zone1 -o yaml

Expected output (stuck):

status:
  phase: CreatingDRBD

Check the DRBD resource on both sides:

oc --kubeconfig ~/.kube/config-primary \
  get drbdresource -n default -l vm=my-vm

oc --kubeconfig ~/.kube/config-dr \
  get drbdresource -n default -l vm=my-vm

If the DRBDResource does not exist on either side, check the workload control plane logs on the cluster where it is missing:

oc --kubeconfig ~/.kube/config-primary \
  logs -n dr-zone1 \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --tail=100

Look for errors referencing the VM name or namespace. A missing LVM thin pool on a worker node is the most common cause—verify that every worker node has a block device and thin pool configured as required by the prerequisites.

Example 2: TestFailover stuck in `CreatingVolumes`

oc --kubeconfig ~/.kube/config-quorum \
  get testfailover my-tf -n dr-zone1 -o yaml

Expected output (stuck):

status:
  phase: CreatingVolumes
  testNamespace: site-recovery-web-tier-pg-a3f9b

Inspect the test namespace the controller created:

oc --kubeconfig ~/.kube/config-dr \
  get pvc -n site-recovery-web-tier-pg-a3f9b

If PVCs are Pending, the DR cluster's storage provisioner cannot fulfil the request. Check that the storage class referenced in the DRBDReplicationPolicy exists on the DR cluster and that the DRBD node agent DaemonSet is healthy:

oc --kubeconfig ~/.kube/config-dr \
  get daemonset -n dr-zone1 \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

Example 3: FailoverRequest stuck in `InProgress`

oc --kubeconfig ~/.kube/config-quorum \
  get failoverrequest my-fr -n dr-zone1 -o yaml

Expected output (stuck):

status:
  phase: InProgress
  vmStatuses:
    - name: prod-vm-1
      status: Stopping
    - name: prod-vm-2
      status: Stopping

Per-VM detail is in status.vmStatuses. A VM that stays in Stopping is not shutting down cleanly. (spec.drainTimeoutSeconds is accepted by the API but not consumed in Tech Preview, so it is not the threshold being applied.) Check the VM on the source cluster:

oc --kubeconfig ~/.kube/config-primary \
  get vmi prod-vm-1 -n <vm-namespace>

For an unplanned failover where the primary is unavailable, the failover controller force-promotes DRBD volumes without waiting for the primary—if the request is still stuck, check that spec.failoverType is set to unplanned (lowercase):

oc --kubeconfig ~/.kube/config-quorum \
  get failoverrequest my-fr -n dr-zone1 \
  -o jsonpath='{.spec.failoverType}'

Valid values are exactly planned and unplanned. A capitalised or misspelled value will cause the admission webhook to reject the resource.

Example 4: ProtectionGroup in `Degraded` state blocking failover readiness

oc --kubeconfig ~/.kube/config-primary \
  get protectiongroup web-tier-pg -n default

Expected output:

NAME          STATE    VMS   REPLICATION   HEALTH    AGE
web-tier-pg   Active   3     synchronous   Degraded  4h

Get the detailed status:

oc --kubeconfig ~/.kube/config-primary \
  describe protectiongroup web-tier-pg -n default

Then check for associated RPO events:

oc --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-zone1 \
  --sort-by=.spec.timestamp | tail -10

A Degraded ProtectionGroup with recent critical RPOEvent records indicates the DRBD replication link between primary and DR worker nodes has degraded. Verify TCP port connectivity on ports 7000–8000 between worker nodes on both clusters.

Troubleshooting

Use the following reference for the most common reconciliation failures. Each entry lists the symptom you observe, the likely cause, and the steps to resolve it.

Issue: ProtectionRequest stuck in `Pending`

Symptom: A ProtectionRequest remains in the Pending phase for more than 5 minutes after creation.

Likely cause: The quorum control plane pod is not running, has lost leader election, or cannot reach the workload cluster API server.

Fix:

Check that the quorum control plane pod is 1/1 Running:

oc --kubeconfig ~/.kube/config-quorum get pods \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

If the pod is in CrashLoopBackOff, read its logs:

oc --kubeconfig ~/.kube/config-quorum logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager --previous

Verify the leader lease is held by a live pod:
```
oc --kubeconfig ~/.kube/config-quorum get lease -n dr-<zone-name>
```
If no pod is renewing the lease, delete the lease to force re-election.
Verify the ProtectionZone spec.clusters entries contain valid kubeconfig references for both workload clusters. An unreachable API server causes the protection controller to requeue indefinitely.

Issue: ProtectionRequest stuck in `Validating`

Symptom: The ProtectionRequest entered Validating but does not advance to CreatingDRBD.

Likely cause: The protection controller found an issue with the VM or its PVCs—for example, the VM does not exist in the specified namespace, a PVC is not bound, or the VM is already partially protected.

Fix:

Confirm the VM exists on the primary cluster:

oc --kubeconfig ~/.kube/config-primary \
  get vm <vm-name> -n <vm-namespace>

Confirm all PVCs attached to the VM are Bound:

oc --kubeconfig ~/.kube/config-primary \
  get pvc -n <vm-namespace>

Read the protection controller logs and filter for the VM name to find the validation error:

oc --kubeconfig ~/.kube/config-quorum logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager --tail=300 | grep <vm-name>

Correct the underlying issue (bind the PVC, fix the VM name in the spec) and the controller will requeue automatically.

Issue: ProtectionRequest stuck in `CreatingDRBD`

Symptom: The ProtectionRequest has been in CreatingDRBD for more than 10 minutes.

Likely cause: The DRBD node agent cannot create the backing volume on one or both clusters. Common sub-causes: missing LVM thin pool on a worker node, TCP ports 7000–8000 not open between clusters, or the workload control plane DaemonSet has unhealthy pods.

Fix:

Check the workload DaemonSet on both clusters:

oc --kubeconfig ~/.kube/config-primary get daemonset \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

oc --kubeconfig ~/.kube/config-dr get daemonset \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

Check DRBD node agent pod logs on any unhealthy node:

oc --kubeconfig ~/.kube/config-primary logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --tail=200

Verify network connectivity on DRBD ports from a worker node on the primary to a worker node on DR:
```
# From a primary worker node
nc -zv <dr-worker-node-ip> 7000
```
Verify LVM thin pool presence on each worker node that will host DRBD volumes.

Issue: ProtectionRequest stuck in `Syncing`

Symptom: The ProtectionRequest is in Syncing and the DRBDResource shows less than 100 % sync completion for an extended period.

Likely cause: Initial synchronisation of large volumes over a limited-bandwidth link takes time. However, if progress has not advanced at all, the replication link may be broken.

Fix:

Check sync progress on both sides:

oc --kubeconfig ~/.kube/config-primary \
  get drbdresource -n <vm-namespace> -o wide

oc --kubeconfig ~/.kube/config-dr \
  get drbdresource -n <vm-namespace> -o wide

Check ReplicationGroupStatus for the ProtectionGroup:

oc --kubeconfig ~/.kube/config-quorum \
  get replicationgroupstatus -n dr-<zone-name>

If progress is completely frozen, check for recent RPOEvent records with Critical severity:

oc --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-<zone-name> --sort-by=.spec.timestamp

For Protocol C (synchronous), verify round-trip latency between primary and DR worker nodes is below 50 ms. Latency above this threshold causes DRBD to stall writes and halt synchronisation.

Issue: TestFailover stuck in `CreatingSnapshots`

Symptom: A TestFailover does not advance past CreatingSnapshots.

Likely cause: The volume snapshot class is not available on the DR cluster, or the ProtectionGroup is not in a Consistent state at the time the snapshot is requested.

Fix:

Check that the ProtectionGroup replication is healthy:
```
oc --kubeconfig ~/.kube/config-primary \
  get protectiongroup <pg-name> -n <vm-namespace>
```
A Degraded ProtectionGroup prevents the test-failover controller from taking a consistent snapshot.

Check for volume snapshot resources on the DR cluster in the test namespace from status.testNamespace:

TF_NS=$(oc --kubeconfig ~/.kube/config-quorum \
  get testfailover <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')

oc --kubeconfig ~/.kube/config-dr \
  get volumesnapshot -n $TF_NS

If snapshots are in a ReadyToUse: false state, check the volume snapshot controller logs on the DR cluster.

Issue: TestFailover stuck in `CleaningUp` and never reaching `Cleaned`

Symptom: A TestFailover with spec.cleanupPolicy: Automatic stays in CleaningUp.

Likely cause: A resource in the test namespace (a PVC with a finalizer, a running VMI) is blocking deletion.

Fix:

Get the test namespace:

TF_NS=$(oc --kubeconfig ~/.kube/config-quorum \
  get testfailover <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')

List resources in the test namespace:

oc --kubeconfig ~/.kube/config-dr get all,pvc -n $TF_NS

If a VMI is still running, force-stop it:

oc --kubeconfig ~/.kube/config-dr \
  delete vmi --all -n $TF_NS --force --grace-period=0

If PVCs have stuck finalizers, patch them:

oc --kubeconfig ~/.kube/config-dr \
  patch pvc <pvc-name> -n $TF_NS \
  --type json \
  -p '[{"op":"remove","path":"/metadata/finalizers"}]'

The test-failover controller will detect the cleared resources and advance to Cleaned.

Issue: FailoverRequest stuck in `InProgress` with VMs in `Stopping`

Symptom: A planned FailoverRequest stays in InProgress and status.vmStatuses shows one or more VMs stuck in Stopping. (spec.drainTimeoutSeconds is accepted by the API but not consumed in Tech Preview, so there is no timeout window being enforced.)

Likely cause: A VM is not shutting down gracefully—its guest OS may be hung, or KubeVirt cannot send the shutdown signal.

Fix:

Check the VMI on the source cluster:

oc --kubeconfig ~/.kube/config-primary \
  get vmi <vm-name> -n <vm-namespace>

If the VMI is still Running, force-stop it:

oc --kubeconfig ~/.kube/config-primary \
  delete vmi <vm-name> -n <vm-namespace> --force --grace-period=0

The failover controller will detect the VM is stopped and advance the reconciliation. If the primary cluster is completely unreachable, consider whether the correct approach is an unplanned failover—set spec.failoverType: unplanned in a new FailoverRequest so the controller force-promotes DRBD volumes without waiting for the primary.

Issue: All reconcilers appear paused simultaneously

Symptom: Multiple CRs across different types all stop advancing at approximately the same time.

Likely cause: The quorum control plane pod lost the leader lease—possibly due to a pod restart, node eviction, or a temporary API server disruption—and the new pod has not yet acquired the lease.

Fix:

Check the current leader lease state:

oc --kubeconfig ~/.kube/config-quorum \
  get lease -n dr-<zone-name>

Check that the quorum control plane pod is running:

oc --kubeconfig ~/.kube/config-quorum get pods \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

If the pod is running but the lease is stale (not renewed in the last 15+ seconds), delete the lease to force immediate acquisition:
```
oc --kubeconfig ~/.kube/config-quorum \
  delete lease <lease-name> -n dr-<zone-name>
```
The running pod will acquire the lease within seconds and all reconcilers will resume.

Issue: Admission webhook rejecting CR creation

Symptom: oc apply returns a webhook validation error immediately; the CR is never created and therefore never enters a reconciliation loop.

Likely cause: A spec field value is invalid. Common cases: storageBackendMode set to a value other than DRBD; FailoverRequest.spec.failoverType capitalised (Planned instead of planned); FailoverRequest.spec.targetCluster set to a value other than cluster1 or cluster2; a required field missing.

Fix:

Read the full webhook rejection message from the oc apply output—it identifies the exact field and constraint.
Correct the field in your manifest. Key constraints:
- storageBackendMode must be DRBD
- spec.failoverType must be lowercase planned or unplanned
- spec.targetCluster must be exactly cluster1 or cluster2
- TestFailover.spec.cleanupPolicy must be Manual or Automatic
Re-apply the corrected manifest.

Reconciliation Issues

Log level

Metadata provisioning deadline

Leader election

Webhook certificate

Locating the controller responsible for a stuck resource

Reading controller logs

Inspecting conditions on a stuck resource

Checking ProtectionRequest phase

Checking FailoverRequest phase and per-VM status

Checking TestFailover phase

Filtering ReplicationGroupStatus and RPOEvents by protection group

Checking DRBDResource placement readiness

Checking ProtectionZone reachability

Example 1: ProtectionRequest stuck in CreatingDRBD

Example 2: TestFailover stuck in CreatingVolumes

Example 3: FailoverRequest stuck in InProgress

Example 4: ProtectionGroup in Degraded state blocking failover readiness

Issue: ProtectionRequest stuck in Pending

Issue: ProtectionRequest stuck in Validating

Issue: ProtectionRequest stuck in CreatingDRBD

Issue: ProtectionRequest stuck in Syncing

Issue: TestFailover stuck in CreatingSnapshots

Issue: TestFailover stuck in CleaningUp and never reaching Cleaned

Issue: FailoverRequest stuck in InProgress with VMs in Stopping

Issue: All reconcilers appear paused simultaneously

Issue: Admission webhook rejecting CR creation

Example 1: ProtectionRequest stuck in `CreatingDRBD`

Example 2: TestFailover stuck in `CreatingVolumes`

Example 3: FailoverRequest stuck in `InProgress`

Example 4: ProtectionGroup in `Degraded` state blocking failover readiness

Issue: ProtectionRequest stuck in `Pending`

Issue: ProtectionRequest stuck in `Validating`

Issue: ProtectionRequest stuck in `CreatingDRBD`

Issue: ProtectionRequest stuck in `Syncing`

Issue: TestFailover stuck in `CreatingSnapshots`

Issue: TestFailover stuck in `CleaningUp` and never reaching `Cleaned`

Issue: FailoverRequest stuck in `InProgress` with VMs in `Stopping`