Trilio Site Recovery for Kubernetes
Docs/site-recovery/Troubleshooting/Reconciliation Issues
Guide

Reconciliation Issues

Resources stuck in pending or in-progress states, reconciliation loop errors

Overview

This page helps you diagnose and resolve situations where Trilio Site Recovery custom resources become stuck in pending or in-progress states, or where reconciliation loops emit repeated errors. These conditions typically surface as a ProtectionRequest that never advances past Syncing, a FailoverRequest that stays InProgress indefinitely, a ProtectionZone that does not reach Available, or a TestFailover that stalls between phases. Because every DR operation is CRD-driven, understanding how each controller processes its resource — and what conditions, phases, and label selectors to inspect — is the fastest path to resolution.

Prerequisites

Before following any procedure on this page, confirm you have:

  • oc CLI authenticated to the TSR-Manager cluster (required; kubectl-only operation is not a supported mode)
  • Kubeconfig files for all three clusters (TSR-Manager, primary workload, DR workload)
  • Read access to the dr-<name> namespace on the TSR-Manager cluster and to the trilio-site-recovery-system namespace on both workload clusters
  • Familiarity with the CRD phase values documented in the ProtectionRequest, FailoverRequest, and TestFailover references; exact phase names are required when filtering logs

Namespace reminder: TSR-Manager-side components and every TSR-Manager CR (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent) live in dr-<name>. Workload-side components (DRBDResource, node agent DaemonSet) live in trilio-site-recovery-system on each workload cluster. Querying the wrong namespace returns empty results, not an error.

Installation

No additional software installation is required to follow this guide. All diagnostic commands use oc or kubectl with existing cluster access. If you need to collect a full diagnostic bundle for Trilio support, the tsr-gather tool is run via oc adm must-gather against the TSR-Manager cluster:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  --dest-dir=./tsr-bundle

Replace <release-tag> with the pinned release tag for your installed version. The gather binary inside the image is the entrypoint; no -- <command> override is needed. The bundle contains logs, CRD state, and configuration from all clusters in the DR estate.

Configuration

The following configuration knobs are most relevant when diagnosing reconciliation issues.

Runtime log levels

Log verbosity is controlled per namespace by the site-recovery-logging ConfigMap in the zone namespace (dr-<name> on the TSR-Manager cluster and trilio-site-recovery-system on workload clusters). Accepted levels are error, info, debug, and trace (there is no warn level).

ConfigMap keyScope
levelGlobal fallback for all components in the namespace
level.control-planeTSR-Manager controller manager only
level.drbd-node-agentDRBD node agent DaemonSet only
level.protectionzoneProtectionZone controller only

Raise the level at runtime without restarting pods:

# Increase the TSR-Manager controller to debug on the TSR-Manager cluster
oc patch configmap site-recovery-logging \
  -n dr-<name> \
  --type merge \
  -p '{"data":{"level.control-plane":"debug"}}'

# Increase the node agent to debug on a workload cluster
oc patch configmap site-recovery-logging \
  -n trilio-site-recovery-system \
  --context <workload-context> \
  --type merge \
  -p '{"data":{"level.drbd-node-agent":"debug"}}'

Revert to info when troubleshooting is complete to avoid excessive log volume.

Leader election

Leader election is enabled by default (leaderElect: true). If the manager pod restarts mid-reconciliation, a requeue occurs after the new leader is elected. This is normal; allow 30–60 seconds before concluding a reconciliation loop has stalled.

Webhook certificate

Webhook serving certificates are not managed by the operator. The secret named by webhook.certSecretName (default: webhook-server-cert) must exist in the zone namespace before the admission webhooks become available. A missing or expired certificate causes all CR admission to fail with a TLS handshake error, which appears as every new CR remaining Pending.

Usage

General inspection workflow

For any stuck resource, the sequence is:

  1. Check the resource's status.phase and status.conditions
  2. Identify which controller owns the resource and inspect its logs
  3. Inspect dependent resources on the correct cluster
  4. Resolve the root cause and allow the controller to requeue

All TSR-Manager-side controllers run as reconcilers inside a single manager container in the controller manager Deployment. Because there is one container, pod output shows 1/1.

# Locate the controller manager pod on the TSR-Manager cluster
oc get pods \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane

# Stream logs from the manager container
oc logs -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane \
  -c manager -f

For node-agent issues on a workload cluster:

# List all node agent pods on a workload cluster
oc get pods \
  -n trilio-site-recovery-system \
  --context <workload-context> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent

# Fetch logs from the agent on a specific node (two-step: get pod name, then logs)
oc get pods \
  -n trilio-site-recovery-system \
  --context <workload-context> \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}'

oc logs -n trilio-site-recovery-system \
  --context <workload-context> \
  <pod-name-from-above>

Inspecting ProtectionRequest phases

Valid status.phase values are: Pending, Validating, CreatingDRBD, Syncing, ReadyToActivate, Activating, Protected, Unprotecting, Failed.

# View all ProtectionRequests and their phase
oc get pr -n dr-<name>

# Describe a specific ProtectionRequest for conditions
oc describe pr <name> -n dr-<name>

Inspecting FailoverRequest phases

Valid status.phase values are: Initializing, Validating, RunningPreHooks, StoppingSource, WaitingForLocks, RemovingTaints, StartingTarget, Monitoring, RunningPostHooks, Completed, Failed. Per-VM detail is in status.vmStatuses; there is no status.message or status.failedVMs field.

oc get fr -n dr-<name>
oc describe fr <name> -n dr-<name>

# View per-VM status detail
oc get fr <name> -n dr-<name> -o jsonpath='{.status.vmStatuses}'

Inspecting TestFailover phases

Valid status.phase values are: Queued, PreparingEnvironment, CreatingSnapshots, CreatingVolumes, CreatingVMs, StartingVMs, Verifying, Succeeded, Failed, CleaningUp, Cleaned.

oc get tfo -n dr-<name>

# The test namespace is controller-generated; read it from status, never construct it manually
oc get tfo <name> -n dr-<name> -o jsonpath='{.status.testNamespace}'

Filtering ReplicationGroupStatus and RPOEvents by protection group

Field selectors on spec.* and status.* fields do not work on custom resources. Use the siterecovery.trilio.io/protection-group label instead:

oc get rgs -n dr-<name> -l siterecovery.trilio.io/protection-group=<pg-name>
oc get rpoe -n dr-<name> -l siterecovery.trilio.io/protection-group=<pg-name>
Examples

Example 1: ProtectionRequest stuck in CreatingDRBD

A ProtectionRequest that stays in CreatingDRBD for more than 10 minutes indicates the protection controller cannot provision the DRBDResource pair.

oc --kubeconfig ~/.kube/config-quorum \
  get protectionrequest my-vm-pr -n dr-zone1 -o yaml

Expected output (stuck):

status:
  phase: CreatingDRBD

Check the DRBD resource on both sides:

oc --kubeconfig ~/.kube/config-primary \
  get drbdresource -n default -l vm=my-vm

oc --kubeconfig ~/.kube/config-dr \
  get drbdresource -n default -l vm=my-vm

If the DRBDResource does not exist on either side, check the workload control plane logs on the cluster where it is missing:

oc --kubeconfig ~/.kube/config-primary \
  logs -n dr-zone1 \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --tail=100

Look for errors referencing the VM name or namespace. A missing LVM thin pool on a worker node is the most common cause—verify that every worker node has a block device and thin pool configured as required by the prerequisites.

Example 2: TestFailover stuck in CreatingVolumes

oc --kubeconfig ~/.kube/config-quorum \
  get testfailover my-tf -n dr-zone1 -o yaml

Expected output (stuck):

status:
  phase: CreatingVolumes
  testNamespace: site-recovery-web-tier-pg-a3f9b

Inspect the test namespace the controller created:

oc --kubeconfig ~/.kube/config-dr \
  get pvc -n site-recovery-web-tier-pg-a3f9b

If PVCs are Pending, the DR cluster's storage provisioner cannot fulfil the request. Check that the storage class referenced in the DRBDReplicationPolicy exists on the DR cluster and that the DRBD node agent DaemonSet is healthy:

oc --kubeconfig ~/.kube/config-dr \
  get daemonset -n dr-zone1 \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

Example 3: FailoverRequest stuck in InProgress

oc --kubeconfig ~/.kube/config-quorum \
  get failoverrequest my-fr -n dr-zone1 -o yaml

Expected output (stuck):

status:
  phase: InProgress
  vmStatuses:
    - name: prod-vm-1
      status: Stopping
    - name: prod-vm-2
      status: Stopping

Per-VM detail is in status.vmStatuses. A VM that stays in Stopping is not shutting down cleanly. (spec.drainTimeoutSeconds is accepted by the API but not consumed in Tech Preview, so it is not the threshold being applied.) Check the VM on the source cluster:

oc --kubeconfig ~/.kube/config-primary \
  get vmi prod-vm-1 -n <vm-namespace>

For an unplanned failover where the primary is unavailable, the failover controller force-promotes DRBD volumes without waiting for the primary—if the request is still stuck, check that spec.failoverType is set to unplanned (lowercase):

oc --kubeconfig ~/.kube/config-quorum \
  get failoverrequest my-fr -n dr-zone1 \
  -o jsonpath='{.spec.failoverType}'

Valid values are exactly planned and unplanned. A capitalised or misspelled value will cause the admission webhook to reject the resource.

Example 4: ProtectionGroup in Degraded state blocking failover readiness

oc --kubeconfig ~/.kube/config-primary \
  get protectiongroup web-tier-pg -n default

Expected output:

NAME          STATE    VMS   REPLICATION   HEALTH    AGE
web-tier-pg   Active   3     synchronous   Degraded  4h

Get the detailed status:

oc --kubeconfig ~/.kube/config-primary \
  describe protectiongroup web-tier-pg -n default

Then check for associated RPO events:

oc --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-zone1 \
  --sort-by=.spec.timestamp | tail -10

A Degraded ProtectionGroup with recent critical RPOEvent records indicates the DRBD replication link between primary and DR worker nodes has degraded. Verify TCP port connectivity on ports 7000–8000 between worker nodes on both clusters.

Troubleshooting

Use the following reference for the most common reconciliation failures. Each entry lists the symptom you observe, the likely cause, and the steps to resolve it.

Issue: ProtectionRequest stuck in Pending

Symptom: A ProtectionRequest remains in the Pending phase for more than 5 minutes after creation.

Likely cause: The quorum control plane pod is not running, has lost leader election, or cannot reach the workload cluster API server.

Fix:

  1. Check that the quorum control plane pod is 1/1 Running: 
    oc --kubeconfig ~/.kube/config-quorum get pods \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
  2. If the pod is in CrashLoopBackOff, read its logs: 
    oc --kubeconfig ~/.kube/config-quorum logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager --previous
  3. Verify the leader lease is held by a live pod: 
    oc --kubeconfig ~/.kube/config-quorum get lease -n dr-<zone-name>
    If no pod is renewing the lease, delete the lease to force re-election.
  4. Verify the ProtectionZone spec.clusters entries contain valid kubeconfig references for both workload clusters. An unreachable API server causes the protection controller to requeue indefinitely.

Issue: ProtectionRequest stuck in Validating

Symptom: The ProtectionRequest entered Validating but does not advance to CreatingDRBD.

Likely cause: The protection controller found an issue with the VM or its PVCs—for example, the VM does not exist in the specified namespace, a PVC is not bound, or the VM is already partially protected.

Fix:

  1. Confirm the VM exists on the primary cluster: 
    oc --kubeconfig ~/.kube/config-primary \
  get vm <vm-name> -n <vm-namespace>
  2. Confirm all PVCs attached to the VM are Bound: 
    oc --kubeconfig ~/.kube/config-primary \
  get pvc -n <vm-namespace>
  3. Read the protection controller logs and filter for the VM name to find the validation error: 
    oc --kubeconfig ~/.kube/config-quorum logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager --tail=300 | grep <vm-name>
  4. Correct the underlying issue (bind the PVC, fix the VM name in the spec) and the controller will requeue automatically.

Issue: ProtectionRequest stuck in CreatingDRBD

Symptom: The ProtectionRequest has been in CreatingDRBD for more than 10 minutes.

Likely cause: The DRBD node agent cannot create the backing volume on one or both clusters. Common sub-causes: missing LVM thin pool on a worker node, TCP ports 7000–8000 not open between clusters, or the workload control plane DaemonSet has unhealthy pods.

Fix:

  1. Check the workload DaemonSet on both clusters: 
    oc --kubeconfig ~/.kube/config-primary get daemonset \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

oc --kubeconfig ~/.kube/config-dr get daemonset \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane
  2. Check DRBD node agent pod logs on any unhealthy node: 
    oc --kubeconfig ~/.kube/config-primary logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  --tail=200
  3. Verify network connectivity on DRBD ports from a worker node on the primary to a worker node on DR: 
    # From a primary worker node
nc -zv <dr-worker-node-ip> 7000
  4. Verify LVM thin pool presence on each worker node that will host DRBD volumes.

Issue: ProtectionRequest stuck in Syncing

Symptom: The ProtectionRequest is in Syncing and the DRBDResource shows less than 100 % sync completion for an extended period.

Likely cause: Initial synchronisation of large volumes over a limited-bandwidth link takes time. However, if progress has not advanced at all, the replication link may be broken.

Fix:

  1. Check sync progress on both sides: 
    oc --kubeconfig ~/.kube/config-primary \
  get drbdresource -n <vm-namespace> -o wide

oc --kubeconfig ~/.kube/config-dr \
  get drbdresource -n <vm-namespace> -o wide
  2. Check ReplicationGroupStatus for the ProtectionGroup: 
    oc --kubeconfig ~/.kube/config-quorum \
  get replicationgroupstatus -n dr-<zone-name>
  3. If progress is completely frozen, check for recent RPOEvent records with Critical severity: 
    oc --kubeconfig ~/.kube/config-quorum \
  get rpoevent -n dr-<zone-name> --sort-by=.spec.timestamp
  4. For Protocol C (synchronous), verify round-trip latency between primary and DR worker nodes is below 50 ms. Latency above this threshold causes DRBD to stall writes and halt synchronisation.

Issue: TestFailover stuck in CreatingSnapshots

Symptom: A TestFailover does not advance past CreatingSnapshots.

Likely cause: The volume snapshot class is not available on the DR cluster, or the ProtectionGroup is not in a Consistent state at the time the snapshot is requested.

Fix:

  1. Check that the ProtectionGroup replication is healthy: 
    oc --kubeconfig ~/.kube/config-primary \
  get protectiongroup <pg-name> -n <vm-namespace>
    A Degraded ProtectionGroup prevents the test-failover controller from taking a consistent snapshot.
  2. Check for volume snapshot resources on the DR cluster in the test namespace from status.testNamespace: 
    TF_NS=$(oc --kubeconfig ~/.kube/config-quorum \
  get testfailover <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')

oc --kubeconfig ~/.kube/config-dr \
  get volumesnapshot -n $TF_NS
  3. If snapshots are in a ReadyToUse: false state, check the volume snapshot controller logs on the DR cluster.

Issue: TestFailover stuck in CleaningUp and never reaching Cleaned

Symptom: A TestFailover with spec.cleanupPolicy: Automatic stays in CleaningUp.

Likely cause: A resource in the test namespace (a PVC with a finalizer, a running VMI) is blocking deletion.

Fix:

  1. Get the test namespace: 
    TF_NS=$(oc --kubeconfig ~/.kube/config-quorum \
  get testfailover <name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')
  2. List resources in the test namespace: 
    oc --kubeconfig ~/.kube/config-dr get all,pvc -n $TF_NS
  3. If a VMI is still running, force-stop it: 
    oc --kubeconfig ~/.kube/config-dr \
  delete vmi --all -n $TF_NS --force --grace-period=0
  4. If PVCs have stuck finalizers, patch them: 
    oc --kubeconfig ~/.kube/config-dr \
  patch pvc <pvc-name> -n $TF_NS \
  --type json \
  -p '[{"op":"remove","path":"/metadata/finalizers"}]'
    The test-failover controller will detect the cleared resources and advance to Cleaned.

Issue: FailoverRequest stuck in InProgress with VMs in Stopping

Symptom: A planned FailoverRequest stays in InProgress and status.vmStatuses shows one or more VMs stuck in Stopping. (spec.drainTimeoutSeconds is accepted by the API but not consumed in Tech Preview, so there is no timeout window being enforced.)

Likely cause: A VM is not shutting down gracefully—its guest OS may be hung, or KubeVirt cannot send the shutdown signal.

Fix:

  1. Check the VMI on the source cluster: 
    oc --kubeconfig ~/.kube/config-primary \
  get vmi <vm-name> -n <vm-namespace>
  2. If the VMI is still Running, force-stop it: 
    oc --kubeconfig ~/.kube/config-primary \
  delete vmi <vm-name> -n <vm-namespace> --force --grace-period=0
  3. The failover controller will detect the VM is stopped and advance the reconciliation. If the primary cluster is completely unreachable, consider whether the correct approach is an unplanned failover—set spec.failoverType: unplanned in a new FailoverRequest so the controller force-promotes DRBD volumes without waiting for the primary.

Issue: All reconcilers appear paused simultaneously

Symptom: Multiple CRs across different types all stop advancing at approximately the same time.

Likely cause: The quorum control plane pod lost the leader lease—possibly due to a pod restart, node eviction, or a temporary API server disruption—and the new pod has not yet acquired the lease.

Fix:

  1. Check the current leader lease state: 
    oc --kubeconfig ~/.kube/config-quorum \
  get lease -n dr-<zone-name>
  2. Check that the quorum control plane pod is running: 
    oc --kubeconfig ~/.kube/config-quorum get pods \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
  3. If the pod is running but the lease is stale (not renewed in the last 15+ seconds), delete the lease to force immediate acquisition: 
    oc --kubeconfig ~/.kube/config-quorum \
  delete lease <lease-name> -n dr-<zone-name>
    The running pod will acquire the lease within seconds and all reconcilers will resume.

Issue: Admission webhook rejecting CR creation

Symptom: oc apply returns a webhook validation error immediately; the CR is never created and therefore never enters a reconciliation loop.

Likely cause: A spec field value is invalid. Common cases: storageBackendMode set to a value other than DRBD; FailoverRequest.spec.failoverType capitalised (Planned instead of planned); FailoverRequest.spec.targetCluster set to a value other than cluster1 or cluster2; a required field missing.

Fix:

  1. Read the full webhook rejection message from the oc apply output—it identifies the exact field and constraint.
  2. Correct the field in your manifest. Key constraints:
    • storageBackendMode must be DRBD
    • spec.failoverType must be lowercase planned or unplanned
    • spec.targetCluster must be exactly cluster1 or cluster2
    • TestFailover.spec.cleanupPolicy must be Manual or Automatic
  3. Re-apply the corrected manifest.