Troubleshooting

Diagnose and resolve common controller errors, log analysis failures, and stuck or degraded states in Trilio Site Recovery — use this runbook when a ProtectionRequest, FailoverRequest, TestFailover, or ReplicationGroupStatus resource is stuck, failed, or showing unexpected behaviour.

This runbook covers:

• Reading controller logs on the quorum and workload clusters
• Diagnosing stuck or failed ProtectionRequests, FailoverRequests, and TestFailovers
• Interpreting DRBD replication health from DRBDResource and ReplicationGroupStatus
• Collecting a diagnostic bundle with tsr-gather
• Runtime log-level tuning

This runbook does not cover:

• Initial installation or zone provisioning (see the deployment runbooks)
• Node loss recovery (see docs/runbooks/drbdresource-node-loss.md)
• Re-protecting a VM after unprotect (see docs/runbooks/drbdresource-reprotect.md)
• Zone-cluster lease conflicts (see docs/runbooks/tsr-52-zone-cluster-lease.md)
• DRBD kernel module loading failures (see docs/runbooks/drbd-module-loading.md)

Before using this runbook, ensure:

• You have oc CLI access to the quorum cluster (OpenShift 4.17 or later — the supported floor for all three clusters)
• You hold sufficient RBAC to read pod logs, CRs, ConfigMaps, and Events across the relevant namespaces. The trilio-site-recovery-admin ClusterRole (or a RoleBinding to it in the zone namespace) is the minimum required.
• You know your zone name; quorum-side components and CRs live in the namespace dr-<zone-name>, not in trilio-site-recovery-system. Workload-side components (DRBD node agents, DRBDResource CRs) live in trilio-site-recovery-system on each workload cluster.
• You have kubeconfig credentials for the quorum cluster and, where noted, for the primary and DR workload clusters.
• tsr-gather is available — it is shipped as an oc adm must-gather image at quay.io/triliovault/site-recovery-must-gather:<tag>.

1. Collect a support bundle first

Collect a diagnostic bundle from the quorum cluster before making any changes. Pod logs are bounded and a restarted container loses its previous log.

# Whole estate — quorum + both workload clusters for every zone
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag>

# Tighter log window (faster; use when the incident is recent)
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

# One zone only
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --zone <zone-name>

Read SUMMARY.md first (overall zone state, replication health, per-VM consistency, recent failovers, warnings), then manifest.yaml (per-cluster inventory and collection status).

2. Identify the quorum manager pod

All quorum-side reconcilers — failover, protection, pg-sync, replication-monitor, test-failover — run in a single manager container in the quorum control-plane Deployment. The Deployment lives in the zone namespace (dr-<zone-name>).

# List quorum control-plane pods — output is 1/1 (one container named manager)
oc -n dr-<zone-name> get pods \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

Expected output shape:

NAME                                                        READY   STATUS    RESTARTS
<release>-site-recovery-quorum-control-plane-xxxx-yyyy      1/1     Running   0

If status is not Running, proceed to step 3.

3. Read quorum controller logs

# Stream logs from the quorum manager container
oc -n dr-<zone-name> logs \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager --tail=200 -f

# For a specific CR (e.g. a stuck ProtectionRequest named protect-my-vm)
oc -n dr-<zone-name> logs \
  -l app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep protect-my-vm

What to look for:

• ERROR lines with a reconciler name (protectionrequest, failoverrequest, testfailover)
• Repeated reconcile loops on the same object name (indicates a retrying error)
• phase= values — these show which lifecycle phase the controller last processed

4. Read workload node-agent logs

The DRBD node agent runs as a DaemonSet on every worker node of each workload cluster. Use two commands: get pods with a field selector to find the pod on a specific node, then logs.

# 1. Find the agent pod on a specific node
oc --kubeconfig <workload-kubeconfig> -n trilio-site-recovery-system \
  get pods --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}'

# 2. Read its logs
oc --kubeconfig <workload-kubeconfig> -n trilio-site-recovery-system \
  logs <pod-name-from-step-1> --tail=200

What to look for:

• DRBD connection state changes (Connected, Disconnected, SyncSource, SyncTarget)
• drbdadm errors or exit codes
• not-ready taint application or removal
• Metadata PVC provisioning failures

5. Increase log verbosity at runtime (no restart required)

Log levels are tuned at runtime via the site-recovery-logging ConfigMap in the zone namespace. Valid levels: error, info, debug, trace. Changes take effect within approximately 15 seconds without restarting pods.

# Increase all zone components to debug
oc -n dr-<zone-name> patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"debug"}}'

# Increase only the control plane to debug, leaving agents at info
oc -n dr-<zone-name> patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"info","level.control-plane":"debug"}}'

# Increase only the DRBD node agents to debug
oc -n dr-<zone-name> patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"info","level.drbd-node-agent":"debug"}}'

# Restore to info across all components
oc -n dr-<zone-name> patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"info"}}'

Alternatively, set the log level persistently via the ProtectionZone CR (the controller propagates this to the ConfigMap):

oc -n dr-<zone-name> patch protectionzone <zone-name> --type=merge \
  -p '{"spec":{"logging":{"level":"debug"}}}'

6. Diagnose a stuck or failed ProtectionRequest

6a. Read the phase and conditions

# Summary view — columns: VM, Namespace, Phase, Sync, Age
oc -n dr-<zone-name> get pr

# Full status for a specific request
oc -n dr-<zone-name> get pr <name> -o yaml

Valid status.phase values (in lifecycle order): Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected Teardown path: Unprotecting Terminal failure: Failed

There is no SwitchingPVCs, DrainFailed, or CreatingDRBDResource phase. If you see those names in older notes, they are stale.

6b. Phase-specific diagnostics

6c. Check the Consistent condition

The Consistent condition on a ProtectionRequest is the authoritative per-VM verdict that the DR copy is usable (Connected + every volume UpToDate on both sides). It is tri-state: always gate on True, never on "not False" — Unknown means the controller could not observe, not that everything is fine.

oc -n dr-<zone-name> get pr <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="Consistent")]}{.status}: {.reason} — {.message}{"\n"}{end}'

Caveat: the Consistent condition is only refreshed while the request is in Syncing or Protected. In ReadyToActivate or Activating it carries the last value Syncing computed and may be stale.

6d. Check DRBDResource on the workload cluster

DRBDResource CRs live on workload clusters only — oc get dres against the quorum cluster returns a server doesn't have a resource type error.

# On the primary workload cluster, in the VM's own namespace
oc --kubeconfig <primary-kubeconfig> -n <vm-namespace> get dres
# Columns: Side, Role, Node, State (connectionState), Phase

# Full detail
oc --kubeconfig <primary-kubeconfig> -n <vm-namespace> get dres <name> -o yaml

DRBDResource naming is deterministic: drbd-<namespace>-<vm-name>-<hash8>. Do not append -primary, -dr, or -src — those suffixes do not exist.

Healthy state: status.connectionState: Connected, status.phase: Consistent, status.role: Primary (on the active cluster) or Secondary (on the DR cluster).

7. Diagnose a stuck or failed FailoverRequest

7a. Read the state and phase

# Summary view — columns: Protection Group, Target, State, Phase, Age
oc -n dr-<zone-name> get fr

# Per-VM detail (no `message` or `failedVMs` field exists — use vmStatuses)
oc -n dr-<zone-name> get fr <name> \
  -o jsonpath='{range .status.vmStatuses[*]}{.name}: {.phase} (source={.sourceStatus}, target={.targetStatus}){"\n"}{end}'

Valid status.state values: Pending, InProgress, Completed, Failed, Cancelled

Valid status.phase values: Initializing, Validating, StoppingSource, WaitingForLocks, RemovingTaints, StartingTarget, Monitoring, Completed, Failed

spec.failoverType must be lowercase planned or unplanned. spec.targetCluster must be exactly cluster1 or cluster2 — no free-form cluster names.

spec.drainTimeoutSeconds is accepted and range-checked by the admission webhook but has no effect on drain or failover behaviour in Tech Preview. Do not include it in example manifests or rely on it for timing control.

7b. Phase-specific diagnostics

7c. Safety constraints — do not violate these

⚠️ Do not delete a stuck FailoverRequest while VMs are in an unknown state. The failover controller is re-entrant, not transactional — it resumes from where it stopped. Deleting the CR while VMs are mid-transition leaves VMs in an unrecoverable split-brain state.

⚠️ Do not start VMs on the primary cluster after the DR cluster has been promoted. DRBD will have two Primary copies of the same volume, causing data corruption. Force-promoting the DR side without gracefully shutting down the source is the unplanned failover path — use it only when the primary is genuinely unreachable.

8. Diagnose a stuck or failed TestFailover

8a. Read the phase and conditions

# Summary view — columns: Protection Group, Phase, Passed, Age
oc -n dr-<zone-name> get tfo

# Full status
oc -n dr-<zone-name> get tfo <name> -o yaml

Valid status.phase values (in order): Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded Cleanup path: CleaningUp → Cleaned Terminal failure: Failed

There is no Pending phase or VerifyingData phase. tfo is the correct short name — not tf.

8b. Find the generated test namespace

The test namespace is generated by the controller as site-recovery-<protection-group>-<random> and is never user-provided. Read it from the CR — do not construct or guess it:

oc -n dr-<zone-name> get tfo <name> \
  -o jsonpath='{.status.testNamespace}{"\n"}'

Resources on the target workload cluster (test VMs, PVCs, snapshots) live in this namespace.

8c. Phase-specific diagnostics

# Read verification check results
oc -n dr-<zone-name> get tfo <name> \
  -o jsonpath='{range .status.verificationResults.checks[*]}{.vm}: {.type} passed={.passed} — {.message}{"\n"}{end}'

# Check target cluster selection signal
oc -n dr-<zone-name> get tfo <name> \
  -o jsonpath='{.status.targetCluster} (reason: {.status.targetClusterReason}){"\n"}'

8d. Cleanup

To clean up test resources, delete the TestFailover CR. The controller runs cleanup before releasing its finalizer — do not patch the finalizer directly.

oc -n dr-<zone-name> delete tfo <name>

For a TestFailover with cleanupPolicy: Automatic, resources are cleaned up automatically after retentionTime (default 2h) has elapsed since completion. With cleanupPolicy: Manual, deletion of the CR triggers cleanup.

9. Diagnose replication health

9a. ReplicationGroupStatus

The replication monitor auto-creates exactly one ReplicationGroupStatus per ProtectionGroup, named <protection-group-name>-replication. Use the short name rgs.

# Summary — columns: Protection-Group, RPO-Status, Out-Of-Sync, Synced, Connection, Last-Sync, Age
oc -n dr-<zone-name> get rgs

# Aggregated health for a specific group
oc -n dr-<zone-name> get rgs <pg-name>-replication -o yaml

Key fields:

• status.aggregated.connectionState: Connected (healthy), Degraded, or Disconnected
• status.aggregated.outOfSyncBytes: non-zero means replication lag
• status.aggregated.syncedVolumeCount vs status.aggregated.totalVolumeCount
• status.rpoStatus: Met, Degraded, Violated, or Unknown

Filter ReplicationGroupStatus objects by protection group using the label siterecovery.trilio.io/protection-group=<pg-name> — field selectors on spec.* or status.* do not work on custom resources.

oc -n dr-<zone-name> get rgs \
  -l siterecovery.trilio.io/protection-group=<pg-name>

9b. RPOEvents

RPO violations and connection events are recorded as immutable RPOEvent CRs. Use the short name rpoe.

# All RPO events for a zone
oc -n dr-<zone-name> get rpoe \
  -l siterecovery.trilio.io/protection-group=<pg-name>

# Event details — columns: Protection-Group, Event-Type, RPO-Seconds, Out-Of-Sync, Reason, Timestamp
oc -n dr-<zone-name> get rpoe

9c. DRBDResource replication state

Do not query DRBDResources against the quorum cluster — the drbdresources CRD is only registered on workload clusters.

# Check per-volume replication state on the primary workload cluster
oc --kubeconfig <primary-kubeconfig> -n <vm-namespace> get dres <name> \
  -o jsonpath='{range .status.volumes[*]}vol{.volumeNumber}: diskState={.diskState} replicationState={.replicationState} syncProgress={.syncProgress.percentage}%{"\n"}{end}'

Healthy per-volume replicationState is Established (not Connected — that was renamed in the TSR-90 release and older triage notes using Connected as a healthy state are stale on current deployments).

10. Diagnose the ProtectionZone

# Zone status — columns: Display Name, Backend, Clusters, Status, Age
oc -n dr-<zone-name> get pz

# Cluster reachability
oc -n dr-<zone-name> get pz <zone-name> \
  -o jsonpath='{range .status.clusters[*]}{.name} ({.clusterID}): {.status} — {.message}{"\n"}{end}'

# Conditions
oc -n dr-<zone-name> get pz <zone-name> \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.reason} — {.message}{"\n"}{end}'

Expected status.phase when healthy: Available.

If the zone shows Degraded or Unavailable, check:

1. That the kubeconfig Secrets for both clusters exist and are valid in the zone namespace
2. That trilio-site-recovery-system namespace exists on both workload clusters (the zone controller probes this namespace by name to confirm reachability)
3. Quorum manager logs for ClustersValidated=False reason

11. Check the DRBD node-agent DaemonSet

# On each workload cluster — DaemonSet name pattern: <release>-site-recovery-workload-control-plane-agent
oc --kubeconfig <workload-kubeconfig> -n trilio-site-recovery-system \
  get daemonset -l app.kubernetes.io/component=drbd-node-agent

# Check desired vs ready count
oc --kubeconfig <workload-kubeconfig> -n trilio-site-recovery-system \
  get daemonset \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].status.desiredNumberScheduled}/{.items[0].status.numberReady}{"\n"}'

⚠️ The DaemonSet object name is <release>-site-recovery-workload-control-plane-agent. Do not use drbd-node-agent as an object name — kubectl get daemonset drbd-node-agent returns NotFound. drbd-node-agent is the component label value and the binary name, not the object name.

If agents are in Init:Error or Init:CrashLoopBackOff, the DRBD kernel module loader failed. Read the init container log:

# Find the affected pod first (see step 4 above), then:
oc --kubeconfig <workload-kubeconfig> -n trilio-site-recovery-system \
  logs <pod-name> -c drbd-module-loader

See docs/runbooks/drbd-module-loading.md for module-loading failure remediation.

A node carrying the siterecovery.trilio.io/not-ready:NoSchedule taint means DRBD has not yet converged on that node. This is applied by the agent at startup and removed once all DRBD resources on the node are connected. Workloads that do not tolerate this taint will not schedule until the taint is lifted.

12. Check for protected VMs

The label siterecovery.trilio.io/protected-vm=true is stamped on the KubeVirt VirtualMachine object only — on workload clusters, in the VM's namespace. It is not present on DRBDResource, ProtectionRequest, or any other Site Recovery CR.

# List all protected VMs on a workload cluster
oc --kubeconfig <workload-kubeconfig> get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

13. Common admission webhook errors

If a CR creation is rejected, the webhook error text is the first diagnostic:

14. Force-delete a stuck DRBDResource (emergency only)

If a DRBDResource is stuck in Deleting because its bound node is gone, follow docs/runbooks/drbdresource-node-loss.md. In an emergency, the siterecovery.trilio.io/force-delete annotation overrides the in-use deletion guard:

# Read the runbook before using this — removing the wrong finalizer leaks storage
oc --kubeconfig <workload-kubeconfig> -n <vm-namespace> annotate dres <name> \
  siterecovery.trilio.io/force-delete=true

Do not patch both finalizers away simultaneously. See docs/runbooks/drbdresource-node-loss.md §3 and §4 for the exact procedure.

After applying any change in this runbook, confirm the expected outcome:

ProtectionRequest recovery:

# Watch until phase reaches Protected
oc -n dr-<zone-name> get pr <name> -w
# Expected: phase=Protected, Sync=100

FailoverRequest completion:

# Watch state and phase
oc -n dr-<zone-name> get fr <name> -w
# Expected: state=Completed, phase=Completed

TestFailover success:

# Watch until Succeeded or Cleaned
oc -n dr-<zone-name> get tfo <name> -w
# Expected: phase=Succeeded, Passed=true

Replication health:

# ReplicationGroupStatus should show Connected and Met
oc -n dr-<zone-name> get rgs
# Expected: Connection=Connected, RPO-Status=Met, Out-Of-Sync=0

Zone health:

oc -n dr-<zone-name> get pz
# Expected: Status=Available

Node agent convergence:

# All nodes should show the taint removed
oc --kubeconfig <workload-kubeconfig> get nodes \
  -o jsonpath='{range .items[*]}{.metadata.name}: {range .spec.taints[*]}{.key}={.effect} {end}{"\n"}{end}' \
  | grep not-ready
# Expected: no output (taint lifted on all nodes)

Most operations in this runbook are read-only (log collection, status inspection). For the changes that can be undone:

Log level change via ConfigMap:

# Restore to info
oc -n dr-<zone-name> patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"info"}}'

# Or restore via the ProtectionZone CR
oc -n dr-<zone-name> patch protectionzone <zone-name> --type=merge \
  -p '{"spec":{"logging":{"level":"info"}}}'

Stuck FailoverRequest: If a failover was started in error and has not yet reached StoppingSource, delete the FailoverRequest:

oc -n dr-<zone-name> delete fr <name>

⚠️ Do not delete a FailoverRequest once VMs are in an intermediate state (stopping, stopped, or starting on the target). The controller is re-entrant but not transactional — deletion mid-flight leaves VMs in an unknown state. If this has occurred, create a new FailoverRequest to fail back to the original cluster.

Force-delete annotation: If you applied siterecovery.trilio.io/force-delete to a DRBDResource and it was incorrect, remove it immediately:

oc --kubeconfig <workload-kubeconfig> -n <vm-namespace> annotate dres <name> \
  siterecovery.trilio.io/force-delete-

TestFailover: A TestFailover can always be cleaned up by deletion — the finalizer ensures workload resources are removed first:

oc -n dr-<zone-name> delete tfo <name>

If this runbook does not resolve the issue, escalate to Trilio Support with the following information:

1. Support bundle collected from the quorum cluster (step 1 of this runbook). Include the SUMMARY.md and manifest.yaml files along with the full bundle archive.

2. Reproduction steps: the exact sequence of kubectl/oc commands run and the CRs applied, including full YAML for any ProtectionRequest, FailoverRequest, TestFailover, or ProtectionZone involved.

3. Cluster details:

   • OpenShift version on the quorum cluster and both workload clusters (oc version)
   • Output of oc -n dr-<zone-name> get pz -o yaml
   • Output of oc -n dr-<zone-name> get pr,fr,tfo,pg -o wide

4. Symptoms and timeline: when the issue first appeared, what changed immediately before (upgrades, node reboots, network events), and what errors are visible on the affected CRs.

5. Quorum manager logs for at least 30 minutes around the time of failure:

   Copy

   oc -n dr-<zone-name> logs \
     -l app.kubernetes.io/component=quorum-control-plane \
     -c manager --since 30m > quorum-manager.log
   

6. Node-agent logs for any node showing DRBD issues (step 4 of this runbook).

7. For replication issues: the full YAML of the affected ReplicationGroupStatus and any RPOEvent CRs:

   Copy

   oc -n dr-<zone-name> get rgs -o yaml > rgs.yaml
   oc -n dr-<zone-name> get rpoe -o yaml > rpoe.yaml