Logging

This page explains how Trilio Site Recovery emits, configures, and manages structured logs across its three binary components: the quorum control plane, the workload control plane, and the DRBD node agent. Understanding the logging system helps you diagnose stuck failovers, degraded replication, and controller errors without restarting pods. Log verbosity is controlled at two levels: a startup flag that sets the initial level, and a per-namespace ConfigMap (site-recovery-logging) that lets you change verbosity at runtime on any running component without a pod restart.

Before working with Site Recovery logging, you need:

• A functioning Site Recovery deployment with at least one ProtectionZone in Available phase
• oc CLI access to the quorum cluster (OpenShift 4.17 or later)
• Cluster-admin or equivalent RBAC on the quorum cluster namespace (dr-<name>) and, for node agent logs, on the workload cluster namespace (trilio-site-recovery-system)
• Familiarity with kubectl logs and JSON log parsing (all Site Recovery logs are structured JSON on stdout)

No separate installation is required. The logging subsystem is built into every Site Recovery binary and starts automatically when the operator deploys the control plane components. The site-recovery-logging ConfigMap is created in each zone namespace (dr-<name>) by the ProtectionZone controller when the zone first reaches Available. To verify the ConfigMap exists:

# On the quorum cluster, in the zone namespace
oc get configmap site-recovery-logging -n dr-<zone-name>

If the ConfigMap is absent, the zone has not yet reached Available phase, or it was deleted. Re-reconciliation recreates it — you can force this by making a benign annotation change to the ProtectionZone CR.

To verify logs are flowing from each component:

# Quorum control plane (all reconcilers run in a single manager pod)
oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery \
  --tail=20

# Workload control plane (on the primary or DR cluster)
oc logs -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane,app.kubernetes.io/part-of=site-recovery \
  --tail=20

# DRBD node agent on a specific node (two-step — field selectors only work with get)
POD=$(oc get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
oc logs -n trilio-site-recovery-system "$POD" --tail=20

Log levels

Site Recovery uses a four-level vocabulary. There is no warn level — the underlying logr model uses two classes (error and info), with verbosity integers for sub-levels:

trace can only be set at startup via --zap-log-level=2. All other levels can be set at startup or at runtime via the ConfigMap.

Startup log level (flag)

The initial log level is set by the --zap-log-level flag, controlled by the logging.level Helm value for each chart. Valid startup values are debug, info, error, or an integer (1 = debug, 2 = trace).

The default is info. To change the startup level, update the ProtectionZone spec logging field — the ProtectionZone controller propagates this to the underlying chart:

spec:
  logging:
    level: debug           # applies to all components in this zone
    controlPlaneLevel: debug   # overrides for the quorum/workload control plane only
    nodeAgentLevel: info       # overrides for the DRBD node agent only

Valid values for all logging.* fields: error, info, debug, trace.

Changing the ProtectionZone logging spec triggers a reconcile that updates the site-recovery-logging ConfigMap, which the LevelWatcher in each component reads on its next poll (default: every 15 seconds).

Runtime log level (ConfigMap)

Every component polls site-recovery-logging in its own namespace every 15 seconds and applies the effective level without a pod restart.

ConfigMap name: site-recovery-logging

Location:

• Quorum control plane: namespace dr-<zone-name>
• Workload control plane and DRBD node agent: namespace trilio-site-recovery-system on each workload cluster

ConfigMap keys:

A per-component key wins over the global level key. If the ConfigMap or a key is absent, the component falls back to its startup-flag level. An unrecognized value (for example verbose) is ignored and the current level is kept; a warning is logged once per distinct bad value.

Polling behavior:

• When the ConfigMap sets a level, the component re-asserts that level on every poll, overriding any out-of-band change.
• When the ConfigMap is absent or the key is missing, the component restores the startup-flag level only when transitioning away from a previously ConfigMap-set level. This means that on ConfigMap-less deployments the startup flag remains the effective control.

Log output format

All three binaries emit structured JSON on stdout. Non-JSON output (panics, pre-logger klog, flag errors) goes to stderr. Every line includes at minimum:

Additional structured key-value pairs (e.g., "vm", "namespace", "phase") are appended by the emitting controller and vary by context.

Zone-config tunables (advanced)

The site-recovery-zone-config ConfigMap (distinct from site-recovery-logging) stores per-zone controller timing tunables. These are not log-level settings, but they affect how long controllers wait before surfacing failure conditions in logs:

Edit values in place; controllers re-read on every reconcile (no restart needed). Values are Go durations (e.g., "90s", "2m"). Values below the minimum are clamped up silently; unparseable values fall back to the default.

Changing log level at runtime

To raise the quorum control plane to debug for a troubleshooting session on the quorum cluster:

# Edit the ConfigMap in the zone namespace
oc edit configmap site-recovery-logging -n dr-<zone-name>

Set the level.control-plane key to debug:

data:
  level: info
  level.control-plane: debug

The change takes effect within 15 seconds. To confirm the level changed, look for the log line:

{"level":"info","msg":"runtime log level applied","level":"debug","source":"configmap", ...}

To restore the default, remove the key or set it back to info.

Raising node agent verbosity on a specific workload cluster

The DRBD node agent reads site-recovery-logging from the trilio-site-recovery-system namespace on its own cluster. To raise all node agents on the DR cluster to debug:

# On the DR workload cluster
oc edit configmap site-recovery-logging -n trilio-site-recovery-system

data:
  level.drbd-node-agent: debug

Each agent pod picks up the change independently on its next 15-second poll.

Following logs for a specific controller

All reconcilers (failover, protection, pg-sync, replication-monitor, test-failover) run inside a single manager container in the quorum control plane pod. Filter by the controller key in the JSON output:

oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery \
  -f | grep '"controller":"failoverrequest"'

Following logs for a specific VM or ProtectionRequest

Most log lines emitted by the protection and failover controllers include structured keys for the VM name and namespace. Filter on those:

oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery \
  -f | grep '"virtualMachine":"my-vm"'

Collecting a diagnostic bundle

For support cases, use tsr-gather (the oc adm must-gather compatible tool) to collect logs, CRD state, and configuration from all clusters in the DR estate:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --dest-dir=./tsr-bundle

This collects from the quorum cluster by default and follows kubeconfig references to gather from the workload clusters as well. No -- <command> override is needed; the entrypoint runs automatically.

Updating zone-level logging via the ProtectionZone API

You can also set the global log level for a zone by patching the ProtectionZone CR. The ProtectionZone controller propagates the value to the site-recovery-logging ConfigMap:

oc patch protectionzone <zone-name> -n dr-<zone-name> \
  --type=merge \
  -p '{"spec":{"logging":{"level":"debug"}}}'

To set different levels per component:

oc patch protectionzone <zone-name> -n dr-<zone-name> \
  --type=merge \
  -p '{"spec":{"logging":{"level":"info","controlPlaneLevel":"debug","nodeAgentLevel":"info"}}}'

Example 1: View current effective log level

Read the logging ConfigMap to see what level is currently configured:

oc get configmap site-recovery-logging -n dr-<zone-name> -o yaml

Expected output:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-myzone
data:
  level: info
  level.control-plane: debug

Example 2: Raise quorum control plane to debug, verify, then restore

# Step 1: raise to debug
oc patch configmap site-recovery-logging -n dr-myzone \
  --type=merge \
  -p '{"data":{"level.control-plane":"debug"}}'

# Step 2: wait up to 15s, then confirm in logs
oc logs -n dr-myzone \
  -l app.kubernetes.io/component=quorum-control-plane,app.kubernetes.io/part-of=site-recovery \
  --since=30s | grep 'runtime log level applied'

Expected log line (JSON, formatted for readability):

{
  "level": "info",
  "ts": "2026-07-18T12:00:15.123456789Z",
  "caller": "logging/levelwatcher.go:142",
  "msg": "runtime log level applied",
  "level": "debug",
  "source": "configmap",
  "configMap": "site-recovery-logging"
}

# Step 3: restore to info after troubleshooting
oc patch configmap site-recovery-logging -n dr-myzone \
  --type=merge \
  -p '{"data":{"level.control-plane":"info"}}'

Example 3: Get logs from the node agent on a specific node

# On the workload cluster (primary or DR)
# Step 1: find the agent pod for the node
POD=$(oc get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=worker-0.example.com \
  -l app.kubernetes.io/component=drbd-node-agent,app.kubernetes.io/part-of=site-recovery \
  -o jsonpath='{.items[0].metadata.name}')

# Step 2: tail logs from that pod
oc logs -n trilio-site-recovery-system "$POD" --tail=50 -f

Expected output format (one JSON object per line):

{"level":"info","ts":"2026-07-18T12:01:00.000000000Z","caller":"reconciler/reconciler.go:212","msg":"reconciling DRBDResource","drbdResource":"drbd-appns-myvm-a1b2c3d4","node":"worker-0.example.com"}

Example 4: Collect a full diagnostic bundle

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --dest-dir=./tsr-bundle-$(date +%Y%m%d)

The bundle is written to ./tsr-bundle-<date>/ and contains:

• Logs from all Site Recovery pods across all clusters
• CRD state (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, DRBDResource, ReplicationGroupStatus, RPOEvent)
• ConfigMaps (site-recovery-logging, site-recovery-zone-config)
• Events from all Site Recovery namespaces

Example 5: Adjust zone-config deadline for slow CSI provisioners

If your storage provisioner is slower than the default 10-minute deadline, increase the failure deadline to avoid false protection failures appearing in logs:

oc edit configmap site-recovery-zone-config -n dr-myzone

data:
  drbdMetadataReadyGracePeriod: "3m"
  drbdMetadataProvisioningFailureDeadline: "20m"
  README: "..."

Controllers pick up the change on the next reconcile — no restart needed.

Issue: Log level does not change after editing the ConfigMap

Symptom: You edited site-recovery-logging but the component continues emitting at the old level after 15 seconds.

Likely cause 1: The ConfigMap key contains an unrecognized value (e.g., verbose instead of debug). The component ignores unrecognized values and logs a warning once.

Fix: Check the component logs for a line containing "unrecognized log level value". Set the key to one of the four valid values: error, info, debug, trace.

Likely cause 2: You edited the ConfigMap in the wrong namespace. The quorum control plane reads from dr-<zone-name>; the workload control plane and node agent read from trilio-site-recovery-system on their respective clusters.

Fix: Verify the namespace:

# Quorum control plane reads from:
oc get configmap site-recovery-logging -n dr-<zone-name>

# Workload components read from (on each workload cluster):
oc get configmap site-recovery-logging -n trilio-site-recovery-system

Issue: site-recovery-logging ConfigMap does not exist

Symptom: oc get configmap site-recovery-logging -n dr-<zone-name> returns NotFound.

Likely cause: The ProtectionZone has not reached Available phase, or the ConfigMap was deleted manually.

Fix: Check the ProtectionZone phase:

oc get pz -n dr-<zone-name>

If the phase is not Available, resolve any zone condition errors first. If the zone is Available but the ConfigMap is missing, trigger a reconcile by patching the zone:

oc annotate protectionzone <zone-name> -n dr-<zone-name> \
  siterecovery.trilio.io/reconcile-trigger=$(date +%s) --overwrite

Issue: Cannot set trace level via ConfigMap

Symptom: You set level: trace in the ConfigMap but the component logs at debug, not trace.

Likely cause: trace (logr V(2), zap level -2) is reachable via the ConfigMap and the LevelWatcher applies it correctly. However, this is only possible if the component was started at a level that permits it. If the startup flag was --zap-log-level=error, the AtomicLevel may not descend to trace via the CM path on some controller-runtime versions.

Fix: For trace level, set the startup flag explicitly. Update the ProtectionZone logging spec:

oc patch protectionzone <zone-name> -n dr-<zone-name> \
  --type=merge \
  -p '{"spec":{"logging":{"level":"trace"}}}'

This triggers a rollout with --zap-log-level=2, which sets trace at startup. The ConfigMap then re-asserts it on each poll.

Issue: Node agent logs are missing or sparse

Symptom: oc logs for a node agent pod returns very little output even at info level.

Likely cause 1: The node agent pod has not started because the siterecovery.trilio.io/not-ready:NoSchedule taint on the node is blocking pod scheduling on this node. The taint is applied by the agent during DRBD initialization and removed once DRBD converges.

Fix: Check whether the agent pod is actually running:

oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent,app.kubernetes.io/part-of=site-recovery \
  -o wide

If the pod is Pending, check for taint-related scheduling events:

oc describe pod <pod-name> -n trilio-site-recovery-system

Likely cause 2: You are looking at the wrong pod. Use the two-step field-selector approach to find the pod on your target node:

POD=$(oc get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent,app.kubernetes.io/part-of=site-recovery \
  -o jsonpath='{.items[0].metadata.name}')
echo "Agent pod: $POD"
oc logs -n trilio-site-recovery-system "$POD" --tail=100

Issue: Status patch rejection events appear in kubectl describe

Symptom: kubectl describe on a DRBDResource or ProtectionRequest shows Warning StatusPatchRejected events.

Likely cause: The API server rejected a status write from the controller, typically due to a CRD schema drift (a field the controller tried to write is not recognized by the installed CRD version).

Fix: The event message contains the API server's verbatim rejection text, including the field name. Collect the full diagnostic bundle to share with support:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --dest-dir=./tsr-bundle

Then check whether the CRD versions on the quorum cluster match the installed operator version:

oc get crd protectionrequests.siterecovery.trilio.io -o jsonpath='{.metadata.annotations.site-recovery-pz-controller}'

A mismatch indicates the CRDs were not updated during the last operator upgrade.

Issue: oc adm must-gather fails to pull the image

Symptom: oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather fails with an image pull error.

Likely cause: The quorum cluster nodes do not have egress access to quay.io/triliovault.

Fix: Ensure your cluster's image pull configuration permits access to quay.io/triliovault. If you are using a private registry mirror, substitute the mirrored image reference. Do not attempt to pull from eu.gcr.io — that is a Trilio-internal registry not accessible to customers.