Logging

This page describes how Trilio Site Recovery emits structured logs, how you control log verbosity at runtime, and how to integrate those logs with your existing aggregation pipeline. Every component — the manager control plane, the DRBD node agent, and the protection zone controller — writes single-line JSON to stdout using a shared logging stack (zap on the Go side, structlog on the Python Flask backend). Because all three stacks share the same output channel and key vocabulary, your collector can treat the entire DR estate as a homogeneous log source. Understanding log levels and the runtime adjustment mechanism is especially important when diagnosing a stuck failover or degraded replication link, where switching a running pod from info to debug without a restart is the fastest way to capture per-phase detail.

Before configuring logging for Trilio Site Recovery, confirm the following:

• Trilio Site Recovery operator installed via OLM on your TSR-Manager cluster (OpenShift 4.17 or later).
• A ProtectionZone CR in Available phase — the manager control plane and DRBD node agents must be running before log-level changes take effect.
• oc CLI authenticated to the TSR-Manager cluster and, where agent logs are needed, to each workload cluster.
• Read/write access to ConfigMaps in the zone namespace (dr-<name>) on the TSR-Manager cluster — required to modify the site-recovery-logging ConfigMap.
• Familiarity with oc logs and, optionally, your organisation's log aggregation tooling (for example, OpenShift Logging / Loki, Elasticsearch, or Splunk).

Logging is built into every Trilio Site Recovery component. There is nothing to install separately. The steps below describe how to verify that structured logging is active and where to direct your collector.

Step 1 — Confirm the manager control plane is emitting JSON

Run this against your TSR-Manager cluster, substituting your zone namespace:

oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  --tail=5

Each line should be a single-line JSON object with at minimum the keys ts, level, msg, logger, and caller. If you see console-format (human-readable) output instead, check the ProtectionZone.spec.logging field — the json encoder is the production default and reverts automatically on the next reconcile.

Step 2 — Confirm the DRBD node agent is emitting JSON

Run this against a workload cluster worker node:

# Identify the agent pod on a specific node
AGENT_POD=$(oc get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<worker-node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')

oc logs -n trilio-site-recovery-system "$AGENT_POD" --tail=5

Output should be single-line JSON to stdout, matching the same key vocabulary as the manager.

Step 3 — Point your log collector at stdout

All three stacks (Go manager, Go node agent, Python Flask backend) write structured JSON exclusively to stdout. Anything on stderr is abnormal (panics, pre-logger output, flag errors) and should trigger an alert in its own right. Configure your collector sidecar or DaemonSet to scrape stdout from the following label selectors:

Log verbosity is controlled at two layers: a startup flag baked into the pod spec, and a runtime ConfigMap polled by every running component. The ProtectionZone CR provides the authoritative knob for both.

Log levels

Trilio Site Recovery uses a four-level vocabulary:

Note: there is no warn level. The logging stack uses a two-class model (info/error), with debug and trace expressed as logr verbosity integers (V(1) and V(2) respectively).

Setting the level via ProtectionZone (recommended)

The ProtectionZone.spec.logging field is the preferred way to control verbosity across all components in a zone. The ProtectionZone controller propagates the setting to the site-recovery-logging ConfigMap in the zone namespace, from which every component polls the effective level.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: my-zone
  namespace: dr-my-zone
spec:
  displayName: "My DR Zone"
  storageBackendMode: DRBD
  logging:
    level: info           # global default: error | info | debug | trace
    controlPlaneLevel: debug  # override for the manager control plane only
    nodeAgentLevel: info      # override for the DRBD node agent only
  clusters:
    # ...

spec.logging.level sets the global default for all components in the zone. spec.logging.controlPlaneLevel and spec.logging.nodeAgentLevel override the global default for their respective components. An absent override key inherits the global level.

Valid values for all three fields: error | info | debug | trace.

Setting the level via the ConfigMap directly

If you need to change the level without editing the ProtectionZone CR — for example, during an incident when you want the fastest possible path — edit the site-recovery-logging ConfigMap in the zone namespace on the TSR-Manager cluster:

oc edit configmap site-recovery-logging -n dr-<zone-name>

The ConfigMap data keys are:

Example data section:

data:
  level: info
  level.control-plane: debug

Every running component polls this ConfigMap every 15 seconds using an uncached API read. A level change propagates within one poll interval without any pod restart. If the ConfigMap is absent or the relevant key is missing, the component falls back to its startup flag level.

An invalid level value (anything other than error, info, debug, or trace) is logged as an error once per distinct bad value and the current level is preserved.

Important: When the ProtectionZone controller reconciles, it overwrites the site-recovery-logging ConfigMap from ProtectionZone.spec.logging. If you have edited the ConfigMap directly, update the ProtectionZone CR as well to prevent your change from being reverted.

Setting the level via the TSR web console API

The TSR web console backend exposes a logging endpoint for each protection zone:

PUT /api/protection-zones/<zone_id>/logging

This is the same operation the web console UI performs when you change the log level from the zone settings page. It propagates the change to the underlying site-recovery-logging ConfigMap. Browser access to the console is supported on OpenShift only (via the oauth-proxy Route created by the ProtectionZone controller).

Trace level at startup

The trace level (logr V(2)) can be set at runtime via the ConfigMap. It can also be set at startup by passing --zap-log-level=2 to the component binary, which requires a pod rollout. Under normal operations, reaching trace via the ConfigMap is preferable because it avoids a restart.

Log output format

All Go components use the production zap preset:

• Encoder: JSON (single-line per event)
• Output: stdout
• Timestamp key: ts, RFC3339Nano precision (sub-second ordering is preserved)
• Level key: level, lowercase: error | info | debug | trace
• Caller key: caller, short file:line form
• Stack traces: attached at error level under the stacktrace key

The Python Flask backend (site-recovery-backend) uses structlog with the same cross-stack contract:

• Encoder: JSON to stdout (console renderer only when FLASK_ENV=development)
• Timestamp key: timestamp, RFC3339 UTC
• Level key: level, lowercase
• Logger key: logger
• Request-scoped key: request_id
• Stack trace key: stacktrace (not exception)

Gunicorn error records flow through the same structlog pipeline, so they appear as JSON on stdout rather than raw text on stderr.

Increasing verbosity to diagnose an incident

The most common reason to change log verbosity is to diagnose a stuck failover or degraded replication link. The recommended workflow is:

1. Patch ProtectionZone.spec.logging.controlPlaneLevel to debug on the TSR-Manager cluster.
2. Wait up to 15 seconds for the level watcher to apply the change.
3. Reproduce or observe the failing operation.
4. Collect logs.
5. Restore controlPlaneLevel to info (or remove the field to inherit level).

# Increase the manager control plane to debug without restarting pods
oc patch protectionzone my-zone -n dr-my-zone --type=merge \
  -p '{"spec":{"logging":{"controlPlaneLevel":"debug"}}}'

# Confirm the level was applied (within ~15 seconds)
oc logs -n dr-my-zone \
  -l app.kubernetes.io/component=manager-control-plane \
  --since=30s | grep 'runtime log level applied'

# Restore to info when done
oc patch protectionzone my-zone -n dr-my-zone --type=merge \
  -p '{"spec":{"logging":{"controlPlaneLevel":"info"}}}'

Tailing manager control plane logs

oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  -f

The manager pod runs a single container named manager. The label selector above targets all manager pods in the zone namespace.

Tailing DRBD node agent logs for a specific node

Field selectors on Pod spec.nodeName are supported. Use two commands: one to resolve the pod name, one to tail:

# Step 1: find the agent pod on the target node (workload cluster)
AGENT_POD=$(oc get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<worker-node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')

# Step 2: tail its logs
oc logs -n trilio-site-recovery-system "$AGENT_POD" -f

Note: Run this against the workload cluster kubeconfig, not the TSR-Manager cluster. The DaemonSet object name follows the pattern <release>-site-recovery-workload-control-plane-agent; kubectl get daemonset drbd-node-agent returns NotFound.

Collecting a full diagnostic bundle

For a support escalation, use oc adm must-gather with the Trilio must-gather image. This collects logs, CRD state, and configuration from all clusters in the DR estate into a single bundle:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -n dr-<zone-name>

The gather binary inside the image runs automatically as the ENTRYPOINT; no -- <command> override is needed. Replace <release-tag> with the pinned tag for your installed version.

Filtering logs by protection group

Logs for operations on a specific protection group carry the protection group name as a structured field. Use your log aggregation tool's JSON filter or jq to narrow the output:

oc logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  --since=1h \
| jq 'select(.protectionGroup == "<pg-name>")"

Adjust the field name to match the key emitted by the specific reconciler you are inspecting.

Example 1 — ProtectionZone with per-component log levels

This zone sets the global level to info but bumps the manager control plane to debug for a deployment that is actively being tested:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod-zone
  namespace: dr-prod-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  logging:
    level: info
    controlPlaneLevel: debug
    nodeAgentLevel: info
  clusters:
    - name: primary
      clusterID: cluster1
      kubeconfigSecretRef:
        name: primary-kubeconfig
        key: kubeconfig
    - name: dr-site
      clusterID: cluster2
      kubeconfigSecretRef:
        name: dr-kubeconfig
        key: kubeconfig

Example 2 — site-recovery-logging ConfigMap after controller propagation

After the ProtectionZone controller reconciles the zone above, the ConfigMap in dr-prod-zone will contain:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-prod-zone
data:
  level: info
  level.control-plane: debug
  level.drbd-node-agent: info

Example 3 — Patching to trace for a severe incident

# Enable trace on the node agent for both workload clusters simultaneously
# (edit the ProtectionZone; the controller propagates to both clusters)
oc patch protectionzone prod-zone -n dr-prod-zone --type=merge \
  -p '{"spec":{"logging":{"nodeAgentLevel":"trace"}}}'

Warning: trace generates high log volume. Restore to info as soon as you have captured the data you need.

Example 4 — Confirming a level change was applied

After patching the ProtectionZone or ConfigMap, confirm the watcher applied the change by searching for the transition log line:

oc logs -n dr-prod-zone \
  -l app.kubernetes.io/component=manager-control-plane \
  --since=1m \
| grep 'runtime log level applied'

The log line is emitted by the level watcher after each successful level transition and includes the level, source, and configMap fields.

Example 5 — Changing log level via the backend API (OpenShift only)

If you have access to the TSR web console backend (OpenShift, authenticated via oauth-proxy), you can adjust logging for a zone over the API:

curl -X PUT \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <token>' \
  -d '{"level": "debug"}' \
  https://<console-route-host>/api/protection-zones/<zone_id>/logging

Replace <console-route-host> with the Route hostname created by the ProtectionZone controller, and <zone_id> with your zone's name.

Issue 1: Log output is not JSON — lines are human-readable or mixed format

Symptom: oc logs shows coloured, human-readable lines instead of single-line JSON objects.

Likely cause: The Flask backend is running with FLASK_ENV=development, which enables the structlog console renderer. On the Go side this cannot happen in production because Development: false is the hardcoded default.

Fix: Check the site-recovery-backend pod environment:

oc set env pod/<backend-pod> -n dr-<zone-name> --list | grep FLASK_ENV

If FLASK_ENV=development is set, remove or override it. The production default (production) enables JSON output.

Issue 2: Level change not taking effect after editing the ConfigMap

Symptom: You edited site-recovery-logging directly, but pod log verbosity has not changed after several minutes.

Likely cause (a): The ProtectionZone controller has reverted the ConfigMap from ProtectionZone.spec.logging. This happens on every reconcile.

Fix: Update ProtectionZone.spec.logging to match your desired level so the controller propagates the correct value and stops overwriting your change.

Likely cause (b): The level value you wrote is not one of the four accepted values (error, info, debug, trace). The level watcher logs the bad value once as an error and keeps the current level.

Fix: Check the manager logs for a line containing unrecognized log level value and correct the site-recovery-logging ConfigMap (or ProtectionZone.spec.logging) to use a valid value.

Issue 3: Cannot reach trace level at runtime — ConfigMap shows trace but verbosity appears unchanged

Symptom: You set level: trace in the ConfigMap or ProtectionZone, but log output does not show trace-level lines.

Likely cause: trace corresponds to logr V(2) / zap level -2. This level is reachable at runtime via the ConfigMap (the level watcher calls SetLevel with the numeric value). If the watcher is not running — for example, because the pod could not resolve its own namespace from the POD_NAMESPACE environment variable — it silently falls back to flag-only level control.

Fix: Confirm the POD_NAMESPACE downward-API environment variable is set on the manager pod:

oc exec -n dr-<zone-name> <manager-pod> -- printenv POD_NAMESPACE

If it is missing, the level watcher returns nil and is not started. Check the Helm chart templates for the workload side or contact Trilio support.

Issue 4: Log lines from the DRBD node agent are missing from the aggregator

Symptom: The log aggregator receives manager and Flask logs but not agent logs.

Likely cause: The collector is configured to scrape pods in dr-<zone-name> only, but the DRBD node agent DaemonSet runs in trilio-site-recovery-system on each workload cluster.

Fix: Extend your collector configuration to include:

• Namespace trilio-site-recovery-system on each workload cluster.
• Label selector app.kubernetes.io/component=drbd-node-agent.

Issue 5: oc logs shows no output for the manager pod

Symptom: oc logs -l app.kubernetes.io/component=manager-control-plane -n dr-<zone-name> returns nothing.

Likely cause (a): The manager pod has not started because the ProtectionZone has not reached Available phase.

Fix: Check zone status:

oc get pz -n dr-<zone-name>

If the phase is Provisioning or Degraded, inspect the ProtectionZone conditions for the root cause before expecting log output.

Likely cause (b): You are running the command against the TSR-Manager cluster kubeconfig but the pods are on a workload cluster, or vice versa.

Fix: Ensure the correct --context or kubeconfig is active. Manager and protection zone controller pods live in dr-<zone-name> on the TSR-Manager cluster; workload control plane and node agent pods live in trilio-site-recovery-system on each workload cluster.

Issue 6: site-recovery-logging ConfigMap is absent

Symptom: oc get configmap site-recovery-logging -n dr-<zone-name> returns NotFound.

Likely cause: The ProtectionZone has not yet reached Available phase, so the controller has not created the ConfigMap. Alternatively, the zone's spec.logging field is absent and the controller has not been triggered to create it.

Fix: Either wait for the zone to reach Available, or set at least one logging field on the ProtectionZone to trigger the controller to create the ConfigMap. The level watcher treats an absent ConfigMap as "keep startup flag level" and does not error.