Logging

This page explains how to work with logs produced by Trilio Site Recovery's Kubernetes operators and agents. It covers the log levels available across the quorum and workload control planes, the structured JSON format emitted by each controller, and recommendations for aggregating logs from a multi-cluster DR estate into a centralised observability stack. Because Site Recovery runs as a set of Kubernetes operators distributed across your quorum cluster, primary cluster, and DR cluster, understanding where logs originate and how to query them efficiently is essential for diagnosing replication failures, tracing failover operations end-to-end, and meeting audit requirements.

Before following this guide, confirm you have:

• Trilio Site Recovery deployed across your clusters (quorum control plane chart site-recovery-quorum-control-plane, workload control plane chart site-recovery-workload-control-plane)
• kubectl or oc CLI configured with kubeconfig contexts for your quorum cluster, primary cluster, and DR cluster
• Sufficient RBAC permissions to read pod logs across the DR namespaces (dr-<name>) on the quorum cluster and across the default and application namespaces on the workload clusters
• OpenShift 4.14 or later on all clusters
• (Optional) A log aggregation platform such as OpenShift Logging (Loki/Vector), Elasticsearch, or Splunk, reachable from all three clusters

No separate installation is required to enable logging — all Site Recovery controllers emit structured logs by default as soon as they are deployed. However, you can adjust verbosity and configure log aggregation at deploy time or afterward.

1. Set the log level at deploy time

Pass the desired log level as a Helm value when installing or upgrading either chart. Both charts accept the same logLevel key.

For the quorum control plane (run on the quorum cluster):

helm upgrade --install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  --namespace dr-<name> \
  --set logLevel=info

For the workload control plane (run on the primary and DR clusters):

helm upgrade --install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  --namespace site-recovery-system \
  --set logLevel=info

Valid values for logLevel are debug, info, warn, and error. The default is info.

2. Change the log level without redeployment

You can patch the log level on a running deployment by editing the controller manager's environment variable. Replace dr-<name> with your actual DR namespace and info with the desired level:

# Quorum control plane — quorum cluster
kubectl --context <quorum-context> \
  set env deployment/site-recovery-quorum-control-plane \
  LOG_LEVEL=debug \
  -n dr-<name>

# Workload control plane — primary cluster
kubectl --context <primary-context> \
  set env daemonset/drbd-node-agent \
  LOG_LEVEL=debug \
  -n site-recovery-system

Controllers reload the level without requiring a pod restart; the drbd-node-agent DaemonSet pods will roll one at a time.

3. Verify controllers are emitting logs

# Quorum control plane
kubectl --context <quorum-context> \
  logs -n dr-<name> \
  -l app=site-recovery-quorum-control-plane \
  --since=5m

# DRBD node agent on the primary cluster
kubectl --context <primary-context> \
  logs -n site-recovery-system \
  -l app=drbd-node-agent \
  --since=5m

If either command returns no output, confirm the pods are running:

kubectl --context <quorum-context> get pods -n dr-<name>
kubectl --context <primary-context> get pods -n site-recovery-system

All Site Recovery controllers emit structured JSON logs over stdout. Each log line is a single JSON object, making lines directly ingestible by Loki, Fluentd, Vector, and most SIEM products without a parsing step.

Log level

Set via the logLevel Helm value (see Installation) or the LOG_LEVEL environment variable on the controller deployment or DaemonSet.

Structured log fields

Every log line includes the following standard fields:

Additional context-specific fields appear depending on the controller:

ProtectionZone logging configuration

The ProtectionZone CRD accepts a spec.logging object that lets you tune logging behaviour per DR zone without redeploying the chart:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: my-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: drbd-operator   # required
  clusters:
    - name: primary
    - name: dr
  logging:
    level: info          # overrides chart-level LOG_LEVEL for this zone
    auditEvents: true    # emit an info-level log for every CRD status transition
    rpoViolations: true  # emit a warn-level log for every RPOEvent recorded

The logging field is optional. When omitted, the chart-level logLevel applies.

Tailing logs during an active operation

During a failover, protection, or test failover operation, the most useful pattern is to watch the quorum control plane in one terminal and the workload cluster agent in another.

Watch the failover controller during a FailoverRequest:

kubectl --context <quorum-context> \
  logs -n dr-<name> \
  -l app=site-recovery-quorum-control-plane \
  -c failover-controller \
  --follow

Watch the DRBD node agent on the DR cluster (volume promotion):

kubectl --context <dr-context> \
  logs -n site-recovery-system \
  -l app=drbd-node-agent \
  --follow

Filtering logs for a specific CRD instance

Because all log lines are structured JSON, you can pipe kubectl logs through jq to isolate events for a single FailoverRequest or ProtectionGroup:

kubectl --context <quorum-context> \
  logs -n dr-<name> \
  -l app=site-recovery-quorum-control-plane \
  --since=1h \
| jq 'select(.name == "my-failover-request")"

# Show only error-level lines across all controllers
kubectl --context <quorum-context> \
  logs -n dr-<name> \
  -l app=site-recovery-quorum-control-plane \
  --since=1h \
| jq 'select(.level == "error")'

Correlating a full reconciliation pass

Every reconciliation pass emits a unique reconcileID. Capture a failing request's reconcileID from the status event or from an initial error line, then filter all log lines for that ID to see the complete execution path:

kubectl --context <quorum-context> \
  logs -n dr-<name> \
  -l app=site-recovery-quorum-control-plane \
  --since=2h \
| jq 'select(.reconcileID == "a1b2c3d4-e5f6-7890-abcd-ef1234567890")'

Collecting a full diagnostic bundle

For support cases or post-incident review, use the tsr-gather must-gather tool instead of manually collecting logs from each cluster. It collects logs, CRD state, and configuration from the quorum cluster and all associated workload clusters into a single bundle:

oc adm must-gather \
  --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle

The resulting tsr-bundle/ directory contains per-cluster log directories and a CRD snapshot for every DR namespace.

Log aggregation: recommended approach for multi-cluster estates

Because Site Recovery spans at minimum two clusters (primary and DR) and typically three (including the quorum cluster), centralising logs is strongly recommended. The following architecture works with OpenShift Logging:

1. Deploy OpenShift Logging (Loki stack) on each cluster independently.
2. Label DR namespaces for easy filtering. All quorum-side logs originate in dr-<name> namespaces; workload-side logs originate in site-recovery-system.
3. Forward workload cluster logs to the quorum cluster's Loki instance using a ClusterLogForwarder pointing at the quorum Loki endpoint. This creates a single query target for the entire estate.
4. Add a cluster label to each forwarder so log lines from the primary cluster and DR cluster are distinguishable in aggregate queries — this matches the cluster field already emitted in every structured log line.

For non-OpenShift Logging stacks (Splunk, Elastic, Datadog), the same pattern applies: run a log-shipping agent (Fluentd, Vector, or the vendor's equivalent) on each cluster's site-recovery-system and dr-<name> namespaces, and ship to a central index. Because all logs are JSON, no custom parsing configuration is required.

Example 1: Successful ProtectionRequest — protection-controller log sequence

When a ProtectionRequest is submitted and processed successfully, you will see the following progression in the quorum control plane logs:

kubectl --context <quorum-context> \
  logs -n dr-prod \
  -l app=site-recovery-quorum-control-plane \
  --since=10m \
| jq 'select(.name == "protect-vm-web-01")'

Expected output (one line per log event, pretty-printed here for readability):

{"timestamp":"2026-01-15T10:01:00Z","level":"info","controller":"protection-controller","msg":"ProtectionRequest received","namespace":"dr-prod","name":"protect-vm-web-01","phase":"Pending","reconcileID":"aabbccdd-1122"}
{"timestamp":"2026-01-15T10:01:02Z","level":"info","controller":"protection-controller","msg":"VirtualMachine validated","namespace":"dr-prod","name":"protect-vm-web-01","virtualMachine":"web-01","reconcileID":"aabbccdd-1122"}
{"timestamp":"2026-01-15T10:01:05Z","level":"info","controller":"protection-controller","msg":"DRBDResource pair provisioned","namespace":"dr-prod","name":"protect-vm-web-01","drdbResource":"web-01","reconcileID":"aabbccdd-1122"}
{"timestamp":"2026-01-15T10:01:08Z","level":"info","controller":"protection-controller","msg":"VM switched to frontend PVC","namespace":"dr-prod","name":"protect-vm-web-01","virtualMachine":"web-01","phase":"Protected","reconcileID":"aabbccdd-1122"}

Example 2: RPO violation warning — replication-monitor log

When replication lag for a ProtectionGroup exceeds the RPO objective configured in the associated ReplicationGroupStatus, the replication monitor emits a warn-level event and records an RPOEvent CRD:

kubectl --context <quorum-context> \
  logs -n dr-prod \
  -l app=site-recovery-quorum-control-plane \
  --since=1h \
| jq 'select(.controller == "replication-monitor" and .level == "warn")'

Expected output:

{"timestamp":"2026-01-15T14:22:11Z","level":"warn","controller":"replication-monitor","msg":"RPO violation detected","namespace":"dr-prod","protectionGroup":"pg-databases","rpoSeconds":47,"violationSeverity":"warning","reconcileID":"cc112233-4455"}

Corroborate by checking the RPOEvent CRD:

kubectl --context <quorum-context> \
  get rpoevents -n dr-prod \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.timestamp}{"\t"}{.spec.rpoAtEvent}{"s\n"}{end}'

Expected output:

rpoevent-pg-databases-1705325331   2026-01-15T14:22:11Z   47s

Example 3: Unplanned failover — failover-controller log sequence

An unplanned failover force-promotes DRBD volumes without waiting for the primary. Filter the failover-controller logs to trace the operation:

kubectl --context <quorum-context> \
  logs -n dr-prod \
  -l app=site-recovery-quorum-control-plane \
  --since=30m \
| jq 'select(.controller == "failover-controller" and .name == "emergency-failover-01")'

Expected output:

{"timestamp":"2026-01-15T18:05:00Z","level":"info","controller":"failover-controller","msg":"FailoverRequest accepted","name":"emergency-failover-01","failoverType":"unplanned","protectionGroup":"pg-databases","phase":"Pending"}
{"timestamp":"2026-01-15T18:05:01Z","level":"warn","controller":"failover-controller","msg":"Primary cluster unreachable; proceeding with force promotion","name":"emergency-failover-01","failoverType":"unplanned","protectionGroup":"pg-databases","phase":"InProgress"}
{"timestamp":"2026-01-15T18:05:45Z","level":"info","controller":"failover-controller","msg":"DRBD volumes promoted on DR cluster","name":"emergency-failover-01","protectionGroup":"pg-databases","phase":"InProgress"}
{"timestamp":"2026-01-15T18:07:30Z","level":"info","controller":"failover-controller","msg":"VMs started on DR cluster","name":"emergency-failover-01","protectionGroup":"pg-databases","phase":"Completed"}

Example 4: TestFailover — full phase progression

kubectl --context <quorum-context> \
  logs -n dr-prod \
  -l app=site-recovery-quorum-control-plane \
  --since=20m \
| jq 'select(.controller == "test-failover-controller")'

Expected output:

{"timestamp":"2026-01-15T09:00:00Z","level":"info","controller":"test-failover-controller","msg":"TestFailover initiated","name":"tf-weekly-check","phase":"CreatingSnapshots","snapshotCount":0}
{"timestamp":"2026-01-15T09:00:30Z","level":"info","controller":"test-failover-controller","msg":"Snapshots created","name":"tf-weekly-check","phase":"CreatingSnapshots","snapshotCount":3}
{"timestamp":"2026-01-15T09:01:00Z","level":"info","controller":"test-failover-controller","msg":"Test VMs provisioned in isolated namespace","name":"tf-weekly-check","testNamespace":"tf-weekly-check-isolation","phase":"VerifyingData"}
{"timestamp":"2026-01-15T09:03:45Z","level":"info","controller":"test-failover-controller","msg":"Verification checks passed","name":"tf-weekly-check","phase":"Succeeded"}
{"timestamp":"2026-01-15T09:04:00Z","level":"info","controller":"test-failover-controller","msg":"Cleaning up test namespace and snapshots","name":"tf-weekly-check","phase":"CleaningUp"}

Issue 1: No logs visible from the quorum control plane

Symptom: kubectl logs -n dr-<name> -l app=site-recovery-quorum-control-plane returns no output or Error from server (NotFound).

Likely cause: The site-recovery-quorum-control-plane deployment is not running, or you are querying the wrong namespace. The quorum control plane is deployed per DR namespace (dr-<name>), not in a global namespace.

Fix:

1. Confirm the namespace: kubectl --context <quorum-context> get ns | grep dr-
2. Confirm the pods are running: kubectl --context <quorum-context> get pods -n dr-<name>
3. If pods are in CrashLoopBackOff, inspect startup logs: kubectl --context <quorum-context> logs -n dr-<name> -l app=site-recovery-quorum-control-plane --previous
4. If the namespace does not exist, the quorum control plane chart has not been installed for this DR deployment. Re-run the Helm install targeting the quorum cluster.

Issue 2: DRBD node agent logs are absent on a specific worker node

Symptom: kubectl logs -n site-recovery-system -l app=drbd-node-agent returns logs from some nodes but not all. The node in question is running but its agent pod is missing.

Likely cause: The drbd-node-agent DaemonSet has a node selector or toleration that does not match the node, or the node joined after the DaemonSet was created and a taint is blocking scheduling.

Fix:

1. Check which nodes have a running agent pod: kubectl --context <primary-context> get pods -n site-recovery-system -l app=drbd-node-agent -o wide
2. Check whether the missing node has an unexpected taint: kubectl --context <primary-context> describe node <node-name> | grep -A5 Taints
3. If the node has a NoSchedule taint not tolerated by the DaemonSet, add a matching toleration via the workload control plane Helm values and upgrade: helm upgrade site-recovery-workload-control-plane trilio/site-recovery-workload-control-plane --reuse-values --set agent.tolerations[0].key=<taint-key> ...

Issue 3: Log lines are not JSON — output is unstructured text

Symptom: kubectl logs output contains plain-text lines rather than JSON objects, causing jq to fail with parse error.

Likely cause: The controller is emitting startup or panic-level messages before the structured logger initialises, or you are reading logs from an init container rather than the main controller container.

Fix:

1. Confirm you are reading the correct container. The quorum control plane pod may contain an init container; specify -c site-recovery-quorum-control-plane explicitly: kubectl --context <quorum-context> logs -n dr-<name> -l app=site-recovery-quorum-control-plane -c site-recovery-quorum-control-plane
2. Filter out non-JSON lines before piping to jq: kubectl logs ... | grep -E '^\{' | jq ...
3. If unstructured output persists across all lines, check whether LOG_LEVEL has been set to an unrecognised value, which causes the logger to fall back to a plain-text mode: kubectl --context <quorum-context> get deployment site-recovery-quorum-control-plane -n dr-<name> -o jsonpath='{.spec.template.spec.containers[0].env}'

Issue 4: RPOEvent CRDs are created but no warn-level logs appear

Symptom: kubectl get rpoevents -n dr-<name> shows violation records, but querying logs for warn or error level lines returns nothing for the replication-monitor controller.

Likely cause: The log level on the quorum control plane is set to error, suppressing warn-level output. RPOEvents are written as CRDs regardless of log level, but the corresponding log line is emitted at warn.

Fix:

1. Check the current log level: kubectl --context <quorum-context> get deployment site-recovery-quorum-control-plane -n dr-<name> -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="LOG_LEVEL")'
2. Raise the level to warn or info: kubectl --context <quorum-context> set env deployment/site-recovery-quorum-control-plane LOG_LEVEL=warn -n dr-<name>
3. Alternatively, check spec.logging.rpoViolations in your ProtectionZone CR and ensure it is set to true.

Issue 5: Cannot correlate logs across primary, DR, and quorum clusters for a single failover

Symptom: A failover completed but you cannot reconstruct the full event timeline because logs are spread across three separate kubectl sessions with no shared identifier.

Likely cause: Logs are not yet centralised, and the reconcileID field is only consistent within a single controller on a single cluster.

Fix:

1. Use tsr-gather to collect a unified bundle from all clusters in one step: oc adm must-gather --image=<tsr-gather-image> --dest-dir=./tsr-bundle — the bundle includes a cross-cluster log directory ordered by timestamp.
2. To correlate without must-gather, use the FailoverRequest name as the common key. The failover-controller on the quorum cluster and the drbd-node-agent DaemonSet on the workload clusters both emit the FailoverRequest name in the name field. Collect logs from all three clusters filtered on that name and merge by timestamp:
   Copy

   # Collect and tag each cluster's logs
   kubectl --context <quorum-context> logs -n dr-<name> -l app=site-recovery-quorum-control-plane --since=1h \
     | jq --arg c quorum '. + {clusterSource: $c}' > /tmp/quorum.ndjson
   kubectl --context <primary-context> logs -n site-recovery-system -l app=drbd-node-agent --since=1h \
     | jq --arg c primary '. + {clusterSource: $c}' > /tmp/primary.ndjson
   kubectl --context <dr-context> logs -n site-recovery-system -l app=drbd-node-agent --since=1h \
     | jq --arg c dr '. + {clusterSource: $c}' > /tmp/dr.ndjson
   # Merge and sort by timestamp
   cat /tmp/quorum.ndjson /tmp/primary.ndjson /tmp/dr.ndjson \
     | jq 'select(.name == "<failover-request-name>")' \
     | jq -s 'sort_by(.timestamp)[]'
   

3. For a long-term fix, configure a ClusterLogForwarder on the primary and DR clusters to ship logs to the quorum cluster's Loki instance, and add a cluster label matching the CLUSTER_ROLE env var so all three clusters' logs are queryable in one place.