Trilio Site Recovery for OpenStackOpenStack native DR service
Guide

Logging

Log locations, levels, and format


Overview

This page describes how Trilio Site Recovery for OpenStack records service activity: where log files are written, how to adjust verbosity, what the log format looks like, and how to retrieve logs under each deployment model (Juju, Kolla-Ansible, and OpenShift). Understanding the logging surface is essential for diagnosing failed DR operations, degraded replication, and service startup problems. Because the service emits no oslo.messaging notifications, logs and polled API state — dr replication health, dr replication audit, dr recovery point list, and operation records — are the primary observability surface.


Prerequisites

Before working with service logs you need:

  • Trilio Site Recovery deployed via Juju, Kolla-Ansible, or OpenShift (RHOSO18).
  • For Juju deployments: Juju 2.9+ or Juju 3.x client with access to the model, and SSH access through the Juju controller.
  • For Kolla deployments: Docker CLI access on the host running the protector_api and protector_engine containers.
  • For OpenShift deployments: oc CLI authenticated to the RHOSO18 cluster, with access to the trilio-site-recovery namespace.
  • Read access to /var/log/protector/ on Juju and Kolla nodes (the directory is mode 0750, owned by the protector system user; use sudo).

Installation

No separate installation is required to access logs. The logging subsystem is part of the trilio-site-recovery Debian package (Juju/Kolla) or the container image (OpenShift). The steps below describe how to reach the logs for each deployment model.

Juju

Logs are written to /var/log/protector/ on each unit. The charm also forwards service output to the systemd journal under the protector-api and protector-engine units. Use juju ssh to reach them:

# List recent API log
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-api.log

# List recent engine log
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-engine.log

# Journal for the API service
juju ssh trilio-site-recovery/0 -- sudo journalctl -u protector-api --since '1h ago' --no-pager

# Journal for the engine service
juju ssh trilio-site-recovery/0 -- sudo journalctl -u protector-engine --since '1h ago' --no-pager

The charm debug log captures hook events and charm-level errors:

juju debug-log --replay --include trilio-site-recovery --no-tail --limit 2000

Kolla-Ansible

The protector_api and protector_engine containers write to stdout/stderr, which Docker captures. Use:

# API container logs
docker logs protector_api

# Engine container logs (follow)
docker logs -f protector_engine

# Last 200 lines from each
docker logs --tail 200 protector_api
docker logs --tail 200 protector_engine

OpenShift (RHOSO18)

On OpenShift the containers write to stdout only — no log files exist inside the pod. oc logs is the only path to service output. Both the API and engine deployments share the label app.kubernetes.io/name=trilio-site-recovery; the app.kubernetes.io/component label (api or engine) separates them:

# API pod logs
oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api' \
  --tail=200

# Engine pod logs (follow)
oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Important: Do not use -l app=trilio-site-recovery-api or -l app=trilio-site-recovery-engine. Those labels do not exist on the OpenShift manifests and will return "No resources found" — leaving you with no path to the logs.


Configuration

Log behaviour is controlled by oslo.log options registered in protector.conf. All options below live in the [DEFAULT] section unless stated otherwise.

Verbosity

OptionTypeDefaultEffect
debugbooleanfalseEmit DEBUG-level messages. Enables per-request tracing, RPC call arguments, and storage driver interactions. High volume; use only when diagnosing a specific problem.
verbosebooleanfalseLegacy oslo.log flag, off by default. INFO-level output is emitted whenever debug = false; this option does not gate it.

Log destination

OptionTypeDefaultEffect
log_dirstring/var/log/protectorDirectory for log files. The Juju charm and Kolla deployments write here. On OpenShift, log_dir is unset and use_stderr = true sends all output to stdout for oc logs.
log_filestring(none)Write logs to a specific file path instead of log_dir. Rarely set directly; log_dir is the standard form.
use_stderrbooleanfalseAlso emit to stderr (and therefore the systemd journal or container stdout). The charm template sets this for all units after a certain charm revision; older rendered configs may not have it, in which case the journal contains only startup and gunicorn access events while full application output is in the file.

Log rotation

Logrotate manages files in /var/log/protector/. The default configuration retains 14 days of rotated history. Compressed (.gz) and plain rotated files coexist in the directory.

Annotated sample (/etc/protector/protector.conf)

[DEFAULT]
# Standard operational verbosity. Flip debug = true when tracing a failure.
debug = false
verbose = false

# Write logs to this directory. The 'protector' user must own it.
log_dir = /var/log/protector

# Also mirror to stderr so the systemd journal picks up tracebacks.
use_stderr = true

Adjusting log level in Juju without editing the conf

The charm exposes debug and verbose as first-class config options. To enable debug logging on a running unit:

juju config trilio-site-recovery debug=true

To disable it again:

juju config trilio-site-recovery debug=false

The charm rewrites protector.conf on every config-changed hook, so editing the file directly on the unit will be overwritten at the next hook. Use juju config for all persistent changes.

For options the charm does not model directly, use config-flags. The separator is a colon between section and key (section:key=value); a dot form such as engine.cross_site_timeout is parsed as a single literal key under [DEFAULT] and silently does nothing:

# Example: raise the cross-site call timeout to 60 s (not a charm-native option)
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"

Caution: Do not use config-flags to repeat a key that the charm template already renders (such as engine:topic). The charm appends the extra section, oslo.config tolerates duplicate sections but configparser may not, and you can cause a startup failure.


Usage

Reading the operational log

In normal operation both services emit INFO messages as they handle requests and execute DR workflows. A healthy idle engine logs background activity — coverage refreshes, floating IP refreshes, and (if configured) scheduled recovery-point snapshots — roughly every few minutes.

A failover in progress produces a dense sequence of INFO and DEBUG messages from the engine: storage snapshot scheduling, volume import, instance creation, and metadata synchronisation to the peer site. If a step fails the engine logs the exception at ERROR level with a traceback.

Identifying a failed operation

When a DR operation fails, correlate the operation ID from the CLI with the log:

# 1. Find the failed operation
openstack dr operation list --status failed

# 2. Note its ID, then search the engine log
juju ssh trilio-site-recovery/0 -- \
  sudo grep '<operation-id>' /var/log/protector/protector-engine.log

On Kolla:

docker logs protector_engine 2>&1 | grep '<operation-id>'

On OpenShift:

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine' \
  | grep '<operation-id>'

Watching a running operation

Follow the engine log while an operation is in progress:

# Juju
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-engine.log

# Kolla
docker logs -f protector_engine

# OpenShift
oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

In parallel, poll operation progress through the CLI:

watch -n 5 openstack dr operation show <operation-id>

The progress field is an integer 0–100 updated by the engine as it completes steps. There is no estimated completion time — the service does not compute one.

Checking service startup

Startup problems — missing config, database schema mismatch, failed Keystone registration — appear in the first few lines of the log or in the systemd journal. On Juju:

juju ssh trilio-site-recovery/0 -- \
  sudo journalctl -u protector-api -u protector-engine --since '30min ago' --no-pager

After a Juju upgrade the unit may enter BLOCKED with the message:

set local-site, pure-primary-site-name, pure-secondary-site-name

This is expected: the charm requires local-site to be set explicitly (and the Pure site names when a Pure array is configured). It is not a failed upgrade. Set the missing options and the unit will return to active:

juju config trilio-site-recovery local-site=<site-name>

Collecting a support bundle

The repository ships scripts/collect-support-bundle.sh to gather all relevant logs, DB state, Juju status, and redacted configuration into a single directory for support escalation:

bash scripts/collect-support-bundle.sh \
  --app trilio-site-recovery \
  --since 4h \
  --pg <protection-group-id> \
  --openrc /path/to/openrc \
  --output ./support-bundle

The script collects via juju ssh and needs no direct SSH access. Credentials in the collected configuration are redacted automatically.


Examples

Enable debug logging on a Juju unit

juju config trilio-site-recovery debug=true
# After capturing what you need:
juju config trilio-site-recovery debug=false

Tail the engine log on a Juju unit

juju ssh trilio-site-recovery/0 -- sudo tail -100f /var/log/protector/protector-engine.log

View the last 200 lines of the API container log on Kolla

docker logs --tail 200 protector_api

Follow the engine container log on Kolla

docker logs -f protector_engine

View API pod logs on OpenShift

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api' \
  --tail=200

Follow engine pod logs on OpenShift

oc -n trilio-site-recovery logs -f \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Search engine log for a specific operation ID

# Juju
juju ssh trilio-site-recovery/0 -- \
  sudo grep 'a1b2c3d4-e5f6-7890-abcd-ef1234567890' \
  /var/log/protector/protector-engine.log

# Kolla
docker logs protector_engine 2>&1 \
  | grep 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'

# OpenShift
oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine' \
  | grep 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'

View rotated log history on a Juju unit (last 14 days)

juju ssh trilio-site-recovery/0 -- \
  'sudo sh -c "for f in /var/log/protector/*.log*; do [ -f \"$f\" ] || continue; echo \"===== $f ====="; zcat -f \"$f\" 2>/dev/null; done"'

Collect the charm debug log

juju debug-log --replay --include trilio-site-recovery --no-tail --limit 2000

Run the health-check charm action and review output

# Juju 3.x
juju run trilio-site-recovery/leader health-check

# Juju 2.9
juju run-action trilio-site-recovery/leader health-check --wait

Collect a full support bundle

bash scripts/collect-support-bundle.sh \
  --app trilio-site-recovery \
  --since 4h \
  --pg <protection-group-id> \
  --openrc ~/openrc \
  --output ./support-bundle-$(date +%Y%m%d)

Raise cross-site call timeout via config-flags (slow inter-site link)

juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"

Troubleshooting

Journal is thin; application log file is thick

Symptom: journalctl -u protector-engine shows only startup and gunicorn access entries, but /var/log/protector/protector-engine.log has full output including tracebacks.

Cause: The charm template sets use_stderr = true from a certain revision onwards. Units whose protector.conf was rendered before that upgrade did not pick up the option, so oslo.log writes only to the file.

Fix: Trigger a config-changed hook to re-render the conf:

juju config trilio-site-recovery verbose=true   # no-op change is enough

After the hook completes both destinations will carry the full output.


OpenShift: oc logs returns "No resources found"

Symptom: Running oc logs -l app=trilio-site-recovery-api -n trilio-site-recovery returns nothing or "No resources found".

Cause: The label app does not exist on the OpenShift manifests. The correct selector requires both app.kubernetes.io/name and app.kubernetes.io/component.

Fix:

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api'

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Unit BLOCKED after upgrade: "set local-site, pure-primary-site-name, pure-secondary-site-name"

Symptom: After a charm upgrade the unit enters BLOCKED with this message. Logs may show the service not starting.

Cause: The charm now requires local-site to be set explicitly to the name of the DR site this unit belongs to. This is expected behaviour, not a failed upgrade. Without it the engine cannot determine which side of the pair it is on.

Fix: Set the required options:

juju config trilio-site-recovery local-site=<site-name-as-registered>
# If using Pure Storage:
juju config trilio-site-recovery \
  pure-primary-site-name=<primary-site-name> \
  pure-secondary-site-name=<secondary-site-name>

No log output at all from protector-api or protector-engine

Symptom: Log files are empty or missing; journal shows no entries for the units.

Cause (Juju): The services may have failed to start. Check systemd status:

juju ssh trilio-site-recovery/0 -- \
  sudo systemctl status protector-api protector-engine --no-pager

Common reasons: local_site not set in protector.conf, database unreachable, or schema not migrated (protector-manage db sync not run).

Fix: Resolve the startup error indicated in the journal, then restart:

juju ssh trilio-site-recovery/0 -- sudo systemctl restart protector-api protector-engine

Cause (OpenShift): The pod may be in CrashLoopBackOff. Check pod state first:

oc -n trilio-site-recovery get pods
oc -n trilio-site-recovery describe pod <pod-name>

Then read its logs (even a crashing pod retains the last log buffer):

oc -n trilio-site-recovery logs <pod-name> --previous

Log fills with "Cross-site call timed out" at every operation

Symptom: The engine log shows repeated timeout errors on cross-site Keystone or API calls.

Cause: The default cross_site_timeout of 30 seconds is too short for the inter-site network latency.

Fix (Juju): Raise the timeout via config-flags (there is no native charm option for this):

juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"

Fix (Kolla / bare metal): Add to protector.conf under [engine]:

[engine]
cross_site_timeout = 60

Then restart the engine service or container.


Operation shows progress stalled; no new engine log entries

Symptom: openstack dr operation show <id> returns a non-zero progress value that has not changed for many minutes, and the engine log has gone quiet.

Cause: The engine may be waiting for a volume detach (failback waits up to 120 seconds per volume for Nova to release the Cinder attachment) or for a cross-site call that is slower than expected but within the timeout window.

What to check:

# Is the engine still running?
juju ssh trilio-site-recovery/0 -- sudo systemctl is-active protector-engine

# Any recent ERROR entries?
juju ssh trilio-site-recovery/0 -- \
  sudo grep -i 'error\|exception\|traceback' \
  /var/log/protector/protector-engine.log | tail -30

If the engine is running and no errors appear, allow more time. If it has stopped or logged an unrecoverable error, check the operation result and consider running openstack dr protection group reconcile <pg> to repair an interrupted failover or failback.


Database password visible in log output

Symptom: A log line or the output of protector-manage db sync contains the database URL with the password in clear text.

Cause: Some earlier versions of the alembic env.py masked the host instead of the password. The collect-support-bundle.sh script redacts userinfo on output, but raw log files are not redacted.

Fix: Restrict read access to /var/log/protector/ (it should already be mode 0750, owned by protector). When sharing logs for support, use the bundle script, which redacts credentials automatically:

bash scripts/collect-support-bundle.sh --app trilio-site-recovery --output ./bundle