Logging
Log locations, levels, and format
This page describes how Trilio Site Recovery for OpenStack records service activity: where log files are written, how to adjust verbosity, what the log format looks like, and how to retrieve logs under each deployment model (Juju, Kolla-Ansible, and OpenShift). Understanding the logging surface is essential for diagnosing failed DR operations, degraded replication, and service startup problems. Because the service emits no oslo.messaging notifications, logs and polled API state — dr replication health, dr replication audit, dr recovery point list, and operation records — are the primary observability surface.
Before working with service logs you need:
- Trilio Site Recovery deployed via Juju, Kolla-Ansible, or OpenShift (RHOSO18).
- For Juju deployments: Juju 2.9+ or Juju 3.x client with access to the model, and SSH access through the Juju controller.
- For Kolla deployments: Docker CLI access on the host running the
protector_apiandprotector_enginecontainers. - For OpenShift deployments:
ocCLI authenticated to the RHOSO18 cluster, with access to thetrilio-site-recoverynamespace. - Read access to
/var/log/protector/on Juju and Kolla nodes (the directory is mode 0750, owned by theprotectorsystem user; usesudo).
No separate installation is required to access logs. The logging subsystem is part of the trilio-site-recovery Debian package (Juju/Kolla) or the container image (OpenShift). The steps below describe how to reach the logs for each deployment model.
Juju
Logs are written to /var/log/protector/ on each unit. The charm also forwards service output to the systemd journal under the protector-api and protector-engine units. Use juju ssh to reach them:
# List recent API log
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-api.log
# List recent engine log
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-engine.log
# Journal for the API service
juju ssh trilio-site-recovery/0 -- sudo journalctl -u protector-api --since '1h ago' --no-pager
# Journal for the engine service
juju ssh trilio-site-recovery/0 -- sudo journalctl -u protector-engine --since '1h ago' --no-pager
The charm debug log captures hook events and charm-level errors:
juju debug-log --replay --include trilio-site-recovery --no-tail --limit 2000
Kolla-Ansible
The protector_api and protector_engine containers write to stdout/stderr, which Docker captures. Use:
# API container logs
docker logs protector_api
# Engine container logs (follow)
docker logs -f protector_engine
# Last 200 lines from each
docker logs --tail 200 protector_api
docker logs --tail 200 protector_engine
OpenShift (RHOSO18)
On OpenShift the containers write to stdout only — no log files exist inside the pod. oc logs is the only path to service output. Both the API and engine deployments share the label app.kubernetes.io/name=trilio-site-recovery; the app.kubernetes.io/component label (api or engine) separates them:
# API pod logs
oc -n trilio-site-recovery logs \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api' \
--tail=200
# Engine pod logs (follow)
oc -n trilio-site-recovery logs -f \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'
Important: Do not use
-l app=trilio-site-recovery-apior-l app=trilio-site-recovery-engine. Those labels do not exist on the OpenShift manifests and will return "No resources found" — leaving you with no path to the logs.
Log behaviour is controlled by oslo.log options registered in protector.conf. All options below live in the [DEFAULT] section unless stated otherwise.
Verbosity
| Option | Type | Default | Effect |
|---|---|---|---|
debug | boolean | false | Emit DEBUG-level messages. Enables per-request tracing, RPC call arguments, and storage driver interactions. High volume; use only when diagnosing a specific problem. |
verbose | boolean | false | Legacy oslo.log flag, off by default. INFO-level output is emitted whenever debug = false; this option does not gate it. |
Log destination
| Option | Type | Default | Effect |
|---|---|---|---|
log_dir | string | /var/log/protector | Directory for log files. The Juju charm and Kolla deployments write here. On OpenShift, log_dir is unset and use_stderr = true sends all output to stdout for oc logs. |
log_file | string | (none) | Write logs to a specific file path instead of log_dir. Rarely set directly; log_dir is the standard form. |
use_stderr | boolean | false | Also emit to stderr (and therefore the systemd journal or container stdout). The charm template sets this for all units after a certain charm revision; older rendered configs may not have it, in which case the journal contains only startup and gunicorn access events while full application output is in the file. |
Log rotation
Logrotate manages files in /var/log/protector/. The default configuration retains 14 days of rotated history. Compressed (.gz) and plain rotated files coexist in the directory.
Annotated sample (/etc/protector/protector.conf)
[DEFAULT]
# Standard operational verbosity. Flip debug = true when tracing a failure.
debug = false
verbose = false
# Write logs to this directory. The 'protector' user must own it.
log_dir = /var/log/protector
# Also mirror to stderr so the systemd journal picks up tracebacks.
use_stderr = true
Adjusting log level in Juju without editing the conf
The charm exposes debug and verbose as first-class config options. To enable debug logging on a running unit:
juju config trilio-site-recovery debug=true
To disable it again:
juju config trilio-site-recovery debug=false
The charm rewrites protector.conf on every config-changed hook, so editing the file directly on the unit will be overwritten at the next hook. Use juju config for all persistent changes.
For options the charm does not model directly, use config-flags. The separator is a colon between section and key (section:key=value); a dot form such as engine.cross_site_timeout is parsed as a single literal key under [DEFAULT] and silently does nothing:
# Example: raise the cross-site call timeout to 60 s (not a charm-native option)
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Caution: Do not use
config-flagsto repeat a key that the charm template already renders (such asengine:topic). The charm appends the extra section, oslo.config tolerates duplicate sections but configparser may not, and you can cause a startup failure.
Reading the operational log
In normal operation both services emit INFO messages as they handle requests and execute DR workflows. A healthy idle engine logs background activity — coverage refreshes, floating IP refreshes, and (if configured) scheduled recovery-point snapshots — roughly every few minutes.
A failover in progress produces a dense sequence of INFO and DEBUG messages from the engine: storage snapshot scheduling, volume import, instance creation, and metadata synchronisation to the peer site. If a step fails the engine logs the exception at ERROR level with a traceback.
Identifying a failed operation
When a DR operation fails, correlate the operation ID from the CLI with the log:
# 1. Find the failed operation
openstack dr operation list --status failed
# 2. Note its ID, then search the engine log
juju ssh trilio-site-recovery/0 -- \
sudo grep '<operation-id>' /var/log/protector/protector-engine.log
On Kolla:
docker logs protector_engine 2>&1 | grep '<operation-id>'
On OpenShift:
oc -n trilio-site-recovery logs \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine' \
| grep '<operation-id>'
Watching a running operation
Follow the engine log while an operation is in progress:
# Juju
juju ssh trilio-site-recovery/0 -- sudo tail -f /var/log/protector/protector-engine.log
# Kolla
docker logs -f protector_engine
# OpenShift
oc -n trilio-site-recovery logs -f \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'
In parallel, poll operation progress through the CLI:
watch -n 5 openstack dr operation show <operation-id>
The progress field is an integer 0–100 updated by the engine as it completes steps. There is no estimated completion time — the service does not compute one.
Checking service startup
Startup problems — missing config, database schema mismatch, failed Keystone registration — appear in the first few lines of the log or in the systemd journal. On Juju:
juju ssh trilio-site-recovery/0 -- \
sudo journalctl -u protector-api -u protector-engine --since '30min ago' --no-pager
After a Juju upgrade the unit may enter BLOCKED with the message:
set local-site, pure-primary-site-name, pure-secondary-site-name
This is expected: the charm requires local-site to be set explicitly (and the Pure site names when a Pure array is configured). It is not a failed upgrade. Set the missing options and the unit will return to active:
juju config trilio-site-recovery local-site=<site-name>
Collecting a support bundle
The repository ships scripts/collect-support-bundle.sh to gather all relevant logs, DB state, Juju status, and redacted configuration into a single directory for support escalation:
bash scripts/collect-support-bundle.sh \
--app trilio-site-recovery \
--since 4h \
--pg <protection-group-id> \
--openrc /path/to/openrc \
--output ./support-bundle
The script collects via juju ssh and needs no direct SSH access. Credentials in the collected configuration are redacted automatically.
Enable debug logging on a Juju unit
juju config trilio-site-recovery debug=true
# After capturing what you need:
juju config trilio-site-recovery debug=false
Tail the engine log on a Juju unit
juju ssh trilio-site-recovery/0 -- sudo tail -100f /var/log/protector/protector-engine.log
View the last 200 lines of the API container log on Kolla
docker logs --tail 200 protector_api
Follow the engine container log on Kolla
docker logs -f protector_engine
View API pod logs on OpenShift
oc -n trilio-site-recovery logs \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api' \
--tail=200
Follow engine pod logs on OpenShift
oc -n trilio-site-recovery logs -f \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'
Search engine log for a specific operation ID
# Juju
juju ssh trilio-site-recovery/0 -- \
sudo grep 'a1b2c3d4-e5f6-7890-abcd-ef1234567890' \
/var/log/protector/protector-engine.log
# Kolla
docker logs protector_engine 2>&1 \
| grep 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'
# OpenShift
oc -n trilio-site-recovery logs \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine' \
| grep 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'
View rotated log history on a Juju unit (last 14 days)
juju ssh trilio-site-recovery/0 -- \
'sudo sh -c "for f in /var/log/protector/*.log*; do [ -f \"$f\" ] || continue; echo \"===== $f ====="; zcat -f \"$f\" 2>/dev/null; done"'
Collect the charm debug log
juju debug-log --replay --include trilio-site-recovery --no-tail --limit 2000
Run the health-check charm action and review output
# Juju 3.x
juju run trilio-site-recovery/leader health-check
# Juju 2.9
juju run-action trilio-site-recovery/leader health-check --wait
Collect a full support bundle
bash scripts/collect-support-bundle.sh \
--app trilio-site-recovery \
--since 4h \
--pg <protection-group-id> \
--openrc ~/openrc \
--output ./support-bundle-$(date +%Y%m%d)
Raise cross-site call timeout via config-flags (slow inter-site link)
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Journal is thin; application log file is thick
Symptom: journalctl -u protector-engine shows only startup and gunicorn access entries, but /var/log/protector/protector-engine.log has full output including tracebacks.
Cause: The charm template sets use_stderr = true from a certain revision onwards. Units whose protector.conf was rendered before that upgrade did not pick up the option, so oslo.log writes only to the file.
Fix: Trigger a config-changed hook to re-render the conf:
juju config trilio-site-recovery verbose=true # no-op change is enough
After the hook completes both destinations will carry the full output.
OpenShift: oc logs returns "No resources found"
Symptom: Running oc logs -l app=trilio-site-recovery-api -n trilio-site-recovery returns nothing or "No resources found".
Cause: The label app does not exist on the OpenShift manifests. The correct selector requires both app.kubernetes.io/name and app.kubernetes.io/component.
Fix:
oc -n trilio-site-recovery logs \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api'
oc -n trilio-site-recovery logs \
-l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'
Unit BLOCKED after upgrade: "set local-site, pure-primary-site-name, pure-secondary-site-name"
Symptom: After a charm upgrade the unit enters BLOCKED with this message. Logs may show the service not starting.
Cause: The charm now requires local-site to be set explicitly to the name of the DR site this unit belongs to. This is expected behaviour, not a failed upgrade. Without it the engine cannot determine which side of the pair it is on.
Fix: Set the required options:
juju config trilio-site-recovery local-site=<site-name-as-registered>
# If using Pure Storage:
juju config trilio-site-recovery \
pure-primary-site-name=<primary-site-name> \
pure-secondary-site-name=<secondary-site-name>
No log output at all from protector-api or protector-engine
Symptom: Log files are empty or missing; journal shows no entries for the units.
Cause (Juju): The services may have failed to start. Check systemd status:
juju ssh trilio-site-recovery/0 -- \
sudo systemctl status protector-api protector-engine --no-pager
Common reasons: local_site not set in protector.conf, database unreachable, or schema not migrated (protector-manage db sync not run).
Fix: Resolve the startup error indicated in the journal, then restart:
juju ssh trilio-site-recovery/0 -- sudo systemctl restart protector-api protector-engine
Cause (OpenShift): The pod may be in CrashLoopBackOff. Check pod state first:
oc -n trilio-site-recovery get pods
oc -n trilio-site-recovery describe pod <pod-name>
Then read its logs (even a crashing pod retains the last log buffer):
oc -n trilio-site-recovery logs <pod-name> --previous
Log fills with "Cross-site call timed out" at every operation
Symptom: The engine log shows repeated timeout errors on cross-site Keystone or API calls.
Cause: The default cross_site_timeout of 30 seconds is too short for the inter-site network latency.
Fix (Juju): Raise the timeout via config-flags (there is no native charm option for this):
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Fix (Kolla / bare metal): Add to protector.conf under [engine]:
[engine]
cross_site_timeout = 60
Then restart the engine service or container.
Operation shows progress stalled; no new engine log entries
Symptom: openstack dr operation show <id> returns a non-zero progress value that has not changed for many minutes, and the engine log has gone quiet.
Cause: The engine may be waiting for a volume detach (failback waits up to 120 seconds per volume for Nova to release the Cinder attachment) or for a cross-site call that is slower than expected but within the timeout window.
What to check:
# Is the engine still running?
juju ssh trilio-site-recovery/0 -- sudo systemctl is-active protector-engine
# Any recent ERROR entries?
juju ssh trilio-site-recovery/0 -- \
sudo grep -i 'error\|exception\|traceback' \
/var/log/protector/protector-engine.log | tail -30
If the engine is running and no errors appear, allow more time. If it has stopped or logged an unrecoverable error, check the operation result and consider running openstack dr protection group reconcile <pg> to repair an interrupted failover or failback.
Database password visible in log output
Symptom: A log line or the output of protector-manage db sync contains the database URL with the password in clear text.
Cause: Some earlier versions of the alembic env.py masked the host instead of the password. The collect-support-bundle.sh script redacts userinfo on output, but raw log files are not redacted.
Fix: Restrict read access to /var/log/protector/ (it should already be mode 0750, owned by protector). When sharing logs for support, use the bundle script, which redacts credentials automatically:
bash scripts/collect-support-bundle.sh --app trilio-site-recovery --output ./bundle