Logging

This page explains where Trilio Site Recovery writes its logs, how to control verbosity, what the log format looks like, and how to retrieve logs under each deployment model — Juju charm, Kolla-Ansible, and OpenShift. Understanding the logging surface matters because the service emits no oslo.messaging notifications: the only way to observe DR operation progress, replication health, and engine activity is by reading logs and polling the API with commands such as openstack dr replication health, openstack dr replication audit, openstack dr recovery point list, and openstack dr operation show.

Before using this guide, you need:

• A running deployment of Trilio Site Recovery for OpenStack (Juju, Kolla-Ansible, or OpenShift)
• For Juju deployments: the juju client (v2 or v3) with access to the model containing the trilio-site-recovery application
• For Kolla deployments: Docker access to the nodes running protector_api and protector_engine containers
• For OpenShift deployments: oc or kubectl access to the trilio-site-recovery namespace
• Sufficient permissions to read /var/log/protector on the service nodes (requires sudo under Juju and Kolla)
• The python3-triliositerecovery package installed on any machine where you run openstack dr CLI commands

No additional packages are required to access logs. The logging configuration is part of the standard service deployment. If you need to adjust log verbosity after deployment, see the Configuration section below.

Juju deployments — the charm writes the log directory and manages log rotation automatically as part of unit installation. No manual setup is required.

Kolla deployments — logging is configured by deploy/kolla/install.sh when you run it:

bash deploy/kolla/install.sh

OpenShift deployments — logging is configured by deploy/openshift/scripts/install.sh:

bash deploy/openshift/scripts/install.sh

On OpenShift the service logs to stdout only (the ConfigMap sets use_stderr = true with no log_dir). There are no log files under /var/log/protector on OpenShift nodes — oc logs is the only path to service output.

Logging is controlled by oslo.log options in /etc/protector/protector.conf. On Juju-managed deployments the charm writes this file on every hook; do not edit it by hand. Use juju config instead.

Log verbosity

The two top-level options that control log volume are in the [DEFAULT] section:

To enable debug logging on a Juju deployment:

juju config trilio-site-recovery debug=true

To enable verbose logging on a Juju deployment:

juju config trilio-site-recovery verbose=true

Remember to turn debug off after diagnosis — sustained DEBUG output fills /var/log/protector rapidly and can mask the events you care about.

To set options that the charm does not model directly, use config-flags. For example, to adjust the cross-site call timeout (which affects log output when remote calls are slow):

juju config trilio-site-recovery config-flags="engine.cross_site_timeout=60"

Do not set a key in config-flags that the charm already renders in its template — doing so causes a duplicate section error at oslo.config parse time.

Log file location

Under Juju and Kolla deployments, oslo.log writes to /var/log/protector. The directory contains one file per service process:

logrotate caps these files at 14 days of rotated history. Rotated files are compressed with gzip (.log.1.gz, .log.2.gz, …).

On OpenShift there is no /var/log/protector directory. All output goes to stdout/stderr and is available only through oc logs.

Journal output

When use_stderr is set in the rendered protector.conf (introduced in charm version 1.0.20 and always present in the OpenShift ConfigMap), oslo.log also writes to stderr, which means all application log lines appear in the systemd journal. On units whose config predates that upgrade, the journal contains only startup messages and the gunicorn access log — tracebacks are in the file only. If /var/log/protector/protector-engine.log is substantially larger than the journal output for the same unit, the config has not been re-rendered since the upgrade; triggering a config-changed hook resolves this.

Juju deployments

Read the live API log from the leader unit:

juju ssh trilio-site-recovery/leader "sudo journalctl -u protector-api -f --no-pager"

Read the live engine log from the leader unit:

juju ssh trilio-site-recovery/leader "sudo journalctl -u protector-engine -f --no-pager"

Read the last hour of engine output from the file (includes rotated history):

juju ssh trilio-site-recovery/leader "sudo zcat -f /var/log/protector/protector-engine.log* | grep -E '$(date -u +%Y-%m-%dT%H)'"

Run the built-in health check charm action:

juju run trilio-site-recovery/leader health-check

Collect a complete support bundle (includes logs, config, database state, and replication health from all units):

bash scripts/collect-support-bundle.sh --app trilio-site-recovery --since 4h

Pass --pg <protection-group-id> to include per-group state, and --openrc <file> to include CLI-based DR state capture.

Kolla deployments

Stream the API container log:

docker logs -f protector_api

Stream the engine container log:

docker logs -f protector_engine

Read the on-disk log files (mounted into the container and also written to the host):

sudo tail -f /var/log/protector/protector-engine.log

OpenShift deployments

On OpenShift, oc logs is the only path to service output. There are no files under /var/log/protector.

Stream the API pod log:

oc -n trilio-site-recovery logs -f -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api'

Stream the engine pod log:

oc -n trilio-site-recovery logs -f -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Show previous container logs (useful after a crash):

oc -n trilio-site-recovery logs --previous -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Observing DR operations

The service emits no oslo.messaging notifications. To observe operation progress, poll the API:

# List running operations
openstack dr operation list --status running

# Show progress for a specific operation (progress is an integer 0–100)
openstack dr operation show <operation-id>

# Check replication health for a protection group
openstack dr replication health <protection-group-id>

# Probe whether secondary-site resources resolve (slower — contacts the remote site)
openstack dr replication health <protection-group-id> --probe

# List available recovery points
openstack dr recovery point list <protection-group-id>

# Run a cross-site replication audit
openstack dr replication audit

The --probe flag on dr replication health queries the recovery site's Neutron and Nova through the same trust-scoped session a real recovery would use. It checks whether the protection group's networks, security groups, and flavors resolve in the project they would recover into — it does not test storage replication lag.

Example 1 — Enable debug logging on a Juju unit and follow the engine log

juju config trilio-site-recovery debug=true
juju ssh trilio-site-recovery/leader "sudo journalctl -u protector-engine -f --no-pager"

Disable debug logging when done:

juju config trilio-site-recovery debug=false

Example 2 — Search the engine log file for a specific protection group ID

juju ssh trilio-site-recovery/leader \
  "sudo zcat -f /var/log/protector/protector-engine.log* | grep 'a1b2c3d4-'"

This searches both the current log and all rotated, compressed copies.

Example 3 — Collect a support bundle covering the last 8 hours for a specific protection group

bash scripts/collect-support-bundle.sh \
  --app trilio-site-recovery \
  --since 8h \
  --pg a1b2c3d4-0000-0000-0000-000000000001 \
  --openrc ~/primary-site-openrc.sh \
  --output ./bundle-incident-20240601

The bundle directory will contain per-unit log files, journal excerpts, redacted configuration, database state, and site registry information.

Example 4 — Read engine logs on OpenShift

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine' \
  --since=1h

Example 5 — Confirm the schema migration version from the manage command output

On Juju deployments you run migrations via the charm action:

juju run trilio-site-recovery/leader db-sync

For manual or Kolla deployments, use the protector-manage command directly:

protector-manage --config-file /etc/protector/protector.conf db sync

Example 6 — Filter journal logs by a specific time window on a Kolla node

docker logs protector_engine 2>&1 | grep '2024-06-01T14'

Example 7 — Check what the API and engine packages are reporting from inside a Juju unit

juju ssh trilio-site-recovery/leader \
  "dpkg -l trilio-site-recovery python3-triliositerecovery"

No log output from the engine after a failover

Symptom: openstack dr operation show <id> shows the operation is running but there are no recent entries in /var/log/protector/protector-engine.log.

Likely cause: The protector-engine systemd service has stopped, or the unit's rendered config predates the use_stderr addition and the journal is sparse.

Fix:

1. Check the service status: juju ssh trilio-site-recovery/leader "sudo systemctl status protector-engine"
2. If stopped, check the journal for the reason: juju ssh trilio-site-recovery/leader "sudo journalctl -u protector-engine -n 100 --no-pager"
3. If the journal is thin but the log file is not, the config has not been re-rendered. Trigger a hook or upgrade the charm to force a re-render.

Log file not found at /var/log/protector/protector-engine.log

Symptom: sudo ls /var/log/protector/ returns nothing, or the directory does not exist.

Likely cause (OpenShift): On OpenShift the service logs to stdout only. There are no files under /var/log/protector.

Fix: Use oc logs instead:

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

oc logs returns "No resources found"

Symptom: Running oc logs -l app=trilio-site-recovery-engine returns "No resources found in trilio-site-recovery namespace."

Likely cause: The label selector is wrong. The pods use app.kubernetes.io/name and app.kubernetes.io/component, not a bare app label.

Fix: Use the correct selector:

oc -n trilio-site-recovery logs \
  -l 'app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine'

Log files grow very fast and fill disk

Symptom: /var/log/protector is consuming unexpected disk space.

Likely cause: debug=true is set in the Juju config and has not been turned off after diagnosis.

Fix:

juju config trilio-site-recovery debug=false

Check disk use on the unit:

juju ssh trilio-site-recovery/leader "df -h /var/log"

Engine log shows LocalSiteNotConfigured on every operation

Symptom: Every DR operation fails immediately. The engine log contains LocalSiteNotConfigured with HTTP 500.

Likely cause: The local_site option in [DEFAULT] is not set, or it names a site that is not registered.

Fix: Set local_site to the exact name given to this site when it was registered:

# Find the site name
openstack dr site list

# Set it on the Juju unit
juju config trilio-site-recovery local-site=<site-name-from-list>

After a charm upgrade to 1.0.20 or later, the unit enters BLOCKED state with the message set local-site, pure-primary-site-name, pure-secondary-site-name until this is done. This is expected behaviour, not a failed upgrade.

Operation log shows InvalidState (HTTP 409) on a protection group action

Symptom: An attempt to start a failover or failback returns HTTP 409 InvalidState.

Likely cause: The protection group is in a state that does not permit the requested operation (for example, attempting a failover on a group that is already in failed_over state).

Fix: Check the group's current state:

openstack dr protection group show <pg-id>

Then consult the operations list to see whether a previous operation is still in running state:

openstack dr operation list --protection-group <pg-id> --status running

Support bundle collection fails to redact config secrets

Symptom: Running collect-support-bundle.sh prints config redaction failed; withheld and the juju-config.json file contains a notice rather than config data.

Likely cause: python3 is not available on the Juju client machine, or the Juju client returned non-JSON output.

Fix: Run juju config trilio-site-recovery --format json manually on the client to confirm the output is valid JSON, then ensure Python 3 is available:

python3 --version

If Python 3 is missing, install it with apt-get install python3 on the client machine before re-running the script.

Cross-site calls time out and the engine logs show no response from the peer

Symptom: The engine log shows a timeout on every call to the secondary site. Operations hang and then fail.

Likely cause: The default cross_site_timeout of 30 seconds is too short for the inter-site link.

Fix: Increase the timeout via config-flags (there is no dedicated charm option for this key):

juju config trilio-site-recovery config-flags="engine.cross_site_timeout=60"

Do not set this key in both config-flags and the charm template — it would cause a duplicate section error.