Logging

This page explains how Abacá (Trilio Share Protection) structures its log output, how to control verbosity across services, and how to ship logs to a central aggregation platform. Understanding the logging configuration helps you diagnose backup and restore failures quickly, correlate log lines across the control plane and data plane, and satisfy audit requirements for secret access and administrative actions. Every Abacá service — abaca-api, abaca-conductor, and abaca-worker-agent — emits structured log records that follow the same format, so a single aggregation pipeline handles all three.

Before configuring or consuming Abacá logs, ensure you have:

• Abacá ≥ 4.12 deployed (minimum supported version)
• OpenStack ≥ 2023.1 (Antelope) / RHOSO 18 — Abacá services use oslo.log, which integrates with the standard OpenStack logging stack
• oslo.config available in your Python ≥ 3.11 environment (installed as part of python-abacaclient or the server packages)
• Access to the service configuration file (abaca.conf) or the environment variables accepted by abaca/common/config.py for the deployment you are operating
• For log aggregation: a running log collector such as Fluentd, Fluentbit, Loki, or a SIEM that accepts JSON or syslog input
• For OpenShift deployments: oc CLI access to the namespace where abaca-api and abaca-conductor pods run
• For Docker Compose (local evaluation): Docker or Podman with access to the docker-compose.dev.yml stack

Abacá uses oslo.log for all services. No separate log-shipping agent is bundled with the product; you integrate with your existing infrastructure tooling. The steps below cover enabling structured JSON logging and, optionally, forwarding logs to an aggregator.

Step 1 — Confirm the services are running

For OpenShift:

oc get pods -n <abaca-namespace> -l app.kubernetes.io/part-of=abaca

For Docker Compose (local dev):

docker compose -f docker-compose.dev.yml ps

Expect containers named after the services: abaca-api, abaca-conductor. Worker VMs appear as Nova instances, not containers.

Step 2 — Locate or create abaca.conf

All services read the same config file. On an OpenShift deployment the file is mounted via a ConfigMap. On a Compose stack it is typically bind-mounted into each container. Identify its path:

# OpenShift: inspect the mount
oc describe pod <abaca-api-pod> -n <abaca-namespace> | grep -A5 'Mounts'

# Compose: inspect the service definition
grep -A10 'abaca-api' docker-compose.dev.yml | grep volumes

Step 3 — Add the [oslo_log] section to abaca.conf

Open abaca.conf and add or update the [oslo_log] section:

[oslo_log]
# Emit one JSON object per log record — required for machine parsing.
log_format_json = true

# Overall verbosity. Use DEBUG during incident investigation; INFO in production.
debug = false

# Write to a file as well as stderr (optional; omit on OpenShift where
# the container runtime captures stderr).
# log_file = /var/log/abaca/abaca-api.log
# log_dir  = /var/log/abaca

# Rotate log files (only meaningful when log_file is set).
# log_rotation_type = interval
# log_rotate_interval = 1
# log_rotate_interval_type = Days
# max_logfile_count = 7

Restart the affected service after editing:

# OpenShift — rolling restart
oc rollout restart deployment/abaca-api -n <abaca-namespace>
oc rollout restart deployment/abaca-conductor -n <abaca-namespace>

# Compose
docker compose -f docker-compose.dev.yml restart abaca-api abaca-conductor

Step 4 — (Optional) Deploy a log forwarder

For OpenShift, enable the OpenShift Logging operator and configure a ClusterLogForwarder to route the abaca-namespace namespace logs to your SIEM or Loki instance. For Compose, attach a Fluentbit sidecar or use the Docker logging driver. These are infrastructure steps outside Abacá itself; refer to your platform documentation.

Abacá inherits the [oslo_log] configuration group from the oslo.log library. The options below are the most operationally significant. Set them in the [oslo_log] section of abaca.conf (or via oslo.config overrides in your deployment tooling).

Verbosity

Set per-library verbosity with default_log_levels. For example, to quiet the oslo.messaging and SQLAlchemy noise at INFO while keeping Abacá at DEBUG:

[oslo_log]
debug = true
default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,keystone=WARN,
    oslo.messaging=INFO,oslo_messaging=INFO,sqlalchemy=WARN,
    abaca=DEBUG

Log format

Log destination

Log rotation (file-based deployments only)

Audit-relevant settings

Audit events (secret access, admin actions, enrollment acknowledgments) are logged at INFO level by the abaca-api and abaca-conductor services. No separate audit log file is written — route these records in your aggregator by filtering on the event_type field (see Usage). Ensure your SIEM pipeline consumes the namespace or file that receives Abacá output.

Tailing live logs

OpenShift:

# Follow abaca-api logs (all pods in the deployment)
oc logs -f deployment/abaca-api -n <abaca-namespace>

# Follow abaca-conductor logs
oc logs -f deployment/abaca-conductor -n <abaca-namespace>

Worker agents run inside short-lived Nova VMs, not pods. Retrieve their logs via the Nova console or by SSH if your abaca-dev worker provisioning injects a keypair:

# List running worker VMs (requires the service-project credentials)
openstack server list --project abaca

# Retrieve console output for a specific worker
openstack console log show <worker-vm-id>

Docker Compose (local dev):

docker compose -f docker-compose.dev.yml logs -f abaca-api abaca-conductor

Log format

When log_format_json = false (the default), each line follows the oslo.log human-readable template:

2025-06-01 12:34:56.789 12345 INFO abaca.conductor.manager [-] Job 3fa85f64 → transferring

When log_format_json = true, each line is a self-contained JSON object:

{
  "timestamp": "2025-06-01T12:34:56.789Z",
  "level": "INFO",
  "logger": "abaca.conductor.manager",
  "pid": 12345,
  "request_id": "req-7c2e1a3b-...",
  "project_id": "<tenant-project-id>",
  "message": "Job 3fa85f64 → transferring"
}

The request_id field propagates through the entire call chain — from the abaca-api HTTP handler through the conductor RPC call to the worker agent — so you can correlate a single user operation across all three services with one filter.

Correlating a job across services

Every asynchronous operation is tracked as a job with a UUID. Log lines emitted during a job include the job ID in the message or in a structured job_id field (when JSON logging is enabled). To follow a backup job from end to end:

1. Retrieve the job ID from the API response or the openstack share protection job show command.
2. Filter your aggregator for that job ID across all three service log streams.
3. The job state machine progresses through: queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available (or error). Each transition is logged at INFO level with the previous and next state.

Identifying actionable errors

Failed jobs carry an error_category field in the API response: tenant_action_required or operator_action_required. The corresponding log record (at ERROR level) includes the same categorization, so you can build alerting rules that page the right team:

# Example log filter for operator-paging alerts
level = ERROR AND error_category = "operator_action_required"

Audit log consumption

Audit events for secret access, enrollment acknowledgments, and administrative actions are emitted at INFO level by abaca-api and abaca-conductor. In your aggregator, filter by logger prefix abaca.api or abaca.conductor and event_type field presence to isolate the audit stream and forward it to your SIEM.

Prometheus metrics vs. logs

Abacá exposes Prometheus metrics that cover every red state in the admin panel: heartbeat loss, coverage breach, overdue maintenance, target authentication failure, and queue latency. These metrics are the primary signal for alerting; logs provide the narrative detail. Use metrics for automated alerting and dashboards, and logs for post-incident root-cause analysis. The coverage metric (backup-did-not-run) is first-class in Prometheus — do not attempt to derive it from log counts alone.

Example 1 — Enable JSON logging in abaca.conf

Add the following to your existing abaca.conf:

[oslo_log]
debug = false
log_format_json = true
use_stderr = true

Restart both control-plane services:

oc rollout restart deployment/abaca-api deployment/abaca-conductor -n <abaca-namespace>

Expected output (one line per event on stderr, captured by the container runtime):

{"timestamp": "2025-06-01T12:00:01.123Z", "level": "INFO", "logger": "abaca.api.app", "pid": 1, "message": "Starting abaca-api on 0.0.0.0:9797"}
{"timestamp": "2025-06-01T12:00:03.456Z", "level": "INFO", "logger": "abaca.conductor.manager", "pid": 1, "message": "Conductor scheduler started; interval=60s"}

Example 2 — Tail logs for a specific backup job

Assume you triggered an on-demand backup and received job ID 3fa85f64-1234-5678-abcd-ef0123456789.

# OpenShift: stream conductor logs and filter by job ID
oc logs -f deployment/abaca-conductor -n <abaca-namespace> \
  | grep '3fa85f64-1234-5678-abcd-ef0123456789'

Expected output (with JSON logging enabled):

{"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 queued", ...}
{"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → provisioning_network", ...}
{"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → connecting_repository", ...}
{"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → transferring", ...}
{"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → available", ...}

Example 3 — Increase verbosity for the conductor only (development)

[oslo_log]
debug = false
log_format_json = true
default_log_levels = abaca.conductor=DEBUG,abaca.api=INFO,amqp=WARN,sqlalchemy=WARN

This logs every RPC dispatch, reconciliation loop tick, and worker heartbeat from the conductor at DEBUG, while keeping the API service at INFO to avoid credential-adjacent noise.

Example 4 — Retrieve a worker agent's log via console output

# List worker VMs in the Abacá service project
export OS_CLOUD=abaca-service
openstack server list --format value -c ID -c Name | grep abaca-worker

# Dump the console log for the specific worker
openstack console log show <worker-vm-id> | tail -200

Expected output (abridged):

[  OK  ] Started abaca-worker-agent.service.
abaca-worker-agent[856]: INFO abaca.worker.agent Registered with conductor; topic=abaca-worker
abaca-worker-agent[856]: INFO abaca.worker.agent Received job 3fa85f64: mount share → kopia snapshot
abaca-worker-agent[856]: INFO abaca.worker.agent Kopia snapshot complete; bytes_transferred=2147483648

Example 5 — File-based logging with weekly rotation (bare-metal / VM deployment)

[oslo_log]
debug = false
log_format_json = true
use_stderr = false
log_file = /var/log/abaca/abaca-conductor.log
log_rotation_type = interval
log_rotate_interval = 1
log_rotate_interval_type = Days
max_logfile_count = 7

This retains seven days of logs. Pair with an abaca-manage database backup and a log-ship cron job to satisfy your retention policy.

Use the following patterns to diagnose common logging problems. Each entry lists the symptom, its most likely cause, and the remediation step.

Symptom: No log output appears from abaca-api or abaca-conductor

Likely cause: use_stderr is false and log_file is unset (or points to a path the container cannot write to), so output goes nowhere.

Fix: Set use_stderr = true in [oslo_log] or provide an absolute, writable log_file path. On OpenShift, always use use_stderr = true — do not configure a file path inside a read-only container filesystem.

Symptom: Logs appear but contain no JSON — output is plain text despite log_format_json = true

Likely cause: The config change has not been picked up. The service reads abaca.conf at start-up; in-flight processes ignore edits.

Fix: Restart the service after editing the config file:

oc rollout restart deployment/abaca-api deployment/abaca-conductor -n <abaca-namespace>

Symptom: Log volume is extremely high and fills disk / overwhelms the aggregator

Likely cause: debug = true is set in production. The DEBUG level logs every RPC message, every reconciliation tick (default every 300 s), every scheduler pass (default every 60 s), and every Kopia subprocess invocation.

Fix: Set debug = false and use default_log_levels to target DEBUG only at the specific subsystem you are investigating:

[oslo_log]
debug = false
default_log_levels = abaca.conductor.reconciliation=DEBUG,abaca=INFO

Symptom: Cannot find log lines for a specific job — only some states appear

Likely cause: The job spans two services (abaca-api records the API call; abaca-conductor records all state transitions; abaca-worker-agent records the data-transfer steps). If you are only tailing one service stream, you see only part of the picture.

Fix: Aggregate all three log streams and filter by job UUID across all of them. On OpenShift:

# If your cluster has OpenShift Logging, query Loki/Elasticsearch with the job UUID
# Otherwise, query each deployment separately:
for svc in abaca-api abaca-conductor; do
  echo "=== $svc ==="
  oc logs deployment/$svc -n <abaca-namespace> | grep '<job-uuid>'
done
# Worker agent: use console log (see Example 4 above)

Symptom: Worker agent logs show nothing after the VM boots

Likely cause: The abaca-worker-agent systemd unit failed to start, or the VM did not reach the conductor's oslo.messaging topic (abaca-worker).

Fix:

1. Retrieve the console log: openstack console log show <worker-vm-id>
2. Check for abaca-worker-agent.service failure messages.
3. Verify the worker VM can reach the RabbitMQ transport URL configured in ABACA_TRANSPORT_URL.
4. Confirm the worker heartbeat deadline has not expired (conductor.worker_heartbeat_deadline_seconds, default 60 s) — if the conductor marked the worker DEAD, a new job will not be dispatched to it.

Symptom: Audit events are not appearing in the SIEM

Likely cause: The log forwarder is not configured to include the abaca-namespace namespace, or the aggregation pipeline drops INFO-level records.

Fix:

1. Confirm your ClusterLogForwarder (OpenShift) or Fluentbit input section includes the correct namespace and does not apply a severity filter that drops INFO.
2. Verify the pipeline is running: oc get pods -n openshift-logging.
3. Test by injecting a known audit event (for example, trigger an enrollment) and search your SIEM immediately afterward.

Symptom: log_file rotation is not happening even though log_rotation_type = interval is configured

Likely cause: On containerized deployments, log rotation configured inside the process is superseded by the container runtime's own log management. The log_rotation_type setting only takes effect when the process owns its log file (bare-metal or VM deployments with use_stderr = false).

Fix: On OpenShift or Docker Compose, remove log_rotation_type and rely on your platform's log management (OpenShift log rotation, docker log driver limits, or an external Logrotate job). On bare-metal, confirm the service process has write and rename permission on log_dir.