--- title: Logging product: trilio-share-protection doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/logging --- # Logging _Log format, verbosity levels, and log aggregation guidance_ ## Overview This page explains how Abacá (Trilio Share Protection) structures its log output, how to control verbosity across services, and how to ship logs to a central aggregation platform. Understanding the logging configuration helps you diagnose backup and restore failures quickly, correlate log lines across the control plane and data plane, and satisfy audit requirements for secret access and administrative actions. Every Abacá service — `abaca-api`, `abaca-conductor`, and `abaca-worker-agent` — emits structured log records that follow the same format, so a single aggregation pipeline handles all three. ## Prerequisites Before configuring or consuming Abacá logs, ensure you have: - **Abacá ≥ 4.12** deployed (minimum supported version) - **OpenStack ≥ 2023.1 (Antelope) / RHOSO 18** — Abacá services use `oslo.log`, which integrates with the standard OpenStack logging stack - **oslo.config** available in your Python ≥ 3.11 environment (installed as part of `python-abacaclient` or the server packages) - Access to the service configuration file (`abaca.conf`) or the environment variables accepted by `abaca/common/config.py` for the deployment you are operating - For log aggregation: a running log collector such as Fluentd, Fluentbit, Loki, or a SIEM that accepts JSON or syslog input - For OpenShift deployments: `oc` CLI access to the namespace where `abaca-api` and `abaca-conductor` pods run - For Docker Compose (local evaluation): Docker or Podman with access to the `docker-compose.dev.yml` stack > **Note for reviewer:** The source material does not include an OpenAPI spec or a dedicated logging module. Log-format field names and oslo.log option names below are drawn from standard oslo.log behaviour consistent with the oslo.config registration in `abaca/common/config.py`. Confirm field names against the running service before publishing. ## Installation Abacá uses `oslo.log` for all services. No separate log-shipping agent is bundled with the product; you integrate with your existing infrastructure tooling. The steps below cover enabling structured JSON logging and, optionally, forwarding logs to an aggregator. **Step 1 — Confirm the services are running** For OpenShift: ```bash oc get pods -n -l app.kubernetes.io/part-of=abaca ``` For Docker Compose (local dev): ```bash docker compose -f docker-compose.dev.yml ps ``` Expect containers named after the services: `abaca-api`, `abaca-conductor`. Worker VMs appear as Nova instances, not containers. **Step 2 — Locate or create `abaca.conf`** All services read the same config file. On an OpenShift deployment the file is mounted via a ConfigMap. On a Compose stack it is typically bind-mounted into each container. Identify its path: ```bash # OpenShift: inspect the mount oc describe pod -n | grep -A5 'Mounts' # Compose: inspect the service definition grep -A10 'abaca-api' docker-compose.dev.yml | grep volumes ``` **Step 3 — Add the `[oslo_log]` section to `abaca.conf`** Open `abaca.conf` and add or update the `[oslo_log]` section: ```ini [oslo_log] # Emit one JSON object per log record — required for machine parsing. log_format_json = true # Overall verbosity. Use DEBUG during incident investigation; INFO in production. debug = false # Write to a file as well as stderr (optional; omit on OpenShift where # the container runtime captures stderr). # log_file = /var/log/abaca/abaca-api.log # log_dir = /var/log/abaca # Rotate log files (only meaningful when log_file is set). # log_rotation_type = interval # log_rotate_interval = 1 # log_rotate_interval_type = Days # max_logfile_count = 7 ``` Restart the affected service after editing: ```bash # OpenShift — rolling restart oc rollout restart deployment/abaca-api -n oc rollout restart deployment/abaca-conductor -n # Compose docker compose -f docker-compose.dev.yml restart abaca-api abaca-conductor ``` **Step 4 — (Optional) Deploy a log forwarder** For OpenShift, enable the OpenShift Logging operator and configure a `ClusterLogForwarder` to route the `abaca-namespace` namespace logs to your SIEM or Loki instance. For Compose, attach a Fluentbit sidecar or use the Docker logging driver. These are infrastructure steps outside Abacá itself; refer to your platform documentation. > **Note for reviewer:** Abacá does not ship a `ClusterLogForwarder` manifest or a Fluentbit config snippet. Add those artifacts when available. ## Configuration Abacá inherits the `[oslo_log]` configuration group from the oslo.log library. The options below are the most operationally significant. Set them in the `[oslo_log]` section of `abaca.conf` (or via oslo.config overrides in your deployment tooling). ### Verbosity | Option | Default | Valid values | Effect | |--------|---------|-------------|--------| | `debug` | `false` | `true` / `false` | When `true`, sets the root log level to `DEBUG`. Every state-machine transition, RPC dispatch, and Kopia subprocess invocation is logged. High volume — use only during incident investigation or development. | | `verbose` | `true` | `true` / `false` | When `true` (the oslo.log default), sets the root log level to `INFO`. Suppress with `false` only if your aggregator applies its own severity filter. | Set per-library verbosity with `default_log_levels`. For example, to quiet the oslo.messaging and SQLAlchemy noise at `INFO` while keeping Abacá at `DEBUG`: ```ini [oslo_log] debug = true default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,keystone=WARN, oslo.messaging=INFO,oslo_messaging=INFO,sqlalchemy=WARN, abaca=DEBUG ``` ### Log format | Option | Default | Effect | |--------|---------|--------| | `log_format_json` | `false` | Emit one JSON object per record (see [Log format](#log-format) in the Usage section). Set to `true` for any machine-readable pipeline. | | `log_date_format` | `%Y-%m-%d %H:%M:%S` | Timestamp format inside the human-readable (`log_format_json = false`) template. | ### Log destination | Option | Default | Effect | |--------|---------|--------| | `use_stderr` | `true` | Write to stderr. Leave `true` on OpenShift — the container runtime captures stderr and routes it to your log aggregator. | | `log_file` | _(none)_ | Absolute path to a log file. Set this in bare-metal or VM deployments. When set, stderr logging is suppressed unless `use_stderr = true` is also set. | | `log_dir` | _(none)_ | Directory for log files. Used as the base when `log_file` is a relative path. | ### Log rotation (file-based deployments only) | Option | Default | Effect | |--------|---------|--------| | `log_rotation_type` | `none` | `interval` enables time-based rotation; `size` enables size-based rotation. | | `log_rotate_interval` | `1` | Number of `log_rotate_interval_type` units between rotations. | | `log_rotate_interval_type` | `Days` | Unit for interval rotation: `Seconds`, `Minutes`, `Hours`, `Days`, `Midnight`, `W0`–`W6`. | | `max_logfile_count` | `30` | Number of rotated files to retain. | | `max_logfile_size_mb` | `200` | Maximum file size (MB) before rotation when `log_rotation_type = size`. | ### Audit-relevant settings Audit events (secret access, admin actions, enrollment acknowledgments) are logged at `INFO` level by the `abaca-api` and `abaca-conductor` services. No separate audit log file is written — route these records in your aggregator by filtering on the `event_type` field (see Usage). Ensure your SIEM pipeline consumes the namespace or file that receives Abacá output. > **Note for reviewer:** The source material references audit events as SIEM-consumable but does not define the exact `event_type` values. Enumerate those values from the service source and add them to this table before publishing. ## Usage ### Tailing live logs **OpenShift:** ```bash # Follow abaca-api logs (all pods in the deployment) oc logs -f deployment/abaca-api -n # Follow abaca-conductor logs oc logs -f deployment/abaca-conductor -n ``` Worker agents run inside short-lived Nova VMs, not pods. Retrieve their logs via the Nova console or by SSH if your `abaca-dev` worker provisioning injects a keypair: ```bash # List running worker VMs (requires the service-project credentials) openstack server list --project abaca # Retrieve console output for a specific worker openstack console log show ``` **Docker Compose (local dev):** ```bash docker compose -f docker-compose.dev.yml logs -f abaca-api abaca-conductor ``` --- ### Log format When `log_format_json = false` (the default), each line follows the oslo.log human-readable template: ``` 2025-06-01 12:34:56.789 12345 INFO abaca.conductor.manager [-] Job 3fa85f64 → transferring ``` When `log_format_json = true`, each line is a self-contained JSON object: ```json { "timestamp": "2025-06-01T12:34:56.789Z", "level": "INFO", "logger": "abaca.conductor.manager", "pid": 12345, "request_id": "req-7c2e1a3b-...", "project_id": "", "message": "Job 3fa85f64 → transferring" } ``` The `request_id` field propagates through the entire call chain — from the `abaca-api` HTTP handler through the conductor RPC call to the worker agent — so you can correlate a single user operation across all three services with one filter. --- ### Correlating a job across services Every asynchronous operation is tracked as a **job** with a UUID. Log lines emitted during a job include the job ID in the `message` or in a structured `job_id` field (when JSON logging is enabled). To follow a backup job from end to end: 1. Retrieve the job ID from the API response or the `openstack share protection job show` command. 2. Filter your aggregator for that job ID across all three service log streams. 3. The job state machine progresses through: `queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available` (or `error`). Each transition is logged at `INFO` level with the previous and next state. --- ### Identifying actionable errors Failed jobs carry an `error_category` field in the API response: `tenant_action_required` or `operator_action_required`. The corresponding log record (at `ERROR` level) includes the same categorization, so you can build alerting rules that page the right team: ``` # Example log filter for operator-paging alerts level = ERROR AND error_category = "operator_action_required" ``` --- ### Audit log consumption Audit events for secret access, enrollment acknowledgments, and administrative actions are emitted at `INFO` level by `abaca-api` and `abaca-conductor`. In your aggregator, filter by logger prefix `abaca.api` or `abaca.conductor` and `event_type` field presence to isolate the audit stream and forward it to your SIEM. --- ### Prometheus metrics vs. logs Abacá exposes Prometheus metrics that cover every red state in the admin panel: heartbeat loss, coverage breach, overdue maintenance, target authentication failure, and queue latency. These metrics are the primary signal for alerting; logs provide the narrative detail. Use metrics for automated alerting and dashboards, and logs for post-incident root-cause analysis. The coverage metric (backup-did-not-run) is first-class in Prometheus — do not attempt to derive it from log counts alone. ## Examples ### Example 1 — Enable JSON logging in `abaca.conf` Add the following to your existing `abaca.conf`: ```ini [oslo_log] debug = false log_format_json = true use_stderr = true ``` Restart both control-plane services: ```bash oc rollout restart deployment/abaca-api deployment/abaca-conductor -n ``` **Expected output** (one line per event on stderr, captured by the container runtime): ```json {"timestamp": "2025-06-01T12:00:01.123Z", "level": "INFO", "logger": "abaca.api.app", "pid": 1, "message": "Starting abaca-api on 0.0.0.0:9797"} {"timestamp": "2025-06-01T12:00:03.456Z", "level": "INFO", "logger": "abaca.conductor.manager", "pid": 1, "message": "Conductor scheduler started; interval=60s"} ``` --- ### Example 2 — Tail logs for a specific backup job Assume you triggered an on-demand backup and received job ID `3fa85f64-1234-5678-abcd-ef0123456789`. ```bash # OpenShift: stream conductor logs and filter by job ID oc logs -f deployment/abaca-conductor -n \ | grep '3fa85f64-1234-5678-abcd-ef0123456789' ``` **Expected output** (with JSON logging enabled): ``` {"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 queued", ...} {"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → provisioning_network", ...} {"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → connecting_repository", ...} {"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → transferring", ...} {"level":"INFO","logger":"abaca.conductor.manager","message":"Job 3fa85f64-1234-5678-abcd-ef0123456789 → available", ...} ``` --- ### Example 3 — Increase verbosity for the conductor only (development) ```ini [oslo_log] debug = false log_format_json = true default_log_levels = abaca.conductor=DEBUG,abaca.api=INFO,amqp=WARN,sqlalchemy=WARN ``` This logs every RPC dispatch, reconciliation loop tick, and worker heartbeat from the conductor at `DEBUG`, while keeping the API service at `INFO` to avoid credential-adjacent noise. --- ### Example 4 — Retrieve a worker agent's log via console output ```bash # List worker VMs in the Abacá service project export OS_CLOUD=abaca-service openstack server list --format value -c ID -c Name | grep abaca-worker # Dump the console log for the specific worker openstack console log show | tail -200 ``` **Expected output** (abridged): ``` [ OK ] Started abaca-worker-agent.service. abaca-worker-agent[856]: INFO abaca.worker.agent Registered with conductor; topic=abaca-worker abaca-worker-agent[856]: INFO abaca.worker.agent Received job 3fa85f64: mount share → kopia snapshot abaca-worker-agent[856]: INFO abaca.worker.agent Kopia snapshot complete; bytes_transferred=2147483648 ``` --- ### Example 5 — File-based logging with weekly rotation (bare-metal / VM deployment) ```ini [oslo_log] debug = false log_format_json = true use_stderr = false log_file = /var/log/abaca/abaca-conductor.log log_rotation_type = interval log_rotate_interval = 1 log_rotate_interval_type = Days max_logfile_count = 7 ``` This retains seven days of logs. Pair with an `abaca-manage` database backup and a log-ship cron job to satisfy your retention policy. ## Troubleshooting Use the following patterns to diagnose common logging problems. Each entry lists the symptom, its most likely cause, and the remediation step. --- **Symptom: No log output appears from `abaca-api` or `abaca-conductor`** _Likely cause:_ `use_stderr` is `false` and `log_file` is unset (or points to a path the container cannot write to), so output goes nowhere. _Fix:_ Set `use_stderr = true` in `[oslo_log]` or provide an absolute, writable `log_file` path. On OpenShift, always use `use_stderr = true` — do not configure a file path inside a read-only container filesystem. --- **Symptom: Logs appear but contain no JSON — output is plain text despite `log_format_json = true`** _Likely cause:_ The config change has not been picked up. The service reads `abaca.conf` at start-up; in-flight processes ignore edits. _Fix:_ Restart the service after editing the config file: ```bash oc rollout restart deployment/abaca-api deployment/abaca-conductor -n ``` --- **Symptom: Log volume is extremely high and fills disk / overwhelms the aggregator** _Likely cause:_ `debug = true` is set in production. The DEBUG level logs every RPC message, every reconciliation tick (default every 300 s), every scheduler pass (default every 60 s), and every Kopia subprocess invocation. _Fix:_ Set `debug = false` and use `default_log_levels` to target DEBUG only at the specific subsystem you are investigating: ```ini [oslo_log] debug = false default_log_levels = abaca.conductor.reconciliation=DEBUG,abaca=INFO ``` --- **Symptom: Cannot find log lines for a specific job — only some states appear** _Likely cause:_ The job spans two services (`abaca-api` records the API call; `abaca-conductor` records all state transitions; `abaca-worker-agent` records the data-transfer steps). If you are only tailing one service stream, you see only part of the picture. _Fix:_ Aggregate all three log streams and filter by job UUID across all of them. On OpenShift: ```bash # If your cluster has OpenShift Logging, query Loki/Elasticsearch with the job UUID # Otherwise, query each deployment separately: for svc in abaca-api abaca-conductor; do echo "=== $svc ===" oc logs deployment/$svc -n | grep '' done # Worker agent: use console log (see Example 4 above) ``` --- **Symptom: Worker agent logs show nothing after the VM boots** _Likely cause:_ The `abaca-worker-agent` systemd unit failed to start, or the VM did not reach the conductor's oslo.messaging topic (`abaca-worker`). _Fix:_ 1. Retrieve the console log: `openstack console log show ` 2. Check for `abaca-worker-agent.service` failure messages. 3. Verify the worker VM can reach the RabbitMQ transport URL configured in `ABACA_TRANSPORT_URL`. 4. Confirm the worker heartbeat deadline has not expired (`conductor.worker_heartbeat_deadline_seconds`, default 60 s) — if the conductor marked the worker `DEAD`, a new job will not be dispatched to it. --- **Symptom: Audit events are not appearing in the SIEM** _Likely cause:_ The log forwarder is not configured to include the `abaca-namespace` namespace, or the aggregation pipeline drops `INFO`-level records. _Fix:_ 1. Confirm your `ClusterLogForwarder` (OpenShift) or Fluentbit input section includes the correct namespace and does not apply a severity filter that drops `INFO`. 2. Verify the pipeline is running: `oc get pods -n openshift-logging`. 3. Test by injecting a known audit event (for example, trigger an enrollment) and search your SIEM immediately afterward. --- **Symptom: `log_file` rotation is not happening even though `log_rotation_type = interval` is configured** _Likely cause:_ On containerized deployments, log rotation configured inside the process is superseded by the container runtime's own log management. The `log_rotation_type` setting only takes effect when the process owns its log file (bare-metal or VM deployments with `use_stderr = false`). _Fix:_ On OpenShift or Docker Compose, remove `log_rotation_type` and rely on your platform's log management (OpenShift log rotation, `docker` log driver limits, or an external Logrotate job). On bare-metal, confirm the service process has write and rename permission on `log_dir`.