Observability

This page describes how to observe the health and behavior of Trilio Share Protection (Abacá) in production. It covers the Prometheus metrics that drive alerting and the admin panel, structured logging emitted by each service, and the health endpoints you can poll from your monitoring stack. Because a missed backup is the worst silent failure mode in a backup service, Abacá treats coverage as a first-class metric — not a derived report — so your alerting and the operator dashboard never disagree about what is protected.

Before configuring observability, ensure you have:

• Trilio Share Protection deployed with abaca-api and abaca-conductor running (OpenShift ≥ 4.12 or a Docker Compose dev stack)
• A Prometheus-compatible scrape target configured and reachable from your monitoring stack
• Access to the abaca-api service port (default 9797) from your Prometheus instance or load balancer
• (Optional) A SIEM or log aggregation platform (for audit-event ingestion)
• (Optional) CloudKitty or a compatible billing adapter (for chargeback integration with usage metrics)
• Python ≥ 3.11 and the python-abacaclient plugin installed if you want to inspect usage metrics from the CLI
• Operator-level Keystone credentials scoped to the service project if you intend to call authenticated endpoints

Observability in Abacá is built in — no separate agent or sidecar is required. The steps below connect your monitoring stack to the running services.

Step 1 — Verify the API service is reachable

Confirm that abaca-api is listening on its configured port:

curl -s http://<abaca-api-host>:9797/v1/service_info

Expected response (unauthenticated — this endpoint is public):

{
  "service_user": {
    "id": "<keystone-user-id>",
    "name": "abaca"
  },
  "worker_project": {
    "id": "<keystone-project-id>",
    "name": "abaca"
  }
}

If this call fails, check that the api.bind_host and api.bind_port configuration values match what your network rules allow.

Step 2 — Configure Prometheus to scrape Abacá metrics

Add a scrape job to your prometheus.yml. Abacá exposes a pull-based usage and operational metrics endpoint:

scrape_configs:
  - job_name: abaca
    static_configs:
      - targets:
          - '<abaca-api-host>:9797'
    metrics_path: /metrics
    scheme: http   # or https if TLS termination is in front

Step 3 — Reload Prometheus

curl -X POST http://<prometheus-host>:9090/-/reload

Verify targets appear healthy in the Prometheus UI under Status → Targets.

Step 4 — (Optional) Configure log forwarding

Abacá services emit structured logs via oslo.log. Forward them to your SIEM or log aggregator using your standard OpenShift log forwarding pipeline (e.g., the OpenShift Logging operator with a ClusterLogForwarder custom resource) or your Kolla-Ansible log driver.

Audit events — secret access, admin actions, and enrollment acknowledgments — are included in the structured log stream in a SIEM-consumable form. No separate audit log sidecar is required.

Observability-relevant configuration lives in the [conductor] and [api] sections of your Abacá config file, plus a small number of environment variables for container deployments. All options follow the oslo.config convention: [section] key = value.

Scheduler and reconciliation cadence

These settings control how often the conductor checks for missed backups and stale state — directly affecting the freshness of your coverage and queue-latency metrics.

API pagination (affects metric query volume)

Environment variable overrides

For container and Docker Compose deployments, the following environment variables override config file values without requiring a mounted config file:

Checking coverage and fleet health from the CLI

Using python-abacaclient, you can inspect the current health of your backup fleet and coverage posture from the OpenStack CLI.

List all protection policies and their last-run status to spot coverage gaps:

openstack share protection policy list

List recent jobs to see queue latency and error distribution:

openstack share protection job list --limit 50

Inspect a specific failed job to read its error_category and decide who needs to act:

openstack share protection job show <job-id>

The error_category field tells you:

• tenant_action_required — the tenant must fix a configuration (for example, a bad S3 bucket URL or expired Barbican credentials).
• operator_action_required — the infrastructure or service needs attention (for example, a worker that cannot reach the Manila service).

Pulling usage metrics for chargeback

Abacá exposes a pull-based usage API compatible with CloudKitty and other external billing integrators. The key billable dimensions are:

• Protected capacity (GB-month): The logical size of the most recent successful backup per share, sampled on the usage_sample_interval_seconds cadence.
• GB transferred per job: Bytes sent to S3 during each backup, covering both initial ingest and incremental churn.
• Restore count: Metered from day one; zero-rated by default (billing becomes a configuration flip — data cannot be collected retroactively).

Sample the current usage stats for a specific backup target:

openstack share protection target show <target-id>

The usage_stats field in the response reflects the most recent kopia content stats sample — post-dedup, post-compression bytes on S3. This is more accurate than summing individual backup record sizes, because Kopia's cross-snapshot deduplication means the sum would double-count shared content.

Interpreting Prometheus metrics

Every red state visible in the admin panel has a corresponding Prometheus metric, so your automated alerts and the human-readable dashboard always reflect the same data. Key signal categories:

• Heartbeat loss: A worker has not checked in within worker_heartbeat_deadline_seconds.
• Coverage breach: A share has a protection policy attached but its most recent successful backup is overdue.
• Overdue maintenance: A backup target has not had a successful kopia maintenance run --full within maintenance_interval_seconds.
• Target auth failure: Abacá could not authenticate against a target's S3 endpoint or Barbican reference.
• Queue latency: The age of the oldest job in the queued state, bounded by queued_job_deadline_seconds.

Alert rules live on the operator side — Abacá ships the metrics; you write the thresholds appropriate for your SLA.

Reading audit events

Abacá emits audit events in a structured, SIEM-consumable format for:

• Barbican secret access (reading S3 credentials and repository passwords)
• Administrative actions performed via abaca-api
• Enrollment acknowledgments (including the Keystone trust delegation)

Forward these from your log aggregation pipeline to your SIEM as you would any oslo.log-structured output. No additional configuration is needed to enable audit events — they are always emitted.

Example 1 — Query the unauthenticated health/identity endpoint

Verify the API is up and read the service account identity without needing credentials:

curl -s http://abaca-api.example.com:9797/v1/service_info | python3 -m json.tool

Expected output:

{
  "service_user": {
    "id": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
    "name": "abaca"
  },
  "worker_project": {
    "id": "b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5",
    "name": "abaca"
  }
}

This endpoint is intentionally unauthenticated. Tenants use it during enrollment to look up the Keystone user ID they must delegate a trust to — without needing user-list privileges.

Example 2 — Inspect a failed job and determine who must act

A backup job has moved to the error state. Retrieve it and read the error_category:

openstack share protection job show d5e6f7a8-b9c0-1234-5678-d5e6f7a8b9c0

Expected output (truncated):

+------------------+---------------------------------------------------+
| Field            | Value                                             |
+------------------+---------------------------------------------------+
| id               | d5e6f7a8-b9c0-1234-5678-d5e6f7a8b9c0              |
| status           | error                                             |
| error_category   | tenant_action_required                            |
| error_message    | S3 endpoint unreachable: connection refused       |
| job_type         | backup                                            |
| created_at       | 2024-10-15T03:00:01Z                              |
| updated_at       | 2024-10-15T03:02:14Z                              |
+------------------+---------------------------------------------------+

tenant_action_required means the tenant must correct their backup target configuration — in this case the S3 endpoint URL stored in Barbican. The operator does not need to intervene.

Example 3 — List jobs with high queue latency

Filter for jobs still in the queued state to spot dispatch failures:

openstack share protection job list --status queued

If jobs appear here consistently beyond queued_job_deadline_seconds (default 300 s), the reconciliation loop will re-dispatch them. If the fleet has no available workers, those jobs will walk to error with operator_action_required.

Example 4 — View usage statistics on a backup target

Read the post-dedup, post-compression bytes stored in a target's Kopia repository:

openstack share protection target show f8a9b0c1-d2e3-4567-89ab-f8a9b0c1d2e3

Expected output (relevant field):

+----------------------------+----------------------------------------------+
| Field                      | Value                                        |
+----------------------------+----------------------------------------------+
| id                         | f8a9b0c1-d2e3-4567-89ab-f8a9b0c1d2e3         |
| name                       | tenant-prod-bucket                           |
| status                     | available                                    |
| usage_stats.bytes_on_s3    | 42949672960                                  |
| usage_stats.sampled_at     | 2024-10-15T06:00:00Z                         |
+----------------------------+----------------------------------------------+

This sample is updated every usage_sample_interval_seconds (default 6 h). Use it as the authoritative figure for chargeback — not the sum of individual backup sizes.

Example 5 — Verify Prometheus scrape is working

After adding Abacá to your prometheus.yml, confirm the scrape target is healthy:

curl -s 'http://<prometheus-host>:9090/api/v1/targets' \
  | python3 -m json.tool \
  | grep -A5 'abaca'

Expected (relevant fragment):

{
  "job": "abaca",
  "health": "up",
  "lastScrape": "2024-10-15T06:05:23Z"
}

Issue: /v1/service_info returns a connection error

Symptom: curl: (7) Failed to connect to abaca-api.example.com port 9797: Connection refused

Likely cause: The abaca-api service is not running, or api.bind_host / api.bind_port do not match the address you are targeting.

Fix:

1. Check the [api] section of your config: bind_host defaults to 0.0.0.0 and bind_port defaults to 9797.
2. Verify the abaca-api pod or container is in a running state.
3. Check that any network policy or security group permits traffic on port 9797 from your monitoring host.

Issue: Prometheus scrape target shows health: down

Symptom: Abacá appears in Prometheus targets but health is down or last_error shows a timeout.

Likely cause: The metrics endpoint is not reachable from Prometheus — either a firewall rule blocks the scrape, or the path is incorrect.

Fix:

1. Confirm the metrics_path in your prometheus.yml matches the actual endpoint path. See the note in the Installation section — the exact path requires source confirmation.
2. Test reachability directly: curl -s http://<abaca-api-host>:9797/metrics.
3. If using TLS termination in front of abaca-api, ensure scheme: https is set in the scrape config.

Issue: Jobs remain in queued state indefinitely

Symptom: openstack share protection job list --status queued shows jobs older than 5 minutes that are not progressing.

Likely cause: No active worker VMs are available to accept jobs, or the RabbitMQ transport is unhealthy.

Fix:

1. Check that at least one abaca-worker-agent VM is registered and heartbeating. If conductor.min_workers is 0 (the default), workers must be booted manually using abaca-dev worker-boot.
2. Verify ABACA_TRANSPORT_URL points to a healthy RabbitMQ instance and that the conductor can reach it.
3. After queued_job_deadline_seconds (default 300 s), the reconciliation loop will attempt to re-dispatch. If the fleet remains empty the job will move to error with error_category: operator_action_required.

Issue: Worker heartbeat loss alerts fire but the worker appears active

Symptom: Your monitoring alerts on heartbeat loss, but the worker VM exists and is processing a job.

Likely cause: worker_heartbeat_deadline_seconds (default 60 s) is too close to worker_heartbeat_interval_seconds (default 10 s) for your network conditions, or a brief network interruption ate one heartbeat cycle.

Fix:

1. Increase conductor.worker_heartbeat_deadline_seconds to give a wider margin — for example, 90 if heartbeats are occasionally delayed by 15–20 s.
2. Ensure worker_heartbeat_interval_seconds remains significantly less than worker_heartbeat_deadline_seconds.
3. Check for network congestion between the worker VM network and the RabbitMQ broker.

Issue: usage_stats.bytes_on_s3 is stale or missing

Symptom: The usage stats on a backup target show a sampled_at timestamp that is many hours old, or the field is empty.

Likely cause: The reconciliation loop's usage sampling pass has not run, or it encountered an error connecting to the Kopia repository during sampling.

Fix:

1. Check conductor.usage_sample_interval_seconds (default 21600 s / 6 h) — the field is only updated at this cadence.
2. Review abaca-conductor logs for errors during the kopia content stats invocation for the affected target. An S3 authentication failure or an unreachable endpoint will cause the sample to be skipped.
3. Confirm the target's Barbican credentials are valid: openstack share protection target show <target-id> — if status is not available, re-enroll or update credentials.

Issue: Audit events are missing from the SIEM

Symptom: Secret-access or enrollment events are not appearing in your log aggregation pipeline.

Likely cause: Log forwarding is not configured for the abaca-api or abaca-conductor pods, or your log aggregator is filtering out oslo.log-formatted entries.

Fix:

1. Verify your OpenShift ClusterLogForwarder (or equivalent Kolla log driver) is configured to forward logs from the abaca namespace.
2. Confirm your SIEM ingest parser handles oslo.log's structured JSON format. Audit events are emitted inline in the same log stream — there is no separate audit log file.
3. Check that you are not filtering by log level in a way that drops INFO-level audit records.