--- title: Observability product: trilio-share-protection doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/observability --- # Observability _Logging, metrics, distributed tracing, and health check endpoints_ ## Overview This page describes how to observe the health and behavior of Trilio Share Protection (Abacá) in production. It covers the Prometheus metrics that drive alerting and the admin panel, structured logging emitted by each service, and the health endpoints you can poll from your monitoring stack. Because a missed backup is the worst silent failure mode in a backup service, Abacá treats coverage as a first-class metric — not a derived report — so your alerting and the operator dashboard never disagree about what is protected. ## Prerequisites Before configuring observability, ensure you have: - Trilio Share Protection deployed with **abaca-api** and **abaca-conductor** running (OpenShift ≥ 4.12 or a Docker Compose dev stack) - A Prometheus-compatible scrape target configured and reachable from your monitoring stack - Access to the abaca-api service port (default **9797**) from your Prometheus instance or load balancer - (Optional) A SIEM or log aggregation platform (for audit-event ingestion) - (Optional) CloudKitty or a compatible billing adapter (for chargeback integration with usage metrics) - Python ≥ 3.11 and the **python-abacaclient** plugin installed if you want to inspect usage metrics from the CLI - Operator-level Keystone credentials scoped to the service project if you intend to call authenticated endpoints ## Installation Observability in Abacá is built in — no separate agent or sidecar is required. The steps below connect your monitoring stack to the running services. **Step 1 — Verify the API service is reachable** Confirm that **abaca-api** is listening on its configured port: ```bash curl -s http://:9797/v1/service_info ``` Expected response (unauthenticated — this endpoint is public): ```json { "service_user": { "id": "", "name": "abaca" }, "worker_project": { "id": "", "name": "abaca" } } ``` If this call fails, check that the `api.bind_host` and `api.bind_port` configuration values match what your network rules allow. **Step 2 — Configure Prometheus to scrape Abacá metrics** Add a scrape job to your `prometheus.yml`. Abacá exposes a pull-based usage and operational metrics endpoint: ```yaml scrape_configs: - job_name: abaca static_configs: - targets: - ':9797' metrics_path: /metrics scheme: http # or https if TLS termination is in front ``` > **Note for reviewer:** The exact `/metrics` path and whether Prometheus exposition is served by abaca-api, abaca-conductor, or both should be confirmed against the source. The design document specifies pull-based Prometheus metrics; the path above is the conventional default. Update this step once the metrics endpoint path is confirmed. **Step 3 — Reload Prometheus** ```bash curl -X POST http://:9090/-/reload ``` Verify targets appear healthy in the Prometheus UI under **Status → Targets**. **Step 4 — (Optional) Configure log forwarding** Abacá services emit structured logs via oslo.log. Forward them to your SIEM or log aggregator using your standard OpenShift log forwarding pipeline (e.g., the OpenShift Logging operator with a `ClusterLogForwarder` custom resource) or your Kolla-Ansible log driver. Audit events — secret access, admin actions, and enrollment acknowledgments — are included in the structured log stream in a SIEM-consumable form. No separate audit log sidecar is required. ## Configuration Observability-relevant configuration lives in the `[conductor]` and `[api]` sections of your Abacá config file, plus a small number of environment variables for container deployments. All options follow the oslo.config convention: `[section] key = value`. --- ### Scheduler and reconciliation cadence These settings control how often the conductor checks for missed backups and stale state — directly affecting the freshness of your coverage and queue-latency metrics. | Option | Section | Default | Effect | |---|---|---|---| | `scheduler_interval` | `[conductor]` | `60` (seconds) | How often the conductor scans cron policies and enqueues overdue backups. Lower values reduce the window between a missed schedule and a metric alert. | | `reconciliation_interval` | `[conductor]` | `300` (seconds) | How often the reconciliation loop checks for stuck jobs, dead workers, and orphaned queue entries. This is also the cadence at which worker heartbeat state is evaluated. | | `worker_heartbeat_deadline_seconds` | `[conductor]` | `60` (seconds) | Seconds since a worker's last heartbeat before the fleet manager marks it `DEAD`. Should stay aligned with `reconciliation_interval`. | | `worker_heartbeat_interval_seconds` | `[conductor]` | `10` (seconds) | How often a running **abaca-worker-agent** sends a heartbeat RPC cast to the conductor. Must be significantly less than `worker_heartbeat_deadline_seconds`. | | `maintenance_interval_seconds` | `[conductor]` | `604800` (weekly) | Cadence for scheduled `kopia maintenance run --full` per enrolled backup target. Until maintenance runs, expired backups do not free S3 storage — relevant to bytes-at-rest metrics. | | `usage_sample_interval_seconds` | `[conductor]` | `21600` (6 hours) | How often the conductor samples `kopia content stats` per enrolled target. The result is the authoritative post-dedup, post-compression bytes-on-S3 figure and is more accurate than summing individual backup sizes. | | `queued_job_deadline_seconds` | `[conductor]` | `300` (seconds) | Seconds a job may remain in the `queued` state with no worker assigned before the reconciliation loop re-dispatches it. This bounds the worst-case queue latency visible in metrics. | --- ### API pagination (affects metric query volume) | Option | Section | Default | Effect | |---|---|---|---| | `max_limit` | `[api]` | `1000` | Maximum page size for list endpoints. Relevant if your monitoring scripts page through large backup lists. | | `default_limit` | `[api]` | `100` | Default page size when no `limit` query parameter is supplied. | --- ### Environment variable overrides For container and Docker Compose deployments, the following environment variables override config file values without requiring a mounted config file: | Variable | Config equivalent | Notes | |---|---|---| | `ABACA_DATABASE_CONNECTION` | `[database] connection` | SQLAlchemy URL for the conductor's state store. | | `ABACA_TRANSPORT_URL` | oslo.messaging `transport_url` | RabbitMQ URL for conductor ↔ worker-agent messaging. | | `ABACA_KOPIA_EXECUTOR` | `[kopia] executor` | Set to `ephemeral_container` for local dev; always `worker_rpc` in production. | | `ABACA_SERVICE_USER_ID` | `[abaca] service_user_id` | Keystone user ID of the service account; surfaced on `/v1/service_info`. | | `ABACA_WORKER_PROJECT_ID` | `[abaca] worker_project_id` | Keystone project ID where worker VMs run; surfaced on `/v1/service_info`. | ## Usage ### Checking coverage and fleet health from the CLI Using **python-abacaclient**, you can inspect the current health of your backup fleet and coverage posture from the OpenStack CLI. List all protection policies and their last-run status to spot coverage gaps: ```bash openstack share protection policy list ``` List recent jobs to see queue latency and error distribution: ```bash openstack share protection job list --limit 50 ``` Inspect a specific failed job to read its `error_category` and decide who needs to act: ```bash openstack share protection job show ``` The `error_category` field tells you: - `tenant_action_required` — the tenant must fix a configuration (for example, a bad S3 bucket URL or expired Barbican credentials). - `operator_action_required` — the infrastructure or service needs attention (for example, a worker that cannot reach the Manila service). --- ### Pulling usage metrics for chargeback Abacá exposes a pull-based usage API compatible with CloudKitty and other external billing integrators. The key billable dimensions are: - **Protected capacity (GB-month):** The logical size of the most recent successful backup per share, sampled on the `usage_sample_interval_seconds` cadence. - **GB transferred per job:** Bytes sent to S3 during each backup, covering both initial ingest and incremental churn. - **Restore count:** Metered from day one; zero-rated by default (billing becomes a configuration flip — data cannot be collected retroactively). Sample the current usage stats for a specific backup target: ```bash openstack share protection target show ``` The `usage_stats` field in the response reflects the most recent `kopia content stats` sample — post-dedup, post-compression bytes on S3. This is more accurate than summing individual backup record sizes, because Kopia's cross-snapshot deduplication means the sum would double-count shared content. --- ### Interpreting Prometheus metrics Every red state visible in the admin panel has a corresponding Prometheus metric, so your automated alerts and the human-readable dashboard always reflect the same data. Key signal categories: - **Heartbeat loss:** A worker has not checked in within `worker_heartbeat_deadline_seconds`. - **Coverage breach:** A share has a protection policy attached but its most recent successful backup is overdue. - **Overdue maintenance:** A backup target has not had a successful `kopia maintenance run --full` within `maintenance_interval_seconds`. - **Target auth failure:** Abacá could not authenticate against a target's S3 endpoint or Barbican reference. - **Queue latency:** The age of the oldest job in the `queued` state, bounded by `queued_job_deadline_seconds`. Alert rules live on the operator side — Abacá ships the metrics; you write the thresholds appropriate for your SLA. --- ### Reading audit events Abacá emits audit events in a structured, SIEM-consumable format for: - Barbican secret access (reading S3 credentials and repository passwords) - Administrative actions performed via abaca-api - Enrollment acknowledgments (including the Keystone trust delegation) Forward these from your log aggregation pipeline to your SIEM as you would any oslo.log-structured output. No additional configuration is needed to enable audit events — they are always emitted. ## Examples ### Example 1 — Query the unauthenticated health/identity endpoint Verify the API is up and read the service account identity without needing credentials: ```bash curl -s http://abaca-api.example.com:9797/v1/service_info | python3 -m json.tool ``` Expected output: ```json { "service_user": { "id": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4", "name": "abaca" }, "worker_project": { "id": "b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5", "name": "abaca" } } ``` This endpoint is intentionally unauthenticated. Tenants use it during enrollment to look up the Keystone user ID they must delegate a trust to — without needing user-list privileges. --- ### Example 2 — Inspect a failed job and determine who must act A backup job has moved to the `error` state. Retrieve it and read the `error_category`: ```bash openstack share protection job show d5e6f7a8-b9c0-1234-5678-d5e6f7a8b9c0 ``` Expected output (truncated): ``` +------------------+---------------------------------------------------+ | Field | Value | +------------------+---------------------------------------------------+ | id | d5e6f7a8-b9c0-1234-5678-d5e6f7a8b9c0 | | status | error | | error_category | tenant_action_required | | error_message | S3 endpoint unreachable: connection refused | | job_type | backup | | created_at | 2024-10-15T03:00:01Z | | updated_at | 2024-10-15T03:02:14Z | +------------------+---------------------------------------------------+ ``` `tenant_action_required` means the tenant must correct their backup target configuration — in this case the S3 endpoint URL stored in Barbican. The operator does not need to intervene. --- ### Example 3 — List jobs with high queue latency Filter for jobs still in the `queued` state to spot dispatch failures: ```bash openstack share protection job list --status queued ``` If jobs appear here consistently beyond `queued_job_deadline_seconds` (default 300 s), the reconciliation loop will re-dispatch them. If the fleet has no available workers, those jobs will walk to `error` with `operator_action_required`. --- ### Example 4 — View usage statistics on a backup target Read the post-dedup, post-compression bytes stored in a target's Kopia repository: ```bash openstack share protection target show f8a9b0c1-d2e3-4567-89ab-f8a9b0c1d2e3 ``` Expected output (relevant field): ``` +----------------------------+----------------------------------------------+ | Field | Value | +----------------------------+----------------------------------------------+ | id | f8a9b0c1-d2e3-4567-89ab-f8a9b0c1d2e3 | | name | tenant-prod-bucket | | status | available | | usage_stats.bytes_on_s3 | 42949672960 | | usage_stats.sampled_at | 2024-10-15T06:00:00Z | +----------------------------+----------------------------------------------+ ``` This sample is updated every `usage_sample_interval_seconds` (default 6 h). Use it as the authoritative figure for chargeback — not the sum of individual backup sizes. --- ### Example 5 — Verify Prometheus scrape is working After adding Abacá to your `prometheus.yml`, confirm the scrape target is healthy: ```bash curl -s 'http://:9090/api/v1/targets' \ | python3 -m json.tool \ | grep -A5 'abaca' ``` Expected (relevant fragment): ```json { "job": "abaca", "health": "up", "lastScrape": "2024-10-15T06:05:23Z" } ``` > **Note for reviewer:** The exact metric names exposed by abaca-api and abaca-conductor (e.g., `abaca_jobs_queued_total`, `abaca_coverage_breach_count`) should be confirmed from source and added here as concrete PromQL examples once available. ## Troubleshooting ### Issue: `/v1/service_info` returns a connection error **Symptom:** `curl: (7) Failed to connect to abaca-api.example.com port 9797: Connection refused` **Likely cause:** The **abaca-api** service is not running, or `api.bind_host` / `api.bind_port` do not match the address you are targeting. **Fix:** 1. Check the `[api]` section of your config: `bind_host` defaults to `0.0.0.0` and `bind_port` defaults to `9797`. 2. Verify the abaca-api pod or container is in a running state. 3. Check that any network policy or security group permits traffic on port 9797 from your monitoring host. --- ### Issue: Prometheus scrape target shows `health: down` **Symptom:** Abacá appears in Prometheus targets but `health` is `down` or `last_error` shows a timeout. **Likely cause:** The metrics endpoint is not reachable from Prometheus — either a firewall rule blocks the scrape, or the path is incorrect. **Fix:** 1. Confirm the `metrics_path` in your `prometheus.yml` matches the actual endpoint path. See the note in the Installation section — the exact path requires source confirmation. 2. Test reachability directly: `curl -s http://:9797/metrics`. 3. If using TLS termination in front of abaca-api, ensure `scheme: https` is set in the scrape config. --- ### Issue: Jobs remain in `queued` state indefinitely **Symptom:** `openstack share protection job list --status queued` shows jobs older than 5 minutes that are not progressing. **Likely cause:** No active worker VMs are available to accept jobs, or the RabbitMQ transport is unhealthy. **Fix:** 1. Check that at least one **abaca-worker-agent** VM is registered and heartbeating. If `conductor.min_workers` is `0` (the default), workers must be booted manually using `abaca-dev worker-boot`. 2. Verify `ABACA_TRANSPORT_URL` points to a healthy RabbitMQ instance and that the conductor can reach it. 3. After `queued_job_deadline_seconds` (default 300 s), the reconciliation loop will attempt to re-dispatch. If the fleet remains empty the job will move to `error` with `error_category: operator_action_required`. --- ### Issue: Worker heartbeat loss alerts fire but the worker appears active **Symptom:** Your monitoring alerts on heartbeat loss, but the worker VM exists and is processing a job. **Likely cause:** `worker_heartbeat_deadline_seconds` (default 60 s) is too close to `worker_heartbeat_interval_seconds` (default 10 s) for your network conditions, or a brief network interruption ate one heartbeat cycle. **Fix:** 1. Increase `conductor.worker_heartbeat_deadline_seconds` to give a wider margin — for example, `90` if heartbeats are occasionally delayed by 15–20 s. 2. Ensure `worker_heartbeat_interval_seconds` remains significantly less than `worker_heartbeat_deadline_seconds`. 3. Check for network congestion between the worker VM network and the RabbitMQ broker. --- ### Issue: `usage_stats.bytes_on_s3` is stale or missing **Symptom:** The usage stats on a backup target show a `sampled_at` timestamp that is many hours old, or the field is empty. **Likely cause:** The reconciliation loop's usage sampling pass has not run, or it encountered an error connecting to the Kopia repository during sampling. **Fix:** 1. Check `conductor.usage_sample_interval_seconds` (default 21600 s / 6 h) — the field is only updated at this cadence. 2. Review **abaca-conductor** logs for errors during the `kopia content stats` invocation for the affected target. An S3 authentication failure or an unreachable endpoint will cause the sample to be skipped. 3. Confirm the target's Barbican credentials are valid: `openstack share protection target show ` — if `status` is not `available`, re-enroll or update credentials. --- ### Issue: Audit events are missing from the SIEM **Symptom:** Secret-access or enrollment events are not appearing in your log aggregation pipeline. **Likely cause:** Log forwarding is not configured for the abaca-api or abaca-conductor pods, or your log aggregator is filtering out oslo.log-formatted entries. **Fix:** 1. Verify your OpenShift `ClusterLogForwarder` (or equivalent Kolla log driver) is configured to forward logs from the `abaca` namespace. 2. Confirm your SIEM ingest parser handles oslo.log's structured JSON format. Audit events are emitted inline in the same log stream — there is no separate audit log file. 3. Check that you are not filtering by log level in a way that drops `INFO`-level audit records.