Observability

This page describes how to observe, monitor, and audit Trilio Share Protection for OpenStack (Abacá). It covers the six signal domains that Abacá exposes — fleet health, backup coverage, restore activity, repository maintenance, target reachability, and control-plane health — along with the Prometheus metrics and alert rules that map to each, the three administrative health endpoints, the structured log invariants that make SIEM correlation possible, and the audit event pipeline that records every operator-actionable state transition. Operators who instrument these signals can ensure that the most critical failure mode — a backup silently not running — is detected and alerted before it becomes a data-loss event.

Before configuring observability for Abacá, ensure you have:

• Abacá control plane deployed on RHOSO 18+ (abaca-api and abaca-conductor running as OpenShift pods in the abaca namespace)
• Access to the OpenShift cluster to inspect pod logs (oc logs)
• A Prometheus-compatible scraping infrastructure (Prometheus 2.x or OpenShift Monitoring) able to reach the abaca-api and abaca-conductor pods
• A log aggregation backend (Loki, Elasticsearch, Splunk, or equivalent) configured to collect structured JSON logs from OpenShift pod stdout
• Operator-level Keystone credentials to call /v1/admin/* endpoints
• The python-abacaclient OSC plugin installed so you can issue openstack share protection commands
• (Optional) A SIEM configured to ingest events from the /v1/admin/events pull endpoint
• (Optional) An alerting layer (Alertmanager or equivalent) to route Prometheus alerts

Abacá's observability surface is built into the control-plane pods — no separate exporter process is required. The steps below cover wiring Prometheus scraping to the pods, deploying the shipped alert rule file, and pointing your log collector at pod stdout.

Step 1 — Verify the API and conductor pods are running

oc -n abaca get pods

Expected output includes pods whose names begin with abaca-api- and abaca-conductor-. Both must be in Running state before metrics and health endpoints are reachable.

Step 2 — Confirm the metrics endpoint is reachable

Abacá exposes Prometheus metrics on the same port as the REST API (default 9797). Scrape the /metrics path.

# From inside the cluster, or via oc port-forward:
oc -n abaca port-forward svc/abaca-api 9797:9797 &
curl -s http://localhost:9797/metrics | head -40

You should see metric families such as abaca_fleet_workers_active, abaca_backup_coverage_ratio, and abaca_amqp_reachable.

Step 3 — Add a Prometheus scrape configuration

Add a scrape job to your Prometheus configuration (or create a ServiceMonitor if you use the Prometheus Operator):

# prometheus-scrape-abaca.yaml (static config example)
scrape_configs:
  - job_name: abaca
    static_configs:
      - targets:
          - abaca-api.abaca.svc.cluster.local:9797
    metrics_path: /metrics
    scheme: https        # adjust to http if TLS is not configured
    tls_config:
      ca_file: /path/to/ca.crt   # omit if using a trusted CA

If you use the OpenShift Monitoring stack, create a ServiceMonitor in the abaca namespace:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: abaca
  namespace: abaca
spec:
  selector:
    matchLabels:
      app: abaca-api
  endpoints:
    - port: api
      path: /metrics

Step 4 — Deploy the shipped Prometheus alert rules

The alert rule file is bundled with the Abacá operator package under deploy/rhoso/. Apply it to your Prometheus or Alertmanager configuration:

# Copy the rule file to your Prometheus rules directory
cp deploy/rhoso/abaca-alerts.yaml /etc/prometheus/rules/abaca-alerts.yaml
# Reload Prometheus
curl -X POST http://localhost:9090/-/reload

On OpenShift Monitoring, create a PrometheusRule resource:

oc -n abaca apply -f deploy/rhoso/abaca-prometheusrule.yaml

Customize alert thresholds via the overrides: mechanism in your Prometheus configuration — do not edit the shipped file directly, so that upgrades do not overwrite your customizations.

Step 5 — Configure log collection from pod stdout

Abacá logs structured JSON to stdout at INFO and above. Point your log collector (Loki, Fluentd, Filebeat, etc.) at the abaca namespace:

# Verify structured output from the conductor
oc -n abaca logs deployment/abaca-conductor --tail=20

Each log line at INFO or above includes the JSON keys job_id, worker_id, domain_id, project_id, and target_id where applicable, making SIEM field extraction straightforward without regex.

Step 6 — Configure the audit event pull endpoint (optional SIEM integration)

Set your SIEM or log collector to poll the audit event endpoint periodically. Use a Keystone token with operator-level access:

# Retrieve events since a timestamp (ISO 8601)
curl -s -H "X-Auth-Token: $OS_TOKEN" \
  "https://abaca.example.com/v1/admin/events?since=2024-01-15T00:00:00Z"

Store the occurred_at of the last ingested row and pass it as since= on the next poll to avoid duplicate ingestion.

Observability behaviour in Abacá is governed by a small set of [conductor] timing options and the global [DEFAULT] endpoint configuration. All options follow oslo.config conventions and are set in the Abacá INI-style .conf file.

[conductor] — timing that affects alert thresholds

[DEFAULT] — endpoint and catalog

[api] — API server binding

Annotated observability-relevant .conf snippet

[DEFAULT]
# Use the internal endpoint for operator health scripts running inside OpenShift
endpoint_type = internalURL

[api]
bind_host = 0.0.0.0
bind_port = 9797
# Allow bulk pulls from the audit event endpoint
max_limit = 1000
default_limit = 100

[conductor]
# Sweep every 5 minutes; orphaned jobs detected within this window
reconciliation_interval = 300

# Workers must heartbeat within 60 s or be declared dead
worker_heartbeat_deadline_seconds = 60
worker_heartbeat_interval_seconds = 10

# Warn if Kopia maintenance hasn't run in more than 2 days (alert threshold)
maintenance_interval_seconds = 604800

# Sample protected capacity every 6 hours
usage_sample_interval_seconds = 21600

# Raise a reconciliation job if a backup has been queued for > 5 min
queued_job_deadline_seconds = 300

Note: Debug-level logging is off by default. Enabling DEBUG in oslo.log configuration is not recommended in production because debug output is free-text and not shipped to operators in a structured form.

Checking system health from the CLI

Use the openstack share protection OSC plugin to query the three administrative health roll-ups. Each returns a status of healthy, degraded, or critical, plus a per-scope breakdown.

# Fleet health: worker count, queued jobs, scaling state
openstack share protection admin fleet health show

# Backup coverage: per-policy coverage ratios and last-run ages
openstack share protection admin coverage show

# Control-plane health: AMQP, DB replication lag, disk, worker image freshness
openstack share protection admin system health show

Querying Prometheus metrics directly

The metrics below correspond directly to the alert rules shipped with Abacá. Query them to understand current state before or after an alert fires.

# Are any workers active?
curl -sg 'http://prometheus:9090/api/v1/query?query=abaca_fleet_workers_active' | jq '.data.result'

# Coverage ratio for all policies (1.0 = fully covered, 0.0 = no backups ran)
curl -sg 'http://prometheus:9090/api/v1/query?query=abaca_backup_coverage_ratio' | jq '.data.result'

# How long has the oldest queued job been waiting?
curl -sg 'http://prometheus:9090/api/v1/query?query=abaca_fleet_oldest_queued_seconds' | jq '.data.result'

# Is the AMQP broker reachable?
curl -sg 'http://prometheus:9090/api/v1/query?query=abaca_amqp_reachable' | jq '.data.result'

Tailing structured logs from the control-plane pods

# Follow conductor logs — includes job state transitions, reconciliation sweeps
oc -n abaca logs -f deployment/abaca-conductor

# Follow API logs — includes request/response for health endpoint calls
oc -n abaca logs -f deployment/abaca-api

# Filter for a specific job_id using jq (structured JSON output)
oc -n abaca logs deployment/abaca-conductor | \
  jq -c 'select(.job_id == "<job-uuid>")

Pulling audit events for SIEM ingestion

Audit events accumulate in the abaca_events table and are exposed for pull-based ingestion:

# Pull all events since a checkpoint timestamp
curl -s \
  -H "X-Auth-Token: $(openstack token issue -f value -c id)" \
  "https://abaca.example.com/v1/admin/events?since=2024-01-15T06:00:00Z"

The response is a paginated list. Use the occurred_at of the last row as the since= value on the next poll. Relevant kind values for security monitoring include worker_dead, target_auth_failed, backup_did_not_run, fleet_health_critical, restore_stuck, maintenance_overdue, and control_plane_degraded.

Understanding metric labels for multi-tenant alerting

In Domain-scoped deployments all fleet and coverage metrics carry domain_id and project_id labels. Write alert rules that aggregate by these labels so a red state in one tenant's Domain does not disappear into a cluster-wide average:

# Alert fires per Domain, not cluster-wide
min by (domain_id) (abaca_fleet_workers_active) == 0
  and
max by (domain_id) (abaca_fleet_jobs_queued) > 0

Example 1 — Verify fleet health and identify a stalled Domain

openstack share protection admin fleet health show

Expected output (healthy):

+------------------+----------+
| Field            | Value    |
+------------------+----------+
| status           | healthy  |
| active_workers   | 4        |
| queued_jobs      | 0        |
+------------------+----------+

Expected output (critical — backups stalled):

+--------------------------------------+-----------+
| Field                                | Value     |
+--------------------------------------+-----------+
| status                               | critical  |
| active_workers                       | 0         |
| queued_jobs                          | 7         |
| degraded_domain_ids                  | <uuid>    |
+--------------------------------------+-----------+

If status is critical and active_workers is 0 while queued_jobs is positive, the AbacaFleetCritical alert rule should also be firing. Proceed to the troubleshooting section.

Example 2 — Check backup coverage for all policies

openstack share protection admin coverage show

Expected output:

+--------------------------------------+----------------+---------------------+
| policy_id                            | coverage_ratio | last_backup_age_s   |
+--------------------------------------+----------------+---------------------+
| aaaaaaaa-0000-0000-0000-000000000001 | 1.0            | 3540                |
| aaaaaaaa-0000-0000-0000-000000000002 | 0.0            | 90120               |
+--------------------------------------+----------------+---------------------+

A coverage_ratio of 0.0 means no successful backup has run in the last policy interval. A value below 0.5 for 24 hours triggers the AbacaBackupCoverageBreach page-level alert.

Example 3 — Query the coverage metric in Prometheus for a specific Domain

abaca_backup_coverage_ratio{domain_id="<domain-uuid>"}

Expected Prometheus query result:

{
  "metric": {
    "__name__": "abaca_backup_coverage_ratio",
    "domain_id": "<domain-uuid>",
    "policy_id": "<policy-uuid>",
    "project_id": "<project-uuid>"
  },
  "value": [1700000000, "1.0"]
}

Example 4 — Pull recent audit events and filter for target authentication failures

TOKEN=$(openstack token issue -f value -c id)
curl -s \
  -H "X-Auth-Token: $TOKEN" \
  "https://abaca.example.com/v1/admin/events?since=2024-01-15T00:00:00Z" \
  | jq '.events[] | select(.kind == "target_auth_failed")'

Expected output (one matching event):

{
  "id": "bbbbbbbb-1111-1111-1111-000000000001",
  "occurred_at": "2024-01-15T04:22:11Z",
  "actor": "system",
  "kind": "target_auth_failed",
  "severity": "error",
  "domain_id": "<domain-uuid>",
  "project_id": "<project-uuid>",
  "subject_id": "<target-uuid>",
  "subject_kind": "target",
  "payload": {
    "reason": "S3 credentials rejected by endpoint",
    "endpoint": "https://s3.example.com"
  }
}

Example 5 — Confirm AMQP and database health from the control-plane endpoint

curl -s \
  -H "X-Auth-Token: $TOKEN" \
  "https://abaca.example.com/v1/admin/system/health" | jq .

Expected output (healthy):

{
  "status": "healthy",
  "components": {
    "amqp": { "reachable": true },
    "database": { "replication_lag_seconds": 0.3 },
    "disk": { "free_bytes": 42949672960 },
    "worker_image": { "fresh": true }
  }
}

If amqp.reachable is false or database.replication_lag_seconds exceeds 60, the AbacaControlPlaneDegraded page-level alert should fire within five minutes.

Example 6 — Check whether the Prometheus metric for repository maintenance is overdue

abaca_repo_maintenance_overdue_seconds > 172800

A result greater than 172800 (two days) means at least one Kopia repository has not had maintenance run within the expected window and the AbacaMaintenanceOverdue warn-level alert will fire.

Use the symptom→cause→fix pattern below. For each issue, check the relevant Prometheus metric, health endpoint, and pod logs in parallel — structured JSON logs at INFO+ will contain job_id, worker_id, domain_id, project_id, and target_id to speed up correlation.

Issue: AbacaFleetCritical alert is firing but you see active workers in Nova

Symptom: Prometheus reports abaca_fleet_workers_active == 0 for a Domain, but openstack server list shows worker VMs in ACTIVE state.

Likely cause: The worker VMs have stopped sending heartbeats to the conductor. This can happen if the worker's API URL (worker_api_url in [worker]) is misconfigured after a control-plane move, or if the TLS CA used by workers (api_ca_file / worker_api_ca_file) has rotated without rolling the worker image.

Fix:

1. Check conductor logs for heartbeat or worker_dead events:
   Copy

   oc -n abaca logs deployment/abaca-conductor | jq -c 'select(.kind == "worker_dead")'
   

2. Verify worker_heartbeat_deadline_seconds in [conductor] is not set too low for your network latency.
3. If the CA has rotated, build and register a new worker image (see the worker image rollout guide) and drain the fleet.
4. If the control-plane URL changed, update worker_api_url in [conductor] and redeploy the conductor pod:
   Copy

   oc -n abaca rollout restart deployment/abaca-conductor
   

Issue: abaca_backup_coverage_ratio is 0.0 for a policy but no jobs appear in error state

Symptom: The coverage endpoint reports coverage_ratio: 0.0 and last_backup_age_s is very large, but openstack share protection job list shows no recent jobs at all — not even failed ones.

Likely cause: The conductor's scheduler is not firing for that policy. Common causes: the policy's cron expression is invalid or far-future, the scheduler_interval in [conductor] is set very high, or the conductor pod has crashed in its scheduling loop.

Fix:

1. Confirm the conductor pod is running and not restarting:
   Copy

   oc -n abaca get pods -l app=abaca-conductor
   oc -n abaca describe pod <conductor-pod>
   

2. Check conductor logs for scheduler activity:
   Copy

   oc -n abaca logs deployment/abaca-conductor | jq -c 'select(.logger | contains("scheduler"))'
   

3. Verify the policy's schedule with the OSC plugin:
   Copy

   openstack share protection policy show <policy-id>
   

4. If the cron expression is malformed, update the policy and allow up to scheduler_interval seconds (default 60) for the next evaluation.

Issue: abaca_target_reachability == 0 for a target

Symptom: The AbacaTargetUnreachable alert is firing. openstack share protection admin fleet health show or the events endpoint shows target_auth_failed events.

Likely cause: S3 credentials stored in Barbican have been rotated at the provider without updating Abacá, the S3 endpoint URL has changed, or the Keystone trust used to read Barbican secrets has been revoked by the tenant.

Fix:

1. Pull the relevant audit event to identify the failure reason:
   Copy

   curl -s -H "X-Auth-Token: $TOKEN" \
     "https://abaca.example.com/v1/admin/events?since=<timestamp>" \
     | jq '.events[] | select(.subject_id == "<target-uuid>")'
   

2. If the reason is S3 credentials rejected, ask the Domain owner to update the S3 credentials via the BackupTarget update API and re-run enrollment.
3. If the reason is trust revoked, ask the tenant to re-issue a Keystone trust for the Abacá service user.
4. If the S3 endpoint is temporarily unreachable (network partition), the alert will auto-resolve once reachability is restored — no action needed on the Abacá side.

Issue: abaca_amqp_reachable == 0 — conductor cannot reach RabbitMQ

Symptom: AbacaControlPlaneDegraded alert fires. New backup and restore jobs queue but never transition out of queued state.

Likely cause: Abacá's dedicated RabbitMQ broker (separate from the platform's shared RabbitMQ) is down or its credentials have changed.

Fix:

1. Verify the broker is running and the [database] connection string and RabbitMQ transport URL in the Abacá .conf are correct.
2. Check conductor logs for connection error messages:
   Copy

   oc -n abaca logs deployment/abaca-conductor | jq -c 'select(.levelname == "ERROR")'
   

3. Restart the conductor pod after fixing connectivity:
   Copy

   oc -n abaca rollout restart deployment/abaca-conductor
   

4. Once AMQP is restored, abaca_amqp_reachable will return to 1 within one reconciliation_interval (default 300 s) and queued jobs will resume.

Issue: Audit events are not appearing in the SIEM despite jobs running

Symptom: Jobs complete successfully but the /v1/admin/events endpoint returns an empty list or stale results.

Likely cause: The since= timestamp on the poll query is set too far in the future (a clock-skew issue), or the page limit is too small and the collector is not following pagination.

Fix:

1. Issue a broad query without a since= filter to confirm events exist:
   Copy

   curl -s -H "X-Auth-Token: $TOKEN" \
     "https://abaca.example.com/v1/admin/events" | jq '.events | length'
   

2. If events exist, adjust your collector's checkpoint timestamp — do not use the local collector clock; use the occurred_at of the last ingested event from the API response.
3. If the result count equals default_limit (default 100), your collector may be truncating. Increase the limit parameter up to max_limit (default 1000) and implement pagination.

Issue: Metrics appear missing or stale in Prometheus

Symptom: Expected metric families (e.g. abaca_backup_coverage_ratio) are absent from Prometheus or have not updated in more than usage_sample_interval_seconds.

Likely cause: The Prometheus scrape job cannot reach the abaca-api pod on port 9797, or the pod has restarted and the scrape target has not recovered.

Fix:

1. Confirm the scrape target is healthy in the Prometheus targets UI.
2. Verify the API pod is running and the port matches bind_port in [api] (default 9797):
   Copy

   oc -n abaca get svc abaca-api -o jsonpath='{.spec.ports}'
   

3. If TLS is configured, confirm the ca_file in the Prometheus scrape config matches the certificate presented by the API pod.
4. Check API pod logs for any startup errors:
   Copy

   oc -n abaca logs deployment/abaca-api | jq -c 'select(.levelname == "ERROR")'