Guide

Notifications & Metrics

oslo.messaging notifications and any exposed metrics

Overview

Abacá surfaces its operational state through two complementary channels: Prometheus metrics scraped from the control plane and audit events written to the abaca_events table and exposed via the admin API. Together they give you a consistent, correlated view of fleet health, backup coverage, repository status, and control-plane vitals — so human dashboards and automated alerting always reflect the same underlying facts. This page explains the metric catalogue, the recommended alert rules, the audit event schema, the health-check endpoints operators and alert routers can poll, and the log-emission invariants that make SIEM correlation tractable.

Prerequisites

Before configuring notifications and metrics, ensure you have:

Abacá control plane deployed and running on RHOSO 18+ (both abaca-api and abaca-conductor pods healthy in the abaca OpenShift namespace)
A Prometheus instance (or compatible scrape target) with network access to the abaca-conductor metrics endpoint
A Prometheus Alertmanager (or equivalent) configured to receive and route alert rules
An oslo.config .conf file for abaca-conductor with [conductor] tuning applied (see Configuration)
Operator-level access to the Abacá admin API (openstack share protection or direct REST calls with an admin-scoped token)
A log aggregation backend capable of ingesting JSON-structured lines (Loki, Elasticsearch, Splunk, or equivalent) if you intend to consume audit events via log collection
Python ≥ 3.11 and python-abacaclient installed if you are querying the events API programmatically

Installation

Prometheus metrics and audit events are built into abaca-conductor and abaca-api — no additional packages are required. The steps below configure scraping, deploy the bundled alert rules, and verify that events are flowing.

Step 1 — Confirm the metrics endpoint is reachable

The conductor exposes its Prometheus metrics on the same host and port as the worker listener. After deploying the control plane, verify the endpoint from within the abaca namespace:

# From inside the OpenShift cluster, or through a port-forward
kubectl -n abaca port-forward deployment/abaca-conductor 9798:9798 &
curl -s http://localhost:9798/metrics | head -30

You should see lines beginning with abaca_ in the output.

Step 2 — Add a Prometheus scrape target

Add the conductor pod(s) as a scrape target. Using the OpenShift ServiceMonitor CRD (if you run the Prometheus Operator):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: abaca-conductor
  namespace: abaca
spec:
  selector:
    matchLabels:
      app: abaca-conductor
  endpoints:
    - port: metrics          # the Service port named 'metrics'
      path: /metrics
      interval: 30s

If you are using a static Prometheus configuration instead:

scrape_configs:
  - job_name: abaca
    static_configs:
      - targets:
          - abaca-conductor.abaca.svc.cluster.local:9798

Step 3 — Deploy the bundled alert rules

Abacá ships a Prometheus rule file with the operator package. Load it into your Prometheus instance:

# Copy the rule file from the operator package to your Prometheus rules directory
cp /usr/share/abaca/prometheus/abaca.rules.yaml /etc/prometheus/rules/

# Reload Prometheus (SIGHUP or the /-/reload endpoint)
curl -X POST http://prometheus:9090/-/reload

Verify the rules loaded without errors:

curl -s http://prometheus:9090/api/v1/rules | \
  python3 -c "import sys,json; rules=json.load(sys.stdin); \
  print('\n'.join(r['name'] for g in rules['data']['groups'] for r in g['rules'] if r['name'].startswith('Abaca')))"

Expected output includes AbacaFleetCritical, AbacaBackupCoverageBreach, AbacaTargetUnreachable, AbacaControlPlaneDegraded, AbacaCircuitBreakerTripped, AbacaFleetBelowFloor, AbacaBootFailuresRising, AbacaMaintenanceOverdue, and AbacaQueueLatencyRising.

Step 4 — Configure your log collector to pull audit events

Abacá does not push events to a bus. Your log collector must poll the events API:

# Example: pull events newer than a given timestamp (ISO-8601)
openstack share protection event list --since "$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"

Or directly via the REST API:

curl -s -H "X-Auth-Token: $OS_TOKEN" \
  "https://abaca.example.com/v1/admin/events?since=2024-01-15T10:00:00Z"

Configure your collector (Fluentd, Vector, Logstash, etc.) to call this endpoint on a schedule and forward rows to your SIEM.

Configuration

All conductor-side metric and scheduling knobs live in abaca.conf under the [conductor] section. There are no dedicated [metrics] or [notifications] sections — observability is governed by the same conductor lifecycle intervals.

`[conductor]` — intervals that directly affect metric freshness

Key	Type	Default	Effect
`scheduler_interval`	integer (seconds)	`60`	How often the scheduler evaluates policy cron expressions and enqueues due backups. A longer interval means `abaca_backup_coverage_ratio` can lag by up to this many seconds.
`reconciliation_interval`	integer (seconds)	`300`	How often the reconciliation sweep runs to detect orphaned jobs, dead workers, and overdue maintenance. Directly controls how quickly `abaca_fleet_workers_active` and job-state metrics reflect reality after a failure.
`worker_heartbeat_deadline_seconds`	integer (seconds)	`60`	A worker that has not sent a heartbeat within this window is considered dead. Drives the `worker_dead` event and decrements `abaca_fleet_workers_active`.
`worker_heartbeat_interval_seconds`	integer (seconds)	`10`	How often each `abaca-worker-agent` sends its heartbeat. Must be well below `worker_heartbeat_deadline_seconds`.
`maintenance_interval_seconds`	integer (seconds)	`604800`	Target maintenance interval (7 days). Exceeding this threshold triggers `abaca_repo_maintenance_overdue_seconds` to rise above zero and fires the `AbacaMaintenanceOverdue` warn rule.
`usage_sample_interval_seconds`	integer (seconds)	`21600`	How often the conductor samples per-share protected-capacity for the billing/usage surface (6 hours). Controls the freshness of the protected-capacity Prometheus gauge.
`catalogue_sync_interval_seconds`	integer (seconds)	`3600`	How often the conductor reconciles its internal target catalogue against Barbican and Kopia repository state. Affects `abaca_target_reachability` probe frequency.
`queued_job_deadline_seconds`	integer (seconds)	`300`	A queued job older than this with no assigned worker fires the `AbacaQueueLatencyRising` warn rule. Also used by the reconciliation sweep to detect permanently stuck jobs.
`worker_boot_max_failures`	integer	`3`	Maximum worker boot failures within `worker_boot_failure_window_seconds` before the circuit breaker trips. Tripping fires `AbacaCircuitBreakerTripped`.
`worker_boot_failure_window_seconds`	integer (seconds)	`7200`	The sliding window in which `worker_boot_max_failures` failures must occur to trip the circuit breaker.
`min_workers`	integer	`1`	Minimum desired active workers per domain. Falling below this fires the `AbacaFleetBelowFloor` warn rule.

Alert rule threshold customization

The shipped rule file uses default thresholds. Override them via the standard overrides: mechanism in your Prometheus rule file rather than editing the bundled file — that way your customizations survive package upgrades. For example, to tighten the coverage breach threshold from 50 % to 75 %:

# /etc/prometheus/rules/abaca-overrides.yaml
groups:
  - name: abaca-overrides
    rules:
      - alert: AbacaBackupCoverageBreach
        expr: abaca_backup_coverage_ratio < 0.75
        for: 24h
        labels:
          severity: page
        annotations:
          summary: "Backup coverage below 75% for 24 hours"

Log level

Set the global log level in your abaca.conf [DEFAULT] section using oslo.config's standard debug flag. Structured JSON is emitted at INFO and above; free-text DEBUG output is suppressed by default and should not be forwarded to your SIEM:

[DEFAULT]
debug = false

Usage

Querying health endpoints

Three admin endpoints give you an immediate operational snapshot without waiting for Prometheus to scrape:

# Fleet: worker count, scaling state, queued jobs
curl -s -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/fleet/health | python3 -m json.tool

# Backup coverage: per-policy ratios and last-run ages
curl -s -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/coverage | python3 -m json.tool

# System: AMQP reachability, DB replication lag, disk pressure
curl -s -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/system/health | python3 -m json.tool

Each response contains a top-level status field (healthy, degraded, or critical) suitable for alert router polling, plus per-scope breakdowns for drill-down.

Using the OSC plugin to list events

Audit events are the primary mechanism for post-hoc investigation and SIEM ingestion:

# List all events in the last hour
openstack share protection event list \
  --since "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)"

# Filter to a specific severity
openstack share protection event list --severity error

# Filter to events about a specific target
openstack share protection event list --subject-kind target --subject-id <target-id>

Prometheus label-based filtering

All fleet and coverage metrics carry domain_id and project_id labels. Use these to scope alert rules so a red state in one tenant's domain does not get lost in a cluster-wide average:

# Active workers per domain
abaca_fleet_workers_active{domain_id="<uuid>"}

# Coverage ratio for a specific project
abaca_backup_coverage_ratio{project_id="<uuid>"}

# Queue depth per domain
abaca_fleet_jobs_queued{domain_id="<uuid>"}

Identifying the worst failure mode: backup-did-not-run

abaca_backup_coverage_ratio is the primary signal for missed backups. It represents the fraction of expected backups (within the last policy interval) that completed successfully. A value below 1.0 means at least one expected backup did not run:

# Which policies have coverage below 80%?
abaca_backup_coverage_ratio < 0.8

Pair this with the backup_did_not_run event kind in the audit log to get the full picture:

openstack share protection event list --kind backup_did_not_run

Checking target reachability

abaca_target_reachability is a gauge that is 1 when the target is reachable and 0 when it is not. Use it in combination with abaca_target_auth_failure_total to distinguish network failures from credential failures:

# Targets currently unreachable
abaca_target_reachability == 0

# Rate of auth failures per target over the last 10 minutes
rate(abaca_target_auth_failure_total[10m])

Examples

Example 1 — Verify the full metric set is being scraped

After adding the scrape target, confirm all expected metric families are present:

curl -s http://localhost:9798/metrics | grep '^abaca_' | awk -F'{' '{print $1}' | sort -u

Expected output (representative subset):

abaca_amqp_reachable
abaca_backup_coverage_ratio
abaca_db_replication_lag_seconds
abaca_disk_free_bytes
abaca_fleet_boot_errors_total
abaca_fleet_jobs_queued
abaca_fleet_oldest_queued_seconds
abaca_fleet_workers_active
abaca_repo_maintenance_overdue_seconds
abaca_restore_duration_seconds
abaca_restore_stuck_total
abaca_target_auth_failure_total
abaca_target_reachability

Example 2 — Fleet health endpoint response (healthy cluster)

curl -s -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/fleet/health

Expected response shape:

{
  "status": "healthy",
  "domains": [
    {
      "domain_id": "d1e2f3a4-...",
      "active_workers": 3,
      "queued_jobs": 0,
      "circuit_breaker_tripped": false
    }
  ]
}

Example 3 — Coverage endpoint showing an overdue policy

curl -s -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/coverage

Expected response shape (one policy overdue):

{
  "status": "degraded",
  "policies": [
    {
      "policy_id": "aabb1122-...",
      "project_id": "proj-uuid",
      "domain_id": "dom-uuid",
      "coverage_ratio": 0.0,
      "last_successful_backup": "2024-01-13T08:00:00Z",
      "next_scheduled": "2024-01-14T08:00:00Z",
      "overdue": true
    }
  ]
}

Example 4 — Pulling recent error-severity audit events for SIEM ingestion

SINCE=$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ)

curl -s -H "X-Auth-Token: $OS_TOKEN" \
  "https://abaca.example.com/v1/admin/events?since=${SINCE}" \
  | python3 -c "
import sys, json
events = json.load(sys.stdin).get('events', [])
for e in events:
    if e['severity'] == 'error':
        print(json.dumps(e))
"

Example output line:

{
  "id": "9f3c2b1a-...",
  "occurred_at": "2024-01-15T10:43:22Z",
  "actor": "system",
  "kind": "target_auth_failed",
  "severity": "error",
  "domain_id": "dom-uuid",
  "project_id": null,
  "subject_id": "target-uuid",
  "subject_kind": "target",
  "payload": {"reason": "Barbican trust revoked"}
}

Example 5 — PromQL: page alert for a completely stalled domain

# Fire when a domain has workers == 0 and jobs queued > 0 for 5+ minutes
(
  abaca_fleet_workers_active == 0
  and on(domain_id)
  abaca_fleet_jobs_queued > 0
)

This corresponds to the shipped AbacaFleetCritical rule. Load it as a Prometheus rule:

groups:
  - name: abaca-page
    rules:
      - alert: AbacaFleetCritical
        expr: |
          (abaca_fleet_workers_active == 0)
          and on(domain_id)
          (abaca_fleet_jobs_queued > 0)
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Domain {{ $labels.domain_id }}: no active workers with jobs queued"
          description: "Backups are stalled. Check fleet health endpoint and worker boot logs."

Troubleshooting

No `abaca_` metrics appear in Prometheus

Symptom: Prometheus shows no series with the abaca_ prefix; the scrape target may show connection refused or timeout.

Likely cause: The abaca-conductor pod is not running, or the metrics port (9798 by default, controlled by [worker] listener_port) is not exposed via a Kubernetes Service.

Fix:

Verify the conductor pod is running: kubectl -n abaca get pods -l app=abaca-conductor
Confirm the Service exists and exposes port 9798: kubectl -n abaca get svc abaca-conductor -o yaml
If using a ServiceMonitor, ensure the port name in the monitor matches the port name in the Service spec.
Port-forward and curl manually to isolate network vs. application issues: kubectl -n abaca port-forward deployment/abaca-conductor 9798:9798

`abaca_backup_coverage_ratio` stays at `0` even after successful backups

Symptom: The coverage metric reads 0 for a policy even though jobs are completing successfully.

Likely cause: The conductor's scheduler_interval or catalogue_sync_interval_seconds is very long, so the coverage gauge has not been recomputed yet. Alternatively, the policy's cron expression has never evaluated within the metric's lookback window.

Fix:

Check [conductor] scheduler_interval (default 60 seconds) and catalogue_sync_interval_seconds (default 3600) in your abaca.conf. For freshly enrolled policies, wait at least one full catalogue_sync_interval_seconds after the first successful backup.
Confirm the policy exists and has at least one available backup job: openstack share protection job list --policy-id <id>
Query the coverage endpoint directly to see whether the API agrees: GET /v1/admin/coverage

`AbacaTargetUnreachable` fires but the S3 bucket is accessible from the tenant

Symptom: The alert fires (abaca_target_reachability == 0), but manually accessing the S3 bucket from the tenant's network succeeds.

Likely cause: The S3 credentials or the Barbican trust used by the worker agent have been rotated or revoked without updating the Abacá target registration. The target_auth_failed audit event will carry the specific reason in its payload.

Fix:

Check the audit log: openstack share protection event list --kind target_auth_failed --subject-id <target-id>
If the trust was revoked, the tenant must re-grant a Keystone trust for this target.
If S3 credentials were rotated, re-register the target with updated credentials and re-run enrollment.
After remediation, wait for the next catalogue_sync_interval_seconds cycle, or trigger a manual reconciliation and observe abaca_target_reachability returning to 1.

`AbacaCircuitBreakerTripped` alert fires; new workers will not boot

Symptom: The circuit-breaker alert is active; the fleet health endpoint shows circuit_breaker_tripped: true for a domain; no new workers are being provisioned.

Likely cause: More than worker_boot_max_failures (default 3) worker boot failures occurred within worker_boot_failure_window_seconds (default 7200). Common root causes: an invalid Nova flavor, an unavailable worker image in Glance, or a network the conductor cannot attach to.

Fix:

Check abaca-conductor logs for worker_dead or boot-error events: kubectl -n abaca logs deployment/abaca-conductor | grep boot_error
Verify the worker image is present in Glance with the abaca_worker_image=1 property: openstack image list --property abaca_worker_image=1
Verify [conductor] worker_boot_flavor (default m1.small) and worker_boot_network are valid for the domain's service project.
After fixing the root cause, the circuit breaker will reset automatically after worker_boot_failure_window_seconds elapses. You can reduce the window in abaca.conf to speed up recovery in non-production environments.

Audit events are not appearing in the SIEM

Symptom: The /v1/admin/events endpoint returns rows, but your SIEM shows no data.

Likely cause: The log collector is either not polling the endpoint, is using an expired token, or the since timestamp is advancing incorrectly (e.g., always set to the current time, so no events fall in the window).

Fix:

Test the endpoint manually with a recent since value: curl -H "X-Auth-Token: $OS_TOKEN" "https://abaca.example.com/v1/admin/events?since=2024-01-01T00:00:00Z"
Ensure your collector stores the occurred_at of the last received event and uses it as the next since value — do not recompute from wall clock.
Confirm that the token used by the collector has not expired; service account tokens must be renewed before worker_token_ttl_seconds (default 1800 seconds).
Verify no secrets appear in logged event payloads — the payload field should contain only IDs and hrefs, never raw credential values. If secrets appear, file a security issue and rotate affected credentials immediately.

Structured log lines are missing `job_id` or `domain_id` fields

Symptom: Some log lines at INFO level are plain text or are missing expected JSON keys such as job_id, worker_id, or domain_id.

Likely cause: The log line originates from a code path that runs outside of a job context (e.g., startup, reconciliation sweep preamble), where no job or domain scope has been established yet. This is expected for lifecycle events.

Fix: This is not an error condition. Structured fields are emitted only when the relevant context (job, worker, domain) is known. For correlation of context-free lines, use the actor and kind fields of the corresponding audit event in abaca_events rather than the log line.

Notifications & Metrics

[conductor] — intervals that directly affect metric freshness

Alert rule threshold customization

Log level

Querying health endpoints

Using the OSC plugin to list events

Prometheus label-based filtering

Identifying the worst failure mode: backup-did-not-run

Checking target reachability

Example 1 — Verify the full metric set is being scraped

Example 2 — Fleet health endpoint response (healthy cluster)

Example 3 — Coverage endpoint showing an overdue policy

Example 4 — Pulling recent error-severity audit events for SIEM ingestion

Example 5 — PromQL: page alert for a completely stalled domain

No abaca_ metrics appear in Prometheus

abaca_backup_coverage_ratio stays at 0 even after successful backups

AbacaTargetUnreachable fires but the S3 bucket is accessible from the tenant

AbacaCircuitBreakerTripped alert fires; new workers will not boot

Audit events are not appearing in the SIEM

Structured log lines are missing job_id or domain_id fields

`[conductor]` — intervals that directly affect metric freshness

No `abaca_` metrics appear in Prometheus

`abaca_backup_coverage_ratio` stays at `0` even after successful backups

`AbacaTargetUnreachable` fires but the S3 bucket is accessible from the tenant

`AbacaCircuitBreakerTripped` alert fires; new workers will not boot

Structured log lines are missing `job_id` or `domain_id` fields