Metrics

Trilio Share Protection (Abacá) exposes two complementary observability surfaces: a REST usage endpoint that returns per-tenant billing metrics and a Prometheus scrape endpoint that surfaces operational health signals for every red state in the admin panel. Together they give you the data you need for chargeback integration, SLA alerting, and capacity planning — without any Ceilometer bus dependency. This page explains both surfaces, how the underlying usage sampling works, and how to wire the metrics into your monitoring stack.

Before using the metrics and usage endpoints:

• Trilio Share Protection ≥ 4.12 deployed with abaca-api and abaca-conductor running
• A valid Keystone token scoped to the project whose usage you want to query (the usage API is tenant-scoped)
• Network access to the abaca-api service endpoint
• For Prometheus scraping: a Prometheus instance (any version supporting pull-based scraping) with network access to the abaca-api metrics endpoint
• At least one backup target in status=available (targets that have not completed enrollment report no usage statistics)
• python-abacaclient installed if you prefer the OpenStack CLI (openstack share protection …) over raw HTTP calls
  Copy

  pip install python-abacaclient
  

• Optional — CloudKitty or another external billing engine if you are wiring usage data into chargeback workflows

No additional components need to be installed to consume metrics — both the usage API and the Prometheus endpoint are served by the abaca-api process that is already running as part of your control plane. The steps below confirm that the endpoints are reachable and optionally configure Prometheus to scrape them.

Step 1 — Obtain a Keystone token

All calls to abaca-api use Keystone authentication. Export a token for the target project:

export OS_AUTH_URL=https://<keystone-endpoint>/v3
export OS_PROJECT_NAME=<your-project>
export OS_USERNAME=<your-username>
export OS_PASSWORD=<your-password>
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default

export TOKEN=$(openstack token issue -f value -c id)

Step 2 — Verify the usage endpoint is reachable

curl -s -H "X-Auth-Token: $TOKEN" \
  https://<abaca-api-endpoint>/v1/usage | python3 -m json.tool

A successful response returns a JSON object with fields such as protected_shares, backup_count, protected_bytes, and stored_bytes. If you receive a 401 the token is invalid or scoped to the wrong project.

Step 3 — Verify the Prometheus metrics endpoint is reachable

curl -s https://<abaca-api-endpoint>/metrics | head -40

You should see # HELP and # TYPE lines for Abacá metric families.

Step 4 — Add a Prometheus scrape job (optional)

Add the following job to your prometheus.yml. Because the usage API is tenant-scoped via Keystone, the Prometheus scrape target should point to the operator-level metrics endpoint, not /v1/usage.

scrape_configs:
  - job_name: abaca
    scheme: https
    static_configs:
      - targets:
          - <abaca-api-endpoint>
    metrics_path: /metrics
    # If the endpoint requires authentication, configure bearer_token or
    # tls_config here according to your deployment's auth setup.
    scrape_interval: 60s

Reload Prometheus:

curl -X POST http://localhost:9090/-/reload

Conductor — usage sampling cadence

The conductor's reconciliation loop periodically calls kopia content stats against each enrolled backup target to populate accurate post-deduplication, post-compression storage figures. This is controlled by a single configuration key:

Set this in your conductor configuration file:

[conductor]
usage_sample_interval_seconds = 21600

Sampling is best-effort: if a target's worker fleet has no free capacity slots, or if a Kopia or Barbican call fails, the target's usage_stats are left unchanged and the next reconciliation tick retries. One failing target does not block the rest of the sweep.

Why two byte fields exist

Abacá tracks two distinct byte counts that represent very different costs:

Summing Backup.size_bytes across all backups on a target would overcount because Kopia's cross-snapshot deduplication means three 15 MB snapshots may occupy only ~15 MB in the bucket. The conductor's kopia content stats sampling captures the real S3 footprint and stores it on BackupTarget.usage_stats, keeping the Target Details view fast (a single DB read rather than a Kopia RPC per page load).

Prometheus alert rules

Alert rule configuration is operator-side — Abacá ships the metrics but does not bundle alert rules. The metrics are designed so that every red state visible in the abaca-dashboard admin panel has a corresponding Prometheus metric, ensuring human views and machine alerting never disagree. Key signals to alert on are described in the Usage section below.

Querying the tenant usage API

The /v1/usage endpoint returns a summary of your project's current protection footprint. It is scoped to the Keystone project in your token — you see only your own data.

curl -s -H "X-Auth-Token: $TOKEN" \
  https://<abaca-api-endpoint>/v1/usage

Or with the OpenStack CLI:

openstack share protection usage show

The response fields map directly to billing dimensions:

Prometheus metrics for operational health

The Prometheus endpoint is intended for operators monitoring fleet health. Key metric families to track:

• Heartbeat loss — worker VMs that have stopped sending heartbeats. Alert when non-zero; indicates a worker that may need to be recycled by the conductor.
• Coverage breach — shares that have an active protection policy but have not produced a successful backup within their scheduled window. This is the most important operational signal: a missed backup is a first-class failure, not a derived report.
• Overdue maintenance — Kopia repository maintenance jobs that are past their scheduled run time. Overdue maintenance can cause repository bloat and should be alerted promptly.
• Target authentication failure — targets whose Barbican credential fetch is failing. If non-zero, backups for affected tenants are silently not running.
• Queue latency — time jobs spend in the queued state before a worker picks them up. Sustained high latency indicates insufficient worker capacity slots.

Wiring into a billing engine

Abacá's usage API is designed to be consumed by pull-based billing systems such as CloudKitty. Poll /v1/usage per project on your billing cycle cadence. The priority billing dimensions are:

1. Protected capacity GB-month — sample protected_bytes at regular intervals and compute the time-weighted average over the billing period.
2. GB transferred per job — available on individual job records; sum transferred_bytes across completed backup jobs in the period.
3. Restore count — count restore jobs in the available state for the period; zero-rated by default but the data is collected from day one so billing can be enabled without retroactive loss.

No Ceilometer bus integration is used or required.

Example 1 — Fetch your project's usage summary

Retrieve a snapshot of your current protection footprint:

export TOKEN=$(openstack token issue -f value -c id)

curl -s \
  -H "X-Auth-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  https://<abaca-api-endpoint>/v1/usage

Expected response:

{
  "project_id": "a3f2c1d4e5b6789012345678abcdef01",
  "protected_shares": 4,
  "backup_count": 27,
  "protected_bytes": 42949672960,
  "stored_bytes": 15032385536
}

In this example the tenant has 4 shares protected, 27 backup records, about 40 GB of logical data under protection, and roughly 14 GB actually stored on S3 after deduplication and compression. The difference reflects Kopia's cross-snapshot deduplication.

Example 2 — Fetch usage via the OpenStack CLI

openstack share protection usage show

Expected output:

+-------------------+-----------------------------+
| Field             | Value                       |
+-------------------+-----------------------------+
| project_id        | a3f2c1d4e5b6789012345678... |
| protected_shares  | 4                           |
| backup_count      | 27                          |
| protected_bytes   | 42949672960                 |
| stored_bytes      | 15032385536                 |
+-------------------+-----------------------------+

Example 3 — Scrape Prometheus metrics and check coverage

Pull a snapshot of the metrics endpoint and filter for coverage-related lines:

curl -s https://<abaca-api-endpoint>/metrics \
  | grep -E '^abaca_coverage'

Example output (metric names are illustrative — confirm against your deployment):

# HELP abaca_coverage_breached_shares Shares with an active policy that have not completed a successful backup within their scheduled window
# TYPE abaca_coverage_breached_shares gauge
abaca_coverage_breached_shares{project_id="a3f2c1d4e5b6789012345678abcdef01"} 0
abaca_coverage_breached_shares{project_id="b1e2d3c4f5a6789012345678bcdef012"} 1

A non-zero value for any project means at least one share is not being backed up on schedule and requires immediate investigation.

Example 4 — Convert bytes to GB for a billing report

import os
import requests

token = os.environ["OS_TOKEN"]
base_url = os.environ["ABACA_API_URL"]  # e.g. https://abaca.example.com

resp = requests.get(
    f"{base_url}/v1/usage",
    headers={"X-Auth-Token": token},
    timeout=10,
)
resp.raise_for_status()
data = resp.json()

GB = 1024 ** 3
print(f"Project:           {data['project_id']}")
print(f"Protected shares:  {data['protected_shares']}")
print(f"Total backups:     {data['backup_count']}")
print(f"Protected (GB):    {data['protected_bytes'] / GB:.2f}")
print(f"Stored on S3 (GB): {data['stored_bytes'] / GB:.2f}")

Expected output:

Project:           a3f2c1d4e5b6789012345678abcdef01
Protected shares:  4
Total backups:     27
Protected (GB):    40.00
Stored on S3 (GB): 14.00

stored_bytes and protected_bytes are both 0

Symptom: The /v1/usage response returns 0 for both byte fields even though backups have completed successfully.

Likely cause: The conductor has not yet completed its first usage sampling sweep for your target, or all of your targets are not in status=available (for example, enrollment is still in progress).

Fix:

1. Confirm your backup target status: openstack share protection target show <target-id> and verify status is available.
2. Wait for the conductor's next sampling tick (default: 6 hours). You can temporarily reduce usage_sample_interval_seconds in the conductor configuration to force a faster sample, then restore the default.
3. Check conductor logs for lines containing target_usage: — WARNING entries will identify the specific failure (Barbican fetch error, no available worker slot, Kopia failure).

stored_bytes value seems unexpectedly low compared to backup_count

Symptom: You have many backups recorded but stored_bytes is much smaller than backup_count × average_backup_size.

Likely cause: This is expected behavior, not an error. Kopia's incremental-forever deduplication means that after the initial backup, subsequent snapshots of unchanged data add very little to S3. stored_bytes reflects actual S3 consumption after deduplication and compression; protected_bytes reflects logical data size. The difference is the deduplication saving.

Fix: No action required. If you need the per-backup logical size for a specific snapshot, inspect individual backup records rather than the aggregate usage summary.

/v1/usage returns 401 Unauthorized

Symptom: The API returns a 401 response.

Likely cause: Your Keystone token is expired, scoped to the wrong project, or missing.

Fix:

1. Re-issue the token: export TOKEN=$(openstack token issue -f value -c id)
2. Confirm the token is scoped to the project you intend to query.
3. Verify the abaca-api Keystone middleware is configured with the correct Keystone endpoint.

Prometheus scrape returns no data or connection refused

Symptom: Prometheus shows the abaca scrape target as DOWN, or curl https://<abaca-api-endpoint>/metrics returns a connection error.

Likely cause: The abaca-api pod is not running, the metrics path is incorrect, or network policy is blocking the Prometheus scraper.

Fix:

1. Confirm abaca-api is running: check the pod status in your OpenShift namespace.
2. Verify the metrics endpoint path against your deployment — the path /metrics is conventional but should be confirmed with your operator.
3. Check OpenShift NetworkPolicy resources to ensure the Prometheus namespace is allowed to reach the abaca-api service port.

Conductor logs show target_usage: no ACTIVE worker with a free slot

Symptom: Usage statistics are not being updated despite targets being in available status. Conductor logs contain the message sample deferred to next tick.

Likely cause: All worker VMs currently have their capacity slots fully occupied by backup or restore jobs, so no slot is available for the read-only kopia content stats call.

Fix: This is a transient condition — the sample will run automatically on the next reconciliation tick once a slot becomes free. If it persists, consider increasing worker fleet capacity or reducing concurrent job load. Usage sampling does not require a dedicated slot and will proceed as soon as any worker has a free slot, so no immediate action is required for correctness.

target_usage: barbican fetch failed in conductor logs

Symptom: Usage statistics for a specific target are stale. Conductor logs show a WARNING with barbican fetch failed for that target.

Likely cause: The Keystone trust associated with the target has expired or been revoked, or the Barbican secrets (S3 credentials or repository password) have been deleted.

Fix:

1. Check the target's error_category field — tenant_action_required means the tenant must re-enroll the target or restore the Barbican secrets; operator_action_required means the infrastructure (Barbican service or Keystone) needs attention.
2. If the trust has expired, the target must be re-enrolled to generate a new Keystone trust.
3. Verify that the Barbican secrets referenced by the target still exist and are readable by the service account.