--- title: Metrics product: trilio-share-protection doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/metrics --- # Metrics _Exposed metrics endpoints and key service-level indicators_ ## Overview Trilio Share Protection (Abacá) exposes two complementary observability surfaces: a REST usage endpoint that returns per-tenant billing metrics and a Prometheus scrape endpoint that surfaces operational health signals for every red state in the admin panel. Together they give you the data you need for chargeback integration, SLA alerting, and capacity planning — without any Ceilometer bus dependency. This page explains both surfaces, how the underlying usage sampling works, and how to wire the metrics into your monitoring stack. ## Prerequisites Before using the metrics and usage endpoints: - **Trilio Share Protection ≥ 4.12** deployed with `abaca-api` and `abaca-conductor` running - A valid Keystone token scoped to the project whose usage you want to query (the usage API is tenant-scoped) - Network access to the `abaca-api` service endpoint - For Prometheus scraping: a Prometheus instance (any version supporting pull-based scraping) with network access to the `abaca-api` metrics endpoint - At least one backup target in `status=available` (targets that have not completed enrollment report no usage statistics) - **python-abacaclient** installed if you prefer the OpenStack CLI (`openstack share protection …`) over raw HTTP calls ``` pip install python-abacaclient ``` - Optional — **CloudKitty** or another external billing engine if you are wiring usage data into chargeback workflows ## Installation No additional components need to be installed to consume metrics — both the usage API and the Prometheus endpoint are served by the `abaca-api` process that is already running as part of your control plane. The steps below confirm that the endpoints are reachable and optionally configure Prometheus to scrape them. **Step 1 — Obtain a Keystone token** All calls to `abaca-api` use Keystone authentication. Export a token for the target project: ```bash export OS_AUTH_URL=https:///v3 export OS_PROJECT_NAME= export OS_USERNAME= export OS_PASSWORD= export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_DOMAIN_NAME=Default export TOKEN=$(openstack token issue -f value -c id) ``` **Step 2 — Verify the usage endpoint is reachable** ```bash curl -s -H "X-Auth-Token: $TOKEN" \ https:///v1/usage | python3 -m json.tool ``` A successful response returns a JSON object with fields such as `protected_shares`, `backup_count`, `protected_bytes`, and `stored_bytes`. If you receive a `401` the token is invalid or scoped to the wrong project. **Step 3 — Verify the Prometheus metrics endpoint is reachable** ```bash curl -s https:///metrics | head -40 ``` You should see `# HELP` and `# TYPE` lines for Abacá metric families. > **Note for reviewer:** The exact path of the Prometheus `/metrics` endpoint is not confirmed in the source material. `/metrics` is documented here as the conventional path; a human reviewer should confirm this against the actual `abaca-api` route registration before publishing. **Step 4 — Add a Prometheus scrape job (optional)** Add the following job to your `prometheus.yml`. Because the usage API is tenant-scoped via Keystone, the Prometheus scrape target should point to the operator-level metrics endpoint, not `/v1/usage`. ```yaml scrape_configs: - job_name: abaca scheme: https static_configs: - targets: - metrics_path: /metrics # If the endpoint requires authentication, configure bearer_token or # tls_config here according to your deployment's auth setup. scrape_interval: 60s ``` Reload Prometheus: ```bash curl -X POST http://localhost:9090/-/reload ``` ## Configuration ### Conductor — usage sampling cadence The conductor's reconciliation loop periodically calls `kopia content stats` against each enrolled backup target to populate accurate post-deduplication, post-compression storage figures. This is controlled by a single configuration key: | Key | Section | Default | Effect | |---|---|---|---| | `usage_sample_interval_seconds` | `[conductor]` | `21600` (6 hours) | How often the conductor re-samples Kopia content statistics for each `available` target. Lower values give fresher figures at the cost of more Kopia RPC calls. | Set this in your conductor configuration file: ```ini [conductor] usage_sample_interval_seconds = 21600 ``` Sampling is **best-effort**: if a target's worker fleet has no free capacity slots, or if a Kopia or Barbican call fails, the target's `usage_stats` are left unchanged and the next reconciliation tick retries. One failing target does not block the rest of the sweep. ### Why two byte fields exist Abacá tracks two distinct byte counts that represent very different costs: | Field | Meaning | Billing relevance | |---|---|---| | `protected_bytes` | Logical size of data under protection (post-dedup across snapshots) | Headline rate: GB-month of protected capacity | | `stored_bytes` | Actual bytes written to S3 (post-compression) | Supplementary: real S3 storage consumed | Summing `Backup.size_bytes` across all backups on a target would overcount because Kopia's cross-snapshot deduplication means three 15 MB snapshots may occupy only ~15 MB in the bucket. The conductor's `kopia content stats` sampling captures the real S3 footprint and stores it on `BackupTarget.usage_stats`, keeping the Target Details view fast (a single DB read rather than a Kopia RPC per page load). ### Prometheus alert rules Alert rule configuration is **operator-side** — Abacá ships the metrics but does not bundle alert rules. The metrics are designed so that every red state visible in the `abaca-dashboard` admin panel has a corresponding Prometheus metric, ensuring human views and machine alerting never disagree. Key signals to alert on are described in the [Usage](#usage) section below. ## Usage ### Querying the tenant usage API The `/v1/usage` endpoint returns a summary of your project's current protection footprint. It is scoped to the Keystone project in your token — you see only your own data. ```bash curl -s -H "X-Auth-Token: $TOKEN" \ https:///v1/usage ``` Or with the OpenStack CLI: ```bash openstack share protection usage show ``` The response fields map directly to billing dimensions: | Field | Type | Description | |---|---|---| | `project_id` | string | The Keystone project this summary covers | | `protected_shares` | integer | Number of distinct shares covered by at least one active protection policy | | `backup_count` | integer | Total number of backup records for this project | | `protected_bytes` | integer | Logical bytes under protection (post-dedup, see Configuration) | | `stored_bytes` | integer | Actual bytes stored on S3 (post-compression) | ### Prometheus metrics for operational health The Prometheus endpoint is intended for operators monitoring fleet health. Key metric families to track: - **Heartbeat loss** — worker VMs that have stopped sending heartbeats. Alert when non-zero; indicates a worker that may need to be recycled by the conductor. - **Coverage breach** — shares that have an active protection policy but have not produced a successful backup within their scheduled window. This is the most important operational signal: a missed backup is a first-class failure, not a derived report. - **Overdue maintenance** — Kopia repository maintenance jobs that are past their scheduled run time. Overdue maintenance can cause repository bloat and should be alerted promptly. - **Target authentication failure** — targets whose Barbican credential fetch is failing. If non-zero, backups for affected tenants are silently not running. - **Queue latency** — time jobs spend in the `queued` state before a worker picks them up. Sustained high latency indicates insufficient worker capacity slots. ### Wiring into a billing engine Abacá's usage API is designed to be consumed by pull-based billing systems such as CloudKitty. Poll `/v1/usage` per project on your billing cycle cadence. The priority billing dimensions are: 1. **Protected capacity GB-month** — sample `protected_bytes` at regular intervals and compute the time-weighted average over the billing period. 2. **GB transferred per job** — available on individual job records; sum `transferred_bytes` across completed backup jobs in the period. 3. **Restore count** — count restore jobs in the `available` state for the period; zero-rated by default but the data is collected from day one so billing can be enabled without retroactive loss. No Ceilometer bus integration is used or required. ## Examples ### Example 1 — Fetch your project's usage summary Retrieve a snapshot of your current protection footprint: ```bash export TOKEN=$(openstack token issue -f value -c id) curl -s \ -H "X-Auth-Token: $TOKEN" \ -H "Content-Type: application/json" \ https:///v1/usage ``` Expected response: ```json { "project_id": "a3f2c1d4e5b6789012345678abcdef01", "protected_shares": 4, "backup_count": 27, "protected_bytes": 42949672960, "stored_bytes": 15032385536 } ``` In this example the tenant has 4 shares protected, 27 backup records, about 40 GB of logical data under protection, and roughly 14 GB actually stored on S3 after deduplication and compression. The difference reflects Kopia's cross-snapshot deduplication. --- ### Example 2 — Fetch usage via the OpenStack CLI ```bash openstack share protection usage show ``` Expected output: ``` +-------------------+-----------------------------+ | Field | Value | +-------------------+-----------------------------+ | project_id | a3f2c1d4e5b6789012345678... | | protected_shares | 4 | | backup_count | 27 | | protected_bytes | 42949672960 | | stored_bytes | 15032385536 | +-------------------+-----------------------------+ ``` --- ### Example 3 — Scrape Prometheus metrics and check coverage Pull a snapshot of the metrics endpoint and filter for coverage-related lines: ```bash curl -s https:///metrics \ | grep -E '^abaca_coverage' ``` Example output (metric names are illustrative — confirm against your deployment): ``` # HELP abaca_coverage_breached_shares Shares with an active policy that have not completed a successful backup within their scheduled window # TYPE abaca_coverage_breached_shares gauge abaca_coverage_breached_shares{project_id="a3f2c1d4e5b6789012345678abcdef01"} 0 abaca_coverage_breached_shares{project_id="b1e2d3c4f5a6789012345678bcdef012"} 1 ``` A non-zero value for any project means at least one share is not being backed up on schedule and requires immediate investigation. > **Note for reviewer:** Exact Prometheus metric names are not confirmed in the source material. The names above (`abaca_coverage_breached_shares`, etc.) are illustrative. A human reviewer must replace them with the actual registered metric names from the `abaca-api` source before publishing. --- ### Example 4 — Convert bytes to GB for a billing report ```python import os import requests token = os.environ["OS_TOKEN"] base_url = os.environ["ABACA_API_URL"] # e.g. https://abaca.example.com resp = requests.get( f"{base_url}/v1/usage", headers={"X-Auth-Token": token}, timeout=10, ) resp.raise_for_status() data = resp.json() GB = 1024 ** 3 print(f"Project: {data['project_id']}") print(f"Protected shares: {data['protected_shares']}") print(f"Total backups: {data['backup_count']}") print(f"Protected (GB): {data['protected_bytes'] / GB:.2f}") print(f"Stored on S3 (GB): {data['stored_bytes'] / GB:.2f}") ``` Expected output: ``` Project: a3f2c1d4e5b6789012345678abcdef01 Protected shares: 4 Total backups: 27 Protected (GB): 40.00 Stored on S3 (GB): 14.00 ``` ## Troubleshooting ### `stored_bytes` and `protected_bytes` are both `0` **Symptom:** The `/v1/usage` response returns `0` for both byte fields even though backups have completed successfully. **Likely cause:** The conductor has not yet completed its first usage sampling sweep for your target, or all of your targets are not in `status=available` (for example, enrollment is still in progress). **Fix:** 1. Confirm your backup target status: `openstack share protection target show ` and verify `status` is `available`. 2. Wait for the conductor's next sampling tick (default: 6 hours). You can temporarily reduce `usage_sample_interval_seconds` in the conductor configuration to force a faster sample, then restore the default. 3. Check conductor logs for lines containing `target_usage:` — `WARNING` entries will identify the specific failure (Barbican fetch error, no available worker slot, Kopia failure). --- ### `stored_bytes` value seems unexpectedly low compared to `backup_count` **Symptom:** You have many backups recorded but `stored_bytes` is much smaller than `backup_count × average_backup_size`. **Likely cause:** This is expected behavior, not an error. Kopia's incremental-forever deduplication means that after the initial backup, subsequent snapshots of unchanged data add very little to S3. `stored_bytes` reflects actual S3 consumption after deduplication and compression; `protected_bytes` reflects logical data size. The difference is the deduplication saving. **Fix:** No action required. If you need the per-backup logical size for a specific snapshot, inspect individual backup records rather than the aggregate usage summary. --- ### `/v1/usage` returns `401 Unauthorized` **Symptom:** The API returns a `401` response. **Likely cause:** Your Keystone token is expired, scoped to the wrong project, or missing. **Fix:** 1. Re-issue the token: `export TOKEN=$(openstack token issue -f value -c id)` 2. Confirm the token is scoped to the project you intend to query. 3. Verify the `abaca-api` Keystone middleware is configured with the correct Keystone endpoint. --- ### Prometheus scrape returns no data or connection refused **Symptom:** Prometheus shows the `abaca` scrape target as `DOWN`, or `curl https:///metrics` returns a connection error. **Likely cause:** The `abaca-api` pod is not running, the metrics path is incorrect, or network policy is blocking the Prometheus scraper. **Fix:** 1. Confirm `abaca-api` is running: check the pod status in your OpenShift namespace. 2. Verify the metrics endpoint path against your deployment — the path `/metrics` is conventional but should be confirmed with your operator. 3. Check OpenShift `NetworkPolicy` resources to ensure the Prometheus namespace is allowed to reach the `abaca-api` service port. --- ### Conductor logs show `target_usage: no ACTIVE worker with a free slot` **Symptom:** Usage statistics are not being updated despite targets being in `available` status. Conductor logs contain the message `sample deferred to next tick`. **Likely cause:** All worker VMs currently have their capacity slots fully occupied by backup or restore jobs, so no slot is available for the read-only `kopia content stats` call. **Fix:** This is a transient condition — the sample will run automatically on the next reconciliation tick once a slot becomes free. If it persists, consider increasing worker fleet capacity or reducing concurrent job load. Usage sampling does not require a dedicated slot and will proceed as soon as any worker has a free slot, so no immediate action is required for correctness. --- ### `target_usage: barbican fetch failed` in conductor logs **Symptom:** Usage statistics for a specific target are stale. Conductor logs show a `WARNING` with `barbican fetch failed` for that target. **Likely cause:** The Keystone trust associated with the target has expired or been revoked, or the Barbican secrets (S3 credentials or repository password) have been deleted. **Fix:** 1. Check the target's `error_category` field — `tenant_action_required` means the tenant must re-enroll the target or restore the Barbican secrets; `operator_action_required` means the infrastructure (Barbican service or Keystone) needs attention. 2. If the trust has expired, the target must be re-enrolled to generate a new Keystone trust. 3. Verify that the Barbican secrets referenced by the target still exist and are readable by the service account.