API Layer

This page describes the abaca-api WSGI application: how requests are routed, authenticated, and dispatched; the structure of the resource tree; and the lifecycle of a request from token validation through response serialization. Understanding this layer is essential whether you are integrating an external system against the REST API, writing policy overrides, or diagnosing unexpected 4xx or 5xx responses. The API is versioned at /v1, follows OpenStack conventions (Keystone tokens, oslo.policy, service catalog discovery), and exposes two consumer surfaces: a tenant-facing resource group and an operator-facing admin resource group.

Before working with the abaca-api layer you need:

• RHOSO ≥ 18 with OpenShift ≥ 4.14 hosting the control plane
• Keystone ≥ 2023.1 with domain-scoped tokens and trusts enabled — all API requests require a valid Keystone token
• OpenStack Barbican — the API stores and resolves S3 credential references as Barbican secret hrefs; the service account must have access
• MySQL or MariaDB (Galera) — the dedicated Abacá database must be reachable and fully migrated (abaca-manage db_sync) before the API pod starts
• RabbitMQ (dedicated Abacá broker) — the API dispatches work to the conductor over oslo.messaging; it must be reachable at startup
• python-abacaclient installed in the client environment if you plan to use openstack share protection … CLI commands
• A valid clouds.yaml or environment-variable credentials (OS_*) pointing at the target OpenStack cluster

The abaca-api service runs as an OpenShift pod in the abaca namespace. You do not install it separately; it is deployed as part of the full control-plane rollout.

Step 1 — Run the identity and catalog bootstrap script

This script creates the Keystone service entry, registers the share-protection endpoints in the service catalog, and provisions the Abacá service account:

bash deploy/rhoso/01-identity.sh

Step 2 — Provision the database and message bus

bash deploy/rhoso/02-database.sh
bash deploy/rhoso/03-messagebus.sh

Step 3 — Apply the kustomize manifests

This deploys the abaca-api Deployment, the abaca-conductor Deployment, the db-sync Job, the OpenShift Service, and the Route:

kubectl apply -k deploy/rhoso/

The db-sync Job runs abaca-manage db_sync to apply any pending Alembic migrations before the API pod reaches Running.

Step 4 — Verify the pod is healthy

kubectl -n abaca get pods
kubectl -n abaca logs deployment/abaca-api

Expect the API pod to log Starting abaca-api on 0.0.0.0:9797 once configuration is valid and the database connection is established.

Step 5 — Install the CLI plugin on operator workstations

python-abacaclient is layered into the environment that already has python-openstackclient:

pip install python-abacaclient

Verify the plugin is registered:

openstack share protection --help

Local developer stack (optional)

For local development, the API and conductor run in --noauth mode via Docker Compose:

docker compose -f deploy/docker-compose.dev.yml up

In this mode Keystone authentication is disabled (noauth = true) and SQLite with in-memory fakes replaces the database and message bus. Do not use this configuration against a production cluster.

Configuration is managed by oslo.config. Options are declared in INI-style .conf files and grouped by section. The sections relevant to the API process are [DEFAULT], [abaca], [api], and [database]. Set these in your abaca.conf before starting the pod.

[DEFAULT]

[abaca]

[api]

[database]

[enrollment]

Annotated sample abaca.conf

[DEFAULT]
catalog_type = share-protection
endpoint_type = internalURL

[abaca]
service_user_id = <keystone-user-id>
service_user_name = abaca
worker_project_id = <service-project-id>
worker_project_name = abaca-service

[api]
bind_host = 0.0.0.0
bind_port = 9797
noauth = false
max_limit = 1000
default_limit = 100

[database]
connection = mysql+pymysql://abaca:SECRET@galera-host/abaca

[enrollment]
dispatch = rpc

All three consumer surfaces — the REST API, the openstack share protection CLI plugin, and the python-abacaclient Python SDK — reach the same abaca-api WSGI process. Each surface is shown below for the most common operations.

Authentication

Every request must carry a valid Keystone token. The CLI plugin handles this automatically via clouds.yaml. For direct HTTP calls, obtain a token first:

export OS_TOKEN=$(openstack token issue -f value -c id)
curl -H "X-Auth-Token: $OS_TOKEN" https://<abaca-api-host>/v1/targets

Tenant-facing resources

The tenant surface is rooted at /v1. All list endpoints are paginated, sorted newest-first by default, and accept limit and marker query parameters.

Backup targets (/v1/targets)

Register an S3 bucket as a backup target. This triggers the preflight conformance check and runs an enrollment job on a worker VM:

openstack share protection target create \
  --template-id <template-id> \
  --bucket-name my-backup-bucket

List registered targets for the current project:

openstack share protection target list

Backup policies (/v1/policies)

Create a policy that schedules daily backups of a share:

openstack share protection policy create \
  --share-id <share-id> \
  --target-template-id <template-id> \
  --schedule "0 2 * * *" \
  --name daily-backup

Backups (/v1/backups)

Request an on-demand backup:

openstack share protection backup create --share-id <share-id>

List backups for a specific share (expired backups are hidden by default; pass --include-expired to surface them):

openstack share protection backup list --share-id <share-id>

Restores (/v1/restores)

Restore a backup to a new share:

openstack share protection restore create \
  --backup-id <backup-id> \
  --mode new_share

In-place restore overwrites the original share and requires explicit confirmation:

openstack share protection restore create \
  --backup-id <backup-id> \
  --mode in_place \
  --force

Restore a sub-path only:

openstack share protection restore create \
  --backup-id <backup-id> \
  --mode new_share \
  --sub-path /data/reports

Jobs (/v1/jobs)

Poll a job's state machine status:

openstack share protection job show <job-id>

Cancel a running job:

openstack share protection job cancel <job-id>

Usage (/v1/usage)

Retrieve consumption metrics for the current tenant:

openstack share protection usage show

Admin resources

Admin endpoints live under /v1/admin and require a Keystone role that satisfies the abaca:admin oslo.policy default rule. Use these to operate the service across all tenants.

Worker fleet (/v1/admin/workers)

openstack share protection admin worker list
openstack share protection admin worker drain <worker-id>
openstack share protection admin worker retire <worker-id>

All-tenant jobs (/v1/admin/jobs)

openstack share protection admin job list
openstack share protection admin job retry <job-id>
openstack share protection admin job cancel <job-id>

Coverage (/v1/admin/coverage)

Shows shares whose last successful backup exceeds their policy interval:

openstack share protection admin coverage show

Scheduling (/v1/admin/scheduling)

Pause all scheduled backups globally (for maintenance windows):

openstack share protection admin scheduling pause
openstack share protection admin scheduling resume

Reconciliation (/v1/admin/reconciliation)

Trigger a manual reconciliation sweep:

openstack share protection admin reconciliation trigger

Pagination

All list responses include a pagination envelope:

{
  "pagination": {
    "count": 250,
    "limit": 100,
    "next_marker": "abc123"
  },
  "backups": [ ... ]
}

Pass next_marker as the marker query parameter to fetch the next page. The server never returns more than max_limit (default 1000) items in one response regardless of what the client requests.

Error responses

Every error response uses the same envelope:

{
  "code": "target_not_found",
  "title": "Target not found",
  "detail": "No target with id 'abc' exists in this project.",
  "category": "tenant_action_required"
}

The category field is always one of tenant_action_required (the tenant must fix credentials, permissions, or request parameters) or operator_action_required (the operator must fix fleet, network, or service configuration). Use this field in alerting and runbooks to route failures to the correct team.

Example 1 — Discover the API endpoint from the service catalog

Before making direct HTTP calls, resolve the correct URL from Keystone:

openstack catalog show share-protection

Expected output (abbreviated):

+-----------+-------------------------------------------------------+
| Field     | Value                                                 |
+-----------+-------------------------------------------------------+
| endpoints | public: https://abaca.example.com/v1                  |
|           | internal: https://abaca.abaca.svc.cluster.local/v1    |
|           | admin: https://abaca.example.com/v1                   |
| name      | abaca                                                 |
| type      | share-protection                                      |
+-----------+-------------------------------------------------------+

The catalog_type config option (share-protection by default) determines what type string to look up.

Example 2 — Create an on-demand backup and poll until complete

# Request the backup
BACKUP=$(openstack share protection backup create \
  --share-id d3f1a2b4-0001-0001-0001-000000000001 \
  -f json)

JOB_ID=$(echo $BACKUP | python3 -c "import sys,json; print(json.load(sys.stdin)['job_id'])")
echo "Job: $JOB_ID"

# Poll until the job reaches a terminal state
while true; do
  STATE=$(openstack share protection job show $JOB_ID -f value -c state)
  echo "State: $STATE"
  [[ "$STATE" == "available" || "$STATE" == "error" ]] && break
  sleep 10
done

Expected terminal output:

Job: 7e3c1a00-dead-beef-0000-000000000042
State: queued
State: provisioning_network
State: provisioning_source
State: connecting_repository
State: transferring
State: finalizing
State: releasing
State: available

If the job ends in error, retrieve the full error detail:

openstack share protection job show $JOB_ID -f json

The response includes error_category (tenant_action_required or operator_action_required) so you know immediately who needs to act.

Example 3 — List backups, show expired entries, filter by share

By default, expired backups are hidden. To include them:

# Newest-first, expired included, filtered to one share
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  "https://abaca.example.com/v1/backups?share_id=d3f1a2b4-0001-0001-0001-000000000001&include_expired=true&limit=20" \
  | python3 -m json.tool

Expected shape:

{
  "pagination": {
    "count": 3,
    "limit": 20,
    "next_marker": null
  },
  "backups": [
    {
      "id": "bkp-001",
      "share_id": "d3f1a2b4-0001-0001-0001-000000000001",
      "status": "available",
      "crash_consistent": true,
      "kopia_snapshot_id": "k1a2b3c4d5",
      "target_bucket_id": "tgt-001"
    },
    {
      "id": "bkp-000",
      "share_id": "d3f1a2b4-0001-0001-0001-000000000001",
      "status": "expired",
      "crash_consistent": true,
      "kopia_snapshot_id": "k0z9y8x7w6",
      "target_bucket_id": "tgt-001"
    }
  ]
}

Example 4 — Restore a single sub-path to a new share via the REST API

curl -s -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "restore": {
      "backup_id": "bkp-001",
      "mode": "new_share",
      "sub_path": "/data/reports/2024"
    }
  }' \
  https://abaca.example.com/v1/restores

Expected response (202 Accepted):

{
  "restore": {
    "id": "rst-042",
    "backup_id": "bkp-001",
    "mode": "new_share",
    "sub_path": "/data/reports/2024",
    "force": false,
    "status": "creating"
  }
}

Example 5 — Pause scheduling before a maintenance window (admin)

# Pause all policy-driven scheduling globally
curl -s -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/scheduling/pause
# Returns 202 Accepted

# ... perform maintenance ...

# Resume
curl -s -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  https://abaca.example.com/v1/admin/scheduling/resume

Example 6 — Using the Python SDK

import openstack
from abacaclient import Client as AbacaClient

conn = openstack.connect(cloud='mycloud')
token = conn.auth_token
endpoint = conn.endpoint_for('share-protection', interface='public')

client = AbacaClient(endpoint=endpoint, token=token)

# List policies
for policy in client.policies.list():
    print(policy.id, policy.name, policy.schedule, policy.enabled)

# Create an on-demand backup
backup = client.backups.create(share_id='d3f1a2b4-0001-0001-0001-000000000001')
print('Backup job:', backup.job_id)

Use the following patterns to diagnose common API-layer failures. Retrieve logs with:

kubectl -n abaca logs deployment/abaca-api

Issue: All API requests return 401 Unauthorized

Symptom: Every request receives a 401 with code: auth_required.

Likely causes:

1. The Keystone token has expired.
2. The OS_AUTH_URL in your environment points at the wrong Keystone endpoint.
3. The abaca-api pod cannot reach Keystone to validate the token (network policy or wrong endpoint_type in config).

Fix:

• Re-issue a token: openstack token issue
• Confirm openstack catalog show identity returns a reachable URL.
• Check endpoint_type in [DEFAULT]; for pod-to-pod traffic inside OpenShift, set endpoint_type = internalURL.
• Check the API pod logs for keystonemiddleware connection errors.

Issue: 403 Forbidden on admin endpoints

Symptom: Requests to /v1/admin/... return 403 even with a valid token.

Likely cause: The requesting user does not hold a role that satisfies the abaca:admin oslo.policy rule for that endpoint.

Fix:

• Confirm the user has the required Keystone role in the target project/domain: openstack role assignment list --user <username> --names
• If you have customized access control, check policy.yaml in the API pod's config volume for an override that is more restrictive than the default.
• The category: tenant_action_required field in the error body confirms the problem is on the client side (credentials/roles), not the service.

Issue: API pod fails to start — database connection refused

Symptom: The abaca-api pod enters CrashLoopBackOff; logs show OperationalError: (2003) Can't connect to MySQL server.

Likely causes:

1. The [database] connection URL in abaca.conf is wrong (hostname, port, password, or database name).
2. The db-sync Job did not complete successfully before the pod started.
3. The Galera cluster is not reachable from the abaca namespace.

Fix:

• Check the db-sync Job: kubectl -n abaca logs job/abaca-db-sync
• Verify the connection string: kubectl -n abaca exec deployment/abaca-api -- python3 -c "import sqlalchemy; e = sqlalchemy.create_engine('$DB_URL'); e.connect()"
• Confirm network policies allow egress from the abaca namespace to the database host on port 3306.

Issue: List endpoints return an empty page even though data exists

Symptom: GET /v1/backups returns an empty backups array and pagination.count = 0, but backups are known to exist.

Likely cause: The request token is scoped to a project that has no backups, or expired backups are filtered out by default.

Fix:

• Confirm the token is scoped to the correct project: openstack token issue -f json | python3 -m json.tool | grep project
• For backups, add ?include_expired=true to surface expired records: GET /v1/backups?include_expired=true
• Admin users can use GET /v1/admin/jobs (with appropriate role) to view cross-tenant data.

Issue: Pagination produces duplicate or missing records

Symptom: Iterating through pages with marker returns an item seen on a previous page, or skips items.

Likely cause: A new backup or restore was created between page fetches, shifting the result window, or the client is constructing the marker value incorrectly.

Fix:

• Always use the next_marker value from the pagination envelope of the previous response — do not construct marker values from resource IDs manually.
• For audit-grade enumeration (e.g. coverage reports), prefer the admin endpoints and process the full result set in a single pass if the record count is below max_limit (1000).

Issue: error_category: operator_action_required on backup jobs

Symptom: A backup job ends in error state and the job detail shows category: operator_action_required.

Likely cause: The conductor could not provision a worker VM for the job — either no worker fleet is available (min_workers not satisfied), no worker has a free capacity slot, or worker boot failures exceeded worker_boot_max_failures.

Fix:

• Check the conductor logs: kubectl -n abaca logs deployment/abaca-conductor
• Check the worker fleet: openstack share protection admin worker list
• Verify worker_boot_max_failures and worker_boot_failure_window_seconds in [conductor] have not been tripped; if they have, resolve the underlying Nova/Neutron issue and restart the conductor pod to reset the failure counter.
• Do not ask the tenant to retry — this is an operator-side failure. Fix the fleet first, then use openstack share protection admin job retry <job-id>.

Issue: noauth = true accidentally set in production

Symptom: Any HTTP request to the API succeeds regardless of whether an X-Auth-Token header is present.

Likely cause: noauth = true in [api] — intended only for the local Docker Compose dev stack.

Fix:

• Set noauth = false in abaca.conf immediately.
• Restart the API pod: kubectl -n abaca rollout restart deployment/abaca-api
• Audit access logs for unauthenticated requests made during the window the setting was active.