Guide

Common Workflows

End-to-end tasks combining CLI / API calls

Overview

This page walks you through the end-to-end operational workflows for Trilio Share Protection for OpenStack (Abacá): registering backup targets, creating protection policies, running on-demand backups, and restoring Manila shares. Each workflow is shown using the OpenStack CLI (openstack share protection …), the REST API, and where relevant the operator CLI (abaca-manage). Work through these tasks in order when bringing up a new deployment; return to individual sections when you need to perform a specific operation in steady state.

Prerequisites

Before running any workflow on this page, confirm the following are in place:

Abacá control plane deployed — abaca-api and abaca-conductor pods are running in the abaca OpenShift namespace and the service is registered in the Keystone catalog under the share-protection catalog type.
Worker image registered — at least one worker VM image is uploaded to Glance with the abaca_worker_image=1 property (built by deploy/rhoso/worker/20-worker-image.sh).
Worker fleet active — at least one worker VM is in ACTIVE state and sending heartbeats; the conductor's [conductor] min_workers threshold is satisfied.
python-abacaclient installed — the openstack share protection subcommands are available in your OpenStack CLI environment.
Keystone credentials sourced — your shell has a valid clouds.yaml or OS_* environment variables for the target project and domain.
Domain owner role — target template and bucket registration require the abaca_domain_owner Keystone role scoped to the relevant domain. Regular tenant operations (policy, backup, restore) require standard project membership.
S3 bucket pre-provisioned — the S3 bucket you intend to register must already exist, with versioning enabled. Enable S3 Object Lock at bucket creation time if you want immutable backups; it cannot be added later.
Manila share available — the share you want to protect must be in available status with at least one export location.
Barbican reachable — the key manager service is accessible from both the conductor and worker VMs; Abacá never stores secret values in its own database.
OpenStack CLI with openstack share protection subcommands ≥ the version shipped with this release of Abacá.
Kopia ≥ 0.17.0 (FIPS-built, version-pinned) baked into the worker image.

Installation

The python-abacaclient package that provides the openstack share protection CLI is layered into your existing OpenStack client environment by the deployment scripts. For operator tooling, abaca-manage is available inside the abaca-api and abaca-conductor pods.

Step 1 — Verify the CLI plugin is available

After deployment, confirm the plugin loaded correctly:

openstack share protection --help

You should see a list of share protection subcommands. If the command is not found, ensure python-abacaclient is installed in the same Python environment as your python-openstackclient.

Step 2 — Confirm the service catalog entry

openstack catalog show share-protection

The output should show publicURL, internalURL, and adminURL endpoints pointing to the abaca-api service (default port 9797).

Step 3 — Verify control plane health (operator)

From inside the OpenShift cluster, confirm both pods are running:

oc -n abaca get pods

Expected output shows abaca-api-* and abaca-conductor-* pods in Running state.

Step 4 — Run the database migration (first deployment only)

The db_sync Kubernetes Job runs automatically before service pods start during initial deployment. To run it manually (for example, after an upgrade):

oc -n abaca create job --from=cronjob/abaca-db-sync abaca-db-sync-manual
oc -n abaca logs job/abaca-db-sync-manual --follow

Or invoke the CLI directly inside an abaca-api pod:

oc -n abaca exec deploy/abaca-api -- abaca-manage db_sync

Step 5 — Register the domain (operator, one-time per domain)

Before any tenant in a domain can use Abacá, the operator must register that domain:

oc -n abaca exec deploy/abaca-api -- abaca-manage register_domain \
  --domain-id <keystone-domain-id> \
  --service-project-id <service-project-id>

This creates the domain_service_projects mapping that tells Abacá which OpenStack project hosts worker VMs, S3 buckets, and Barbican secrets for this domain.

Configuration

Abacá uses oslo.config with an INI-style .conf file. The options most relevant to day-to-day workflows are grouped below. All options are declared in code and read at service startup; changes require a pod restart.

[abaca] — Service identity

Option	Type	Default	Purpose
`service_user_id`	string	—	Keystone user ID of the Abacá service account.
`service_user_name`	string	—	Keystone username of the Abacá service account.
`worker_project_id`	string	—	Project ID where worker VMs are booted.
`worker_project_name`	string	—	Human-readable name of the worker project (for logging).

[api] — API server

Option	Type	Default	Purpose
`bind_host`	string	`0.0.0.0`	Address the API server listens on.
`bind_port`	integer	`9797`	TCP port for the API server.
`noauth`	boolean	`false`	Disable Keystone auth (dev mode only; never enable in production).
`max_limit`	integer	`1000`	Hard cap on collection page size.
`default_limit`	integer	`100`	Default page size when the client omits `limit`.

[conductor] — Orchestration and fleet management

Option	Type	Default	Purpose
`scheduler_interval`	integer	`60`	Seconds between policy schedule evaluations.
`reconciliation_interval`	integer	`300`	Seconds between reconciliation sweep runs.
`worker_heartbeat_deadline_seconds`	integer	`60`	Seconds of silence before a worker is considered dead.
`worker_heartbeat_interval_seconds`	integer	`10`	Expected heartbeat cadence from worker agents.
`queued_job_deadline_seconds`	integer	`300`	Seconds a queued job may wait before the reconciliation loop re-casts it.
`maintenance_interval_seconds`	integer	`604800`	Seconds between scheduled repository maintenance jobs (default: weekly).
`min_workers`	integer	`1`	Minimum desired active workers; the conductor boots new VMs to maintain this floor.
`worker_boot_cooldown_seconds`	integer	`180`	Seconds to wait between consecutive worker boot attempts.
`worker_boot_max_failures`	integer	`3`	Maximum consecutive boot failures before the conductor stops retrying.
`worker_boot_failure_window_seconds`	integer	`7200`	Window over which boot failure count is evaluated.
`worker_boot_grace_seconds`	integer	`1200`	Seconds after boot before a worker is expected to send its first heartbeat.
`worker_boot_os_cloud`	string	`abaca-service`	`clouds.yaml` cloud name the conductor uses to boot workers.
`worker_boot_image`	string	`abaca-worker-0.23.1`	Glance image name or ID used when booting new worker VMs.
`worker_boot_flavor`	string	`m1.small`	Nova flavor used when booting worker VMs.
`worker_boot_network`	string	—	Neutron network ID the conductor attaches workers to.
`worker_api_url`	string	—	Base HTTPS URL the conductor uses to reach worker agents.
`worker_api_ca_file`	string	—	CA certificate file for validating worker TLS.
`worker_api_host_aliases`	list	``	Hostname aliases for worker API URLs (useful in split-DNS environments).
`worker_token_ttl_seconds`	integer	`1800`	Lifetime of the short-lived token issued to a worker for a single job.
`sweep_command_wait_seconds`	integer	`120`	Seconds the reconciliation loop waits for an in-progress command before considering it orphaned.
`catalogue_sync_interval_seconds`	integer	`3600`	Seconds between catalog synchronization runs.
`usage_sample_interval_seconds`	integer	`21600`	Seconds between usage metering samples.
`worker_boot_key_name`	string	—	Nova key pair name injected into worker VMs (for emergency operator SSH access).

[database] — Persistence

Option	Type	Default	Purpose
`connection`	string	—	SQLAlchemy connection string for the dedicated Abacá MySQL/MariaDB database. Example: `mysql+pymysql://abaca:password@db-host/abaca`

[kopia] — Backup engine

Option	Type	Default	Purpose
`binary`	string	`kopia`	Path to the Kopia binary inside worker VMs.
`require_fips_profile`	boolean	`true`	Reject repository initialization unless the FIPS crypto profile (AES-256-GCM-HMAC-SHA256) is used. Set to `false` only in non-regulated dev environments.
`subprocess_timeout`	integer	`3600`	Seconds before a Kopia subprocess is killed.
`executor`	string	—	How the conductor runs Kopia commands: `http` (via worker agent over HTTPS) or `ephemeral_container` (local container, dev only).
`image`	string	—	Container image used when `executor=ephemeral_container`.
`container_engine`	string	—	`docker` or `podman` when `executor=ephemeral_container`.
`container_network`	string	—	Network name for ephemeral containers.

[worker] — Worker agent (set in the worker image's config, not the control plane)

Option	Type	Default	Purpose
`capacity_slots`	integer	—	Maximum concurrent jobs this worker VM may run simultaneously.
`mount_base`	string	`/var/lib/abaca/mnt`	Directory under which the agent mounts Manila shares.
`command_timeout_seconds`	integer	`300`	Seconds before an individual command is considered timed out.
`command_lease_seconds`	integer	`120`	Duration of the lease the agent holds on a claimed command.
`max_command_duration_seconds`	integer	`21600`	Hard ceiling on how long any single command may run.
`claim_poll_interval_seconds`	float	`2`	Seconds between command claim polls (short-poll fallback).
`long_poll_seconds`	integer	`20`	Seconds the agent holds a long-poll connection open when waiting for commands.
`progress_interval_seconds`	integer	`30`	Seconds between progress heartbeats sent back to the conductor.
`listener_port`	integer	`9798`	Port the worker agent listens on for inbound HTTPS from the conductor.
`api_url`	string	—	URL the worker agent uses to reach the Abacá API.
`token`	string	—	Short-lived token for authenticating the worker to the API (injected at boot).
`api_ca_file`	string	—	CA certificate for validating the Abacá API TLS certificate.

[enrollment] — Target enrollment dispatch

Option	Type	Default	Purpose
`dispatch`	string	—	`rpc` (production: enrolls via the conductor job queue) or `inline` (dev: runs synchronously in the API process).

Annotated sample configuration

[DEFAULT]
catalog_type = share-protection
endpoint_type = publicURL

[abaca]
service_user_name = abaca
worker_project_name = abaca-service

[api]
bind_host = 0.0.0.0
bind_port = 9797
noauth = false
max_limit = 1000
default_limit = 100

[conductor]
scheduler_interval = 60
reconciliation_interval = 300
worker_heartbeat_deadline_seconds = 60
queued_job_deadline_seconds = 300
maintenance_interval_seconds = 604800
min_workers = 2
worker_boot_os_cloud = abaca-service
worker_boot_image = abaca-worker-0.23.1
worker_boot_flavor = m1.large
worker_boot_network = <neutron-network-uuid>
worker_api_url = https://abaca-workers.example.com
worker_token_ttl_seconds = 1800

[database]
connection = mysql+pymysql://abaca:changeme@mariadb.example.com/abaca

[kopia]
binary = kopia
require_fips_profile = true
subprocess_timeout = 3600
executor = http

[worker]
capacity_slots = 4
mount_base = /var/lib/abaca/mnt
command_lease_seconds = 120
long_poll_seconds = 20
listener_port = 9798

[enrollment]
dispatch = rpc

Usage

The workflows below are presented in the order you typically perform them. A domain owner completes the first two; tenant users perform the remaining steps.

Workflow 1 — Register a backup target template (domain owner)

A BackupTargetTemplate describes the S3 endpoint, region, provider, and how buckets are laid out across projects in a domain (shared or per_project). Create the template before registering any buckets.

CLI:

openstack share protection target template create \
  --name my-template \
  --s3-endpoint https://s3.example.com \
  --s3-region us-east-1 \
  --provider s3 \
  --bucket-scheme per_project

API:

curl -s -X POST https://abaca.example.com/v1/target-templates \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "target_template": {
      "name": "my-template",
      "s3_endpoint": "https://s3.example.com",
      "s3_region": "us-east-1",
      "provider": "s3",
      "bucket_scheme": "per_project"
    }
  }'

Record the id from the response — you need it when registering buckets and creating policies.

Workflow 2 — Register an S3 bucket (domain owner)

Bucket registration runs a bucket_enroll job: a worker VM validates the bucket's conformance (reachability, read/write access, Object Lock, absence of lifecycle expiration rules), generates and stores the Kopia repository password in Barbican, and initializes the encrypted repository. The bucket must already exist in S3.

CLI:

openstack share protection target bucket register \
  --template-id <template-uuid> \
  --bucket-name my-project-backups \
  --access-key-secret <barbican-secret-href> \
  --project-id <tenant-project-uuid>

API:

curl -s -X POST https://abaca.example.com/v1/target-templates/<template-uuid>/buckets \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bucket": {
      "bucket_name": "my-project-backups",
      "access_key_secret_ref": "<barbican-secret-href>",
      "project_id": "<tenant-project-uuid>"
    }
  }'

A job is created and the response includes a job object. Poll the job until it reaches available:

openstack share protection job show <job-uuid>

If enrollment fails, the job error_category field tells you whether the domain owner (domain_owner_action_required) or the operator (operator_action_required) needs to act. Common enrollment failures and remediation are in the Troubleshooting section.

Workflow 3 — Create a protection policy (tenant user)

A policy binds a Manila share to a target template, defines the backup schedule as a cron expression, and sets retention rules. Once created, the conductor's scheduler evaluates the cron expression every [conductor] scheduler_interval seconds and queues backup jobs automatically.

CLI:

openstack share protection policy create \
  --share-id <manila-share-uuid> \
  --target-template-id <template-uuid> \
  --schedule "0 2 * * *" \
  --retention-days 30 \
  --name nightly-backup

API:

curl -s -X POST https://abaca.example.com/v1/policies \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "name": "nightly-backup",
      "share_id": "<manila-share-uuid>",
      "target_template_id": "<template-uuid>",
      "schedule": "0 2 * * *",
      "retention_days": 30
    }
  }'

The schedule field is a standard cron expression in UTC. The first policy-driven backup runs at the next scheduled time; use Workflow 4 to trigger an immediate backup without waiting.

Workflow 4 — Request an on-demand backup (tenant user)

You can back up a share immediately regardless of any policy schedule. Abacá creates a Backup record and a Job record, then dispatches the job to an available worker.

CLI:

openstack share protection backup create \
  --share-id <manila-share-uuid> \
  --target-template-id <template-uuid>

API:

curl -s -X POST https://abaca.example.com/v1/backups \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "backup": {
      "share_id": "<manila-share-uuid>",
      "target_template_id": "<template-uuid>"
    }
  }'

The response includes both a backup object (with its id and initial status) and a job object. Track progress by polling the backup:

openstack share protection backup show <backup-uuid>

The backup transitions through the job state machine: queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available. A backup in error status means the job failed; inspect the job's error_category and error_detail fields.

openstack share protection job show <job-uuid>

Workflow 5 — List and inspect backups (tenant user)

CLI:

# List all backups in your project
openstack share protection backup list

# Show details for a specific backup
openstack share protection backup show <backup-uuid>

API:

curl -s https://abaca.example.com/v1/backups \
  -H "X-Auth-Token: $OS_TOKEN"

Key fields in a backup record:

status — available, error, or expired
kopia_snapshot_id — the Kopia manifest ID in the repository; populated only when status=available
crash_consistent — whether the backup reflects a point-in-time snapshot
size_bytes — deduplicated bytes uploaded to S3
files_count — number of files captured

Workflow 6 — Restore a share from backup (tenant user)

Abacá supports two restore modes:

new_share (default) — restores into a new Manila share of the same type and adequate size. The target_share_id field on the restore record holds the ID of the newly created share.
in_place — restores over the existing share in place; requires the force flag and should only be used when you accept that current share content will be overwritten.

You can restore the whole snapshot, a sub-path, or a single file.

CLI — whole-share restore to a new share:

openstack share protection restore create \
  --backup-id <backup-uuid> \
  --mode new_share

CLI — sub-path restore to a new share:

openstack share protection restore create \
  --backup-id <backup-uuid> \
  --mode new_share \
  --sub-path /data/reports/2025

CLI — in-place restore (use with caution):

openstack share protection restore create \
  --backup-id <backup-uuid> \
  --mode in_place \
  --force

API:

curl -s -X POST https://abaca.example.com/v1/restores \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "restore": {
      "backup_id": "<backup-uuid>",
      "mode": "new_share"
    }
  }'

Poll the restore until it reaches available:

openstack share protection restore show <restore-uuid>

When mode=new_share, the target_share_id field identifies the Manila share that was created for you. Mount it using Manila's standard export locations.

Workflow 7 — Monitor job health (operator and tenant)

List all jobs (operators see all projects; tenants see their own):

openstack share protection job list

Filter by state to find failures:

openstack share protection job list --state error

The error_category field on a failed job tells you who must act:

tenant_action_required — the tenant must fix something (for example, invalid S3 credentials, missing access rules).
operator_action_required — the operator must act (for example, no worker fleet available, network misconfiguration).

Workflow 8 — Rebuild the catalog after database loss (operator)

If the Abacá database is lost but the Kopia repositories in S3 are intact, you can reconstruct the catalog using abaca-manage:

oc -n abaca exec deploy/abaca-api -- abaca-manage rebuild-from-repository \
  --target-id <target-uuid>

This reads the Kopia repository manifests in S3 and reconstructs the Backup and related rows in the database. Re-run for each enrolled target. After the rebuild, verify the catalog matches expectations:

openstack share protection backup list

Examples

Example 1 — Full backup and restore round-trip

This example backs up a Manila share, waits for the backup to complete, and restores it to a new share.

Step 1: Create an on-demand backup

openstack share protection backup create \
  --share-id d1f2e3c4-0000-0000-0000-000000000001 \
  --target-template-id a1b2c3d4-0000-0000-0000-000000000001

Expected output:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| id                | b0000000-aaaa-bbbb-cccc-000000000001 |
| status            | queued                               |
| share_id          | d1f2e3c4-0000-0000-0000-000000000001 |
| kopia_snapshot_id |                                      |
| job_id            | j0000000-aaaa-bbbb-cccc-000000000001 |
+-------------------+--------------------------------------+

Step 2: Poll until available

watch -n 10 openstack share protection backup show b0000000-aaaa-bbbb-cccc-000000000001

When the backup completes, status becomes available and kopia_snapshot_id is populated:

+-------------------+------------------------------------------+
| Field             | Value                                    |
+-------------------+------------------------------------------+
| id                | b0000000-aaaa-bbbb-cccc-000000000001     |
| status            | available                                |
| kopia_snapshot_id | k9c3a1b2e4f5d6a7b8c9d0e1f2a3b4c5d6e7  |
| size_bytes        | 1048576                                  |
| files_count       | 42                                       |
| crash_consistent  | true                                     |
+-------------------+------------------------------------------+

Step 3: Restore to a new share

openstack share protection restore create \
  --backup-id b0000000-aaaa-bbbb-cccc-000000000001 \
  --mode new_share

Expected output:

+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| id               | r0000000-aaaa-bbbb-cccc-000000000001 |
| status           | queued                               |
| mode             | new_share                            |
| target_share_id  |                                      |
+------------------+--------------------------------------+

Step 4: Poll until the restore is available

watch -n 10 openstack share protection restore show r0000000-aaaa-bbbb-cccc-000000000001

When complete:

+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| id               | r0000000-aaaa-bbbb-cccc-000000000001 |
| status           | available                            |
| mode             | new_share                            |
| target_share_id  | e9f8a7b6-0000-0000-0000-000000000099 |
+------------------+--------------------------------------+

The target_share_id is the newly created Manila share. Use openstack share show e9f8a7b6-... to retrieve its export locations and mount it.

Example 2 — Create a nightly backup policy and verify it

Create the policy:

openstack share protection policy create \
  --share-id d1f2e3c4-0000-0000-0000-000000000001 \
  --target-template-id a1b2c3d4-0000-0000-0000-000000000001 \
  --schedule "0 2 * * *" \
  --retention-days 14 \
  --name nightly-14d

Expected output:

+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| id                 | p0000000-aaaa-bbbb-cccc-000000000001 |
| name               | nightly-14d                          |
| share_id           | d1f2e3c4-0000-0000-0000-000000000001 |
| target_template_id | a1b2c3d4-0000-0000-0000-000000000001 |
| schedule           | 0 2 * * *                            |
| retention_days     | 14                                   |
+--------------------+--------------------------------------+

Verify the policy is listed:

openstack share protection policy list

Example 3 — Check for failed jobs and read error details

openstack share protection job list --state error

Expected output:

+--------------------------------------+--------+-------+------------------------------+------------------------------+
| id                                   | type   | state | error_category               | error_detail                 |
+--------------------------------------+--------+-------+------------------------------+------------------------------+
| j0000000-ffff-bbbb-cccc-000000000099 | backup | error | operator_action_required     | WorkerUnavailable: no active |
|                                      |        |       |                              | workers with free slots      |
+--------------------------------------+--------+-------+------------------------------+------------------------------+

Show full detail for a specific failed job:

openstack share protection job show j0000000-ffff-bbbb-cccc-000000000099

Example 4 — Register a target template and bucket (domain owner)

Create the template:

openstack share protection target template create \
  --name wasabi-per-project \
  --s3-endpoint https://s3.wasabisys.com \
  --s3-region us-east-1 \
  --provider s3 \
  --bucket-scheme per_project

Register a bucket for a specific tenant project:

openstack share protection target bucket register \
  --template-id a1b2c3d4-0000-0000-0000-000000000001 \
  --bucket-name tenant-alpha-backups \
  --access-key-secret https://barbican.example.com/v1/secrets/abcdef01-... \
  --project-id 7f8e9d0c-0000-0000-0000-000000000007

Check the enrollment job:

openstack share protection job show <enrollment-job-uuid>

When state=available, the bucket is enrolled and the target is ready to receive backups for that project.

Example 5 — Rebuild the catalog after database loss (operator DR)

If the Abacá database has been lost and restored from an S3-backed Kopia repository:

oc -n abaca exec deploy/abaca-api -- \
  abaca-manage rebuild-from-repository \
  --target-id a1b2c3d4-0000-0000-0000-000000000001

Verify the catalog was rebuilt:

openstack share protection backup list

Troubleshooting

Use this section to diagnose common failures. Each issue shows the symptom, the most likely cause, and the remediation steps.

Job stuck in `queued` state

Symptom: A backup or restore job remains in queued state for longer than 5 minutes ([conductor] queued_job_deadline_seconds = 300).

Likely causes:

No active worker VMs with free capacity slots.
The conductor's RabbitMQ connection is unhealthy and the initial dispatch cast was dropped.
The conductor pod is not running.

Remediation:

Check worker fleet status:

openstack share protection job list --state error
oc -n abaca logs deploy/abaca-conductor --tail=100

Confirm the conductor pod is running:
```
oc -n abaca get pods
```
The reconciliation loop ([conductor] reconciliation_interval = 300 seconds) will re-cast dropped jobs automatically. If the job is still queued after two reconciliation intervals, check [conductor] min_workers and whether the conductor can boot new VMs:
```
oc -n abaca logs deploy/abaca-conductor | grep -i "worker_boot"
```
If error_category=operator_action_required and error_detail mentions WorkerUnavailable, the fleet is exhausted. Increase [conductor] min_workers or reduce [worker] capacity_slots per worker.

Backup job fails in `provisioning_network` state

Symptom: A backup job reaches error with the last recorded state provisioning_network.

Likely causes:

The worker VM cannot attach a Neutron port to the share's network (DHSS=true backend).
The share's export is not reachable from the worker's static network (DHSS=false backend).
The Keystone trust used to scope Nova/Neutron calls has expired or been revoked by the tenant.

Remediation:

Inspect the job's error_detail field:

openstack share protection job show <job-uuid>

If the error mentions trust or token issues (error_category=tenant_action_required), the tenant must re-authorize Abacá. The trust can be revoked by the tenant at any time; ask the tenant to re-issue the authorization.
For network attachment failures, verify the worker VMs can reach the Manila share network — check Neutron security groups and router configurations.

Bucket enrollment fails with `mixed_use` error

Symptom: A bucket_enroll job fails with an error mentioning mixed_use.

Cause: The S3 bucket already contains objects that are not part of a Kopia repository and do not match an adoptable existing repository. Abacá requires a dedicated, empty bucket (or a bucket already containing a Kopia repository created by a previous Abacá enrollment).

Remediation:

Create a new, empty, dedicated S3 bucket.
Enable versioning and Object Lock on the new bucket before registering it (these cannot be enabled after creation).
Register the new bucket in place of the existing one.

Bucket enrollment fails with `lifecycle_unverifiable` warning or `domain_owner_action_required`

Symptom: Enrollment succeeds but the bucket health shows a lifecycle_unverifiable warning, or enrollment fails with domain_owner_action_required mentioning lifecycle rules.

Cause: The S3 bucket has lifecycle expiration rules configured. Abacá requires that no lifecycle expiration rules exist, because such rules could silently delete live backup objects within the retention window.

Remediation:

Remove all lifecycle expiration rules from the S3 bucket.
If the S3 IAM policy on the service account does not permit GetBucketLifecycleConfiguration, the check will produce a lifecycle_unverifiable warning rather than a hard failure. Grant the service account the necessary read permission, or acknowledge the risk and monitor manually.
Re-run enrollment after removing the rules.

Worker agent not registering (conductor logs show no heartbeats)

Symptom: Worker VMs are booted (visible in Nova) but the conductor logs show no worker heartbeats and no jobs are dispatched.

Likely causes:

The worker VM cannot reach the Abacá API over HTTPS (network or security group issue).
[conductor] worker_api_url is misconfigured or the TLS certificate is not trusted by the worker (check [conductor] worker_api_ca_file).
The worker image was built without abaca-worker-agent (check the Glance image properties for abaca_worker_image=1).
The worker VM's [worker] token was not injected correctly at boot (check the Nova user-data or cloud-init configuration).

Remediation:

SSH into a worker VM (using the key pair set in [conductor] worker_boot_key_name) and check the agent process:
```
systemctl status abaca-worker-agent
journalctl -u abaca-worker-agent --tail=50
```

From the worker VM, verify reachability of the API:

curl -v --cacert /etc/abaca/api-ca.crt https://<worker_api_url>/v1

Confirm the Glance image has the correct property:
```
openstack image show <worker-image-id> -c properties
```
The abaca_worker_image property must be 1.
If the image property is missing, rebuild the worker image using deploy/rhoso/worker/20-worker-image.sh and re-upload to Glance.

Restore completed but `target_share_id` is empty

Symptom: A new_share restore reaches available status but target_share_id is not populated.

Cause: This indicates the restore dispatcher created the destination Manila share but failed to write the share ID back to the restore record — this is a bug. The restore may have partially succeeded.

Remediation:

Check the conductor logs around the time the restore completed:
```
oc -n abaca logs deploy/abaca-conductor | grep <restore-uuid>
```
Check Manila for a recently created share in your project that matches the source share type and size:
```
openstack share list --sort-key created_at --sort-dir desc
```
Report the issue to the operator with the restore UUID and conductor log output for investigation.

Backup status shows `expired` unexpectedly

Symptom: A backup that should be within the retention window shows status=expired in the Abacá catalog.

Cause: The reconciliation loop's _sweep_expire_backups sweep compares the Abacá database against the Kopia repository manifest. If the Kopia snapshot was pruned by a maintenance job (because the retention policy was met), the backup row is flipped to expired. This is expected behavior; the backup no longer exists in S3.

If the expiry is unexpected:

Verify the retention_days setting on the policy that created the backup:
```
openstack share protection policy show <policy-uuid>
```
Check whether the maintenance interval is set very aggressively ([conductor] maintenance_interval_seconds). The default is weekly (604800 seconds).
If the backup was within its retention window when it expired, check the Kopia repository directly using abaca-manage — this may indicate a retention miscalculation that should be reported as a bug.

Common Workflows

[abaca] — Service identity

[api] — API server

[conductor] — Orchestration and fleet management

[database] — Persistence

[kopia] — Backup engine

[worker] — Worker agent (set in the worker image's config, not the control plane)

[enrollment] — Target enrollment dispatch

Annotated sample configuration

Workflow 1 — Register a backup target template (domain owner)

Workflow 2 — Register an S3 bucket (domain owner)

Workflow 3 — Create a protection policy (tenant user)

Workflow 4 — Request an on-demand backup (tenant user)

Workflow 5 — List and inspect backups (tenant user)

Workflow 6 — Restore a share from backup (tenant user)

Workflow 7 — Monitor job health (operator and tenant)

Workflow 8 — Rebuild the catalog after database loss (operator)

Example 1 — Full backup and restore round-trip

Example 2 — Create a nightly backup policy and verify it

Example 3 — Check for failed jobs and read error details

Example 4 — Register a target template and bucket (domain owner)

Example 5 — Rebuild the catalog after database loss (operator DR)

Job stuck in queued state

Backup job fails in provisioning_network state

Bucket enrollment fails with mixed_use error

Bucket enrollment fails with lifecycle_unverifiable warning or domain_owner_action_required

Worker agent not registering (conductor logs show no heartbeats)

Restore completed but target_share_id is empty

Backup status shows expired unexpectedly

Job stuck in `queued` state

Backup job fails in `provisioning_network` state

Bucket enrollment fails with `mixed_use` error

Bucket enrollment fails with `lifecycle_unverifiable` warning or `domain_owner_action_required`

Restore completed but `target_share_id` is empty

Backup status shows `expired` unexpectedly