Guide

Components

API service, workers/agents, scheduler/conductor, RPC, database

Overview

This page describes the components that make up Trilio Share Protection for OpenStack (Abacá): the control plane services that run as OpenShift pods, the worker VMs that execute data-path operations inside tenant networks, the supporting infrastructure services, and the management and client tooling. Understanding how these components interact—and how each is configured—is essential for deploying and operating the service correctly, diagnosing failures, and integrating with the API.

Prerequisites

Before deploying or operating Abacá, ensure the following are in place:

Red Hat OpenStack Services on OpenShift (RHOSO) ≥ 18 — the control plane runs in the abaca OpenShift namespace
OpenShift ≥ 4.14 — hosts the abaca-api and abaca-conductor pods
OpenStack Keystone ≥ 2023.1 — with trusts enabled and domain-scoped tokens supported; Abacá registers as a service with a catalog entry and dedicated service account
OpenStack Manila — the Shared File System service whose shares Abacá protects
OpenStack Barbican — stores S3 credentials and Kopia repository passwords as secret references; Abacá never writes secret values to its own database
OpenStack Nova and Neutron — required for worker VM lifecycle management and network attachment
OpenStack Glance — stores the worker VM image (uploaded with the abaca_worker_image=1 property)
MySQL or MariaDB (Galera) — a dedicated database instance for Abacá bookkeeping metadata
RabbitMQ — a dedicated broker instance for API-to-conductor oslo.messaging RPC; must be separate from the platform's shared RabbitMQ
S3-compatible object storage (e.g., Wasabi, MinIO, AWS S3) — with versioning and Object Lock enabled at bucket creation time
RHEL base cloud image ≥ 9 — for building the worker VM image; the build host must have libguestfs (virt-customize/guestfish) available
Kopia binary ≥ 0.17.0 — FIPS-built and version-pinned, baked into the worker image
Python ≥ 3.11 — required for all Python packages
Horizon (optional) — required only for the abaca-dashboard plugin
tox ≥ 4.0 (optional) — for local development and testing

Installation

Abacá is deployed as an OpenStack service on RHOSO 18. The deployment is split across two tool sets: kustomize manifests for the OpenShift control plane, and bash install scripts for OpenStack identity, catalog, and supporting infrastructure.

1. Provision identity and infrastructure

Run the RHOSO deploy scripts in order. These scripts register Abacá in Keystone (service entry, endpoints, service account, roles), provision the dedicated database and message bus, upload the Horizon plugin, and register the worker image in Glance.

# From the repository root
cd deploy/rhoso/

# Register Keystone service, catalog endpoints, service user, and roles
bash 01-identity.sh

# Provision dedicated MySQL database and RabbitMQ vhost
bash 02-database.sh
bash 03-rabbitmq.sh

# Register service catalog endpoint
bash 04-catalog.sh

2. Build and upload the worker VM image

The worker image must be built before any enrollment or backup can run. All runtime dependencies (Kopia, NFS utilities, abaca-worker-agent) are baked in at build time. Worker VMs must not fetch packages from the internet at boot.

bash deploy/rhoso/worker/20-worker-image.sh

This script:

Runs virt-customize against a base RHEL 9 cloud image
Installs Kopia (FIPS-built, version-pinned), nfs-common, and abaca-worker-agent into the image
Uploads the image to Glance in raw format with the abaca_worker_image=1 property

Important: The image is uploaded as --disk-format raw (not qcow2) to avoid Nova's qemu-img convert step, which can exhaust disk on constrained controllers.

3. Install the Horizon plugin (optional)

If you are deploying the dashboard plugin, install it into the existing Horizon pod venv:

bash deploy/rhoso/08-dashboard.sh

This layers the abaca-dashboard and python-abacaclient packages into the Horizon pod's Python environment.

4. Deploy the control plane with kustomize

Apply the kustomize manifests to create the abaca namespace and deploy the API, conductor, and supporting resources:

# Run the database schema migration Job first
kubectl apply -k deploy/kustomize/base/

# Wait for the db-sync Job to complete before the service pods start
kubectl wait --for=condition=complete job/abaca-db-sync -n abaca --timeout=120s

The db-sync Job runs abaca-manage db_sync, which applies additive-only Alembic migrations to the dedicated MySQL/MariaDB instance. Migrations are additive-only and safe across RHOSO upgrade rebases.

5. Verify the deployment

Confirm both pods are running and the service is reachable through Keystone:

kubectl get pods -n abaca
# Expected: abaca-api-* and abaca-conductor-* pods in Running state

# Confirm the service is registered in the catalog
openstack catalog show share-protection

6. Register a Domain and provision a backup target

After the control plane is up, use abaca-manage to register a Keystone Domain with the service, then use the CLI or API to create a BackupTargetTemplate and enroll an S3 bucket as a BackupTarget:

# Register a domain (assigns a per-Domain service project)
abaca-manage register-domain --domain-id <keystone-domain-id>

# Use the OpenStack CLI plugin to create a target template
openstack share protection target-template create \
  --domain-id <keystone-domain-id> \
  --endpoint https://s3.example.com \
  --region us-east-1 \
  --bucket-scheme shared \
  my-template

# Enroll an S3 bucket (runs as a job on a worker VM)
openstack share protection target create \
  --template-id <template-id> \
  --bucket my-backup-bucket \
  my-target

Configuration

All Abacá components are configured via a single abaca.conf INI-style file, loaded automatically from /etc/abaca/abaca.conf by oslo.config. Each option belongs to a named section. The following tables document every supported option by section.

`[DEFAULT]`

Option	Type	Default	Purpose
`catalog_type`	string	`share-protection`	The service type used for catalog discovery
`endpoint_type`	string	`publicURL`	Which catalog endpoint to use (`publicURL`, `adminURL`, `internalURL`)
`abaca_enabled`	boolean	`false`	Master switch; set `true` to activate the service
`build_interval`	integer	`3`	Poll interval (seconds) for job status during tempest tests
`build_timeout`	integer	`600`	Maximum wait (seconds) for a job to reach terminal state during tempest tests
`test_target_template_id`	string	`""`	Target template ID used by the Tempest plugin for live tests
`test_share_id`	string	`""`	Share ID used by the Tempest plugin for live tests
`verify_bytes_end_to_end`	boolean	`false`	If `true`, Tempest tests verify byte-level integrity after restore
`mount_base_dir`	string	`/tmp/abaca-tempest`	Temporary mount point used by Tempest integration tests
`auth_url`	string	—	Keystone auth URL (required)
`username`	string	—	Service user name
`password`	string	—	Service user password
`project_name`	string	—	Service project name
`project_domain_name`	string	`Default`	Domain of the service project
`user_domain_name`	string	`Default`	Domain of the service user
`region_name`	string	—	OpenStack region

`[abaca]`

Option	Type	Default	Purpose
`service_user_id`	string	—	Keystone UUID of the Abacá service account
`service_user_name`	string	—	Keystone name of the Abacá service account
`worker_project_id`	string	—	UUID of the service project where worker VMs are booted
`worker_project_name`	string	—	Name of the service project where worker VMs are booted

`[api]`

Option	Type	Default	Purpose
`bind_host`	string	`0.0.0.0`	Address the API process binds to
`bind_port`	integer	`9797`	TCP port the API listens on
`noauth`	boolean	`false`	Disable Keystone auth (dev-only; never set `true` in production)
`max_limit`	integer	`1000`	Maximum number of results per paginated list response
`default_limit`	integer	`100`	Default page size for list responses when the caller omits `limit`

`[conductor]`

Option	Type	Default	Purpose
`scheduler_interval`	integer	`60`	How often (seconds) the scheduler loop checks for due policy backups
`reconciliation_interval`	integer	`300`	How often (seconds) the reconciliation loop sweeps for stuck/orphaned jobs
`worker_heartbeat_deadline_seconds`	integer	`60`	Workers not heard from within this window are marked DEAD
`worker_heartbeat_interval_seconds`	integer	`10`	How often (seconds) worker agents send heartbeats
`sweep_command_wait_seconds`	integer	`120`	How long the reconciliation loop waits for an in-flight command before treating it as stuck
`worker_token_ttl_seconds`	integer	`1800`	Lifetime (seconds) of tokens issued to worker agents
`maintenance_interval_seconds`	integer	`604800`	How often (seconds) to enqueue a maintenance job per enrolled target (default: weekly)
`catalogue_sync_interval_seconds`	integer	`3600`	How often (seconds) to synchronize the internal catalog with Manila
`usage_sample_interval_seconds`	integer	`21600`	How often (seconds) to record usage meter samples
`min_workers`	integer	`1`	Minimum number of workers the fleet must maintain
`worker_boot_cooldown_seconds`	integer	`180`	Minimum gap (seconds) between successive worker boot attempts
`worker_api_host_aliases`	list	`""`	Additional hostnames or IPs workers may use to reach the conductor API
`worker_boot_max_failures`	integer	`3`	Number of consecutive boot failures before the conductor stops retrying
`worker_boot_failure_window_seconds`	integer	`7200`	Window (seconds) over which boot failures are counted
`worker_boot_grace_seconds`	integer	`1200`	Time (seconds) allowed for a booted worker to register before it is considered failed
`worker_boot_os_cloud`	string	`abaca-service`	`clouds.yaml` cloud profile used to boot workers
`worker_boot_image`	string	`abaca-worker-0.23.1`	Glance image name used when booting new workers
`worker_boot_flavor`	string	`m1.small`	Nova flavor used for worker VMs
`worker_boot_network`	string	—	Neutron network ID or name attached to worker VMs at boot
`worker_api_url`	string	—	URL workers use to reach the conductor's command endpoint
`worker_api_ca_file`	string	—	CA bundle file for TLS verification of the conductor API by workers
`worker_boot_key_name`	string	—	Nova keypair name injected into worker VMs (for emergency SSH access)
`queued_job_deadline_seconds`	integer	`300`	Maximum time (seconds) a job may sit in `queued` state before the reconciliation loop flags it

`[database]`

Option	Type	Default	Purpose
`connection`	string	—	SQLAlchemy connection string for the dedicated Abacá MySQL/MariaDB database

`[enrollment]`

Option	Type	Default	Purpose
`dispatch`	string	—	How enrollment jobs are dispatched: `rpc` (production, via worker fleet) or `inline` (dev-only)

`[kopia]`

Option	Type	Default	Purpose
`binary`	string	`kopia`	Path to the Kopia binary inside worker VMs
`require_fips_profile`	boolean	`true`	Reject repository creation or connection unless the FIPS crypto profile is in use
`subprocess_timeout`	integer	`3600`	Maximum time (seconds) for a single Kopia subprocess invocation
`executor`	string	—	How Kopia is invoked: `http` (production, via worker agent) or `ephemeral_container` (docker-compose dev only)
`image`	string	—	Container image used when `executor=ephemeral_container` (dev only)
`container_engine`	string	—	`docker` or `podman` (dev only, used with `ephemeral_container` executor)
`container_network`	string	—	Docker/Podman network for ephemeral containers (dev only)

`[worker]`

These options configure the abaca-worker-agent process running inside each worker VM.

Option	Type	Default	Purpose
`capacity_slots`	integer	—	Number of concurrent jobs this worker VM can run; the conductor's fleet picker only assigns jobs to workers below this limit
`mount_base`	string	`/var/lib/abaca/mnt`	Base directory under which Manila shares are mounted inside the worker
`command_timeout_seconds`	integer	`300`	How long the agent waits for a command to be acknowledged before treating it as timed out
`command_lease_seconds`	integer	`120`	How long the conductor holds a command lease before considering it abandoned
`max_command_duration_seconds`	integer	`21600`	Absolute maximum duration (seconds) for any single command (backup, restore, enroll)
`claim_poll_interval_seconds`	float	`2`	How often (seconds) the agent polls for new commands from the conductor
`id`	string	—	Worker identity (Nova instance UUID); auto-resolved from the metadata service at boot
`api_url`	string	—	Conductor command endpoint URL the agent connects to
`token`	string	—	Auth token issued by the conductor for this worker
`api_ca_file`	string	—	CA bundle for TLS verification of the conductor endpoint
`long_poll_seconds`	integer	`20`	Duration (seconds) for long-poll HTTP requests when waiting for new commands
`progress_interval_seconds`	integer	`30`	How often (seconds) the agent sends job progress updates to the conductor
`listener_port`	integer	`9798`	TCP port the worker agent listens on for inbound conductor probes
`command_poll_interval_seconds`	float	`2`	How often (seconds) the agent retries claiming an available command

Annotated sample `abaca.conf`

[DEFAULT]
catalog_type = share-protection
endpoint_type = publicURL
abaca_enabled = true
auth_url = https://keystone.example.com:5000/v3
username = abaca
password = <service-user-password>
project_name = abaca-service
project_domain_name = Default
user_domain_name = Default
region_name = RegionOne

[abaca]
service_user_name = abaca
worker_project_name = abaca-service

[api]
bind_host = 0.0.0.0
bind_port = 9797
# noauth = false   # Never set true in production
max_limit = 1000
default_limit = 100

[conductor]
scheduler_interval = 60
reconciliation_interval = 300
worker_heartbeat_deadline_seconds = 60
worker_heartbeat_interval_seconds = 10
maintenance_interval_seconds = 604800
min_workers = 2
worker_boot_os_cloud = abaca-service
worker_boot_image = abaca-worker-0.23.1
worker_boot_flavor = m1.medium
worker_boot_network = abaca-mgmt-net
worker_api_url = https://abaca-worker-api.example.com
worker_api_ca_file = /etc/abaca/ca-bundle.crt

[database]
connection = mysql+pymysql://abaca:secret@db.example.com/abaca

[enrollment]
dispatch = rpc

[kopia]
binary = /usr/local/bin/kopia
require_fips_profile = true
subprocess_timeout = 3600
executor = http

[worker]
capacity_slots = 4
mount_base = /var/lib/abaca/mnt
max_command_duration_seconds = 21600
progress_interval_seconds = 30

Usage

The API service (`abaca-api`)

abaca-api is a Keystone-authenticated REST service versioned at /v1. Tenants, the Horizon plugin, and operators interact with it through three surfaces: the openstack share protection CLI plugin, the REST API directly, or the python-abacaclient SDK.

All requests require a Keystone token in the X-Auth-Token header, except in noauth mode (dev only). The service is discovered via the share-protection catalog type.

Using the CLI plugin

The python-abacaclient package adds openstack share protection subcommands to the standard openstack client:

# List all backup targets visible to your project
openstack share protection target list

# Show details of a specific backup target
openstack share protection target show <target-id>

# Create a protection policy for a share
openstack share protection policy create \
  --share-id <manila-share-id> \
  --template-id <target-template-id> \
  --schedule "0 2 * * *" \
  --retention-days 30 \
  my-policy

# Request an on-demand backup
openstack share protection backup create \
  --policy-id <policy-id> \
  my-backup

# List jobs and their states
openstack share protection job list

# Show detailed job status (including error category if failed)
openstack share protection job show <job-id>

# Restore a share from a backup
openstack share protection restore create \
  --backup-id <backup-id> \
  --target-share-id <manila-share-id> \
  my-restore

Using the REST API directly

All resources live under /v1. Authenticate with Keystone and pass the token:

# Discover the API endpoint from the catalog
ABACA_URL=$(openstack catalog show share-protection -f value -c publicURL)

# List backup targets
curl -s -H "X-Auth-Token: $OS_TOKEN" \
  "${ABACA_URL}/v1/targets" | python3 -m json.tool

# Request an on-demand backup
curl -s -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"backup": {"policy_id": "<policy-id>", "name": "my-backup"}}' \
  "${ABACA_URL}/v1/backups" | python3 -m json.tool

Using the Python SDK

from abacaclient import Client
import openstack

conn = openstack.connect(cloud="mycloud")
client = Client(session=conn.session)

# List targets
for target in client.targets.list():
    print(target.id, target.name, target.status)

# Poll a job until it reaches a terminal state
import time
job = client.jobs.get("<job-id>")
while job.state not in ("available", "error"):
    time.sleep(5)
    job = client.jobs.get(job.id)
print("Job finished:", job.state)

The conductor (`abaca-conductor`)

abaca-conductor runs as a single pod in the abaca OpenShift namespace. It owns the job state machine, the scheduler loop, the worker fleet lifecycle, and the reconciliation sweep. You do not interact with it directly — you observe its effects through job states and worker fleet status:

# View conductor logs
kubectl logs -n abaca deployment/abaca-conductor --follow

# Check worker fleet registration state
openstack share protection worker list

The scheduler runs every conductor.scheduler_interval seconds (default: 60) and enqueues backup jobs when a policy's cron expression is due. The reconciliation loop runs every conductor.reconciliation_interval seconds (default: 300) and handles dead workers, stuck jobs, overdue maintenance, and usage sampling.

Worker VMs

Worker VMs are booted automatically by the conductor in the Domain's service project when the fleet falls below conductor.min_workers. You can also boot workers manually during bring-up using abaca-dev:

# Boot a worker VM manually (bring-up / testing only)
abaca-dev worker-boot --cloud abaca-service

# Run a data-path probe against a real share (bring-up validation)
abaca-dev probe --share-id <manila-share-id> --target-id <target-id>

abaca-dev worker-boot refuses to launch any Glance image that does not have the abaca_worker_image=1 property, preventing accidental use of unprepared base images.

Database management with `abaca-manage`

Schema migrations are run automatically as a Kubernetes Job at deploy time, but you can also run them manually:

# Apply schema migrations
abaca-manage db_sync

# Register a Keystone domain with the service
abaca-manage register-domain --domain-id <keystone-domain-id>

# Disaster recovery: rebuild catalog from Kopia repositories
abaca-manage rebuild-from-repository --target-id <target-id>

Checking backup coverage and fleet health

# List all policies and their last-run status
openstack share protection policy list --long

# List workers and their states
openstack share protection worker list

# List recent jobs filtered by state
openstack share protection job list --state error

# Show a failed job's error category and code
openstack share protection job show <job-id>
# Look for: error_category (tenant_action_required or operator_action_required)
#            error_code (e.g. abaca.worker_unavailable)

Examples

Example 1 — Verify the control plane is running after deployment

After applying the kustomize manifests, confirm both pods are healthy and the service is registered:

kubectl get pods -n abaca

Expected output:

NAME                               READY   STATUS    RESTARTS   AGE
abaca-api-7d9f4b8c6-xk2pq          1/1     Running   0          4m
abaca-conductor-6b5c9d7f4-wn8rt    1/1     Running   0          4m

openstack catalog show share-protection

Expected output (truncated):

+-----------+--------------------------------------------------------------+
| Field     | Value                                                        |
+-----------+--------------------------------------------------------------+
| endpoints | RegionOne public: https://abaca.example.com:9797             |
|           | RegionOne internal: http://abaca.abaca.svc:9797              |
| name      | abaca                                                        |
| type      | share-protection                                             |
+-----------+--------------------------------------------------------------+

Example 2 — Create a protection policy and trigger an on-demand backup

This is the primary day-1 workflow after a target has been enrolled.

# Create a policy: back up the share nightly at 02:00 UTC, keep 30 days
openstack share protection policy create \
  --share-id d3a1f290-2c4e-4b77-9b3f-abc123456789 \
  --template-id e5b2c381-3d5f-5c88-ac4g-def234567890 \
  --schedule "0 2 * * *" \
  --retention-days 30 \
  nightly-production

Expected output:

+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| id             | f7c3d492-4e6g-6d99-bd5h-ghi345678901 |
| name           | nightly-production                   |
| share_id       | d3a1f290-2c4e-4b77-9b3f-abc123456789 |
| schedule       | 0 2 * * *                            |
| retention_days | 30                                   |
| enabled        | True                                 |
+----------------+--------------------------------------+

# Trigger an immediate (on-demand) backup outside the schedule
openstack share protection backup create \
  --policy-id f7c3d492-4e6g-6d99-bd5h-ghi345678901 \
  pre-maintenance-backup

Expected output:

+------------+--------------------------------------+
| Field      | Value                                |
+------------+--------------------------------------+
| id         | a1b2c3d4-e5f6-7890-abcd-ef1234567890 |
| name       | pre-maintenance-backup               |
| policy_id  | f7c3d492-4e6g-6d99-bd5h-ghi345678901 |
| state      | queued                               |
+------------+--------------------------------------+

# Poll until the backup job reaches a terminal state
openstack share protection job list --state transferring
# ...wait and re-run until state is 'available'
openstack share protection backup show a1b2c3d4-e5f6-7890-abcd-ef1234567890

Example 3 — Restore a share from backup (full share)

openstack share protection restore create \
  --backup-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --target-share-id 99f12abc-cafe-babe-dead-beef00000001 \
  full-restore-2024

Expected output:

+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| id               | 77aa88bb-cc99-dd00-ee11-ff2233445566 |
| name             | full-restore-2024                    |
| backup_id        | a1b2c3d4-e5f6-7890-abcd-ef1234567890 |
| target_share_id  | 99f12abc-cafe-babe-dead-beef00000001 |
| state            | queued                               |
+------------------+--------------------------------------+

# Monitor the restore job
openstack share protection job show <restore-job-id>

Example 4 — Inspect a failed job and identify who must act

When a job reaches error state, the error_category field indicates whether the tenant or the operator must resolve the issue.

openstack share protection job show <job-id>

Expected output (truncated):

+-------------------+----------------------------------+
| Field             | Value                            |
+-------------------+----------------------------------+
| id                | 55cc66dd-ee77-ff88-0011-22334455 |
| type              | backup                           |
| state             | error                            |
| error_code        | abaca.worker_unavailable         |
| error_category    | operator_action_required         |
| error_detail      | No ACTIVE worker with a free     |
|                   | capacity slot was found.         |
+-------------------+----------------------------------+

An error_category of operator_action_required (such as abaca.worker_unavailable) means the operator must investigate the worker fleet. A tenant_action_required category (such as invalid S3 credentials) means the tenant must correct their configuration.

Example 5 — Disaster recovery: rebuild catalog from Kopia repositories

If the Abacá database is lost but S3 data is intact, reconstruct the catalog:

# After restoring the database schema with db_sync:
abaca-manage db_sync

# Rebuild catalog for each enrolled target
abaca-manage rebuild-from-repository --target-id <target-id>

This reads the Kopia repositories in S3 and reconstructs the backups, restores, and related metadata rows.

Troubleshooting

Use a consistent diagnostic pattern: check the job state and error_category, then examine component logs, then inspect infrastructure.

Issue: `abaca-api` pod is not starting

Symptom: kubectl get pods -n abaca shows abaca-api-* in CrashLoopBackOff or Error state.

Likely causes:

Missing or malformed abaca.conf (wrong database connection string, missing [DEFAULT] auth_url)
The db-sync Job did not complete before the pod started
Keystone endpoint registration is incomplete

Fix:

# View the pod logs
kubectl logs -n abaca deployment/abaca-api --previous

# Confirm db-sync completed
kubectl get jobs -n abaca
kubectl logs -n abaca job/abaca-db-sync

# Verify Keystone registration
openstack service list | grep share-protection
openstack endpoint list --service share-protection

Correct the config volume or re-run the identity and catalog scripts, then delete the pod to force a restart.

Issue: Backup job is stuck in `queued` state

Symptom: openstack share protection job show <id> reports state: queued for longer than conductor.queued_job_deadline_seconds (default: 300 seconds).

Likely causes:

No ACTIVE worker VMs with a free capacity_slots slot
abaca-conductor is not running or has crashed
All workers are DEAD (missed heartbeats)

Fix:

# Check conductor pod
kubectl logs -n abaca deployment/abaca-conductor --follow

# Check worker fleet state
openstack share protection worker list

# If no workers are ACTIVE, boot one manually
abaca-dev worker-boot --cloud abaca-service

If error_category is operator_action_required with error_code: abaca.worker_unavailable, the fleet needs attention. Verify conductor.min_workers, worker_boot_image (must exist in Glance with abaca_worker_image=1), and worker_boot_network.

Issue: Worker VM is DEAD and associated jobs errored

Symptom: openstack share protection worker list shows a worker in DEAD state. Jobs that were running on it transitioned to error.

Likely cause: The worker stopped sending heartbeats. The reconciliation loop marks workers DEAD when last_heartbeat_at is older than conductor.worker_heartbeat_deadline_seconds (default: 60 seconds), then immediately walks their non-terminal jobs to error.

Fix:

# Check conductor logs for the reap event
kubectl logs -n abaca deployment/abaca-conductor | grep -i reap

# Check the Nova instance for the worker
openstack server show <worker-nova-uuid>

# If the worker is genuinely gone, the conductor will boot a replacement
# (subject to min_workers and boot cooldown). Check:
kubectl logs -n abaca deployment/abaca-conductor | grep -i boot

Manually clean up any Neutron ports or Manila access rules the dead worker was holding, as noted in the reap log entry. Re-run the failed backup jobs once a replacement worker is ACTIVE.

Issue: Enrollment fails at the preflight stage

Symptom: A target create job transitions to error with an error related to bucket validation.

Likely causes:

The S3 bucket is not reachable from the worker VM's network
Versioning or Object Lock is not enabled on the bucket (must be enabled at bucket creation time)
S3 credentials stored in Barbican are incorrect
A lifecycle rule is configured that would delete live data

Fix:

openstack share protection job show <enrollment-job-id>
# Read error_detail for the specific preflight check that failed

For network reachability: verify the bucket endpoint is accessible from the worker VM's network scope.
For Object Lock: Object Lock cannot be enabled retroactively — create a new bucket with it enabled from the start.
For credential errors (error_category: tenant_action_required): update the S3 credential secret in Barbican and re-enroll.

Issue: `require_fips_profile` causes repository connection failure

Symptom: A backup or restore job fails with an error indicating the Kopia repository's crypto profile is not FIPS-compliant.

Likely cause: The Kopia repository was initialized without the FIPS crypto profile (e.g., conductor.kopia.require_fips_profile = false was set at enrollment time). The current setting require_fips_profile = true now rejects it.

Fix: The crypto profile is immutable after repository creation. If the repository was created without FIPS mode and you need FIPS compliance, you must enroll a new target with a fresh bucket, run a full re-ingest, and decommission the old target. Do not toggle require_fips_profile on existing repositories.

Issue: Conductor logs show the scheduler is skipping ticks

Symptom: Backup jobs are not being enqueued on schedule. Conductor logs contain WARN messages about the scheduler skipping.

Likely cause: croniter failed to import at startup. The SchedulerLoop fail-closes (skips ticks with a warning) rather than enqueuing jobs incorrectly.

Fix:

kubectl logs -n abaca deployment/abaca-conductor | grep -i croniter

If croniter is missing or the wrong version, rebuild the conductor image to include croniter>=2.0 as a runtime dependency.

Issue: Worker agent cannot reach the conductor API

Symptom: Worker VMs register but then lose heartbeat. Conductor logs show no heartbeats from recently booted workers.

Likely cause: conductor.worker_api_url is not reachable from the tenant network, or the TLS CA bundle (conductor.worker_api_ca_file / worker.api_ca_file) is missing or incorrect.

Fix:

# SSH into the worker VM (if worker_boot_key_name is set) and test connectivity
curl -v --cacert /etc/abaca/ca-bundle.crt https://<conductor-worker-api-url>/healthz

Verify conductor.worker_api_url resolves and is reachable from the worker's network. If using TLS, ensure the CA bundle is baked into the worker image or delivered via cloud-init user-data. Check conductor.worker_api_host_aliases if the conductor is reachable under multiple hostnames.

Components

1. Provision identity and infrastructure

2. Build and upload the worker VM image

3. Install the Horizon plugin (optional)

4. Deploy the control plane with kustomize

5. Verify the deployment

6. Register a Domain and provision a backup target

[DEFAULT]

[abaca]

[api]

[conductor]

[database]

[enrollment]

[kopia]

[worker]

Annotated sample abaca.conf

The API service (abaca-api)

Using the CLI plugin

Using the REST API directly

Using the Python SDK

The conductor (abaca-conductor)

Worker VMs

Database management with abaca-manage

Checking backup coverage and fleet health

Example 1 — Verify the control plane is running after deployment

Example 2 — Create a protection policy and trigger an on-demand backup

Example 3 — Restore a share from backup (full share)

Example 4 — Inspect a failed job and identify who must act

Example 5 — Disaster recovery: rebuild catalog from Kopia repositories

Issue: abaca-api pod is not starting

Issue: Backup job is stuck in queued state

Issue: Worker VM is DEAD and associated jobs errored

Issue: Enrollment fails at the preflight stage

Issue: require_fips_profile causes repository connection failure

Issue: Conductor logs show the scheduler is skipping ticks

Issue: Worker agent cannot reach the conductor API

`[DEFAULT]`

`[abaca]`

`[api]`

`[conductor]`

`[database]`

`[enrollment]`

`[kopia]`

`[worker]`

Annotated sample `abaca.conf`

The API service (`abaca-api`)

The conductor (`abaca-conductor`)

Database management with `abaca-manage`

Issue: `abaca-api` pod is not starting

Issue: Backup job is stuck in `queued` state

Issue: `require_fips_profile` causes repository connection failure