Guide

Configuration (oslo.config)

The .conf file, its option groups, and an annotated sample

Overview

This page describes every configuration option Abacá exposes through oslo.config, explains where each option lives in the INI-style .conf file, and shows how to produce a working configuration for both the API service and the conductor. Understanding this file is prerequisite to deploying either component: without a correctly configured .conf, Abacá cannot authenticate to Keystone, reach the database, talk to the broker, or boot worker VMs. Options are declared in code (abaca/common/config.py) and grouped by section; you should never hand-edit the generated sample — edit the Python source and regenerate with tox -e sample-config.

Prerequisites

Before configuring Abacá, ensure the following are in place:

RHOSO ≥ 18 with OpenShift ≥ 4.14 (for control-plane hosting)
OpenStack services running and reachable: Keystone ≥ 2023.1 (with trusts and domain-scoped tokens enabled), Manila, Nova, Neutron, Barbican, and Glance
Dedicated MySQL / MariaDB (Galera) instance — Abacá's own database; do not share the platform's instance
Dedicated RabbitMQ instance — Abacá's own broker; do not share the platform's instance
S3-compatible object storage with versioning and Object Lock enabled at bucket creation (Wasabi, MinIO, or AWS S3)
Abacá identity objects created — service user, service project, worker project, and Keystone roles — typically provisioned by deploy/rhoso/01-identity.sh
Service user ID and worker project ID known (needed for [abaca] section) — look them up with openstack user show and openstack project show under an admin credential
Python ≥ 3.11 on any host that runs abaca-manage, abaca-api, or abaca-conductor
Kopia binary ≥ 0.17.0 (FIPS-built, version-pinned) available inside worker VM images
For RHOSO deployments, the rendered .conf files are published as a Kubernetes Secret and mounted into pods — you do not place them on a host filesystem directly

Installation

Abacá's configuration is rendered by deploy/rhoso/06-config.sh (RHOSO target) or deploy/kolla/06-config.sh (Kolla reference environment) and is never hand-written from scratch in production. Follow these steps to produce and publish a valid configuration.

Step 1 — Complete the prerequisite deploy steps

Run the earlier deploy scripts in order before touching configuration. The config scripts read secrets and IDs that those steps produce.

# RHOSO path — run from the repo root
bash deploy/rhoso/01-identity.sh   # creates service user, roles, projects
bash deploy/rhoso/03-database.sh   # provisions the dedicated Galera database
bash deploy/rhoso/04-messaging.sh  # provisions the dedicated RabbitMQ instance

Step 2 — Render and publish the configuration

With admin credentials sourced, run the config script. It resolves all IDs and secrets from the cluster, renders two .conf files (abaca-api.conf and abaca-conductor.conf) plus a clouds.yaml for the conductor's warm-pool boots, and publishes them as an OpenShift Secret.

bash deploy/rhoso/06-config.sh

The script creates or updates the Secret named by ABACA_CONFIG_SECRET in the abaca namespace and immediately rolls any already-running abaca-api and abaca-conductor Deployments so they pick up the new configuration.

Step 3 — Verify the Secret contents

oc -n abaca get secret "${ABACA_CONFIG_SECRET}" -o json \
  | jq -r '.data | keys[]'

Expected output:

abaca-api.conf
abaca-conductor.conf
clouds.yaml
my.cnf
worker-api-ca.pem

Step 4 — (Development only) local workstation config

For the Docker Compose developer stack (deploy/docker-compose.dev.yml), copy the sample and override the minimum set of options:

cp etc/abaca.conf.sample /etc/abaca/abaca.conf
# Edit the copy — uncomment and set at minimum:
#   [database] connection
#   [DEFAULT] transport_url
#   [api] noauth = true   # dev only — NEVER in production

Point each binary at the file with --config-file:

abaca-api       --config-file /etc/abaca/abaca.conf
abaca-conductor --config-file /etc/abaca/abaca.conf
abaca-manage    --config-file /etc/abaca/abaca.conf db_sync

Step 5 — Regenerate the sample after changing options

If you add or modify an option in abaca/common/config.py, regenerate the sample before committing:

tox -e sample-config        # regenerate etc/abaca.conf.sample in place
tox -e sample-config-check  # CI check — fails if the sample has drifted

Configuration

All options are grouped into INI sections. Every section below lists only the options that Abacá itself declares; oslo.log and oslo.messaging options (such as debug, transport_url, and rpc_response_timeout) follow standard OpenStack conventions and are documented in those libraries.

`[DEFAULT]`

The [DEFAULT] section carries oslo.log and oslo.messaging top-level options. The two most commonly set:

Option	Type	Default	Purpose
`debug`	boolean	`false`	Raise log level to DEBUG. Can be changed without restarting.
`use_stderr`	boolean	`false`	Write logs to stderr; the typical choice for containerised deployments.
`transport_url`	string	`rabbit://`	AMQP URL for Abacá's dedicated RabbitMQ instance. Use the full `rabbit://user:pass@host:port/vhost` form.

Why a dedicated broker? Abacá's RabbitMQ is separate from the platform's shared broker. It carries only Barbican hrefs and Keystone trust IDs — never secret material — but keeping it isolated bounds blast radius and simplifies access control.

`[database]`

Option	Type	Default	Purpose
`connection`	string	—	SQLAlchemy connection URL for Abacá's dedicated Galera database. On RHOSO, append `?read_default_file=/etc/my.cnf` so PyMySQL reads TLS settings from the mounted `my.cnf`.

Example (RHOSO, with TLS):

[database]
connection = mysql+pymysql://abaca:SECRET@galera.abaca-db.svc/abaca?read_default_file=/etc/my.cnf

Example (Kolla / dev, no TLS):

[database]
connection = mysql+pymysql://abaca:SECRET@192.0.2.10/abaca?charset=utf8mb4

Abacá's database stores only bookkeeping metadata (targets, policies, jobs, worker fleet state). Backup data lives in S3, not here.

`[keystone_authtoken]`

Standard keystonemiddleware options — documented at docs.openstack.org/keystonemiddleware. The options you must set for Abacá:

[keystone_authtoken]
www_authenticate_uri         = https://keystone-internal.<namespace>.svc:5000
auth_url                     = https://keystone-internal.<namespace>.svc:5000
auth_type                    = password
project_domain_name          = Default
user_domain_name             = Default
project_name                 = <abaca-service-project>
username                     = abaca
password                     = <service-password>
service_token_roles          = service
service_token_roles_required = true
region_name                  = <region>
interface                    = internal
memcached_servers            = <memcached-host>:11211
cafile                       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

On RHOSO the cafile is required because the UBI9/Python 312 image trusts the certifi bundle, not the system store. Without it, requests to the internal Keystone endpoint fail TLS verification.

`[abaca]`

Identity data for the Abacá service account. These values are non-sensitive — they are published on the unauthenticated /v1/service_info endpoint so tenants can construct Keystone trusts to the correct trustee without needing user-list privileges.

Option	Type	Default	Purpose
`service_user_id`	string	—	Keystone user ID of the Abacá service account. Resolved at deploy time via `openstack user show`.
`service_user_name`	string	`abaca`	Keystone user name of the Abacá service account.
`worker_project_id`	string	—	Keystone project ID of the project where worker VMs run. Resolved at deploy time.
`worker_project_name`	string	`abaca`	Keystone project name of the worker VM project.

`[api]`

Applies to the abaca-api process only.

Option	Type	Default	Purpose
`bind_host`	string	`0.0.0.0`	IP address on which the API server listens.
`bind_port`	integer	`9797`	TCP port for the API server.
`noauth`	boolean	`false`	Development only. Bypasses keystonemiddleware and injects a fake admin context. The conductor refuses to start with `dispatch = rpc` when `noauth = true` is absent — and will refuse to start in production if `noauth = true` is present. Never enable in production.
`max_limit`	integer	`1000`	Maximum page size for list endpoints.
`default_limit`	integer	`100`	Default page size for list endpoints when the caller does not specify `limit`.

`[conductor]`

Applies to the abaca-conductor process only. Options fall into four groups.

Scheduler and reconciliation cadence

Option	Type	Default	Purpose
`scheduler_interval`	integer	`60`	Seconds between passes that evaluate cron-expression policies and enqueue overdue backup jobs.
`reconciliation_interval`	integer	`300`	Seconds between reconciliation sweeps — the background loop that reaps dead workers, recovers orphaned jobs, and enforces retention.
`sweep_command_wait_seconds`	integer	`120`	How long a reconciliation sweep waits for a worker command response before giving up and retrying next tick. Keep this well below `reconciliation_interval`. A sweep that cannot get an answer should skip — it has no tenant waiting on it — whereas tenant-facing jobs keep the full Kopia subprocess timeout. Setting this too high caused 12-hour sweep periods on a production-equivalent cluster (each hung `snapshot list` blocked for 3600 s, stretching a 300 s loop to over 12 hours).
`queued_job_deadline_seconds`	integer	`300`	How long a job may sit in the `queued` state before the reconciliation sweep considers it orphaned and reschedules it.
`maintenance_interval_seconds`	integer	`604800`	Cadence (seconds) for scheduled `kopia maintenance run --full` per registered BackupTarget. Kopia reclaims S3 storage from orphaned content blobs only during maintenance, so this directly determines when `keep_daily=7` retention actually frees space. Default is one week.
`catalogue_sync_interval_seconds`	integer	`3600`	Cadence for reconciling each target's Backup rows against the snapshots Kopia actually holds. A target never synced is always considered due.
`usage_sample_interval_seconds`	integer	`21600`	Cadence for `kopia content stats` per BackupTarget — the source of the post-dedup, post-compression byte count shown on the Target Details page. Default is 6 hours.

Worker heartbeat and token lifecycle

Option	Type	Default	Purpose
`worker_heartbeat_interval_seconds`	integer	`10`	How often a running `abaca-worker-agent` sends a heartbeat RPC cast to the conductor. Must be comfortably less than `worker_heartbeat_deadline_seconds`.
`worker_heartbeat_deadline_seconds`	integer	`60`	Seconds since a worker's last heartbeat before the fleet manager marks it `DEAD` and reassigns its in-flight job. Match this to your reconciliation loop cadence.
`worker_token_ttl_seconds`	integer	`1800`	Lifetime (seconds) of a worker's bearer token — the credential it presents to `/v1/workers/*`, not a Keystone token. This is a sliding window: every heartbeat pushes expiry out by this amount, so a live worker never loses its credential. A hard-rebooted RHEL worker VM has been measured taking 15–19 minutes to reach its first heartbeat; 1800 s clears that with margin. Set `0` to disable expiry. Do not lower this below a comfortable multiple of `worker_heartbeat_deadline_seconds`.

Warm-pool worker boot

These options control unattended worker VM provisioning by the reconciliation loop. The loop boots one worker per active project per tick until every project with an enabled Policy or a non-terminal Job is at the min_workers floor.

Option	Type	Default	Purpose
`min_workers`	integer	`1`	Minimum warm workers per project that has active work. Set `0` to disable unattended booting and manage the fleet manually. Requires `worker_api_url`, `worker_boot_network`, and `worker_boot_key_name` to be set; without them the sweep stays disabled and logs a warning.
`worker_boot_cooldown_seconds`	integer	`180`	Minimum seconds between warm-pool boot attempts per project. Cloud-init typically finishes registering a worker VM within 30–90 s; 180 s guards against double-booting when the sweep ticks before the previous VM has registered.
`worker_boot_os_cloud`	string	`abaca-service`	The named cloud in `clouds.yaml` that the conductor uses to call Nova when booting workers. On RHOSO this must match the cloud name rendered by `06-config.sh` (`abaca-worker` by convention).
`worker_boot_image`	string	`abaca-worker-0.23.1`	Glance image name used when booting worker VMs. In RHOSO deployments this is discovered automatically by the `abaca_worker_image=1` property (newest wins); pin `ABACA_WORKER_IMAGE` in the environment to override.
`worker_boot_flavor`	string	`m1.small`	Nova flavor for worker VMs.
`worker_boot_network`	string	—	Neutron network name or ID on which worker VMs are booted. Required for unattended booting.
`worker_boot_key_name`	string	—	Nova keypair name injected into worker VMs for emergency SSH access. Required for unattended booting.
`worker_boot_max_failures`	integer	`3`	Number of consecutive boot failures before the sweep stops attempting for a project, preventing runaway Nova quota consumption.
`worker_boot_failure_window_seconds`	integer	`7200`	Lookback window (seconds) in which `worker_boot_max_failures` is counted.
`worker_boot_grace_seconds`	integer	`1200`	How long after a worker VM is booted before the conductor considers it overdue to register. A VM still in cloud-init is not yet `DEAD`.
`worker_api_url`	string	—	The URL (`https://…`) that booted worker VMs use to reach the Abacá API. The conductor embeds this in each worker's cloud-init payload at boot time. Without it, no worker can register and `min_workers > 0` is ineffective.
`worker_api_ca_file`	string	—	Path to a CA bundle file that the worker agent uses to verify TLS on `worker_api_url`. Required wherever the Route is signed by a non-publicly-trusted CA (the default on RHOSO). Omitting it causes silent total failure: workers boot, fail TLS on their first API call, never register, are reaped, and are re-booted — consuming Nova quota with nothing visible in the API access log.
`worker_api_host_aliases`	list	—	One or more `IP=HOSTNAME` pairs written into each booted worker's `/etc/hosts`. Required when the worker subnet's resolver cannot answer for the `worker_api_url` hostname — on RHOSO the Route sits on a private wildcard domain while workers use a public resolver. Format: `192.0.2.5=abaca.apps.cluster.example.com`.

`[enrollment]`

Option	Type	Valid values	Purpose
`dispatch`	string	`rpc`, `inline`	Controls how bucket enrollment jobs are dispatched. `rpc` is the production path: the API casts the job, and the conductor drives it on a worker VM. `inline` runs the enrollment synchronously in the API process — only valid when `[api] noauth = true`, and the conductor refuses to start with `dispatch = inline` on a real cluster.

`[kopia]`

Controls how Abacá invokes Kopia inside worker VMs.

Option	Type	Default	Valid values	Purpose
`binary`	string	`kopia`	—	Path or name of the Kopia binary.
`require_fips_profile`	boolean	`true`	—	When `true`, Abacá refuses to create a repository unless the FIPS cryptographic profile is applied (AES-256-GCM-HMAC-SHA256 encryption, HMAC-SHA256-128 block hash, PBKDF2 key derivation). The profile is set at repository creation and is immutable afterward.
`subprocess_timeout`	integer	`3600`	—	Maximum seconds a Kopia subprocess may run before being killed. This is the tenant-facing job timeout; keep `sweep_command_wait_seconds` well below this value.
`executor`	string	`http`	`http`, `ephemeral_container`	How Abacá invokes Kopia. `http` is the production path on RHOSO (the conductor has no container runtime). `ephemeral_container` is a Kolla/dev-environment path only — the conductor uses a local container engine.
`image`	string	—	—	Container image for Kopia when `executor = ephemeral_container`. Not used on RHOSO.
`container_engine`	string	—	`docker`, `podman`	Container engine when `executor = ephemeral_container`. Not used on RHOSO.
`container_network`	string	—	—	Docker/Podman network for ephemeral Kopia containers. Not used on RHOSO.

`[worker]`

Applies to the abaca-worker-agent process running inside each worker VM.

Option	Type	Default	Purpose
`id`	string	—	Unique identity of this worker instance, assigned at boot and baked into the worker's config by cloud-init.
`api_url`	string	—	URL of the Abacá API this worker registers with and polls for commands. Populated from `[conductor] worker_api_url` at boot time.
`token`	string	—	Bearer token the worker presents to `/v1/workers/*`. Issued at registration and stored in the worker's local config by cloud-init.
`api_ca_file`	string	—	CA bundle file for verifying TLS on `api_url`. Corresponds to `[conductor] worker_api_ca_file` content pushed into the VM at boot.
`capacity_slots`	integer	—	Number of concurrent jobs this worker VM can run. The fleet picker only assigns a new job to a worker whose running-job count is strictly below this limit.
`mount_base`	string	`/var/lib/abaca/mnt`	Base directory under which Manila shares are mounted inside the worker VM.
`command_timeout_seconds`	integer	`300`	Seconds a worker waits for acknowledgment of a command before timing out.
`command_lease_seconds`	integer	`120`	Duration for which a worker holds an exclusive lease on a command — prevents double-execution if the conductor retries.
`max_command_duration_seconds`	integer	`21600`	Hard upper bound (6 hours) on how long any single command (backup, restore) may run.
`claim_poll_interval_seconds`	float	`2`	Seconds between command-claim poll attempts.
`long_poll_seconds`	integer	`20`	How long the worker holds a long-poll connection open waiting for a new command.
`progress_interval_seconds`	integer	`30`	How often the worker reports Kopia transfer progress back to the conductor.
`listener_port`	integer	`9798`	TCP port on which the worker agent's local HTTP listener binds (used by the conductor's `http` executor).
`command_poll_interval_seconds`	float	`2`	Interval between polls for the next command when not in long-poll mode.

Usage

Starting the services

On RHOSO, abaca-api and abaca-conductor run as OpenShift Deployments in the abaca namespace and are managed through oc — not through a host service manager. After 06-config.sh publishes a new Secret, the script automatically restarts both Deployments. To restart them manually:

oc -n abaca rollout restart deployment/abaca-api
oc -n abaca rollout restart deployment/abaca-conductor

To watch rollout progress:

oc -n abaca rollout status deployment/abaca-api
oc -n abaca rollout status deployment/abaca-conductor

Running a database migration

Schema migrations are additive-only and run via abaca-manage db_sync, which executes as a Kubernetes Job before the service pods start:

# Kubernetes Job (managed by kustomize / 07-deploy.sh)
# To trigger manually for debugging:
oc -n abaca create job db-sync-manual --from=cronjob/abaca-db-sync

On a developer workstation:

abaca-manage --config-file /etc/abaca/abaca.conf db_sync

Choosing the right `dispatch` mode

Set [enrollment] dispatch = rpc in every production or staging configuration. The inline value is rejected by the conductor at startup unless [api] noauth = true is also set — which is itself forbidden in production.

Tuning the reconciliation loop

The three most common tuning levers and the interaction you must preserve:

reconciliation_interval > sweep_command_wait_seconds
worker_heartbeat_deadline_seconds > worker_heartbeat_interval_seconds × (some healthy margin)
worker_token_ttl_seconds >> worker_heartbeat_deadline_seconds

The default values satisfy all three. If you increase reconciliation_interval to reduce Kopia round trips in a large fleet, also increase sweep_command_wait_seconds proportionally — but keep it below reconciliation_interval. If you tighten worker_heartbeat_deadline_seconds for faster failure detection, lower worker_heartbeat_interval_seconds to match.

Managing the warm worker pool

Set min_workers = 0 to disable unattended booting and manage workers with abaca-dev. Set min_workers = 1 (the default) to let the conductor maintain at least one idle worker per active project. The conductor requires worker_api_url, worker_boot_network, and worker_boot_key_name to be set before it will attempt any boot; without those three options the warm-pool feature stays disabled regardless of min_workers.

Locating and reading logs

# API logs
oc -n abaca logs deployment/abaca-api --follow

# Conductor logs
oc -n abaca logs deployment/abaca-conductor --follow

# Filter for reconciliation sweep messages
oc -n abaca logs deployment/abaca-conductor | grep reconcil

To enable DEBUG logging without restarting (oslo.log supports live reload for debug):

# Edit the mounted config Secret and restart — oslo.log marks 'debug' as
# reloadable, but on RHOSO the config is immutable in-pod; a rollout is needed.
oc -n abaca rollout restart deployment/abaca-conductor

Examples

Minimal RHOSO production configuration (rendered by `06-config.sh`)

The following is representative of what 06-config.sh renders and publishes as abaca-api.conf. Values in angle brackets are substituted from cluster secrets and Keystone lookups at render time.

[DEFAULT]
debug = false
use_stderr = true
transport_url = rabbit://abaca:<MQ_PASSWORD>@rabbitmq.abaca-mq.svc:5672/abaca

[database]
connection = mysql+pymysql://abaca:<DB_PASSWORD>@galera.abaca-db.svc/abaca?read_default_file=/etc/my.cnf

[keystone_authtoken]
www_authenticate_uri         = https://keystone-internal.openstack.svc:5000
auth_url                     = https://keystone-internal.openstack.svc:5000
auth_type                    = password
project_domain_name          = Default
user_domain_name             = Default
project_name                 = abaca-service
username                     = abaca
password                     = <SERVICE_PASSWORD>
service_token_roles          = service
service_token_roles_required = true
region_name                  = RegionOne
interface                    = internal
memcached_servers            = memcached.openstack.svc:11211
cafile                       = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

[enrollment]
dispatch = rpc

[abaca]
service_user_id     = a1b2c3d4-e5f6-7890-abcd-ef1234567890
service_user_name   = abaca
worker_project_id   = b2c3d4e5-f6a7-8901-bcde-f12345678901
worker_project_name = abaca

[api]
bind_host = 0.0.0.0
bind_port = 9797
noauth    = false

Conductor configuration with warm-pool booting enabled

This is representative of what 06-config.sh renders as abaca-conductor.conf, including the options needed for unattended worker VM provisioning.

# ... [DEFAULT], [database], [keystone_authtoken], [enrollment], [abaca]
# identical to the API config above ...

[conductor]
scheduler_interval          = 60
reconciliation_interval     = 300
sweep_command_wait_seconds  = 120
min_workers                 = 1
worker_boot_cooldown_seconds = 180
worker_api_url              = https://abaca.apps.cluster.example.com
worker_boot_os_cloud        = abaca-worker
worker_boot_network         = abaca-worker-net
worker_boot_key_name        = abaca-worker-key
worker_boot_flavor          = m1.medium
worker_boot_image           = abaca-worker-rhel9-0.23.1
worker_api_ca_file          = /etc/abaca/worker-api-ca.pem
worker_api_host_aliases     = 192.0.2.50=abaca.apps.cluster.example.com

Why worker_api_host_aliases? The Route hostname resolves only inside the cluster. Worker VMs sit on a tenant network whose DNS nameserver (often a public resolver) has never heard of *.apps.cluster.example.com. The alias writes the correct IP into /etc/hosts on each booted VM, eliminating the resolution failure without requiring DNS changes.

Minimal developer workstation configuration

For use with deploy/docker-compose.dev.yml. SQLite and in-memory fakes replace the real services.

[DEFAULT]
debug = true
use_stderr = true

[database]
connection = sqlite:////tmp/abaca-dev.db

[enrollment]
dispatch = inline

[abaca]
service_user_name   = abaca
worker_project_name = abaca

[api]
bind_host = 127.0.0.1
bind_port = 9797
noauth    = true

[kopia]
binary              = kopia
require_fips_profile = false
executor            = ephemeral_container
image               = abaca-kopia:0.17.0
container_engine    = docker
container_network   = host

Warning: noauth = true bypasses all Keystone authentication. Use this only on a local developer workstation. The conductor rejects dispatch = rpc when noauth = true is active; that is why dispatch = inline is required here.

Regenerating the sample configuration file

tox -e sample-config

Expected output:

sample-config run-test: commands[0] | oslo-config-generator --config-file oslo-config-generator/abaca.conf
  ....
sample-config: commands succeeded

The regenerated file appears at etc/abaca.conf.sample. Commit it alongside any option changes.

Troubleshooting

Use the following patterns to diagnose common configuration problems. Each entry follows the format: Symptom → Likely cause → Fix.

Symptom: abaca-api pods crash-loop with oslo_db.exception.DBConnectionError or Can't connect to MySQL server.

Likely cause: The [database] connection URL is incorrect, the Galera service is unreachable, or TLS verification fails because ssl-ca in my.cnf points to a CA bundle that is not mounted at the expected path.

Fix: Verify the Galera service endpoint from inside the pod:

oc -n abaca exec deployment/abaca-api -- \
  python3 -c "import pymysql; pymysql.connect(host='galera.abaca-db.svc', user='abaca', password='SECRET', db='abaca', ssl={'ca': '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'})"

Also confirm the Secret contains my.cnf and that the Deployment mounts it at /etc/my.cnf.

Symptom: API requests return HTTP 401 Unauthorized with WWW-Authenticate: Keystone uri=... even with a valid token.

Likely cause: [keystone_authtoken] cafile is missing or points to the wrong CA. On RHOSO, the UBI9/Python image trusts the certifi bundle rather than the system store, so the internal Keystone endpoint's certificate is not trusted by default.

Fix: Ensure cafile is set to the mounted CA bundle path and that the bundle file is non-empty inside the pod:

oc -n abaca exec deployment/abaca-api -- \
  openssl s_client -connect keystone-internal.openstack.svc:5000 \
  -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem </dev/null

Symptom: Worker VMs boot successfully (visible in openstack server list) but never appear in the conductor's fleet, and no traffic appears in the API access log.

Likely cause: One or both of worker_api_ca_file and worker_api_host_aliases are missing. Workers fail TLS verification or DNS resolution on their first call to worker_api_url and exit before registering. Because TLS handshake failures never become HTTP requests, the API log shows nothing.

Fix:

Confirm worker_api_ca_file names a file that contains the OpenShift ingress CA (not the OpenStack internal CA — these are different).
Confirm worker_api_host_aliases contains an entry resolving the Route hostname to the ingress VIP.
SSH into a booted worker and inspect the agent log:

openstack server ssh <worker-vm-name> -- journalctl -u abaca-worker-agent --no-pager | tail -40

A line containing certificate verify failed or Name or service not known identifies which problem is present.

Symptom: The reconciliation loop runs far more slowly than reconciliation_interval — for example, a 300-second interval stretches to hours.

Likely cause: sweep_command_wait_seconds is set too high (or left at a value inherited from a stale config). Each blocked snapshot list command holds the sweep thread for the full kopia subprocess_timeout (default 3600 s). Twelve orphaned targets × 3600 s = 12+ hours per tick.

Fix: Set sweep_command_wait_seconds well below reconciliation_interval. The default of 120 s is intentional:

[conductor]
reconciliation_interval    = 300
sweep_command_wait_seconds = 120

Confirm the value took effect by checking conductor logs for sweep_command_wait after a restart.

Symptom: The conductor logs worker_boot_image not configured or boots workers from a stale image after uploading a new one.

Likely cause: worker_boot_image names a specific image that no longer exists, or the option is unset and the conductor falls back to its compiled-in default (abaca-worker-0.23.1) which may not exist in Glance.

Fix: On RHOSO, re-run 06-config.sh — it discovers the newest Glance image carrying the abaca_worker_image=1 property (sorted by created_at descending) and writes that name into the conductor's config. To pin a specific image, set ABACA_WORKER_IMAGE in the environment before running the script:

export ABACA_WORKER_IMAGE=abaca-worker-rhel9-0.24.0
bash deploy/rhoso/06-config.sh

Symptom: abaca-manage db_sync fails with Target database is not up to date.

Likely cause: The database was migrated by a newer version of Abacá and you are running an older abaca-manage binary, or the migration Job ran against the wrong database endpoint.

Fix: Confirm the [database] connection URL in the config passed to abaca-manage --config-file points to the correct Galera instance. Abacá's migrations are additive-only — downgrades are not supported. If you need to roll back, restore the database from a backup taken before the upgrade.

Symptom: dispatch = inline causes the conductor to refuse to start.

Likely cause: [enrollment] dispatch = inline is only valid when [api] noauth = true. The conductor calls validate_dispatch_config at startup and rejects this combination on a real cluster.

Fix: Set dispatch = rpc in the production config. The inline value exists only for the local Docker Compose developer stack where noauth = true is also set.

Symptom: The [conductor] warm-pool sweep logs warm pool disabled — missing required options and never boots workers despite min_workers = 1.

Likely cause: One or more of worker_api_url, worker_boot_network, or worker_boot_key_name is unset. The sweep requires all three before it will attempt a boot; omitting any of them is treated as an explicit opt-out.

Fix: Set all three in [conductor]:

[conductor]
worker_api_url       = https://abaca.apps.cluster.example.com
worker_boot_network  = abaca-worker-net
worker_boot_key_name = abaca-worker-key

Then restart the conductor:

oc -n abaca rollout restart deployment/abaca-conductor