Development Guide

This guide walks you through setting up a local Abacá development environment, running the test suite, and iterating on the codebase without requiring a real OpenStack cluster. Abacá ships a Docker Compose stack (deploy/docker-compose.dev.yml) that runs abaca-api and abaca-conductor in --noauth mode with SQLite and in-memory fakes, so you can drive the full API and state machine locally. You will also learn how to run lint, unit tests, the functional smoke test, and how to target a real cluster when you need end-to-end validation.

Before you begin, ensure your workstation has:

• Python 3.11 or 3.12 — 3.12 is the primary development interpreter; 3.11 is the supported floor. Keep code compatible with both.
• tox ≥ 4.0 — used for lint, unit tests, and all CI environments.
• Docker or Podman with Compose support — required for the local dev stack (deploy/docker-compose.dev.yml).
• ruff 0.6.9 and black 25.9.0 — pinned exactly; the CI lint gate compares byte-for-byte output. Install via tox -e lint or the matching pre-commit hooks.
• git — the monorepo contains four installable packages; all must be checked out together.
• libguestfs-tools and qemu-utils (Linux, optional) — only needed if you plan to build the worker VM Glance image locally via deploy/rhoso/worker/20-worker-image.sh.
• A real OpenStack cluster (RHOSO 18+) is only required for the tempest and functional tox environments; all unit and smoke tests run entirely on fakes.

1. Clone the repository and install all packages in editable mode

The monorepo contains four installable Python packages. Install them all into a single virtual environment so imports resolve correctly across package boundaries:

git clone <repo-url> abaca
cd abaca
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -e ./python-abacaclient
pip install -e ./abaca-dashboard
pip install -e ./abaca-tempest-plugin

The abaca package provides the abaca-api, abaca-conductor, abaca-worker-agent, abaca-manage, and abaca-dev console scripts. The python-abacaclient package provides the openstack share protection … CLI plugin.

2. Run a schema migration against SQLite

For local development, SQLite is sufficient for bookkeeping. The ABACA_DATABASE_CONNECTION environment variable sets the connection string:

ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite" abaca-manage db_sync

Important: SQLite does not enforce foreign keys or VARCHAR length limits. Three production bugs shipped through this gap. Always verify DELETE endpoint behaviour against MySQL/MariaDB before merging — the unit suite running green on SQLite is not a substitute.

3. Start the local dev stack

The Docker Compose file runs abaca-api and abaca-conductor in --noauth mode with SQLite and in-memory fakes. No Keystone, RabbitMQ, or real S3 is required:

cd deploy
docker compose -f docker-compose.dev.yml up --build

The API listens on port 9797 by default. In --noauth mode every request is treated as an admin context — suitable for local iteration only, never for production.

Production note: The abaca-api console script (Flask dev server / wsgiref) is for local development only. Production deployments run gunicorn against the WSGI callable: gunicorn --bind 0.0.0.0:9797 -w 4 abaca.api.wsgi:get_application()

4. Install tox and verify the full gate passes

pip install 'tox>=4.0'
python -m tox

This runs the lint and py3 environments. The py3 environment installs all four packages and runs the complete test suite. See the Usage section for how to run individual test subsets.

Abacá uses oslo.config — options are declared in Python (cfg.Opt grouped by config section) and read from an INI-style .conf file. The canonical source of truth is the Python code; the checked-in sample at etc/abaca.conf.sample is generated from it.

Regenerating the config sample

# Regenerate in place:
tox -e sample-config

# Verify the checked-in file is up to date (CI uses this):
tox -e sample-config-check

The sample-config environment pins oslo.config, oslo.log, oslo.messaging, oslo.db, oslo.policy, and keystonemiddleware to exact versions. Bump them deliberately and regenerate in the same commit — any upstream rename or help-string change breaks the byte-for-byte CI diff.

Key options by section

[api] — HTTP server settings

[conductor] — Orchestration, fleet, and reconciliation

Maintenance cadence is a deployment-level knob. Do not make it per-policy — see the maintenance-cadence-not-per-policy design note. The default of one week matches Kopia's own guidance.

[kopia] — Backup engine settings

Security invariant: Secrets (KOPIA_PASSWORD, AWS_*) travel only via environment variables — never via argv, disk, or logs. This is enforced in abaca/worker/kopia.py and must not be relaxed.

[worker] — Worker agent settings (set in the baked worker image, not the control plane)

[database]

[enrollment]

[abaca] — Service identity

[DEFAULT] — Keystone catalog discovery

Running the full test gate

The default tox invocation runs lint and py3 in isolated environments:

python -m tox

For a faster inner loop that reuses your active virtual environment:

python -m pytest abaca/tests python-abacaclient abaca-dashboard abaca-tempest-plugin tests -q

Linting

# Check only (what CI runs):
ruff check . && black --check .

# Auto-fix formatting in place:
python -m tox -e format

Ruff and black are pinned to exact versions (ruff==0.6.9, black==25.9.0). Do not upgrade them independently — bump both in tox.ini and regenerate any affected files in the same commit.

Running the functional smoke test

The smoke test in abaca/tests/functional/test_smoke.py is the end-to-end reference: it drives target → policy → backup → restore through the complete state machine to available using fakes and FakeKopia. Run it whenever you touch the conductor or worker:

python -m pytest abaca/tests/functional/test_smoke.py -v

Regenerating the OpenAPI spec

After any change to abaca/api/openapi.py:build_spec, regenerate the spec — CI verifies it is in sync:

python -m abaca.api.openapi
# writes docs/api-ref/openapi.yaml

Using abaca-manage for database and domain operations

# Apply schema migrations:
ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite" abaca-manage db_sync

# Register a Domain's service project (required for multi-domain installs):
abaca-manage domain_register \
    --domain-id <keystone-domain-id> \
    --service-project-id <project-id>

# List registered domains and their custody state:
abaca-manage domain_list

Multi-domain note: Without a domain_register row, the API falls back to [abaca] worker_project_id — a single-domain bootstrap allowance. GET /v1/domain will report is_service_project=false everywhere, and the dashboard will hide the Register Bucket action in every project. Register every Domain explicitly.

Using abaca-dev to boot a worker VM against a real cluster

abaca-dev is the operator/developer CLI for bring-up and testing. To boot a worker VM in the service project:

OS_CLOUD=abaca-service abaca-dev worker-boot \
    --network <neutron-network-name> \
    --key-name abaca-worker \
    --api-url https://abaca-worker.apps.example.com \
    --serve-project-id <project-id>

--api-url is required (or set $ABACA_WORKER_API_URL): the worker reaches the control plane over one outbound HTTPS connection and needs no message-bus credential. --serve-project-id pins the worker to a tenant; omitting it makes the conductor treat the worker as drain-only and never assign it work.

abaca-dev worker-boot refuses images that lack the abaca_worker_image=1 Glance property — workers must never fetch packages from the internet at boot.

Using the OpenStack CLI plugin

The python-abacaclient package adds openstack share protection … subcommands:

# List registered backup targets:
openstack share protection target list

# Show a protection policy:
openstack share protection policy show <policy-id>

# Request an on-demand backup:
openstack share protection backup create --share-id <share-id> --policy-id <policy-id>

Running Tempest integration tests against a real cluster

The tempest tox environment runs abaca-tempest-plugin smoke tests against a live cluster. Requires a tempest.conf with [share_protection] abaca_enabled = true:

TEMPEST_CONFIG_DIR=/etc/tempest tox -e tempest

The plugin discovers the Abacá API through the Keystone service catalog (share-protection type). It skips itself if abaca_enabled is false.

Running the R1 functional tests against a real cluster

The functional tox environment requires a live deployment and opts in via an environment variable:

export ABACA_FUNCTIONAL=1
export OS_CLOUD=abaca-tenant
tox -e functional

Example 1 — Full test gate (clean environment)

Run lint and the complete unit suite in isolated tox environments, as CI does:

python -m tox

Expected output (abbreviated):

lint: commands succeeded
py3: commands succeeded
congratulations :)

Example 2 — Fast pytest inner loop

During active development, skip tox overhead and run pytest directly against your working virtual environment:

python -m pytest abaca/tests python-abacaclient abaca-dashboard abaca-tempest-plugin tests -q

Expected output (abbreviated):

........................................
128 passed in 4.31s

Example 3 — Auto-format the codebase

python -m tox -e format

Expected output:

format: commands succeeded

This runs ruff check --fix . followed by black . in place. Commit the result.

Example 4 — Schema migration against SQLite

ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite" abaca-manage db_sync

Expected output:

INFO alembic.runtime.migration Running upgrade -> <revision>, initial

The migration is additive-only from v1.0. Revision IDs must be ≤ 32 characters — the alembic_version.version_num column is VARCHAR(32) in MySQL/MariaDB, and SQLite silently ignores the limit.

Example 5 — Start the local dev stack

cd deploy
docker compose -f docker-compose.dev.yml up --build

Expected output (abbreviated):

abaca-api-1      | * Running on http://0.0.0.0:9797
abaca-conductor-1| INFO abaca.conductor starting reconciliation loop

Verify the API is up:

curl -s http://localhost:9797/v1/

Because --noauth is set, no Keystone token is required. Every request is treated as an admin context.

Example 6 — Regenerate the OpenAPI spec

python -m abaca.api.openapi

Expected output:

wrote docs/api-ref/openapi.yaml

Run this after any change to abaca/api/openapi.py:build_spec. CI asserts the file matches a fresh generation — an out-of-date spec fails the build.

Example 7 — Verify the config sample is up to date

tox -e sample-config-check

Expected output on success:

sample-config-check: commands succeeded

On failure, the diff shows which option changed. Regenerate and commit:

tox -e sample-config
git add etc/abaca.conf.sample

Example 8 — Smoke test through the full state machine

python -m pytest abaca/tests/functional/test_smoke.py -v

Expected output (abbreviated):

test_smoke.py::test_backup_restore_roundtrip PASSED
test_smoke.py::test_inplace_restore_requires_force PASSED
1 passed in 2.18s

This test drives target → policy → backup → restore through all eight job states (queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available) using FakeKopia and in-memory OpenStack fakes. Run it after touching the conductor or worker.

Example 9 — Register a domain and list domains

abaca-manage domain_register \
    --domain-id d1e2a3d4-5678-90ab-cdef-111213141516 \
    --service-project-id a0b1c2d3-1234-5678-9abc-def012345678

abaca-manage domain_list

Expected output from domain_list:

+--------------------------------------+--------------------+------------------------------------------+
| domain_id                            | service_project_id | key_custody_project_id                   |
+--------------------------------------+--------------------+------------------------------------------+
| d1e2a3d4-5678-90ab-cdef-111213141516 | a0b1c2d3-...       | None                                     |
+--------------------------------------+--------------------+------------------------------------------+

Re-registering a domain to correct a service-project-id typo preserves key custody unless you explicitly pass --key-custody-project-id or --clear-key-custody — this prevents silently moving repository passwords out from under the domain.

Test suite fails immediately with import errors

Symptom: pytest exits with ModuleNotFoundError for abaca, abacaclient, or abaca_dashboard.

Cause: One or more packages are not installed in editable mode in the active environment.

Fix: Install all four packages:

pip install -e .
pip install -e ./python-abacaclient
pip install -e ./abaca-dashboard
pip install -e ./abaca-tempest-plugin

Or rely on tox, which does this automatically via commands_pre.

black --check or ruff check fails in CI but passes locally

Symptom: CI lint fails; your local run passes.

Cause: Local ruff or black version differs from the pinned versions (ruff==0.6.9, black==25.9.0).

Fix: Use the tox lint environment, which installs the pinned versions:

python -m tox -e lint

Do not upgrade ruff or black independently — bump both in tox.ini and regenerate any affected files in the same commit.

sample-config-check fails after adding a config option

Symptom: CI reports a diff in etc/abaca.conf.sample on a PR that added a new cfg.Opt.

Cause: The checked-in sample is out of date.

Fix: Regenerate and commit:

tox -e sample-config
git add etc/abaca.conf.sample

If the new option reads its default from an environment variable (via os.environ.get), pin that variable in the [testenv:sample-config] setenv block in tox.ini to match the code-level default — otherwise whatever value is in your shell gets baked into the sample.

db_sync migration fails on MySQL but passes locally on SQLite

Symptom: abaca-manage db_sync succeeds locally but fails on the cluster with a column-length or foreign-key error.

Cause: SQLite ignores VARCHAR length limits and does not enforce foreign keys. Two known failure modes:

1. An Alembic revision ID longer than 32 characters — alembic_version.version_num is VARCHAR(32) in MySQL.
2. A DELETE endpoint that cascades silently on SQLite but raises IntegrityError on MySQL.

Fix:

• Keep all revision IDs ≤ 32 characters (enforced by tests/test_monorepo.py).
• Any DELETE endpoint must declare its cascade rule explicitly — return HTTP 409 or null the FK where the column is nullable. Assert the outcome in tests, not just the absence of an exception.

OpenAPI spec is out of date

Symptom: CI fails with a diff in docs/api-ref/openapi.yaml after modifying the API.

Cause: build_spec in abaca/api/openapi.py was changed but the generated file was not updated.

Fix:

python -m abaca.api.openapi
git add docs/api-ref/openapi.yaml

Dev stack API returns 500 with no error envelope

Symptom: curl http://localhost:9797/v1/... returns a bare HTTP 500 with no JSON body.

Cause: A DELETE endpoint hit a foreign-key violation (pymysql IntegrityError) that was not caught and wrapped in the Abacá error envelope ({code, title, detail, category}).

Fix: The endpoint must explicitly handle the FK case — return HTTP 409 with an appropriate AbacaError subclass from abaca/common/exceptions.py. Verify the fix against MySQL/MariaDB, not just the SQLite dev stack.

Worker boots but never registers with the conductor

Symptom: abaca-dev worker-boot succeeds; openstack share protection worker list shows no active workers.

Cause: The worker agent cannot reach the control plane. Common causes:

• --api-url was not passed to abaca-dev worker-boot (it is required).
• The worker VM cannot route outbound HTTPS to the worker_api_url.
• The Glance image lacks the abaca_worker_image=1 property — abaca-dev refuses such images.

Fix:

1. Verify --api-url was supplied. The worker reaches the control plane over a single outbound HTTPS connection; it does not use the message bus.
2. Check network reachability from the worker VM to the control plane URL.
3. Rebuild the worker image if needed: bash deploy/rhoso/worker/20-worker-image.sh.

FakeKopia is not being used and tests invoke a real kopia binary

Symptom: Unit tests fail with FileNotFoundError: kopia or produce unexpected output.

Cause: The KopiaRunner executor was not injected with FakeKopia.

Fix: Use FakeKopia as the executor in all unit and smoke tests — inject it via the executor parameter on KopiaRunner. Never require a real Kopia binary for unit or smoke tests. For strategy tests, use abaca.strategies.FakeWorkerMounter in place of RpcWorkerMounter.

abaca-manage domain_register moved repository passwords to a different project

Symptom: After re-running domain_register to correct a --service-project-id typo, backups fail with Barbican access errors.

Cause: --key-custody-project-id or --clear-key-custody was passed unintentionally, moving key custody away from the project that holds the repository passwords.

Fix: Key custody is preserved by default on re-registration. Only pass --key-custody-project-id or --clear-key-custody when you deliberately intend to move custody. Verify custody state with abaca-manage domain_list before and after any re-registration.