Development Guide

This page walks you through setting up a local Abacá development environment, running the full test suite, and working with the dev stack end-to-end. Because Abacá runs and tests entirely locally with mocks — no real OpenStack, message bus, or Kopia binary required — you can iterate quickly against in-memory fakes before touching a real cluster. Understanding the local setup also gives you the mental model you need when you are ready to deploy against a real Kolla-Ansible or RHOSO environment.

Before you begin, make sure you have the following:

• Python 3.11 or 3.12 — 3.12 is the primary target; 3.11 is the supported floor. Both are tested in CI.
• pip and the ability to create virtual environments (python -m venv)
• tox — used for the full gate (lint + all test environments)
• Docker or Podman — required only for the Docker Compose dev stack; not needed for the local no-container path
• Git — to clone the repository
• ruff 0.6.9 and black 25.9.0 — the linting toolchain (installed automatically by tox)
• MariaDB 10.x or SQLite — SQLite is the default for local dev (sqlite:// in memory); MariaDB is used in the Compose stack and is required for production
• RabbitMQ — required by the Compose stack; not needed for unit tests

Note: For live-cluster work (R1 enrollment or R2 data-path probing), you additionally need an OpenStack environment (RHOSO 18 or Kolla-Ansible ≥ 2023.1 / Antelope), access to a Manila share, a Barbican instance, and an S3-compatible bucket (AWS S3, MinIO, Ceph RGW, or Wasabi).

Follow these steps to get a working local environment.

1. Clone the repository

git clone https://github.com/MuralidharB/abaca.git
cd abaca

2. Create and activate a virtual environment

python -m venv .venv
source .venv/bin/activate

3. Install all four packages in editable mode

The monorepo contains four independently installable packages. Install them together so the test suite can import all of them:

pip install -e ".[test]" -e ./python-abacaclient -e ./abaca-dashboard

This installs:

• abaca — the control plane (API, conductor, worker agent, strategies, DB)
• python-abacaclient — the OpenStack CLI plugin and HTTP SDK
• abaca-dashboard — the Horizon plugin skeleton

The abaca-tempest-plugin package is wired into the test suite automatically; it does not need a separate install step for local unit testing.

4. (Optional) Install tox for the full gate

pip install tox

5. Run the full gate

python -m tox

This runs the lint and py3 environments in isolation, covering lint (ruff + black) and the complete unit test suite across all four packages.

6. (Optional) Start the Docker Compose dev stack

If you want to run the full API and conductor services locally with a real message bus and database:

cd deploy
docker compose -f docker-compose.dev.yml up --build

The stack starts:

• MariaDB on port 3306
• RabbitMQ on ports 5672 / 15672
• MinIO on ports 9000 (API) / 9001 (console)
• abaca-api on port 9797 (in --noauth dev mode)
• abaca-conductor connected to the same broker and database

Two MinIO buckets are provisioned automatically:

• abaca-demo-plain — a standard bucket with no object lock
• abaca-demo-locked — versioning enabled, object lock on, 1-day compliance retention

Important: The abaca-api console script starts a development server (Flask dev server / wsgiref). It is not suitable for production. Production deployments use gunicorn: gunicorn --bind 0.0.0.0:9797 -w 4 abaca.api.wsgi:get_application()

Environment variables

Abacá uses oslo.config for configuration. In development and in the Docker Compose stack, oslo.config options are set via environment variables. Each variable maps directly to a config option.

Generating and checking the sample config file

The authoritative list of all configuration options is in abaca/common/config.py. A human-readable sample is generated from that source:

# Regenerate etc/abaca.conf.sample
tox -e sample-config

# Verify the checked-in sample is up-to-date (CI uses this)
tox -e sample-config-check

When you add or change an oslo.config option, always regenerate the sample and commit the updated file.

Generating the OpenAPI spec

The OpenAPI 3 specification is generated from abaca/api/openapi.py. After any change to build_spec in that module, regenerate the spec:

python -m abaca.api.openapi
# Writes docs/api-ref/openapi.yaml

A test enforces that the checked-in spec matches the generated output, so failing to regenerate will cause a CI failure.

Database setup (local persistent)

For local development with a persistent SQLite database:

ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite" abaca-manage db_sync

For the Compose stack, the api service runs migrations automatically on startup against MariaDB.

FIPS mode

FIPS mode is a supported, validated production configuration. When Kopia creates a repository, Abacá always applies the FIPS crypto profile:

• Encryption: AES256-GCM-HMAC-SHA256
• Block hash: HMAC-SHA256-128
• Key derivation: pbkdf2 (when probed as supported)

Secrets travel only via environment variables (KOPIA_PASSWORD, AWS_*) — never via command-line arguments, disk, or logs. You do not need to configure this manually; it is enforced by the worker internals.

Fast local iteration (no containers)

For the tightest feedback loop during development, run pytest directly against your active venv after installing all packages:

python -m pytest abaca/tests python-abacaclient abaca-dashboard abaca-tempest-plugin tests -q

This runs the entire test suite across all four packages. It uses an in-memory SQLite database, in-memory OpenStack client fakes (Clients.fakes()), and FakeKopia as the Kopia test double — no real OpenStack, broker, or Kopia binary is needed.

Running lint

ruff check . && black --check .

To auto-fix formatting:

python -m tox -e format

Lint settings: line length 88, ruff rules E, F, W, I, UP, B (with E501 and UP042 intentionally ignored).

Running the functional smoke test

The functional smoke test in abaca/tests/functional/test_smoke.py is the end-to-end reference. It drives the full flow — target creation → policy creation → backup → restore — through the complete job state machine to available, using in-memory fakes and FakeKopia:

pytest abaca/tests/functional/test_smoke.py -q

Run this test after touching any conductor or worker code.

Driving the API manually (Compose dev stack)

With the Compose stack running, the API is available at http://localhost:9797 in --noauth mode (no Keystone token required). You can drive a full backup flow with curl:

# 1. Register a backup target
curl -s -XPOST localhost:9797/v1/targets \
  -H 'content-type: application/json' \
  -d '{
    "name": "primary",
    "endpoint": "https://s3.example.com",
    "bucket": "tenant-abaca",
    "barbican_secret_refs": ["https://barbican/secrets/pw"],
    "trust_id": "trust-1"
  }'

# 2. Create a protection policy
curl -s -XPOST localhost:9797/v1/policies \
  -H 'content-type: application/json' \
  -d '{
    "name": "nightly",
    "share_id": "share-uuid-0001",
    "target_id": "<TARGET_ID>",
    "schedule": "0 2 * * *",
    "retention": {"daily": 7}
  }'

# 3. Request an on-demand backup
curl -s -XPOST localhost:9797/v1/backups \
  -H 'content-type: application/json' \
  -d '{
    "share_id": "share-uuid-0001",
    "target_id": "<TARGET_ID>"
  }'

Using the OpenStack CLI plugin

With python-abacaclient installed, you can manage share protection resources from the OpenStack CLI:

openstack share protection target enroll <name> \
  --endpoint <s3-url> \
  --bucket <bucket> \
  --region us-east-1 \
  --wait

All openstack share protection … subcommands are provided by the python-abacaclient OSC plugin. S3 credentials must come from the environment (ABACA_TARGET_ACCESS_KEY / ABACA_TARGET_SECRET_KEY or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) — never from command-line flags.

Using abaca-dev for live enrollment (R1)

abaca-dev enroll is deprecated in favour of the OSC plugin above, but is still available for one release cycle:

export OS_CLOUD=abaca-tenant
export ABACA_TARGET_ACCESS_KEY=<access-key>
export ABACA_TARGET_SECRET_KEY=<secret-key>

abaca-dev enroll \
  --target-name demo \
  --endpoint http://<controller>:9000 \
  --bucket abaca-demo-plain \
  --wait

Booting a worker VM (R2)

Once R1 enrollment is working, boot a worker VM in the abaca service project to exercise the live data path:

export OS_CLOUD=abaca-service

abaca-dev worker-boot \
  --network dr-net \
  --key-name abaca-worker \
  --transport-url rabbit://abaca:<PW>@<controller-ip>:5672/abaca

The --transport-url is rendered into /etc/abaca/abaca.conf on the VM via cloud-init so that abaca-worker-agent can register with the conductor on first boot. Read the correct value from the controller:

grep ^transport_url /etc/kolla/abaca-conductor/abaca.conf

Worker VMs must not depend on the public internet at boot. worker-boot refuses any image that lacks the abaca_worker_image=1 Glance property. Build a compliant image with:

bash deploy/kolla/build-worker-image.sh

Example 1: Run the full test suite with tox

This is the canonical gate command. It runs lint and all unit tests in isolated virtual environments, matching CI exactly.

python -m tox

Expected output (abbreviated):

lint: commands[0]> ruff check .
lint: commands[1]> black --check .
py3: commands[0]> python -m pytest abaca/tests python-abacaclient abaca-dashboard abaca-tempest-plugin tests
...
============================== N passed in X.XXs ==============================
  lint: OK
  py3: OK

Example 2: Run only the functional smoke test

Use this after modifying the conductor state machine or worker to verify the full backup/restore round-trip still works with fakes.

pytest abaca/tests/functional/test_smoke.py -v

Expected output (abbreviated):

abaca/tests/functional/test_smoke.py::test_backup_roundtrip PASSED
abaca/tests/functional/test_smoke.py::test_restore_roundtrip PASSED
============================== 2 passed in X.XXs ==============================

Example 3: Start the dev stack and verify the API is up

cd deploy
docker compose -f docker-compose.dev.yml up --build -d

# Wait for services to be healthy, then probe the root endpoint
curl -s http://localhost:9797/v1/

Expected output (shape — exact fields depend on current API version):

{"version": "v1", "status": "active"}

Example 4: Run a backup end-to-end against the dev stack

This drives the full target → policy → backup flow against the running Compose stack.

# Register a backup target pointing at the local MinIO instance
TARGET=$(curl -s -XPOST http://localhost:9797/v1/targets \
  -H 'content-type: application/json' \
  -d '{
    "name": "local-minio",
    "endpoint": "http://minio:9000",
    "bucket": "abaca-demo-plain",
    "barbican_secret_refs": ["https://barbican/secrets/demo"],
    "trust_id": "trust-local-1"
  }' | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')

echo "Target ID: $TARGET"

# Create a nightly policy
POLICY=$(curl -s -XPOST http://localhost:9797/v1/policies \
  -H 'content-type: application/json' \
  -d "{
    \"name\": \"nightly\",
    \"share_id\": \"share-uuid-0001\",
    \"target_id\": \"$TARGET\",
    \"schedule\": \"0 2 * * *\",
    \"retention\": {\"daily\": 7}
  }" | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')

echo "Policy ID: $POLICY"

# Request an on-demand backup
JOB=$(curl -s -XPOST http://localhost:9797/v1/backups \
  -H 'content-type: application/json' \
  -d "{\"share_id\": \"share-uuid-0001\", \"target_id\": \"$TARGET\"}" \
  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get("job_id",""))')

echo "Job ID: $JOB"

# Poll the job until terminal
curl -s http://localhost:9797/v1/jobs/$JOB

Expected terminal job response shape:

{
  "id": "<job-uuid>",
  "state": "available",
  "error_category": null
}

If the job fails, state will be "error" and error_category will be either "tenant_action_required" or "operator_action_required".

Example 5: Regenerate the OpenAPI spec after an API change

python -m abaca.api.openapi
# Output: writes docs/api-ref/openapi.yaml

Verify the spec is valid:

python -m pytest abaca/tests -k openapi -v

Example 6: Check and auto-fix code formatting

# Check only (what CI runs)
ruff check . && black --check .

# Apply fixes in place
python -m tox -e format

Example 7: Run the live-cluster tempest tests (real cluster)

This requires a real Abacá deployment reachable from your runner and a populated tempest.conf.

TEMPEST_CONFIG_DIR=/etc/tempest tox -e tempest

The plugin discovers the Abacá API through the Keystone service catalog (share-protection type). Tests are skipped automatically if [share_protection] abaca_enabled=False in tempest.conf.

Issue: tox fails with version drift across packages

Symptom:

AssertionError: version drift across packages: {'pyproject.toml': '0.1.0', 'python-abacaclient/pyproject.toml': '0.2.0', ...}

Cause: The monorepo enforces a single shared version across all four pyproject.toml files and abaca.__version__. One or more files has drifted.

Fix: Update every pyproject.toml and the __version__ string in abaca/__init__.py to the same value. The enforcing test is tests/test_monorepo.py::test_single_shared_version_across_all_packages.

Issue: abaca-api or conductor fails to start with a database connection error

Symptom:

OperationalError: (pymysql.err.OperationalError) Can't connect to MySQL server on 'mariadb'

or

No module named 'abaca.db'

Cause: Either MariaDB is not yet healthy when the API starts (Compose timing), or ABACA_DATABASE_CONNECTION is not set.

Fix: In the Compose stack, the api and conductor services declare depends_on: mariadb: condition: service_healthy — if MariaDB takes longer than expected, restart the stack. For local non-Compose use, set the variable explicitly:

export ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite"
abaca-manage db_sync

Issue: sample-config-check tox env fails with a diff

Symptom:

--- etc/abaca.conf.sample
+++ /tmp/abaca.conf.sample.fresh
@@ ...

Cause: An oslo.config option was added or changed in abaca/common/config.py but etc/abaca.conf.sample was not regenerated.

Fix:

tox -e sample-config
git add etc/abaca.conf.sample

Issue: OpenAPI spec test fails

Symptom:

AssertionError: openapi.yaml on disk does not match generated spec

Cause: abaca/api/openapi.py was modified but docs/api-ref/openapi.yaml was not regenerated.

Fix:

python -m abaca.api.openapi
git add docs/api-ref/openapi.yaml

Issue: abaca-dev worker-boot refuses to launch the image

Symptom:

error: image <name> does not have property abaca_worker_image=1

Cause: The Glance image was not built with the required property. Worker VMs must be pre-baked with Kopia and nfs-common — airgap deployments cannot reach public repositories at boot time.

Fix: Build a compliant worker image and upload it to Glance:

bash deploy/kolla/build-worker-image.sh

Then retry worker-boot pointing at the newly uploaded image.

Issue: abaca-dev worker-boot fails with missing transport URL

Symptom:

error: --transport-url not set (also missing $ABACA_TRANSPORT_URL)

Cause: The worker VM needs an oslo.messaging broker URL to register with the conductor. Without it, abaca-worker-agent cannot start.

Fix: Read the correct URL from the conductor's config on the controller, then pass it:

grep ^transport_url /etc/kolla/abaca-conductor/abaca.conf
# Then:
abaca-dev worker-boot \
  --network dr-net \
  --key-name abaca-worker \
  --transport-url rabbit://abaca:<PW>@<controller-ip>:5672/abaca

Alternatively, export ABACA_TRANSPORT_URL before running worker-boot.

Issue: abaca-dev enroll prints a deprecation warning

Symptom:

warning: 'abaca-dev enroll' is deprecated and will be removed in a future release.

Cause: abaca-dev enroll has been superseded by the OSC plugin.

Fix: Switch to the OpenStack CLI command:

openstack share protection target enroll <name> \
  --endpoint <url> \
  --bucket <bucket> \
  --wait

Issue: A job gets stuck in a non-terminal state

Symptom: Polling GET /v1/jobs/<id> keeps returning a state like queued or transferring indefinitely.

Cause: The conductor's reconciliation loop handles stuck jobs, but if the conductor itself is not running or the broker connection is lost, jobs can remain in non-terminal states.

Fix:

1. Verify the conductor is running: check its logs for RPC connectivity errors.
2. Verify the broker URL is correct (ABACA_TRANSPORT_URL).
3. If the job remains stuck after the conductor restarts, the reconciliation loop (abaca/conductor/reconciliation.py) will sweep orphaned queued jobs on its next tick. Check the conductor logs for _sweep_orphan_queued output.
4. If error_category on the failed job is tenant_action_required, the fix is in your configuration (for example, a bad S3 bucket or missing Barbican secret). If it is operator_action_required, the infrastructure or service needs attention.

Issue: test_client_and_dashboard_do_not_import_server_package fails

Symptom:

AssertionError: client/dashboard must not import the server 'abaca' package:
python-abacaclient/abacaclient/foo.py:5: import abaca.common

Cause: Code in python-abacaclient/ or abaca-dashboard/ has imported from the abaca server package. These packages must communicate with the server only over HTTP — they must not take a direct Python dependency on the server.

Fix: Remove the import and replace it with an HTTP call through the SDK, or move shared constants into a module inside abacaclient or abaca_dashboard respectively.