--- title: Development Guide product: trilio-share-protection doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/development-guide --- # Development Guide _Setting up a local development environment and running tests_ ## Overview This page walks you through setting up a local Abacá development environment, running the full test suite, and working with the dev stack end-to-end. Because Abacá runs and tests entirely locally with mocks — no real OpenStack, message bus, or Kopia binary required — you can iterate quickly against in-memory fakes before touching a real cluster. Understanding the local setup also gives you the mental model you need when you are ready to deploy against a real Kolla-Ansible or RHOSO environment. ## Prerequisites Before you begin, make sure you have the following: - **Python 3.11 or 3.12** — 3.12 is the primary target; 3.11 is the supported floor. Both are tested in CI. - **pip** and the ability to create virtual environments (`python -m venv`) - **tox** — used for the full gate (lint + all test environments) - **Docker or Podman** — required only for the Docker Compose dev stack; not needed for the local no-container path - **Git** — to clone the repository - **ruff 0.6.9** and **black 25.9.0** — the linting toolchain (installed automatically by tox) - **MariaDB 10.x or SQLite** — SQLite is the default for local dev (`sqlite://` in memory); MariaDB is used in the Compose stack and is required for production - **RabbitMQ** — required by the Compose stack; not needed for unit tests > **Note:** For live-cluster work (R1 enrollment or R2 data-path probing), you additionally need an OpenStack environment (RHOSO 18 or Kolla-Ansible ≥ 2023.1 / Antelope), access to a Manila share, a Barbican instance, and an S3-compatible bucket (AWS S3, MinIO, Ceph RGW, or Wasabi). ## Installation Follow these steps to get a working local environment. ### 1. Clone the repository ```bash git clone https://github.com/MuralidharB/abaca.git cd abaca ``` ### 2. Create and activate a virtual environment ```bash python -m venv .venv source .venv/bin/activate ``` ### 3. Install all four packages in editable mode The monorepo contains four independently installable packages. Install them together so the test suite can import all of them: ```bash pip install -e ".[test]" -e ./python-abacaclient -e ./abaca-dashboard ``` This installs: - `abaca` — the control plane (API, conductor, worker agent, strategies, DB) - `python-abacaclient` — the OpenStack CLI plugin and HTTP SDK - `abaca-dashboard` — the Horizon plugin skeleton The `abaca-tempest-plugin` package is wired into the test suite automatically; it does not need a separate install step for local unit testing. ### 4. (Optional) Install tox for the full gate ```bash pip install tox ``` ### 5. Run the full gate ```bash python -m tox ``` This runs the `lint` and `py3` environments in isolation, covering lint (ruff + black) and the complete unit test suite across all four packages. ### 6. (Optional) Start the Docker Compose dev stack If you want to run the full API and conductor services locally with a real message bus and database: ```bash cd deploy docker compose -f docker-compose.dev.yml up --build ``` The stack starts: - **MariaDB** on port 3306 - **RabbitMQ** on ports 5672 / 15672 - **MinIO** on ports 9000 (API) / 9001 (console) - **abaca-api** on port 9797 (in `--noauth` dev mode) - **abaca-conductor** connected to the same broker and database Two MinIO buckets are provisioned automatically: - `abaca-demo-plain` — a standard bucket with no object lock - `abaca-demo-locked` — versioning enabled, object lock on, 1-day compliance retention > **Important:** The `abaca-api` console script starts a development server (Flask dev server / wsgiref). It is not suitable for production. Production deployments use gunicorn: `gunicorn --bind 0.0.0.0:9797 -w 4 abaca.api.wsgi:get_application()` ## Configuration ### Environment variables Abacá uses `oslo.config` for configuration. In development and in the Docker Compose stack, oslo.config options are set via environment variables. Each variable maps directly to a config option. | Environment variable | Default (dev) | Effect | |---|---|---| | `ABACA_DATABASE_CONNECTION` | `sqlite://` (in-memory, unit tests) | SQLAlchemy database URL. Use `sqlite:////tmp/abaca.sqlite` for a persistent local file, or `mysql+pymysql://root:abaca@localhost/abaca` for MariaDB. | | `ABACA_TRANSPORT_URL` | `rabbit://guest:guest@rabbitmq:5672/` (Compose) | oslo.messaging broker URL. Required by `abaca-conductor` and `abaca-worker-agent`. | | `ABACA_KOPIA_EXECUTOR` | `ephemeral_container` (Compose), `worker_rpc` (production) | How the conductor runs Kopia. `worker_rpc` dispatches to a worker VM via RPC (production path). `ephemeral_container` launches a Docker container locally (Compose only). | | `ABACA_KOPIA_IMAGE` | `abaca-kopia:0.23.1` | The container image used by the `ephemeral_container` executor. Must contain Kopia ≥ 0.23.1. | | `ABACA_KOPIA_CONTAINER_ENGINE` | `docker` | `docker` or `podman`. | | `ABACA_KOPIA_CONTAINER_NETWORK` | `abaca-dev` | Docker network the Kopia container joins. Must match the Compose network name so containers can reach MinIO. | | `ABACA_ENROLLMENT_DISPATCH` | `rpc` | How enrollment jobs are dispatched. `rpc` sends the job over the message bus to the conductor. | | `ABACA_SERVICE_USER_NAME` | `abaca` | The OpenStack service user that the conductor acts as. | | `ABACA_WORKER_PROJECT_NAME` | `abaca` | The OpenStack project in which worker VMs are booted. | | `ABACA_DEV_LOG_LEVEL` | `INFO` | Log level for the `abaca-dev` CLI. | ### Generating and checking the sample config file The authoritative list of all configuration options is in `abaca/common/config.py`. A human-readable sample is generated from that source: ```bash # Regenerate etc/abaca.conf.sample tox -e sample-config # Verify the checked-in sample is up-to-date (CI uses this) tox -e sample-config-check ``` When you add or change an `oslo.config` option, always regenerate the sample and commit the updated file. ### Generating the OpenAPI spec The OpenAPI 3 specification is generated from `abaca/api/openapi.py`. After any change to `build_spec` in that module, regenerate the spec: ```bash python -m abaca.api.openapi # Writes docs/api-ref/openapi.yaml ``` A test enforces that the checked-in spec matches the generated output, so failing to regenerate will cause a CI failure. ### Database setup (local persistent) For local development with a persistent SQLite database: ```bash ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite" abaca-manage db_sync ``` For the Compose stack, the `api` service runs migrations automatically on startup against MariaDB. ### FIPS mode FIPS mode is a supported, validated production configuration. When Kopia creates a repository, Abacá always applies the FIPS crypto profile: - Encryption: `AES256-GCM-HMAC-SHA256` - Block hash: `HMAC-SHA256-128` - Key derivation: `pbkdf2` (when probed as supported) Secrets travel only via environment variables (`KOPIA_PASSWORD`, `AWS_*`) — never via command-line arguments, disk, or logs. You do not need to configure this manually; it is enforced by the worker internals. ## Usage ### Fast local iteration (no containers) For the tightest feedback loop during development, run pytest directly against your active venv after installing all packages: ```bash python -m pytest abaca/tests python-abacaclient abaca-dashboard abaca-tempest-plugin tests -q ``` This runs the entire test suite across all four packages. It uses an in-memory SQLite database, in-memory OpenStack client fakes (`Clients.fakes()`), and `FakeKopia` as the Kopia test double — no real OpenStack, broker, or Kopia binary is needed. ### Running lint ```bash ruff check . && black --check . ``` To auto-fix formatting: ```bash python -m tox -e format ``` Lint settings: line length 88, ruff rules `E, F, W, I, UP, B` (with `E501` and `UP042` intentionally ignored). ### Running the functional smoke test The functional smoke test in `abaca/tests/functional/test_smoke.py` is the end-to-end reference. It drives the full flow — target creation → policy creation → backup → restore — through the complete job state machine to `available`, using in-memory fakes and `FakeKopia`: ```bash pytest abaca/tests/functional/test_smoke.py -q ``` Run this test after touching any conductor or worker code. ### Driving the API manually (Compose dev stack) With the Compose stack running, the API is available at `http://localhost:9797` in `--noauth` mode (no Keystone token required). You can drive a full backup flow with curl: ```bash # 1. Register a backup target curl -s -XPOST localhost:9797/v1/targets \ -H 'content-type: application/json' \ -d '{ "name": "primary", "endpoint": "https://s3.example.com", "bucket": "tenant-abaca", "barbican_secret_refs": ["https://barbican/secrets/pw"], "trust_id": "trust-1" }' # 2. Create a protection policy curl -s -XPOST localhost:9797/v1/policies \ -H 'content-type: application/json' \ -d '{ "name": "nightly", "share_id": "share-uuid-0001", "target_id": "", "schedule": "0 2 * * *", "retention": {"daily": 7} }' # 3. Request an on-demand backup curl -s -XPOST localhost:9797/v1/backups \ -H 'content-type: application/json' \ -d '{ "share_id": "share-uuid-0001", "target_id": "" }' ``` ### Using the OpenStack CLI plugin With `python-abacaclient` installed, you can manage share protection resources from the OpenStack CLI: ```bash openstack share protection target enroll \ --endpoint \ --bucket \ --region us-east-1 \ --wait ``` All `openstack share protection …` subcommands are provided by the `python-abacaclient` OSC plugin. S3 credentials must come from the environment (`ABACA_TARGET_ACCESS_KEY` / `ABACA_TARGET_SECRET_KEY` or `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`) — never from command-line flags. ### Using abaca-dev for live enrollment (R1) `abaca-dev enroll` is deprecated in favour of the OSC plugin above, but is still available for one release cycle: ```bash export OS_CLOUD=abaca-tenant export ABACA_TARGET_ACCESS_KEY= export ABACA_TARGET_SECRET_KEY= abaca-dev enroll \ --target-name demo \ --endpoint http://:9000 \ --bucket abaca-demo-plain \ --wait ``` ### Booting a worker VM (R2) Once R1 enrollment is working, boot a worker VM in the `abaca` service project to exercise the live data path: ```bash export OS_CLOUD=abaca-service abaca-dev worker-boot \ --network dr-net \ --key-name abaca-worker \ --transport-url rabbit://abaca:@:5672/abaca ``` The `--transport-url` is rendered into `/etc/abaca/abaca.conf` on the VM via cloud-init so that `abaca-worker-agent` can register with the conductor on first boot. Read the correct value from the controller: ```bash grep ^transport_url /etc/kolla/abaca-conductor/abaca.conf ``` Worker VMs **must not** depend on the public internet at boot. `worker-boot` refuses any image that lacks the `abaca_worker_image=1` Glance property. Build a compliant image with: ```bash bash deploy/kolla/build-worker-image.sh ``` ## Examples ### Example 1: Run the full test suite with tox This is the canonical gate command. It runs lint and all unit tests in isolated virtual environments, matching CI exactly. ```bash python -m tox ``` Expected output (abbreviated): ``` lint: commands[0]> ruff check . lint: commands[1]> black --check . py3: commands[0]> python -m pytest abaca/tests python-abacaclient abaca-dashboard abaca-tempest-plugin tests ... ============================== N passed in X.XXs ============================== lint: OK py3: OK ``` --- ### Example 2: Run only the functional smoke test Use this after modifying the conductor state machine or worker to verify the full backup/restore round-trip still works with fakes. ```bash pytest abaca/tests/functional/test_smoke.py -v ``` Expected output (abbreviated): ``` abaca/tests/functional/test_smoke.py::test_backup_roundtrip PASSED abaca/tests/functional/test_smoke.py::test_restore_roundtrip PASSED ============================== 2 passed in X.XXs ============================== ``` --- ### Example 3: Start the dev stack and verify the API is up ```bash cd deploy docker compose -f docker-compose.dev.yml up --build -d # Wait for services to be healthy, then probe the root endpoint curl -s http://localhost:9797/v1/ ``` Expected output (shape — exact fields depend on current API version): ```json {"version": "v1", "status": "active"} ``` --- ### Example 4: Run a backup end-to-end against the dev stack This drives the full target → policy → backup flow against the running Compose stack. ```bash # Register a backup target pointing at the local MinIO instance TARGET=$(curl -s -XPOST http://localhost:9797/v1/targets \ -H 'content-type: application/json' \ -d '{ "name": "local-minio", "endpoint": "http://minio:9000", "bucket": "abaca-demo-plain", "barbican_secret_refs": ["https://barbican/secrets/demo"], "trust_id": "trust-local-1" }' | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') echo "Target ID: $TARGET" # Create a nightly policy POLICY=$(curl -s -XPOST http://localhost:9797/v1/policies \ -H 'content-type: application/json' \ -d "{ \"name\": \"nightly\", \"share_id\": \"share-uuid-0001\", \"target_id\": \"$TARGET\", \"schedule\": \"0 2 * * *\", \"retention\": {\"daily\": 7} }" | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') echo "Policy ID: $POLICY" # Request an on-demand backup JOB=$(curl -s -XPOST http://localhost:9797/v1/backups \ -H 'content-type: application/json' \ -d "{\"share_id\": \"share-uuid-0001\", \"target_id\": \"$TARGET\"}" \ | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d.get("job_id",""))') echo "Job ID: $JOB" # Poll the job until terminal curl -s http://localhost:9797/v1/jobs/$JOB ``` Expected terminal job response shape: ```json { "id": "", "state": "available", "error_category": null } ``` If the job fails, `state` will be `"error"` and `error_category` will be either `"tenant_action_required"` or `"operator_action_required"`. --- ### Example 5: Regenerate the OpenAPI spec after an API change ```bash python -m abaca.api.openapi # Output: writes docs/api-ref/openapi.yaml ``` Verify the spec is valid: ```bash python -m pytest abaca/tests -k openapi -v ``` --- ### Example 6: Check and auto-fix code formatting ```bash # Check only (what CI runs) ruff check . && black --check . # Apply fixes in place python -m tox -e format ``` --- ### Example 7: Run the live-cluster tempest tests (real cluster) This requires a real Abacá deployment reachable from your runner and a populated `tempest.conf`. ```bash TEMPEST_CONFIG_DIR=/etc/tempest tox -e tempest ``` The plugin discovers the Abacá API through the Keystone service catalog (`share-protection` type). Tests are skipped automatically if `[share_protection] abaca_enabled=False` in `tempest.conf`. ## Troubleshooting ### Issue: `tox` fails with `version drift across packages` **Symptom:** ``` AssertionError: version drift across packages: {'pyproject.toml': '0.1.0', 'python-abacaclient/pyproject.toml': '0.2.0', ...} ``` **Cause:** The monorepo enforces a single shared version across all four `pyproject.toml` files and `abaca.__version__`. One or more files has drifted. **Fix:** Update every `pyproject.toml` and the `__version__` string in `abaca/__init__.py` to the same value. The enforcing test is `tests/test_monorepo.py::test_single_shared_version_across_all_packages`. --- ### Issue: `abaca-api` or conductor fails to start with a database connection error **Symptom:** ``` OperationalError: (pymysql.err.OperationalError) Can't connect to MySQL server on 'mariadb' ``` or ``` No module named 'abaca.db' ``` **Cause:** Either MariaDB is not yet healthy when the API starts (Compose timing), or `ABACA_DATABASE_CONNECTION` is not set. **Fix:** In the Compose stack, the `api` and `conductor` services declare `depends_on: mariadb: condition: service_healthy` — if MariaDB takes longer than expected, restart the stack. For local non-Compose use, set the variable explicitly: ```bash export ABACA_DATABASE_CONNECTION="sqlite:////tmp/abaca.sqlite" abaca-manage db_sync ``` --- ### Issue: `sample-config-check` tox env fails with a diff **Symptom:** ``` --- etc/abaca.conf.sample +++ /tmp/abaca.conf.sample.fresh @@ ... ``` **Cause:** An oslo.config option was added or changed in `abaca/common/config.py` but `etc/abaca.conf.sample` was not regenerated. **Fix:** ```bash tox -e sample-config git add etc/abaca.conf.sample ``` --- ### Issue: OpenAPI spec test fails **Symptom:** ``` AssertionError: openapi.yaml on disk does not match generated spec ``` **Cause:** `abaca/api/openapi.py` was modified but `docs/api-ref/openapi.yaml` was not regenerated. **Fix:** ```bash python -m abaca.api.openapi git add docs/api-ref/openapi.yaml ``` --- ### Issue: `abaca-dev worker-boot` refuses to launch the image **Symptom:** ``` error: image does not have property abaca_worker_image=1 ``` **Cause:** The Glance image was not built with the required property. Worker VMs must be pre-baked with Kopia and nfs-common — airgap deployments cannot reach public repositories at boot time. **Fix:** Build a compliant worker image and upload it to Glance: ```bash bash deploy/kolla/build-worker-image.sh ``` Then retry `worker-boot` pointing at the newly uploaded image. --- ### Issue: `abaca-dev worker-boot` fails with missing transport URL **Symptom:** ``` error: --transport-url not set (also missing $ABACA_TRANSPORT_URL) ``` **Cause:** The worker VM needs an oslo.messaging broker URL to register with the conductor. Without it, `abaca-worker-agent` cannot start. **Fix:** Read the correct URL from the conductor's config on the controller, then pass it: ```bash grep ^transport_url /etc/kolla/abaca-conductor/abaca.conf # Then: abaca-dev worker-boot \ --network dr-net \ --key-name abaca-worker \ --transport-url rabbit://abaca:@:5672/abaca ``` Alternatively, export `ABACA_TRANSPORT_URL` before running `worker-boot`. --- ### Issue: `abaca-dev enroll` prints a deprecation warning **Symptom:** ``` warning: 'abaca-dev enroll' is deprecated and will be removed in a future release. ``` **Cause:** `abaca-dev enroll` has been superseded by the OSC plugin. **Fix:** Switch to the OpenStack CLI command: ```bash openstack share protection target enroll \ --endpoint \ --bucket \ --wait ``` --- ### Issue: A job gets stuck in a non-terminal state **Symptom:** Polling `GET /v1/jobs/` keeps returning a state like `queued` or `transferring` indefinitely. **Cause:** The conductor's reconciliation loop handles stuck jobs, but if the conductor itself is not running or the broker connection is lost, jobs can remain in non-terminal states. **Fix:** 1. Verify the conductor is running: check its logs for RPC connectivity errors. 2. Verify the broker URL is correct (`ABACA_TRANSPORT_URL`). 3. If the job remains stuck after the conductor restarts, the reconciliation loop (`abaca/conductor/reconciliation.py`) will sweep orphaned queued jobs on its next tick. Check the conductor logs for `_sweep_orphan_queued` output. 4. If `error_category` on the failed job is `tenant_action_required`, the fix is in your configuration (for example, a bad S3 bucket or missing Barbican secret). If it is `operator_action_required`, the infrastructure or service needs attention. --- ### Issue: `test_client_and_dashboard_do_not_import_server_package` fails **Symptom:** ``` AssertionError: client/dashboard must not import the server 'abaca' package: python-abacaclient/abacaclient/foo.py:5: import abaca.common ``` **Cause:** Code in `python-abacaclient/` or `abaca-dashboard/` has imported from the `abaca` server package. These packages must communicate with the server only over HTTP — they must not take a direct Python dependency on the server. **Fix:** Remove the import and replace it with an HTTP call through the SDK, or move shared constants into a module inside `abacaclient` or `abaca_dashboard` respectively.