---
title: Deployment Guide
product: trilio-share-protection
doc_type: guide
version: master
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/deployment-guide
---

# Deployment Guide

_Steps to deploy abaca to a production or staging environment_

## Overview

This guide walks you through deploying Trilio Share Protection (Abacá) to a production or staging environment. It covers two deployment paths: a Kolla-Ansible-integrated deployment onto an existing OpenStack cluster, and a Kubernetes-native deployment using Kustomize manifests. You will end up with a running `abaca-api` and `abaca-conductor` control plane, a Keystone service catalog entry, and a verified data path through at least one enrolled backup target.

## Prerequisites

Before you begin, ensure the following are in place:

**OpenStack cluster (Kolla-Ansible-deployed):**
- OpenStack ≥ 2023.1 (Antelope) or RHOSO 18, deployed with Kolla-Ansible
- Keystone, Nova, Neutron, Glance, MariaDB, and RabbitMQ — all healthy
- Manila enabled (`enable_manila: yes` in Kolla globals) if you intend to back up Manila shares
- Barbican enabled (`enable_barbican: yes` in Kolla globals) — required for tenant secret storage
- `kolla-venv` present on the controller at `/root/kolla-venv` (Kolla installs this by default)
- ~5 GB free disk on the controller (Docker images + Kolla logs)

**Your workstation:**
- Python 3.11 or 3.12 with `openstackclient` installed (the scripts will bootstrap a local venv at `deploy/cluster1/.venv` if needed)
- Docker (for local wheelhouse builds during the worker image step)
- `libguestfs-tools` and `qemu-utils` if you plan to build the worker VM Glance image (`build-worker-image.sh` installs these on first run)
- An admin OpenRC for the target cluster, sourced in your shell — all deploy scripts require it
- SSH access to the controller as a user with `sudo` (defaults to user `pureuser` / password `pureuser`; override with `ABACA_SSH_USER` and `ABACA_SUDO_PASSWORD`)

**For Kubernetes / Kustomize deployment:**
- OpenShift ≥ 4.12
- `kubectl` or `oc` configured to target your cluster

**S3-compatible object storage** (one of):
- AWS S3
- MinIO
- Ceph RGW
- ODF / Wasabi

**Worker VM requirements:**
- RHEL worker image with Kopia ≥ 0.23.1 baked in
- Host and guest CPUs must expose AES-NI and SHA-NI instruction-set extensions (required for FIPS-mode cryptographic throughput)

## Installation

## Option A — Kolla-Ansible (recommended for OpenStack clusters)

### Step 1 — Source your admin OpenRC

All deploy scripts require an admin OpenRC to be sourced in your shell before running.

```bash
source /path/to/admin-openrc.sh
```

### Step 2 — Run the one-shot installer

From the repository root, run the install script. It executes steps 01–08 in order and is idempotent — safe to re-run.

```bash
bash deploy/kolla/install.sh
```

The installer performs the following steps:

| Step | Script | What it does |
|------|--------|--------------|
| 01 | `01-identity.sh` | Creates the `abaca` Keystone service user (roles: `service`, `admin`) and the `abaca` worker project |
| 02 | `02-catalog.sh` | Registers the `share-protection` service in the Keystone catalog with public, internal, and admin endpoints |
| 03 | `03-database.sh` | Creates the `abaca` MariaDB database and user with a fresh random password |
| 04 | `04-messaging.sh` | Creates the `/abaca` RabbitMQ vhost and user with a fresh random password |
| 05 | `05-build.sh` | Rsyncs source to the controller and builds the `abaca-api` and `abaca-conductor` Docker images there |
| 06 | `06-config.sh` | Renders `/etc/kolla/abaca-api/abaca.conf` and `/etc/kolla/abaca-conductor/abaca.conf` |
| 07 | `07-deploy.sh` | Runs `abaca-manage db_sync`, then starts the `abaca_api` and `abaca_conductor` containers |
| 08 | `08-dashboard.sh` | Installs `abaca-dashboard` into the running `horizon` container |

Generated secrets (DB and MQ passwords) are written to `cluster1-abaca-secrets.env` at the repo root (mode 0600, gitignored).

### Step 3 — Verify the control plane is up

SSH to the controller and confirm both containers are running and the API health endpoint responds:

```bash
# On the controller:
docker ps --filter name=abaca_ --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
curl -fsS http://127.0.0.1:9797/healthcheck
```

You should see `abaca_api` and `abaca_conductor` both in `Up` state, and the healthcheck returning a 200 response.

### Step 4 — Run post-install steps to unlock the full data path

**4a. (Optional) Set up the Manila generic driver** if you do not already have Manila shares available:

```bash
bash deploy/kolla/setup-manila-generic.sh
```

**4b. (Optional) Set up host-NAT egress** if your hypervisor filters non-native source MACs (common on VMware ESXi vSwitch). Without this, worker VMs on the tenant network cannot reach your S3 endpoint:

```bash
TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh
```

> **Note:** This plumbs a veth into `br-int` with iptables MASQUERADE. It is not persistent across a controller reboot — re-run after reboot. See the [Troubleshooting](#troubleshooting) section for diagnosis guidance.

**4c. Build the worker VM Glance image.** Abacá boots short-lived worker VMs to perform data movement. Build the image once (or after any Abacá code change):

```bash
# Idempotent — no-op if the image is already in Glance:
bash deploy/kolla/build-worker-image.sh

# Force a rebuild:
bash deploy/kolla/build-worker-image.sh --force
```

This runs `libguestfs` locally on your workstation. The first run downloads ~700 MB (Ubuntu 24.04 cloud image + Kopia tarball) and uploads the result to Glance as `abaca-worker-<KOPIA_VERSION>`.

**4d. Boot at least one worker VM** before tenants attempt to enroll a backup target:

```bash
# Read the transport URL from the controller config:
export ABACA_TRANSPORT_URL=$(ssh <controller> \
    "sudo grep '^transport_url' /etc/kolla/abaca-conductor/abaca.conf" \
    | cut -d' ' -f3)

# Boot a worker:
OS_CLOUD=abaca-service abaca-dev worker-boot \
    --network dr-net --key-name abaca-worker
```

Confirm the worker registered with the conductor:

```bash
openstack --os-cloud abaca-service share protection worker list
```

You should see one row with `state=active` and a `last_heartbeat_at` timestamp less than one minute old.

---

## Option B — Kustomize (Kubernetes / OpenShift)

The `deploy/kustomize/base/` directory provides Kubernetes manifests for the `abaca-api` and `abaca-conductor` deployments.

### Step 1 — Review the base manifests

The base kustomization deploys both services into the `abaca` namespace:

```bash
ls deploy/kustomize/base/
# kustomization.yaml
# namespace.yaml
# api-deployment.yaml
# conductor-deployment.yaml
# api-service.yaml
```

### Step 2 — Create an overlay for your environment

Create an environment-specific overlay that patches image references and injects your configuration as a Secret or ConfigMap. Then apply:

```bash
kubectl apply -k deploy/kustomize/base/
```

> **Note for reviewer:** The base manifests do not include Secret or ConfigMap definitions for `abaca.conf`. You will need to supply these in your overlay, referencing the configuration options described in the [Configuration](#configuration) section. Confirm the overlay shape against `deploy/kustomize/` in the repository.

### Step 3 — Verify the deployments

```bash
kubectl -n abaca get pods
kubectl -n abaca logs deployment/abaca-api
kubectl -n abaca logs deployment/abaca-conductor
```

---

## Option C — Docker Compose (local development only)

> **Warning:** This stack runs `abaca-api` in `--noauth` mode (no Keystone authentication). It is intended for local development only and must not be used in production.

```bash
# From the repo root:
docker compose -f deploy/docker-compose.dev.yml up --build
```

This brings up MariaDB, RabbitMQ, MinIO (with two demo buckets), `abaca-api`, and `abaca-conductor`. After startup, verify health:

```bash
curl http://localhost:9797/healthcheck
```

To skip individual install steps on a re-run (for example, to rebuild only the containers):

```bash
bash deploy/kolla/install.sh \
    --skip-identity --skip-catalog --skip-db --skip-mq \
    --skip-config --skip-dashboard
```

## Configuration

Abacá uses oslo.config-style configuration files. The canonical paths are:

- API: `/etc/kolla/abaca-api/abaca.conf`
- Conductor: `/etc/kolla/abaca-conductor/abaca.conf`

Both files are rendered by `deploy/kolla/06-config.sh` from sourced credentials and templates. The same options are also accepted as environment variables in the Docker Compose dev stack (prefixed `ABACA_`).

---

### `[DEFAULT]`

| Option | Description | Example |
|--------|-------------|--------|
| `debug` | Enable verbose debug logging. Set `false` in production. | `false` |
| `use_stderr` | Log to stderr (picked up by `docker logs`). | `true` |
| `transport_url` | Oslo.messaging broker URL. Points to the `/abaca` RabbitMQ vhost. | `rabbit://abaca:<pw>@<host>:5672//abaca` |

---

### `[database]`

| Option | Description | Example |
|--------|-------------|--------|
| `connection` | SQLAlchemy connection URL for the `abaca` MariaDB database. Generated by `03-database.sh`. | `mysql+pymysql://abaca:<pw>@<vip>/abaca?charset=utf8mb4` |

---

### `[keystone_authtoken]`

Standard Keystone middleware configuration. Key options:

| Option | Description |
|--------|-------------|
| `www_authenticate_uri` | Keystone public endpoint (returned to clients in 401 responses). |
| `auth_url` | Keystone endpoint used by the service itself to validate tokens. |
| `memcached_servers` | Token cache. |
| `auth_type` | Must be `password`. |
| `project_name` | The Keystone project the service account lives in (Kolla convention: `service`). |
| `username` | The Abacá service user (`abaca`). |
| `password` | Set from `cluster1-abaca-service-openrc.sh`. |
| `service_token_roles` | Set to `service`. |
| `service_token_roles_required` | Must be `true`. |

---

### `[api]`

| Option | Default | Description |
|--------|---------|-------------|
| `bind_host` | `0.0.0.0` | Interface the gunicorn process listens on. |
| `bind_port` | `9797` | Port the API listens on. This must match the Keystone catalog endpoint registration. |
| `noauth` | `false` | Disables Keystone authentication. **Never set `true` in production.** Setting `true` while `dispatch=inline` is also not set will be refused at startup. |

---

### `[conductor]`

| Option | Default | Description |
|--------|---------|-------------|
| `scheduler_interval` | `60` | Seconds between policy-triggered backup schedule checks. |
| `reconciliation_interval` | `300` | Seconds between reconciliation loop runs, which detect stuck jobs and orphaned workers. |
| `worker_agent_topic` | `abaca-worker` | RabbitMQ topic the conductor uses to dispatch RPC calls to worker VMs. |

---

### `[enrollment]`

| Option | Valid values | Default | Description |
|--------|-------------|---------|-------------|
| `dispatch` | `rpc`, `inline` | `rpc` | Controls how enrollment jobs are executed. `rpc` dispatches through the conductor via oslo.messaging — the only valid choice for production. `inline` is a unit-test harness only; any process starting with `dispatch=inline` and `api.noauth=False` will be refused at startup. |

---

### `[kopia]`

| Option | Valid values | Production default | Description |
|--------|--------------|--------------------|-------------|
| `executor` | `worker_rpc`, `ephemeral_container` | `worker_rpc` | How Kopia commands are dispatched. `worker_rpc` sends commands to fleet worker VMs via oslo.messaging — the correct choice for any real cluster. `ephemeral_container` runs Kopia locally in a Docker/Podman container; use only with the Docker Compose dev stack. Any process using `ephemeral_container` logs a startup WARNING. |
| `image` | — | `abaca-kopia:<KOPIA_VERSION>` | The Kopia container image tag. Only used by `ephemeral_container`. |
| `container_engine` | `docker`, `podman` | `docker` | Only used by `ephemeral_container`. |
| `container_network` | — | `abaca-dev` (compose) | Docker network the ephemeral Kopia container joins, so it can reach MinIO. Only used by `ephemeral_container`. |

---

### `[abaca]`

| Option | Description |
|--------|-------------|
| `service_user_id` | UUID of the `abaca` Keystone service user. Served on `/v1/service_info` so tenants can create Keystone trusts without needing user-list privileges. Populated by `06-config.sh`. |
| `service_user_name` | Name of the Abacá service user (`abaca`). |
| `worker_project_id` | UUID of the worker VM project. Served on `/v1/service_info`. |
| `worker_project_name` | Name of the worker VM project (`abaca`). |

---

### Environment variables (Docker Compose dev stack)

In `deploy/docker-compose.dev.yml` the same configuration is supplied as environment variables:

| Variable | Maps to |
|----------|---------|
| `ABACA_DATABASE_CONNECTION` | `[database] connection` |
| `ABACA_TRANSPORT_URL` | `[DEFAULT] transport_url` |
| `ABACA_KOPIA_EXECUTOR` | `[kopia] executor` |
| `ABACA_KOPIA_IMAGE` | `[kopia] image` |
| `ABACA_KOPIA_CONTAINER_ENGINE` | `[kopia] container_engine` |
| `ABACA_KOPIA_CONTAINER_NETWORK` | `[kopia] container_network` |
| `ABACA_ENROLLMENT_DISPATCH` | `[enrollment] dispatch` |

## Usage

Once the control plane is deployed and at least one worker VM is registered, the primary workflow is:

1. **Enroll a backup target** — register your S3 bucket with Abacá. This runs preflight checks and initializes a Kopia repository.
2. **Create a protection policy** — bind a Manila share to the target with a schedule and retention settings.
3. **Run or schedule backups** — backups run automatically per the policy schedule, or you can trigger one on demand.
4. **Restore from backup** — recreate share contents onto a new or the original share.

All operations go through the `abaca-api` REST API (Keystone-authenticated) or via the `openstack share protection …` commands from `python-abacaclient`.

---

### Authenticating as a tenant

Set `OS_CLOUD=abaca-tenant` (or source the appropriate OpenRC) before running CLI commands. Abacá uses Keystone token authentication — the same credential flow as any other OpenStack service.

```bash
export OS_CLOUD=abaca-tenant
```

---

### Enrolling a backup target

Enrollment is a one-time setup that verifies your S3 bucket, stores credentials in Barbican, and initializes a Kopia repository. Use `abaca-dev enroll` (for scripted flows) or the REST API directly.

```bash
# Using abaca-dev (recommended for initial setup and testing):
abaca-dev enroll \
  --target-name my-target \
  --bucket abaca-demo-plain \
  --access-key <s3-access-key> \
  --secret-key <s3-secret-key>
```

The API returns `202 Accepted` with a job ID. Poll the job until it reaches `available`:

```bash
openstack share protection job show <job-id>
```

---

### Creating a protection policy

Once a target is enrolled, attach a share to it with a schedule:

```bash
openstack share protection policy create \
  --share <share-id> \
  --target <target-id> \
  --schedule "0 2 * * *" \
  --retain-daily 7 \
  --retain-weekly 4 \
  --retain-monthly 12
```

---

### Triggering an on-demand backup

```bash
openstack share protection backup create --policy <policy-id>
```

This returns a job ID. Poll it to follow progress through the job state machine:
`queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available`

---

### Restoring from a backup

To restore to a new share (the default):

```bash
openstack share protection restore create --backup <backup-id>
```

To restore in-place (back onto the original share) or restore only a sub-path, pass the appropriate flags. See the API reference for full options.

---

### Checking worker fleet health

```bash
# Via CLI:
openstack share protection worker list

# Via API:
curl -sS -H "X-Auth-Token: $OS_TOKEN" \
  http://<controller>:9797/v1/admin/workers
```

A healthy worker shows `state=active` with a `last_heartbeat_at` timestamp within the last minute.

---

### Development quick loop (fast single-file iteration)

If you are iterating on a single source file, use the hot-patch helper to avoid a full rebuild:

```bash
bash deploy/kolla/hot-patch.sh abaca/api/controllers/backups.py api
```

This SCPs the changed file into the running container and restarts it.

## Examples

### Example 1 — Full one-shot install on a Kolla-Ansible cluster

Source your admin OpenRC, then run the installer from the repo root:

```bash
source /root/admin-openrc.sh
bash deploy/kolla/install.sh
```

Expected output (abbreviated):

```
==> keystone: http://192.168.1.10:5000 region=RegionOne
==> project abaca: creating (worker VM tenant)
==> user abaca: creating in project service
==> role service on service -> abaca
==> role admin on service -> abaca
==> role admin on abaca -> abaca
==> wrote cluster1-abaca-service-openrc.sh (mode 0600)
...
==> registering service 'share-protection' (type share-protection)
==> endpoint public: creating http://192.168.1.10:9797
==> endpoint internal: creating http://192.168.1.10:9797
==> endpoint admin: creating http://192.168.1.10:9797
...
==> running abaca-manage db_sync on 192.168.1.10
==> (re)starting container 'abaca_api'
==> (re)starting container 'abaca_conductor'
==> API /healthcheck
{"status": "ok"}
OK.
```

---

### Example 2 — Rebuild and redeploy containers only (skip identity/catalog/DB/MQ)

Use this during iterative development to skip the idempotent-but-slow identity and infrastructure steps:

```bash
bash deploy/kolla/install.sh \
    --skip-identity \
    --skip-catalog \
    --skip-db \
    --skip-mq \
    --skip-config \
    --skip-dashboard
```

This runs only steps 05 (build) and 07 (deploy), giving you a fast container rebuild and restart cycle.

---

### Example 3 — Build and register the worker VM Glance image

```bash
bash deploy/kolla/build-worker-image.sh
```

Expected output (first run, abbreviated):

```
==> downloading Ubuntu 24.04 cloud image (~700 MB)...
==> downloading kopia 0.23.1 tarball...
==> virt-customize: installing nfs-common, kopia, abaca wheelhouse...
==> uploading to Glance as 'abaca-worker-0.23.1'
==> setting image properties: abaca_worker_image=1, abaca_worker_agent_baked=1
OK.
```

On subsequent runs with the same Kopia version and agent code, the script is a no-op unless you pass `--force`.

---

### Example 4 — Boot a worker VM and verify registration

```bash
# Read the transport URL from the running conductor config:
export ABACA_TRANSPORT_URL=$(ssh pureuser@192.168.1.10 \
    "sudo grep '^transport_url' /etc/kolla/abaca-conductor/abaca.conf" \
    | cut -d' ' -f3)

# Boot a worker into the tenant network:
OS_CLOUD=abaca-service abaca-dev worker-boot \
    --network dr-net \
    --key-name abaca-worker

# Confirm it registered:
openstack --os-cloud abaca-service share protection worker list
```

Expected output:

```
+--------------------------------------+--------+---------------------------+
| id                                   | state  | last_heartbeat_at         |
+--------------------------------------+--------+---------------------------+
| a1b2c3d4-e5f6-7890-abcd-ef1234567890 | active | 2024-07-15T10:42:03Z      |
+--------------------------------------+--------+---------------------------+
```

---

### Example 5 — Enroll a backup target and poll the job to completion

```bash
export OS_CLOUD=abaca-tenant

abaca-dev enroll \
  --target-name demo \
  --bucket abaca-demo-plain \
  --access-key minioadmin \
  --secret-key minioadmin
```

The command prints the job ID. Poll it:

```bash
openstack share protection job show <job-id>
```

Watch the `state` field advance through:
`queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available`

Once `available`, inspect the target's capabilities:

```bash
openstack share protection target show <target-id>
```

Expected capabilities (for an object-lock-enabled bucket):

```
addressing_mode: virtual_hosted
object_lock: true
kopia_encryption: AES256-GCM-HMAC-SHA256
kopia_block_hash: HMAC-SHA256-128
kopia_key_derivation: pbkdf2
```

---

### Example 6 — Deploy the dev stack locally with Docker Compose

```bash
docker compose -f deploy/docker-compose.dev.yml up --build
```

Once up, verify the API:

```bash
curl http://localhost:9797/healthcheck
```

Expected response:

```json
{"status": "ok"}
```

MinIO is available at `http://localhost:9000` (credentials: `abaca` / `abacaabaca`). The two pre-provisioned buckets are `abaca-demo-plain` and `abaca-demo-locked`.

---

### Example 7 — Install the abaca-dashboard Horizon plugin

```bash
bash deploy/kolla/08-dashboard.sh
```

After completion, browse to `http://<controller>/` and log in. The Share Protection panels appear under:
- **Project → Share Protection** (targets, policies, backups, restores)
- **Admin → Share → Share Protection** (fleet and coverage overview)

> **Note:** This install is ephemeral. A `kolla-ansible reconfigure horizon` or horizon image rebuild will remove it. Re-run `08-dashboard.sh` to reinstate.

## Troubleshooting

Use the following format to diagnose and resolve common failures.

---

### API container starts but `/healthcheck` returns 502 or 500

**Symptom:** `curl http://127.0.0.1:9797/healthcheck` returns HTTP 500 or 502.

**Likely causes:**
- Database migration did not complete (step 07 runs `abaca-manage db_sync` first).
- RabbitMQ vhost `/abaca` is missing (step 04 must complete first).
- Config file syntax error in `/etc/kolla/abaca-api/abaca.conf`.

**Fix:**
```bash
# Inspect API logs on the controller:
docker logs abaca_api
docker logs abaca_conductor

# Re-run the deploy step to re-apply db_sync and restart:
bash deploy/kolla/install.sh \
    --skip-identity --skip-catalog --skip-db --skip-mq \
    --skip-config --skip-dashboard
```

---

### `error: openstack CLI not on PATH and no venv at ...`

**Symptom:** Install scripts exit immediately with this message.

**Likely cause:** The workstation-side Python virtual environment has not been bootstrapped.

**Fix:**
```bash
bash deploy/cluster1/bootstrap.sh
```
This installs `openstackclient`, `python-manilaclient`, and related packages into a local venv at `deploy/cluster1/.venv`.

---

### `sudo: a password is required` on the controller

**Symptom:** Scripts that SSH to the controller and attempt `sudo` operations fail with this error.

**Likely cause:** The default SSH user (`pureuser`) or sudo password (`pureuser`) does not match your cluster's convention.

**Fix:** Override the defaults:
```bash
ABACA_SSH_USER=<your-user> ABACA_SUDO_PASSWORD=<your-password> bash deploy/kolla/install.sh
```

---

### Config change not picked up after editing `abaca.conf`

**Symptom:** A change to `/etc/kolla/abaca-api/abaca.conf` or `/etc/kolla/abaca-conductor/abaca.conf` has no visible effect.

**Likely cause:** The config file is bind-mounted read-only into the running container. Editing it in place on the host works, but the container process does not watch for file changes.

**Fix:**
```bash
# On the controller:
docker restart abaca_api abaca_conductor
```

---

### Horizon panels missing after running `08-dashboard.sh`

**Symptom:** The Share Protection panels do not appear in Horizon after the dashboard install script completes, or they disappeared after a Kolla operation.

**Likely cause:** `kolla-ansible reconfigure horizon` or a horizon image rebuild replaced the container, removing the ephemeral plugin install.

**Fix:** Re-run the dashboard install script:
```bash
bash deploy/kolla/08-dashboard.sh
```
> **Note:** A durable deployment path using a custom `abaca-horizon` image is on the roadmap. Until it lands, re-running this script after any Kolla horizon reconfiguration is the correct interim procedure.

---

### Worker boot fails with `abaca_worker_agent_baked=1` gate

**Symptom:** `abaca-dev worker-boot` refuses to boot the worker, citing a missing Glance image property.

**Likely cause:** The Glance image was built before the `abaca-worker-agent` was baked in (a pre-R2 kopia-only image). The `abaca_worker_agent_baked=1` property is absent.

**Fix:** Force a rebuild of the worker image:
```bash
bash deploy/kolla/build-worker-image.sh --force
```

---

### Worker VM boots but never registers with the conductor

**Symptom:** `openstack share protection worker list` shows no workers, or the worker row never reaches `state=active`, even after the VM finishes booting.

**Likely cause (most common):** The transport URL in `/etc/abaca/abaca.conf` on the worker VM does not match the one in `/etc/kolla/abaca-conductor/abaca.conf` on the controller.

**Likely cause (secondary):** The worker VM cannot reach the controller's RabbitMQ port (5672) through the tenant network egress path.

**Fix:**
```bash
# On the worker VM, check the agent service:
journalctl -u abaca-worker-agent

# Verify the transport URL on the worker matches the conductor's:
cat /etc/abaca/abaca.conf | grep transport_url
ssh <controller> "sudo grep '^transport_url' /etc/kolla/abaca-conductor/abaca.conf"

# If the worker can't reach RabbitMQ, check the egress path:
# Either set up host-NAT egress or verify the Neutron router is working.
TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh
```

---

### Enrollment job fails with `error_category: tenant_action_required`

**Symptom:** A target enrollment job reaches `error` state with `error_category: tenant_action_required`.

**Likely causes and fixes:**

| Cause | Fix |
|-------|-----|
| The S3 bucket has lifecycle expiration rules configured | Remove the expiration lifecycle rules from the bucket. Object lock requires versioning and lifecycle rules that delete versions would corrupt the Kopia repository. |
| The bucket contains non-Kopia objects (mixed-use bucket) | Use a dedicated bucket for Abacá backups. |
| The Keystone trust ID is missing or invalid | Re-run enrollment; `abaca-dev enroll` creates the trust automatically. |
| S3 credentials cannot be fetched via the trust | Verify the Barbican secret HREFs are correct and the trust delegation includes Barbican access. |

---

### Enrollment job fails with `error_category: operator_action_required`

**Symptom:** A target enrollment job reaches `error` state with `error_category: operator_action_required`.

**Likely causes:**
- The Kopia repository create command timed out (the kopia subprocess was killed after `[kopia] subprocess_timeout` seconds).
- The conductor cannot reach the S3 endpoint from the worker network.
- Barbican is unavailable or returning errors.

**Fix:** Check conductor logs for the specific error, verify network reachability from worker VMs to the S3 endpoint, and confirm Barbican is healthy:
```bash
docker logs abaca_conductor
curl -fsS http://<barbican-endpoint>:9311/
```

---

### `fips_kdf_gap: true` on an enrolled target

**Symptom:** After successful enrollment, `openstack share protection target show <id>` reports `fips_kdf_gap: true` in the capabilities.

**Likely cause:** The Kopia binary in the worker image does not support `--key-derivation-algorithm` (Kopia < 0.14 or a stripped build). The repository was created with scrypt instead of pbkdf2, which is not FIPS-140 compliant for key derivation.

**Fix:** Update the Kopia version pin in `build/kopia.env`, rebuild the worker image, and re-enroll the target so the new repository uses pbkdf2:
```bash
bash deploy/kolla/build-worker-image.sh --force
# Then re-enroll the target via abaca-dev enroll or the API.
```

---

### Host-NAT egress not working after controller reboot

**Symptom:** Worker VMs cannot reach the S3 endpoint or external hosts after the controller was rebooted.

**Likely cause:** The host-NAT egress setup (veth, `ip_forward` sysctl, and iptables rules) is runtime state and does not survive a reboot.

**Fix:** Re-run the script after every controller reboot:
```bash
TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh
```
Existing worker VMs will not automatically pick up the new gateway IP from their DHCP lease. Either run `dhclient -r && dhclient` inside each affected VM, or reboot the worker VMs.

---

### Teardown

To fully remove Abacá from a cluster (containers, config, DB, MQ, Keystone catalog, and identity):

```bash
bash deploy/kolla/99-teardown.sh
```

This is idempotent and safe to run against a partially-installed cluster.