Deployment Guide

This guide walks you through deploying Trilio Share Protection (Abacá) to a production or staging environment. It covers two deployment paths: a Kolla-Ansible-integrated deployment onto an existing OpenStack cluster, and a Kubernetes-native deployment using Kustomize manifests. You will end up with a running abaca-api and abaca-conductor control plane, a Keystone service catalog entry, and a verified data path through at least one enrolled backup target.

Before you begin, ensure the following are in place:

OpenStack cluster (Kolla-Ansible-deployed):

• OpenStack ≥ 2023.1 (Antelope) or RHOSO 18, deployed with Kolla-Ansible
• Keystone, Nova, Neutron, Glance, MariaDB, and RabbitMQ — all healthy
• Manila enabled (enable_manila: yes in Kolla globals) if you intend to back up Manila shares
• Barbican enabled (enable_barbican: yes in Kolla globals) — required for tenant secret storage
• kolla-venv present on the controller at /root/kolla-venv (Kolla installs this by default)
• ~5 GB free disk on the controller (Docker images + Kolla logs)

Your workstation:

• Python 3.11 or 3.12 with openstackclient installed (the scripts will bootstrap a local venv at deploy/cluster1/.venv if needed)
• Docker (for local wheelhouse builds during the worker image step)
• libguestfs-tools and qemu-utils if you plan to build the worker VM Glance image (build-worker-image.sh installs these on first run)
• An admin OpenRC for the target cluster, sourced in your shell — all deploy scripts require it
• SSH access to the controller as a user with sudo (defaults to user pureuser / password pureuser; override with ABACA_SSH_USER and ABACA_SUDO_PASSWORD)

For Kubernetes / Kustomize deployment:

• OpenShift ≥ 4.12
• kubectl or oc configured to target your cluster

S3-compatible object storage (one of):

• AWS S3
• MinIO
• Ceph RGW
• ODF / Wasabi

Worker VM requirements:

• RHEL worker image with Kopia ≥ 0.23.1 baked in
• Host and guest CPUs must expose AES-NI and SHA-NI instruction-set extensions (required for FIPS-mode cryptographic throughput)

Option A — Kolla-Ansible (recommended for OpenStack clusters)

Step 1 — Source your admin OpenRC

All deploy scripts require an admin OpenRC to be sourced in your shell before running.

source /path/to/admin-openrc.sh

Step 2 — Run the one-shot installer

From the repository root, run the install script. It executes steps 01–08 in order and is idempotent — safe to re-run.

bash deploy/kolla/install.sh

The installer performs the following steps:

Generated secrets (DB and MQ passwords) are written to cluster1-abaca-secrets.env at the repo root (mode 0600, gitignored).

Step 3 — Verify the control plane is up

SSH to the controller and confirm both containers are running and the API health endpoint responds:

# On the controller:
docker ps --filter name=abaca_ --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
curl -fsS http://127.0.0.1:9797/healthcheck

You should see abaca_api and abaca_conductor both in Up state, and the healthcheck returning a 200 response.

Step 4 — Run post-install steps to unlock the full data path

4a. (Optional) Set up the Manila generic driver if you do not already have Manila shares available:

bash deploy/kolla/setup-manila-generic.sh

4b. (Optional) Set up host-NAT egress if your hypervisor filters non-native source MACs (common on VMware ESXi vSwitch). Without this, worker VMs on the tenant network cannot reach your S3 endpoint:

TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh

Note: This plumbs a veth into br-int with iptables MASQUERADE. It is not persistent across a controller reboot — re-run after reboot. See the Troubleshooting section for diagnosis guidance.

4c. Build the worker VM Glance image. Abacá boots short-lived worker VMs to perform data movement. Build the image once (or after any Abacá code change):

# Idempotent — no-op if the image is already in Glance:
bash deploy/kolla/build-worker-image.sh

# Force a rebuild:
bash deploy/kolla/build-worker-image.sh --force

This runs libguestfs locally on your workstation. The first run downloads ~700 MB (Ubuntu 24.04 cloud image + Kopia tarball) and uploads the result to Glance as abaca-worker-<KOPIA_VERSION>.

4d. Boot at least one worker VM before tenants attempt to enroll a backup target:

# Read the transport URL from the controller config:
export ABACA_TRANSPORT_URL=$(ssh <controller> \
    "sudo grep '^transport_url' /etc/kolla/abaca-conductor/abaca.conf" \
    | cut -d' ' -f3)

# Boot a worker:
OS_CLOUD=abaca-service abaca-dev worker-boot \
    --network dr-net --key-name abaca-worker

Confirm the worker registered with the conductor:

openstack --os-cloud abaca-service share protection worker list

You should see one row with state=active and a last_heartbeat_at timestamp less than one minute old.

Option B — Kustomize (Kubernetes / OpenShift)

The deploy/kustomize/base/ directory provides Kubernetes manifests for the abaca-api and abaca-conductor deployments.

Step 1 — Review the base manifests

The base kustomization deploys both services into the abaca namespace:

ls deploy/kustomize/base/
# kustomization.yaml
# namespace.yaml
# api-deployment.yaml
# conductor-deployment.yaml
# api-service.yaml

Step 2 — Create an overlay for your environment

Create an environment-specific overlay that patches image references and injects your configuration as a Secret or ConfigMap. Then apply:

kubectl apply -k deploy/kustomize/base/

Step 3 — Verify the deployments

kubectl -n abaca get pods
kubectl -n abaca logs deployment/abaca-api
kubectl -n abaca logs deployment/abaca-conductor

Option C — Docker Compose (local development only)

Warning: This stack runs abaca-api in --noauth mode (no Keystone authentication). It is intended for local development only and must not be used in production.

# From the repo root:
docker compose -f deploy/docker-compose.dev.yml up --build

This brings up MariaDB, RabbitMQ, MinIO (with two demo buckets), abaca-api, and abaca-conductor. After startup, verify health:

curl http://localhost:9797/healthcheck

To skip individual install steps on a re-run (for example, to rebuild only the containers):

bash deploy/kolla/install.sh \
    --skip-identity --skip-catalog --skip-db --skip-mq \
    --skip-config --skip-dashboard

Abacá uses oslo.config-style configuration files. The canonical paths are:

• API: /etc/kolla/abaca-api/abaca.conf
• Conductor: /etc/kolla/abaca-conductor/abaca.conf

Both files are rendered by deploy/kolla/06-config.sh from sourced credentials and templates. The same options are also accepted as environment variables in the Docker Compose dev stack (prefixed ABACA_).

[DEFAULT]

[database]

[keystone_authtoken]

Standard Keystone middleware configuration. Key options:

[api]

[conductor]

[enrollment]

[kopia]

[abaca]

Environment variables (Docker Compose dev stack)

In deploy/docker-compose.dev.yml the same configuration is supplied as environment variables:

Once the control plane is deployed and at least one worker VM is registered, the primary workflow is:

1. Enroll a backup target — register your S3 bucket with Abacá. This runs preflight checks and initializes a Kopia repository.
2. Create a protection policy — bind a Manila share to the target with a schedule and retention settings.
3. Run or schedule backups — backups run automatically per the policy schedule, or you can trigger one on demand.
4. Restore from backup — recreate share contents onto a new or the original share.

All operations go through the abaca-api REST API (Keystone-authenticated) or via the openstack share protection … commands from python-abacaclient.

Authenticating as a tenant

Set OS_CLOUD=abaca-tenant (or source the appropriate OpenRC) before running CLI commands. Abacá uses Keystone token authentication — the same credential flow as any other OpenStack service.

export OS_CLOUD=abaca-tenant

Enrolling a backup target

Enrollment is a one-time setup that verifies your S3 bucket, stores credentials in Barbican, and initializes a Kopia repository. Use abaca-dev enroll (for scripted flows) or the REST API directly.

# Using abaca-dev (recommended for initial setup and testing):
abaca-dev enroll \
  --target-name my-target \
  --bucket abaca-demo-plain \
  --access-key <s3-access-key> \
  --secret-key <s3-secret-key>

The API returns 202 Accepted with a job ID. Poll the job until it reaches available:

openstack share protection job show <job-id>

Creating a protection policy

Once a target is enrolled, attach a share to it with a schedule:

openstack share protection policy create \
  --share <share-id> \
  --target <target-id> \
  --schedule "0 2 * * *" \
  --retain-daily 7 \
  --retain-weekly 4 \
  --retain-monthly 12

Triggering an on-demand backup

openstack share protection backup create --policy <policy-id>

This returns a job ID. Poll it to follow progress through the job state machine: queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available

Restoring from a backup

To restore to a new share (the default):

openstack share protection restore create --backup <backup-id>

To restore in-place (back onto the original share) or restore only a sub-path, pass the appropriate flags. See the API reference for full options.

Checking worker fleet health

# Via CLI:
openstack share protection worker list

# Via API:
curl -sS -H "X-Auth-Token: $OS_TOKEN" \
  http://<controller>:9797/v1/admin/workers

A healthy worker shows state=active with a last_heartbeat_at timestamp within the last minute.

Development quick loop (fast single-file iteration)

If you are iterating on a single source file, use the hot-patch helper to avoid a full rebuild:

bash deploy/kolla/hot-patch.sh abaca/api/controllers/backups.py api

This SCPs the changed file into the running container and restarts it.

Example 1 — Full one-shot install on a Kolla-Ansible cluster

Source your admin OpenRC, then run the installer from the repo root:

source /root/admin-openrc.sh
bash deploy/kolla/install.sh

Expected output (abbreviated):

==> keystone: http://192.168.1.10:5000 region=RegionOne
==> project abaca: creating (worker VM tenant)
==> user abaca: creating in project service
==> role service on service -> abaca
==> role admin on service -> abaca
==> role admin on abaca -> abaca
==> wrote cluster1-abaca-service-openrc.sh (mode 0600)
...
==> registering service 'share-protection' (type share-protection)
==> endpoint public: creating http://192.168.1.10:9797
==> endpoint internal: creating http://192.168.1.10:9797
==> endpoint admin: creating http://192.168.1.10:9797
...
==> running abaca-manage db_sync on 192.168.1.10
==> (re)starting container 'abaca_api'
==> (re)starting container 'abaca_conductor'
==> API /healthcheck
{"status": "ok"}
OK.

Example 2 — Rebuild and redeploy containers only (skip identity/catalog/DB/MQ)

Use this during iterative development to skip the idempotent-but-slow identity and infrastructure steps:

bash deploy/kolla/install.sh \
    --skip-identity \
    --skip-catalog \
    --skip-db \
    --skip-mq \
    --skip-config \
    --skip-dashboard

This runs only steps 05 (build) and 07 (deploy), giving you a fast container rebuild and restart cycle.

Example 3 — Build and register the worker VM Glance image

bash deploy/kolla/build-worker-image.sh

Expected output (first run, abbreviated):

==> downloading Ubuntu 24.04 cloud image (~700 MB)...
==> downloading kopia 0.23.1 tarball...
==> virt-customize: installing nfs-common, kopia, abaca wheelhouse...
==> uploading to Glance as 'abaca-worker-0.23.1'
==> setting image properties: abaca_worker_image=1, abaca_worker_agent_baked=1
OK.

On subsequent runs with the same Kopia version and agent code, the script is a no-op unless you pass --force.

Example 4 — Boot a worker VM and verify registration

# Read the transport URL from the running conductor config:
export ABACA_TRANSPORT_URL=$(ssh pureuser@192.168.1.10 \
    "sudo grep '^transport_url' /etc/kolla/abaca-conductor/abaca.conf" \
    | cut -d' ' -f3)

# Boot a worker into the tenant network:
OS_CLOUD=abaca-service abaca-dev worker-boot \
    --network dr-net \
    --key-name abaca-worker

# Confirm it registered:
openstack --os-cloud abaca-service share protection worker list

Expected output:

+--------------------------------------+--------+---------------------------+
| id                                   | state  | last_heartbeat_at         |
+--------------------------------------+--------+---------------------------+
| a1b2c3d4-e5f6-7890-abcd-ef1234567890 | active | 2024-07-15T10:42:03Z      |
+--------------------------------------+--------+---------------------------+

Example 5 — Enroll a backup target and poll the job to completion

export OS_CLOUD=abaca-tenant

abaca-dev enroll \
  --target-name demo \
  --bucket abaca-demo-plain \
  --access-key minioadmin \
  --secret-key minioadmin

The command prints the job ID. Poll it:

openstack share protection job show <job-id>

Watch the state field advance through: queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available

Once available, inspect the target's capabilities:

openstack share protection target show <target-id>

Expected capabilities (for an object-lock-enabled bucket):

addressing_mode: virtual_hosted
object_lock: true
kopia_encryption: AES256-GCM-HMAC-SHA256
kopia_block_hash: HMAC-SHA256-128
kopia_key_derivation: pbkdf2

Example 6 — Deploy the dev stack locally with Docker Compose

docker compose -f deploy/docker-compose.dev.yml up --build

Once up, verify the API:

curl http://localhost:9797/healthcheck

Expected response:

{"status": "ok"}

MinIO is available at http://localhost:9000 (credentials: abaca / abacaabaca). The two pre-provisioned buckets are abaca-demo-plain and abaca-demo-locked.

Example 7 — Install the abaca-dashboard Horizon plugin

bash deploy/kolla/08-dashboard.sh

After completion, browse to http://<controller>/ and log in. The Share Protection panels appear under:

• Project → Share Protection (targets, policies, backups, restores)
• Admin → Share → Share Protection (fleet and coverage overview)

Note: This install is ephemeral. A kolla-ansible reconfigure horizon or horizon image rebuild will remove it. Re-run 08-dashboard.sh to reinstate.

Use the following format to diagnose and resolve common failures.

API container starts but /healthcheck returns 502 or 500

Symptom: curl http://127.0.0.1:9797/healthcheck returns HTTP 500 or 502.

Likely causes:

• Database migration did not complete (step 07 runs abaca-manage db_sync first).
• RabbitMQ vhost /abaca is missing (step 04 must complete first).
• Config file syntax error in /etc/kolla/abaca-api/abaca.conf.

Fix:

# Inspect API logs on the controller:
docker logs abaca_api
docker logs abaca_conductor

# Re-run the deploy step to re-apply db_sync and restart:
bash deploy/kolla/install.sh \
    --skip-identity --skip-catalog --skip-db --skip-mq \
    --skip-config --skip-dashboard

error: openstack CLI not on PATH and no venv at ...

Symptom: Install scripts exit immediately with this message.

Likely cause: The workstation-side Python virtual environment has not been bootstrapped.

Fix:

bash deploy/cluster1/bootstrap.sh

This installs openstackclient, python-manilaclient, and related packages into a local venv at deploy/cluster1/.venv.

sudo: a password is required on the controller

Symptom: Scripts that SSH to the controller and attempt sudo operations fail with this error.

Likely cause: The default SSH user (pureuser) or sudo password (pureuser) does not match your cluster's convention.

Fix: Override the defaults:

ABACA_SSH_USER=<your-user> ABACA_SUDO_PASSWORD=<your-password> bash deploy/kolla/install.sh

Config change not picked up after editing abaca.conf

Symptom: A change to /etc/kolla/abaca-api/abaca.conf or /etc/kolla/abaca-conductor/abaca.conf has no visible effect.

Likely cause: The config file is bind-mounted read-only into the running container. Editing it in place on the host works, but the container process does not watch for file changes.

Fix:

# On the controller:
docker restart abaca_api abaca_conductor

Horizon panels missing after running 08-dashboard.sh

Symptom: The Share Protection panels do not appear in Horizon after the dashboard install script completes, or they disappeared after a Kolla operation.

Likely cause: kolla-ansible reconfigure horizon or a horizon image rebuild replaced the container, removing the ephemeral plugin install.

Fix: Re-run the dashboard install script:

bash deploy/kolla/08-dashboard.sh

Note: A durable deployment path using a custom abaca-horizon image is on the roadmap. Until it lands, re-running this script after any Kolla horizon reconfiguration is the correct interim procedure.

Worker boot fails with abaca_worker_agent_baked=1 gate

Symptom: abaca-dev worker-boot refuses to boot the worker, citing a missing Glance image property.

Likely cause: The Glance image was built before the abaca-worker-agent was baked in (a pre-R2 kopia-only image). The abaca_worker_agent_baked=1 property is absent.

Fix: Force a rebuild of the worker image:

bash deploy/kolla/build-worker-image.sh --force

Worker VM boots but never registers with the conductor

Symptom: openstack share protection worker list shows no workers, or the worker row never reaches state=active, even after the VM finishes booting.

Likely cause (most common): The transport URL in /etc/abaca/abaca.conf on the worker VM does not match the one in /etc/kolla/abaca-conductor/abaca.conf on the controller.

Likely cause (secondary): The worker VM cannot reach the controller's RabbitMQ port (5672) through the tenant network egress path.

Fix:

# On the worker VM, check the agent service:
journalctl -u abaca-worker-agent

# Verify the transport URL on the worker matches the conductor's:
cat /etc/abaca/abaca.conf | grep transport_url
ssh <controller> "sudo grep '^transport_url' /etc/kolla/abaca-conductor/abaca.conf"

# If the worker can't reach RabbitMQ, check the egress path:
# Either set up host-NAT egress or verify the Neutron router is working.
TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh

Enrollment job fails with error_category: tenant_action_required

Symptom: A target enrollment job reaches error state with error_category: tenant_action_required.

Likely causes and fixes:

Enrollment job fails with error_category: operator_action_required

Symptom: A target enrollment job reaches error state with error_category: operator_action_required.

Likely causes:

• The Kopia repository create command timed out (the kopia subprocess was killed after [kopia] subprocess_timeout seconds).
• The conductor cannot reach the S3 endpoint from the worker network.
• Barbican is unavailable or returning errors.

Fix: Check conductor logs for the specific error, verify network reachability from worker VMs to the S3 endpoint, and confirm Barbican is healthy:

docker logs abaca_conductor
curl -fsS http://<barbican-endpoint>:9311/

fips_kdf_gap: true on an enrolled target

Symptom: After successful enrollment, openstack share protection target show <id> reports fips_kdf_gap: true in the capabilities.

Likely cause: The Kopia binary in the worker image does not support --key-derivation-algorithm (Kopia < 0.14 or a stripped build). The repository was created with scrypt instead of pbkdf2, which is not FIPS-140 compliant for key derivation.

Fix: Update the Kopia version pin in build/kopia.env, rebuild the worker image, and re-enroll the target so the new repository uses pbkdf2:

bash deploy/kolla/build-worker-image.sh --force
# Then re-enroll the target via abaca-dev enroll or the API.

Host-NAT egress not working after controller reboot

Symptom: Worker VMs cannot reach the S3 endpoint or external hosts after the controller was rebooted.

Likely cause: The host-NAT egress setup (veth, ip_forward sysctl, and iptables rules) is runtime state and does not survive a reboot.

Fix: Re-run the script after every controller reboot:

TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh

Existing worker VMs will not automatically pick up the new gateway IP from their DHCP lease. Either run dhclient -r && dhclient inside each affected VM, or reboot the worker VMs.

Teardown

To fully remove Abacá from a cluster (containers, config, DB, MQ, Keystone catalog, and identity):

bash deploy/kolla/99-teardown.sh

This is idempotent and safe to run against a partially-installed cluster.