Guide

Upgrades

Upgrade procedure, db sync, rolling-upgrade notes

Overview

This page describes how to upgrade a live Abacá deployment — from a routine worker VM image refresh to a full control-plane release upgrade — without losing in-flight backup or restore jobs. Upgrades touch up to six independent surfaces (worker image, Kopia binary, abaca-api, abaca-conductor, abaca-dashboard, and the database schema), each with its own cadence and procedure. Two invariants hold across every surface: no in-flight job is dropped, and schema changes are additive-only so that old code can always read new data for at least one release cycle. Read this page before making any change to a running deployment.

Prerequisites

Before beginning any upgrade:

You have a working Abacá deployment on RHOSO 18 or later with OpenShift 4.14 or later.
You have oc access to the abaca OpenShift namespace and credentials for the OpenStack control plane.
You have the openstack share protection CLI (python-abacaclient) installed and a valid clouds.yaml or environment variables pointing at your cluster.
A build host with libguestfs and virt-customize is available if you are rebuilding the worker VM image.
The S3 bucket and Kopia repository for every backup target are healthy (GET /v1/admin/targets shows all targets in available state).
You have reviewed any release notes for Kopia binary version changes and confirmed the repo-format compatibility matrix (see Kopia version bumps) before upgrading the worker image.
You have operator-level access to the Abacá abaca-manage CLI on the control-plane pod.
Backups of the Abacá MySQL/MariaDB database are current before any schema migration.

Installation

Abacá upgrades follow the same toolchain used for initial deployment. The procedure differs by surface.

Step 1 — Run the database schema migration first

Always migrate the schema before deploying new control-plane images. Migrations are additive-only and idempotent, so running them against an already-migrated database is safe.

# On the new-code container image, run db_sync before starting the service.
# On OpenShift this is the db-sync Kubernetes Job — re-apply it:
oc apply -f deploy/rhoso/manifests/db-sync-job.yaml -n abaca
oc wait --for=condition=complete job/abaca-db-sync -n abaca --timeout=120s

If the job fails, do not proceed. Roll back to the previous image and investigate before retrying.

You can also run abaca-manage db_sync directly from a pod shell:

oc exec -it deployment/abaca-api -n abaca -- abaca-manage db_sync

Step 2 — Upgrade `abaca-api`

abaca-api is stateless. You can restart replicas one at a time with no data loss.

# Rolling restart on OpenShift (updates all replicas sequentially):
oc rollout restart deployment/abaca-api -n abaca

# Wait for the rollout to complete:
oc rollout status deployment/abaca-api -n abaca

After each replica restarts, verify it is serving before the next one is replaced. The first few run_job RPC casts from a freshly restarted replica may be silently dropped due to an AMQP connection warm-up window (see Troubleshooting); the conductor's reconciliation loop recovers stuck-QUEUED jobs within approximately five minutes.

Step 3 — Upgrade `abaca-conductor`

The conductor is currently single-replica. Check that no jobs are mid-transition before restarting:

# Confirm no jobs are in a transitioning state:
openstack share protection job list --state transitioning

If that list is empty, restart:

oc rollout restart deployment/abaca-conductor -n abaca
oc rollout status deployment/abaca-conductor -n abaca

The conductor has approximately 10 seconds of downtime during restart. Jobs already running on worker VMs continue uninterrupted — abaca-worker-agent retries its outbound HTTPS calls during the reconnect window. The scheduler misses at most one scheduling tick (configurable via [conductor] scheduler_interval, default 60 s); the scheduler coalesce logic prevents backup pile-up when the conductor returns.

Step 4 — Upgrade `abaca-dashboard` (Horizon plugin)

The dashboard is upgraded separately from the API and conductor. Run the dedicated install script:

bash deploy/rhoso/08-dashboard.sh

This pip-installs the new abaca-dashboard and python-abacaclient packages into the Horizon venv and restarts the Horizon pod. Do not rely on deploy/rhoso/07-deploy.sh to update the dashboard — that script does not touch it.

Step 5 — Build and register a new worker VM image (when needed)

Worker images are immutable. Every update produces a new image with a new version tag. Run the worker image build script on a host with libguestfs:

bash deploy/rhoso/worker/20-worker-image.sh

The script resizes the base RHEL cloud image, installs Kopia, nfs-common, and abaca-worker-agent, writes the systemd unit and security profiles, and uploads the result to Glance with the abaca_worker_image=1 and abaca_kopia_version=<version> properties. A worker image without the abaca_worker_image=1 property will be refused by abaca-dev worker-boot.

Update [conductor] worker_boot_image in your abaca.conf to point at the new image tag (for example, abaca-worker-0.23.2), then restart abaca-conductor to pick up the change:

# Edit your abaca.conf:
# [conductor]
# worker_boot_image = abaca-worker-0.23.2

oc rollout restart deployment/abaca-conductor -n abaca

Step 6 — Roll out the new worker image to the fleet

Create a rollout descriptor via the admin API to start the controlled per-Domain rollover:

openstack share protection admin rollout create \
  --image abaca-worker-0.23.2 \
  --domains <canary-domain-id>,<domain-b-id>,<domain-c-id>

The conductor's reconciliation loop advances through the Domain list. For each Domain, it drains the oldest-image workers one at a time (at most [conductor] worker_boot_cooldown_seconds apart, default 180 s) and boots replacement workers from the new image. Monitor progress:

openstack share protection admin worker list

To verify that no old-image workers remain after the rollout:

# Substitute the old image tag:
GET /v1/admin/workers?image_version=abaca-worker-0.23.1

The rollout halts automatically if the new image fails to reach ACTIVE within [conductor] worker_boot_grace_seconds (default 1200 s). The old-image worker is un-drained and an alert is emitted. See Troubleshooting for recovery steps.

Step 7 — Upgrade the Kopia repository format (rare, opt-in only)

Kopia's newer binaries can always read repositories written by older binaries. The reverse is not guaranteed. A repository format upgrade is a distinct, explicit action — it is never performed automatically.

Only proceed if the Kopia release notes for your new binary version indicate a required or recommended repository format upgrade.

Confirm the fleet has no old-image workers:

GET /v1/admin/workers?image_version=<old-image-tag>
# Must return an empty list before you proceed.

Trigger the repository format upgrade per target:
```
POST /v1/admin/targets/{target_id}/repo-format-upgrade
```
This requires the abaca_admin role. The operation runs kopia repository upgrade inside a maintenance job slot, holds the target repository lock, and blocks backup and restore operations for that target for the duration. The event target_repo_format_upgraded is emitted on completion.

If you attempt step 2 before completing step 1, the API returns a MixedFleetKopiaFormatUnsafe error.

Configuration

The following configuration options are most relevant to upgrade operations. All options live in abaca.conf (INI format, managed by oslo.config). Changes to the majority of options require a service restart; the two exceptions marked reloadable take effect on SIGHUP.

`[conductor]` — scheduler and fleet behaviour

Option	Type	Default	Effect
`scheduler_interval`	integer (seconds)	`60`	How often the conductor runs the backup scheduler. The scheduler skips at most one tick during a conductor restart.
`reconciliation_interval`	integer (seconds)	`300`	How often the reconciliation sweep runs to recover orphaned queued jobs and dead workers. Reloadable via `SIGHUP`.
`worker_boot_cooldown_seconds`	integer (seconds)	`180`	Minimum time between worker boot requests per project. Controls the rate of fleet rollover during an image upgrade.
`worker_boot_grace_seconds`	integer (seconds)	`1200`	Time allowed for a newly booted worker to reach `ACTIVE` before the rollout is considered failed and the old worker is un-drained.
`worker_boot_image`	string	`abaca-worker-0.23.1`	Glance image name used when booting new worker VMs. Update this to the new image tag as part of a worker image upgrade. Must be an image with `abaca_worker_image=1` set.
`worker_boot_max_failures`	integer	`3`	Maximum consecutive boot failures before the conductor stops attempting to scale out in a project.
`worker_boot_failure_window_seconds`	integer (seconds)	`7200`	Window over which `worker_boot_max_failures` is counted.
`queued_job_deadline_seconds`	integer (seconds)	`300`	Jobs that remain in `queued` state beyond this deadline are picked up by the orphan sweep and re-cast. Governs recovery from the AMQP warm-up issue after `abaca-api` restart.
`worker_heartbeat_deadline_seconds`	integer (seconds)	`60`	Workers that miss heartbeats beyond this deadline are considered dead and their jobs are recovered.
`min_workers`	integer	`1`	Minimum number of active workers the conductor maintains per project.
`maintenance_interval_seconds`	integer (seconds)	`604800`	How often the conductor schedules Kopia repository maintenance (approximately weekly).
`worker_boot_os_cloud`	string	`abaca-service`	Named entry in `clouds.yaml` used when booting worker VMs via Nova.

`[conductor]` — runtime-reloadable options

Two options take effect on SIGHUP without a full restart:

reconciliation_interval — useful for temporarily accelerating sweeps after an upgrade to recover stuck jobs faster.
Log level (via oslo.log) — adjust without restarting.

# Send SIGHUP to the conductor pod to reload these options:
oc exec deployment/abaca-conductor -n abaca -- kill -HUP 1

`[kopia]` — binary and FIPS

Option	Type	Default	Effect
`binary`	string	`kopia`	Path to the Kopia binary inside the worker VM. Pin this to the version-specific path if you maintain multiple Kopia binaries during a transition.
`require_fips_profile`	boolean	`true`	Refuse to create or connect to a repository that was not initialized with the FIPS crypto profile. Do not change this in production.
`subprocess_timeout`	integer (seconds)	`3600`	Maximum time a single Kopia subprocess (backup, restore, maintenance) may run before being killed.

`[worker]` — per-worker agent

Option	Type	Default	Effect
`capacity_slots`	integer	(required)	Number of concurrent jobs this worker VM accepts. The fleet picker will not assign a job to a worker whose running job count equals this value.
`command_timeout_seconds`	integer (seconds)	`300`	Maximum time the agent waits for a command to be acknowledged before it is considered timed out.
`max_command_duration_seconds`	integer (seconds)	`21600`	Hard wall-clock limit on any single job command.

Additive-only schema compatibility

Every Alembic migration adds columns or tables with sensible defaults so that the previous release's code continues to work against the upgraded schema. A NOT NULL constraint on a new column is only added in a subsequent release after a backfill sweep has populated all existing rows. This means you can run abaca-manage db_sync before deploying new control-plane pods without causing service errors in the still-running old pods.

Usage

Typical upgrade sequence

For a standard per-release upgrade (new control-plane images, no worker image change), follow this order:

Run abaca-manage db_sync (or re-apply the db-sync Kubernetes Job) on the new image.
Rolling-restart abaca-api.
Restart abaca-conductor after confirming no jobs are transitioning.
Run deploy/rhoso/08-dashboard.sh to update the Horizon plugin.

For an upgrade that also includes a new worker image:

Build and upload the new worker image via deploy/rhoso/worker/20-worker-image.sh.
Update [conductor] worker_boot_image in abaca.conf and restart the conductor.
Create a rollout descriptor via the admin API to begin the per-Domain fleet rollover.
If the new Kopia binary requires a repository format upgrade, complete the full fleet rollover first, verify no old-image workers remain, then issue POST /v1/admin/targets/{id}/repo-format-upgrade per target.

Checking job health before and after an upgrade

Before restarting the conductor, confirm that no jobs are mid-transition:

openstack share protection job list --state transitioning

After the upgrade, verify the conductor is healthy:

GET /v1/admin/system/health

Expect a 200 response within approximately 30 seconds of the conductor pod becoming ready.

Monitoring a worker image rollout

List all workers and their image versions to track rollover progress:

openstack share protection admin worker list

Filter for workers still on the old image:

GET /v1/admin/workers?image_version=abaca-worker-0.23.1

When this returns an empty list, the fleet is fully migrated.

Applying a config-only change

For options that are not reloadable, edit abaca.conf and restart the affected service:

# After editing abaca.conf on the pod's ConfigMap:
oc rollout restart deployment/abaca-api -n abaca
oc rollout restart deployment/abaca-conductor -n abaca

For the two reloadable options (reconciliation_interval and log level), send SIGHUP instead:

oc exec deployment/abaca-conductor -n abaca -- kill -HUP 1

Per-Domain config changes (no restart required)

Configuration knobs that are scoped to a single Domain can be applied live without a service restart:

PATCH /v1/admin/domains/{domain_id}

The change takes effect on the Domain's next reconciliation tick.

Examples

Example 1 — Standard control-plane upgrade (no worker image change)

This example upgrades abaca-api and abaca-conductor to a new release image, runs the schema migration, and updates the dashboard.

# 1. Apply the new db-sync Job manifest (uses the new image tag):
oc apply -f deploy/rhoso/manifests/db-sync-job.yaml -n abaca
oc wait --for=condition=complete job/abaca-db-sync -n abaca --timeout=120s
# Expected output:
# job.batch/abaca-db-sync condition met

# 2. Rolling-restart abaca-api:
oc rollout restart deployment/abaca-api -n abaca
oc rollout status deployment/abaca-api -n abaca
# Expected output:
# deployment "abaca-api" successfully rolled out

# 3. Confirm no jobs are mid-transition, then restart abaca-conductor:
openstack share protection job list --state transitioning
# Expected: empty table

oc rollout restart deployment/abaca-conductor -n abaca
oc rollout status deployment/abaca-conductor -n abaca
# Expected output:
# deployment "abaca-conductor" successfully rolled out

# 4. Update the Horizon dashboard plugin:
bash deploy/rhoso/08-dashboard.sh
# Expected: pip install output, then Horizon pod restart confirmation

Example 2 — Build and register a new worker VM image

# Build the new image on a host with libguestfs:
bash deploy/rhoso/worker/20-worker-image.sh
# Expected: virt-customize output, then:
# +------------------+--------------------------------------+
# | Field            | Value                                |
# +------------------+--------------------------------------+
# | name             | abaca-worker-0.23.2                  |
# | id               | <new-glance-uuid>                    |
# | abaca_worker_image | 1                                  |
# | abaca_kopia_version | 0.23.2                            |
# +------------------+--------------------------------------+

# Update conductor config to use the new image tag:
# Edit the abaca.conf ConfigMap:
# [conductor]
# worker_boot_image = abaca-worker-0.23.2

# Restart conductor to pick up the change:
oc rollout restart deployment/abaca-conductor -n abaca
oc rollout status deployment/abaca-conductor -n abaca

Example 3 — Initiate a per-Domain worker image rollout

# Start the rollout, canary Domain first:
openstack share protection admin rollout create \
  --image abaca-worker-0.23.2 \
  --domains <canary-domain-id>,<prod-domain-a-id>,<prod-domain-b-id>
# Expected output:
# Rollout descriptor created. Canary Domain will be observed for 72h before proceeding.

# Monitor worker fleet during rollout:
watch -n 30 'openstack share protection admin worker list'

# Check for remaining old-image workers:
GET /v1/admin/workers?image_version=abaca-worker-0.23.1
# Expected when complete: empty list

Example 4 — Opt-in Kopia repository format upgrade (after full fleet rollover)

# Step 1: Verify no old-image workers remain:
GET /v1/admin/workers?image_version=abaca-worker-0.23.1
# Must return: empty list

# Step 2: Trigger repo-format upgrade for each target:
POST /v1/admin/targets/a1b2c3d4-.../repo-format-upgrade
# Expected: 202 Accepted
# The operation runs kopia repository upgrade inside a maintenance job.
# Backup and restore for this target are blocked during the upgrade.
# On completion, event target_repo_format_upgraded is emitted.

# Repeat for each target:
POST /v1/admin/targets/e5f6a7b8-.../repo-format-upgrade

Example 5 — Accelerate reconciliation after upgrade to recover stuck jobs

# Temporarily lower reconciliation_interval to 60 s (reloadable via SIGHUP):
# Edit abaca.conf ConfigMap:
# [conductor]
# reconciliation_interval = 60

# Send SIGHUP to reload without a full restart:
oc exec deployment/abaca-conductor -n abaca -- kill -HUP 1

# After stuck jobs clear, restore the default:
# [conductor]
# reconciliation_interval = 300
oc exec deployment/abaca-conductor -n abaca -- kill -HUP 1

Troubleshooting

Issue 1 — Jobs stuck in `queued` state after `abaca-api` restart

Symptom: Immediately after an abaca-api rolling restart, new backup or restore jobs sit in queued state for up to five minutes and do not progress.

Cause: The first RPC casts from a freshly restarted abaca-api replica may be silently dropped during the AMQP connection warm-up window. The cast appears to succeed on the API side but never reaches abaca-conductor.

Fix: This resolves automatically. The conductor's reconciliation sweep (_sweep_orphan_queued) detects jobs that have been in queued state beyond [conductor] queued_job_deadline_seconds (default 300 s) and re-casts them. If you need faster recovery, temporarily lower reconciliation_interval to 60 s and send SIGHUP to the conductor (see Example 5). No data is lost.

Issue 2 — `abaca-manage db_sync` fails; service does not start

Symptom: The db-sync Kubernetes Job exits with a non-zero code. The new abaca-api or abaca-conductor pods fail to start.

Cause: A schema migration encountered an error — typically a connection failure, a permissions problem on the dedicated Abacá database, or an attempt to run a migration that conflicts with unexpected existing schema state.

Fix:

Inspect the db-sync Job logs: oc logs job/abaca-db-sync -n abaca.
Do not deploy the new control-plane images until the migration succeeds. Roll back to the previous image tag.
Resolve the database connectivity or permissions issue and re-apply the Job.
Migrations are idempotent — re-running a completed migration is safe.

Issue 3 — Worker image rollout halts; `worker_image_rollout_failed` event emitted

Symptom: The conductor emits a worker_image_rollout_failed event. A new-image worker did not reach ACTIVE within [conductor] worker_boot_grace_seconds (default 1200 s). The old-image worker is un-drained and resumes accepting jobs.

Cause: The new worker image fails to boot or the abaca-worker-agent inside it fails to register. Common causes: the Glance image is missing the abaca_worker_image=1 property; a Nova quota or networking issue prevented the VM from booting; the worker image build was incomplete.

Fix:

Check Nova console logs for the failed worker VM.
Verify the Glance image properties: openstack image show abaca-worker-0.23.2 | grep abaca_worker_image — the value must be 1.
Check abaca-conductor logs for the specific boot failure reason: oc logs deployment/abaca-conductor -n abaca | grep worker_image_rollout.
Fix the image or infrastructure issue, then explicitly resume the rollout via the admin API.

Issue 4 — `MixedFleetKopiaFormatUnsafe` error on repo-format upgrade

Symptom: POST /v1/admin/targets/{id}/repo-format-upgrade returns a MixedFleetKopiaFormatUnsafe error.

Cause: At least one worker in the fleet is still running the old image (and therefore the old Kopia binary). Upgrading the repository format while workers on different Kopia versions are active would leave some workers unable to read the repository.

Fix: Complete the full worker image fleet rollover first. Verify that GET /v1/admin/workers?image_version=<old-image-tag> returns an empty list, then retry the repo-format upgrade.

Issue 5 — Dashboard renders stale fields or missing columns after control-plane upgrade

Symptom: The Horizon Share Protection panels are missing new fields introduced in the release, or templates look outdated, while the openstack share protection CLI and the API return the correct new data.

Cause: deploy/rhoso/07-deploy.sh was run to upgrade abaca-api and abaca-conductor but deploy/rhoso/08-dashboard.sh was not run. The abaca-dashboard package in Horizon's venv is still the old version.

Fix: Run bash deploy/rhoso/08-dashboard.sh. This installs the new abaca-dashboard and python-abacaclient packages into the Horizon venv and restarts the Horizon pod. No Abacá API or conductor restart is required.

Issue 6 — Conductor in-memory counters appear wrong immediately after restart

Symptom: Immediately after an abaca-conductor restart, per-Domain cooldown timers or per-project pending-boot counters seem to have reset, and the conductor briefly boots more workers than expected.

Cause: The conductor's in-memory scheduling state (cooldowns, pending-boot counters) is lost on restart by design — these counters are ephemeral and are not persisted to the database. They rebuild from database queries on the first reconciliation tick after startup.

Fix: This is expected behaviour and self-corrects within one reconciliation_interval (default 300 s). The counters are bounded — a brief transient overshoot in worker boot attempts is harmless and self-limiting via worker_boot_max_failures and worker_boot_failure_window_seconds.