Upgrade
This page guides operators through upgrading Trilio Share Protection (Abacá) to a new release. It covers the control plane services running on OpenShift, the worker VM image fleet, the database schema migrations that must be applied before restarting services, and the Kopia repository format considerations that govern when a fleet-wide image rotation is required. Because Abacá's persistence model is deliberately additive-only, upgrades are designed to be non-destructive — your backup data in S3 and the Kopia repositories it contains are never touched by a control plane upgrade.
Before you begin an upgrade, confirm that the following are in place:
- Current installation: A working Abacá deployment on OpenShift ≥ 4.12 with
abaca-apiandabaca-conductorrunning and healthy. - Database access: Operator-level access to the MariaDB or MySQL instance (MariaDB 10.x or the version shipped with Kolla 2024.2+) used by Abacá.
- OpenShift credentials:
ocCLI authenticated to the cluster with sufficient RBAC to update Deployments and ConfigMaps in the Abacá namespace. abaca-managebinary: Theabaca-manageCLI must be available to the operator — either in the new control-plane container image or as a standalone binary — because it is the authoritative tool for database migrations.- Python ≥ 3.11 on any workstation used to run
python-abacaclientorabaca-dev. openstackCLI withpython-abacaclientinstalled if you want to verify the API after upgrade.- RHEL worker image built with the target release's
abaca-worker-agentand the required Kopia version (≥ 0.23.1) staged in your image registry before you begin fleet rotation. - No in-progress jobs: Drain or wait for all running backup, restore, and enrollment jobs to reach a terminal state (
availableorerror) before upgrading the control plane. This prevents jobs from being interrupted mid-state-machine by a conductor restart. - Maintenance window: Schedule a window long enough to cover the database migration and the fleet image rotation. The control plane upgrade itself is fast; fleet rotation duration scales with the number of active workers.
Follow these steps in order. Do not restart services before completing the database migration.
Step 1 — Pull the new container images
Update your OpenShift image references to point to the new release tags for abaca-api and abaca-conductor. Do not apply the Deployment changes yet.
# Example: update the image tag in your Deployment manifests or Kustomize overlay
# Replace <NEW_VERSION> with the target release tag
oc set image deployment/abaca-api \
abaca-api=<registry>/abaca-api:<NEW_VERSION> \
--dry-run=client -o yaml | oc apply -f -
oc set image deployment/abaca-conductor \
abaca-conductor=<registry>/abaca-conductor:<NEW_VERSION> \
--dry-run=client -o yaml | oc apply -f -
Step 2 — Scale down the control plane
Before migrating the database, stop both services to prevent the old binaries from writing to a schema they no longer understand.
oc scale deployment/abaca-conductor --replicas=0
oc scale deployment/abaca-api --replicas=0
# Wait until all pods are terminated
oc wait pod --for=delete \
-l 'app in (abaca-api,abaca-conductor)' \
--timeout=120s
Step 3 — Run database migrations with abaca-manage
Abacá enforces additive-only migrations so that the schema can survive service upgrades and the RHOSO 18→19 rebase without data loss. Run migrations using the abaca-manage CLI from within a temporary pod running the new image, or from a bastion that can reach the database.
# Option A: run from a temporary pod using the new image
oc run abaca-migrate \
--image=<registry>/abaca-api:<NEW_VERSION> \
--restart=Never \
--env-file=<path-to-abaca-env-configmap> \
-- abaca-manage db upgrade
# Wait for the migration pod to complete
oc wait pod/abaca-migrate \
--for=condition=Succeeded \
--timeout=300s
# Inspect logs to confirm success
oc logs abaca-migrate
# Clean up
oc delete pod abaca-migrate
# Option B: if abaca-manage is available on a bastion with DB connectivity
abaca-manage db upgrade
Expect output similar to:
INFO [abaca.manage] Running migrations against abaca database
INFO [alembic] Running upgrade <prev_rev> -> <new_rev>, <description>
INFO [abaca.manage] Migration complete
Step 4 — Scale the control plane back up
Once migrations succeed, apply the updated images and restore replica counts.
oc scale deployment/abaca-api --replicas=<desired>
oc scale deployment/abaca-conductor --replicas=<desired>
# Confirm pods reach Running state
oc rollout status deployment/abaca-api
oc rollout status deployment/abaca-conductor
Step 5 — Verify the API is healthy
Use python-abacaclient or a direct HTTP call to confirm the API is responding and reports the new version.
openstack share protection --help
or with a direct call (substitute your Keystone-issued token and endpoint):
curl -sf \
-H "X-Auth-Token: $OS_TOKEN" \
"$ABACA_ENDPOINT/v1/" | python3 -m json.tool
Step 6 — Rotate the worker VM fleet
See the Usage section for the fleet rotation procedure. Worker image rotation is a separate, rolling operation that can follow the control plane upgrade at your own pace, subject to the Kopia repository format constraints described in Configuration.
Abacá's upgrade-relevant configuration lives in the conductor and in the worker image. No configuration file changes are required for a standard patch-level upgrade; configuration changes are only needed when enabling new features or adjusting operational parameters introduced in the new release.
Conductor configuration — upgrade-relevant options
These keys live in the [conductor] section of your Abacá configuration (consult the ConfigMap or environment injection used by your OpenShift Deployment).
| Key | Default | Effect |
|---|---|---|
maintenance_interval_seconds | weekly (604800) | How often the reconciliation loop enqueues a Kopia maintenance job per enrolled target. After upgrade, verify this is still appropriate — overdue maintenance means retention is not being enforced. |
worker_heartbeat_interval_seconds | 10 | How frequently worker agents heartbeat to the conductor. Unchanged across upgrades; existing workers continue to heartbeat normally during fleet rotation. |
worker_heartbeat_deadline_seconds | 60 | Workers whose last heartbeat is older than this value are marked DEAD by the reconciliation loop. During fleet rotation, old workers are drained before termination, so they should reach a terminal state cleanly before this timer fires. |
reconciliation_interval | 300 | How often the reconciliation loop runs. The loop handles stuck-job cleanup, dead-worker reaping, and maintenance scheduling — all critical during and after upgrade. |
Kopia repository format upgrades
Kopia repository format versions are immutable at creation — a repository created with a given Kopia version retains that format indefinitely. Control plane upgrades do not change existing repositories. A Kopia repository format upgrade requires a fleet-coordinated image rotation event: all worker VMs must be running a Kopia binary that supports the new format before any repository is upgraded. Because Abacá owns the Kopia build pipeline, format upgrades are always announced as explicit upgrade milestones.
Important: Do not upgrade the Kopia version embedded in worker images unless the release notes explicitly authorize it and confirm format compatibility. An incompatible Kopia upgrade will prevent workers from reading existing repositories.
FIPS mode
FIPS mode is a supported, validated production configuration. The cryptographic profile of every Kopia repository is fixed at enrollment:
--encryption=AES256-GCM-HMAC-SHA256--block-hash=HMAC-SHA256-128- Key derivation:
pbkdf2(preferred) orscrypt(fallback, surfacesfips_kdf_gap: trueon the target)
Upgrading to a worker image that ships a Kopia build with PBKDF2 support does not retroactively change existing repositories. Repositories initialized with scrypt will continue to use scrypt. New enrollments after the upgrade will use PBKDF2 if the new binary supports it. Inspect the fips_kdf_gap flag on your backup targets after fleet rotation to confirm which repositories benefited.
No configuration file change is required to maintain FIPS mode across an upgrade — it is governed by the RHEL FIPS mode setting of the worker VM image and the Kopia binary embedded in it.
Additive-only schema guarantee
Abacá's database schema is additive-only: upgrades never drop columns or tables. This means a rolled-back control plane can still read a migrated schema, reducing rollback risk. The field Worker.current_job_id is an example of legacy schema residue that is retained for migration compatibility even though no current code path populates it — do not rely on it in any custom tooling.
Draining the job queue before upgrade
Before scaling down the control plane, allow in-progress jobs to finish naturally or wait for them to reach a terminal state. You can list active jobs using python-abacaclient:
openstack share protection job list --state queued
openstack share protection job list --state transferring
openstack share protection job list --state provisioning_network
openstack share protection job list --state provisioning_source
openstack share protection job list --state connecting_repository
openstack share protection job list --state finalizing
openstack share protection job list --state releasing
Jobs in any of the above states are non-terminal. Wait until the output is empty before proceeding with the control plane shutdown.
Running database migrations with abaca-manage
abaca-manage is the server-side management CLI for database migrations and administrative tasks. It does not go through the REST API. Always run db upgrade with the services stopped:
abaca-manage db upgrade
To inspect the current migration revision without making changes:
abaca-manage db current
To view available migration steps:
abaca-manage db history
Worker fleet rotation
Worker VMs are short-lived by design: the conductor boots them on demand and they register via cloud-init. Fleet rotation is therefore a rolling replacement — you update the image reference in the conductor's fleet configuration so that new workers boot with the new image, then terminate existing workers after their in-progress jobs complete.
Step 1 — Stage the new worker image
Ensure the new RHEL worker image (with the updated abaca-worker-agent and Kopia ≥ 0.23.1) is registered in your OpenStack Glance and accessible to the service project.
Step 2 — Update the worker image reference in conductor configuration
Update the conductor's worker image ID to point to the new image. The conductor will use the new image for all subsequently provisioned workers.
Step 3 — Drain existing workers
Existing workers continue to run jobs against the old image until they are reaped. You can use abaca-dev to probe fleet health and confirm all workers are running the new image version after rotation:
abaca-dev fleet status
Step 4 — Verify worker registration
New workers register themselves with the conductor on cloud-init boot by casting worker_registered with their Nova instance UUID, hostname, and capacity slot count. You can observe registered workers via the API:
openstack share protection worker list
Step 5 — Confirm FIPS KDF status after rotation
If the new Kopia build adds PBKDF2 support, check your backup targets for the fips_kdf_gap flag. Targets enrolled before the upgrade will retain their original KDF; new enrollments will use PBKDF2.
openstack share protection target list
openstack share protection target show <target-id>
A target with fips_kdf_gap: true is still protected — it is using scrypt, which is cryptographically strong — but it is not using PBKDF2 as preferred by strict FIPS 140 interpretations. Re-enrolling the target (which re-initializes the Kopia repository) is the only way to change the KDF for an existing repository.
Post-upgrade maintenance check
After the control plane is running, confirm that the reconciliation loop is scheduling maintenance jobs for all enrolled targets. Overdue maintenance means retention policies are not being enforced:
# List recent maintenance jobs
openstack share protection job list --type maintenance
If maintenance is overdue, you can trigger it by allowing the reconciliation loop to run (it fires every reconciliation_interval seconds, default 300) or by waiting for the next scheduled tick.
Example 1 — Check migration status before upgrading
Before starting the upgrade, confirm the current database revision to have a known baseline.
abaca-manage db current
Expected output:
INFO [abaca.manage] Current revision: a1b2c3d4e5f6 (head)
Example 2 — Full control plane upgrade sequence
This example shows the complete sequence for a patch-level control plane upgrade with no Kopia repository format change.
# 1. Confirm no active jobs
openstack share protection job list --state transferring
# (expect empty output before proceeding)
# 2. Scale down control plane
oc scale deployment/abaca-conductor --replicas=0
oc scale deployment/abaca-api --replicas=0
oc wait pod --for=delete -l 'app in (abaca-api,abaca-conductor)' --timeout=120s
# 3. Run migrations
oc run abaca-migrate \
--image=<registry>/abaca-api:<NEW_VERSION> \
--restart=Never \
--env-file=<path-to-abaca-env-configmap> \
-- abaca-manage db upgrade
oc wait pod/abaca-migrate --for=condition=Succeeded --timeout=300s
oc logs abaca-migrate
oc delete pod abaca-migrate
# 4. Apply new images and scale up
oc set image deployment/abaca-api abaca-api=<registry>/abaca-api:<NEW_VERSION>
oc set image deployment/abaca-conductor abaca-conductor=<registry>/abaca-conductor:<NEW_VERSION>
oc scale deployment/abaca-api --replicas=2
oc scale deployment/abaca-conductor --replicas=1
oc rollout status deployment/abaca-api
oc rollout status deployment/abaca-conductor
# 5. Verify API health
openstack share protection job list
Expected final output (step 5):
+--------------------------------------+--------+-----------+---------------------+
| ID | Type | State | Created At |
+--------------------------------------+--------+-----------+---------------------+
| ... | backup | available | ... |
+--------------------------------------+--------+-----------+---------------------+
Example 3 — Verify fleet image version after worker rotation
After updating the worker image reference and allowing the conductor to cycle workers, confirm all active workers are running the new image.
abaca-dev fleet status
Expected output (illustrative):
Worker Fleet Status
===================
Total workers : 4
Active : 4
Dead : 0
Draining : 0
Worker ID AZ Image Version Slots Running Jobs
-------------------------------------- ---- -------------- ----- ------------
<uuid-1> az1 <NEW_VERSION> 4 0
<uuid-2> az1 <NEW_VERSION> 4 1
<uuid-3> az2 <NEW_VERSION> 4 0
<uuid-4> az2 <NEW_VERSION> 4 0
Example 4 — Inspect a backup target for FIPS KDF gap after fleet rotation
openstack share protection target show 3fa85f64-5717-4562-b3fc-2c963f66afa6
Expected output (illustrative, showing a target with a KDF gap):
+------------------------------+--------------------------------------------------+
| Field | Value |
+------------------------------+--------------------------------------------------+
| id | 3fa85f64-5717-4562-b3fc-2c963f66afa6 |
| name | production-target |
| status | available |
| fips_kdf_gap | true |
| enrollment_status | enrolled |
| ... | ... |
+------------------------------+--------------------------------------------------+
A fips_kdf_gap: true value means this repository was initialized with scrypt KDF because the Kopia binary at enrollment time did not support PBKDF2. The repository remains fully protected; this flag is informational for FIPS compliance auditing.
Migration fails with a database connection error
Symptom: abaca-manage db upgrade exits immediately with a connection refused or authentication error.
Likely cause: The migration pod or bastion cannot reach the database, or the database credentials in the environment are incorrect or missing.
Fix: Confirm that the environment variables or ConfigMap injected into the migration pod include the correct database host, port, database name, and credentials. Verify network reachability from the migration pod to the MariaDB service within the cluster. Check that the database user has ALTER TABLE and CREATE TABLE privileges on the Abacá schema.
API pods crash-loop after upgrade
Symptom: abaca-api pods enter CrashLoopBackOff immediately after the upgraded Deployment is applied.
Likely cause: The database migration was not run before scaling the services back up, causing the new code to encounter an older schema.
Fix: Scale abaca-api back down to zero, run abaca-manage db upgrade successfully, then scale back up. Check pod logs for explicit migration-related errors:
oc scale deployment/abaca-api --replicas=0
# run migration as shown in Installation Step 3
oc scale deployment/abaca-api --replicas=<desired>
oc logs -l app=abaca-api --previous
Jobs remain stuck in a non-terminal state after conductor restart
Symptom: After upgrading and restarting abaca-conductor, some jobs are stuck in provisioning_network, transferring, or another mid-flight state and never progress.
Likely cause: The conductor was restarted while those jobs were in-flight. The reconciliation loop will reap workers whose heartbeat has lapsed and transition their jobs to error.
Fix: Wait for one reconciliation interval (default 300 seconds). The reconciliation loop will call reap_dead_workers followed by reap_stuck_jobs, transitioning orphaned jobs to error with error_code=abaca.worker_unavailable. Inspect the failed jobs to determine whether they require tenant or operator action:
openstack share protection job list --state error
openstack share protection job show <job-id>
# Check the error_category field:
# tenant_action_required — tenant must fix a configuration issue
# operator_action_required — infrastructure or service needs attention
Worker VMs continue to boot with the old image after fleet rotation
Symptom: abaca-dev fleet status shows workers still running the previous image version after the conductor configuration was updated.
Likely cause: The conductor is still assigning jobs to existing workers from the old image because they are healthy and have available capacity slots. New workers are only provisioned when the pool needs to grow.
Fix: To force rotation, reduce the desired fleet size to zero (draining existing workers) and then restore it. Alternatively, use abaca-dev to trigger explicit worker decommissioning. Workers in DRAINING state will not accept new jobs; they will be removed once their current jobs complete.
Maintenance jobs are not running after upgrade
Symptom: Backup targets show overdue maintenance; the openstack share protection job list --type maintenance output shows no recent successful maintenance jobs.
Likely cause: The conductor's reconciliation loop may not have run yet after restart, or the maintenance_interval_seconds threshold has not elapsed since the last successful run.
Fix: Wait for one reconciliation interval (default 300 seconds). If maintenance jobs remain absent, check conductor logs for errors in the _sweep_target_maintenance loop. Confirm that at least one healthy worker is registered and has available capacity slots, because maintenance jobs are dispatched to workers just like backup jobs.
oc logs -l app=abaca-conductor | grep maintenance
fips_kdf_gap appears on all targets after upgrading Kopia
Symptom: After rotating the worker fleet to a new Kopia build, existing backup targets still show fips_kdf_gap: true.
Likely cause: The fips_kdf_gap flag reflects the KDF used when the Kopia repository was initialized at enrollment, not the current worker image. Existing repositories cannot change their KDF without re-initialization.
Fix: This is expected behavior. New enrollments created after the fleet rotation will use PBKDF2 and will not show this flag. To remediate existing targets, you must delete and re-enroll them (which destroys the existing Kopia repository and all backups stored in it). Weigh this against your data retention requirements before proceeding.