--- title: Upgrade product: trilio-share-protection doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/upgrade --- # Upgrade ## Overview This page guides operators through upgrading Trilio Share Protection (Abacá) to a new release. It covers the control plane services running on OpenShift, the worker VM image fleet, the database schema migrations that must be applied before restarting services, and the Kopia repository format considerations that govern when a fleet-wide image rotation is required. Because Abacá's persistence model is deliberately additive-only, upgrades are designed to be non-destructive — your backup data in S3 and the Kopia repositories it contains are never touched by a control plane upgrade. ## Prerequisites Before you begin an upgrade, confirm that the following are in place: - **Current installation**: A working Abacá deployment on OpenShift ≥ 4.12 with `abaca-api` and `abaca-conductor` running and healthy. - **Database access**: Operator-level access to the MariaDB or MySQL instance (MariaDB 10.x or the version shipped with Kolla 2024.2+) used by Abacá. - **OpenShift credentials**: `oc` CLI authenticated to the cluster with sufficient RBAC to update Deployments and ConfigMaps in the Abacá namespace. - **`abaca-manage` binary**: The `abaca-manage` CLI must be available to the operator — either in the new control-plane container image or as a standalone binary — because it is the authoritative tool for database migrations. - **Python ≥ 3.11** on any workstation used to run `python-abacaclient` or `abaca-dev`. - **`openstack` CLI with `python-abacaclient` installed** if you want to verify the API after upgrade. - **RHEL worker image** built with the target release's `abaca-worker-agent` and the required Kopia version (≥ 0.23.1) staged in your image registry before you begin fleet rotation. - **No in-progress jobs**: Drain or wait for all running backup, restore, and enrollment jobs to reach a terminal state (`available` or `error`) before upgrading the control plane. This prevents jobs from being interrupted mid-state-machine by a conductor restart. - **Maintenance window**: Schedule a window long enough to cover the database migration and the fleet image rotation. The control plane upgrade itself is fast; fleet rotation duration scales with the number of active workers. ## Installation Follow these steps in order. Do not restart services before completing the database migration. **Step 1 — Pull the new container images** Update your OpenShift image references to point to the new release tags for `abaca-api` and `abaca-conductor`. Do not apply the Deployment changes yet. ```bash # Example: update the image tag in your Deployment manifests or Kustomize overlay # Replace with the target release tag oc set image deployment/abaca-api \ abaca-api=/abaca-api: \ --dry-run=client -o yaml | oc apply -f - oc set image deployment/abaca-conductor \ abaca-conductor=/abaca-conductor: \ --dry-run=client -o yaml | oc apply -f - ``` > **Note for reviewer:** The exact registry path and image names are not encoded in the available source material. Replace `` with your actual image registry URL. **Step 2 — Scale down the control plane** Before migrating the database, stop both services to prevent the old binaries from writing to a schema they no longer understand. ```bash oc scale deployment/abaca-conductor --replicas=0 oc scale deployment/abaca-api --replicas=0 # Wait until all pods are terminated oc wait pod --for=delete \ -l 'app in (abaca-api,abaca-conductor)' \ --timeout=120s ``` **Step 3 — Run database migrations with `abaca-manage`** Abacá enforces additive-only migrations so that the schema can survive service upgrades and the RHOSO 18→19 rebase without data loss. Run migrations using the `abaca-manage` CLI from within a temporary pod running the new image, or from a bastion that can reach the database. ```bash # Option A: run from a temporary pod using the new image oc run abaca-migrate \ --image=/abaca-api: \ --restart=Never \ --env-file= \ -- abaca-manage db upgrade # Wait for the migration pod to complete oc wait pod/abaca-migrate \ --for=condition=Succeeded \ --timeout=300s # Inspect logs to confirm success oc logs abaca-migrate # Clean up oc delete pod abaca-migrate ``` ```bash # Option B: if abaca-manage is available on a bastion with DB connectivity abaca-manage db upgrade ``` Expect output similar to: ``` INFO [abaca.manage] Running migrations against abaca database INFO [alembic] Running upgrade -> , INFO [abaca.manage] Migration complete ``` > **Note for reviewer:** The exact Alembic revision identifiers and migration descriptions are release-specific and not available in the source material provided. **Step 4 — Scale the control plane back up** Once migrations succeed, apply the updated images and restore replica counts. ```bash oc scale deployment/abaca-api --replicas= oc scale deployment/abaca-conductor --replicas= # Confirm pods reach Running state oc rollout status deployment/abaca-api oc rollout status deployment/abaca-conductor ``` **Step 5 — Verify the API is healthy** Use `python-abacaclient` or a direct HTTP call to confirm the API is responding and reports the new version. ```bash openstack share protection --help ``` or with a direct call (substitute your Keystone-issued token and endpoint): ```bash curl -sf \ -H "X-Auth-Token: $OS_TOKEN" \ "$ABACA_ENDPOINT/v1/" | python3 -m json.tool ``` **Step 6 — Rotate the worker VM fleet** See the **Usage** section for the fleet rotation procedure. Worker image rotation is a separate, rolling operation that can follow the control plane upgrade at your own pace, subject to the Kopia repository format constraints described in **Configuration**. ## Configuration Abacá's upgrade-relevant configuration lives in the conductor and in the worker image. No configuration file changes are required for a standard patch-level upgrade; configuration changes are only needed when enabling new features or adjusting operational parameters introduced in the new release. ### Conductor configuration — upgrade-relevant options These keys live in the `[conductor]` section of your Abacá configuration (consult the ConfigMap or environment injection used by your OpenShift Deployment). | Key | Default | Effect | |-----|---------|--------| | `maintenance_interval_seconds` | weekly (604800) | How often the reconciliation loop enqueues a Kopia maintenance job per enrolled target. After upgrade, verify this is still appropriate — overdue maintenance means retention is not being enforced. | | `worker_heartbeat_interval_seconds` | 10 | How frequently worker agents heartbeat to the conductor. Unchanged across upgrades; existing workers continue to heartbeat normally during fleet rotation. | | `worker_heartbeat_deadline_seconds` | 60 | Workers whose last heartbeat is older than this value are marked DEAD by the reconciliation loop. During fleet rotation, old workers are drained before termination, so they should reach a terminal state cleanly before this timer fires. | | `reconciliation_interval` | 300 | How often the reconciliation loop runs. The loop handles stuck-job cleanup, dead-worker reaping, and maintenance scheduling — all critical during and after upgrade. | ### Kopia repository format upgrades Kopia repository format versions are **immutable at creation** — a repository created with a given Kopia version retains that format indefinitely. Control plane upgrades do not change existing repositories. A Kopia repository format upgrade requires a **fleet-coordinated image rotation event**: all worker VMs must be running a Kopia binary that supports the new format before any repository is upgraded. Because Abacá owns the Kopia build pipeline, format upgrades are always announced as explicit upgrade milestones. > **Important:** Do not upgrade the Kopia version embedded in worker images unless the release notes explicitly authorize it and confirm format compatibility. An incompatible Kopia upgrade will prevent workers from reading existing repositories. ### FIPS mode FIPS mode is a supported, validated production configuration. The cryptographic profile of every Kopia repository is fixed at enrollment: - `--encryption=AES256-GCM-HMAC-SHA256` - `--block-hash=HMAC-SHA256-128` - Key derivation: `pbkdf2` (preferred) or `scrypt` (fallback, surfaces `fips_kdf_gap: true` on the target) Upgrading to a worker image that ships a Kopia build with PBKDF2 support does **not** retroactively change existing repositories. Repositories initialized with scrypt will continue to use scrypt. New enrollments after the upgrade will use PBKDF2 if the new binary supports it. Inspect the `fips_kdf_gap` flag on your backup targets after fleet rotation to confirm which repositories benefited. No configuration file change is required to maintain FIPS mode across an upgrade — it is governed by the RHEL FIPS mode setting of the worker VM image and the Kopia binary embedded in it. ### Additive-only schema guarantee Abacá's database schema is additive-only: upgrades never drop columns or tables. This means a rolled-back control plane can still read a migrated schema, reducing rollback risk. The field `Worker.current_job_id` is an example of legacy schema residue that is retained for migration compatibility even though no current code path populates it — do not rely on it in any custom tooling. ## Usage ### Draining the job queue before upgrade Before scaling down the control plane, allow in-progress jobs to finish naturally or wait for them to reach a terminal state. You can list active jobs using `python-abacaclient`: ```bash openstack share protection job list --state queued openstack share protection job list --state transferring openstack share protection job list --state provisioning_network openstack share protection job list --state provisioning_source openstack share protection job list --state connecting_repository openstack share protection job list --state finalizing openstack share protection job list --state releasing ``` Jobs in any of the above states are non-terminal. Wait until the output is empty before proceeding with the control plane shutdown. ### Running database migrations with `abaca-manage` `abaca-manage` is the server-side management CLI for database migrations and administrative tasks. It does not go through the REST API. Always run `db upgrade` with the services stopped: ```bash abaca-manage db upgrade ``` To inspect the current migration revision without making changes: ```bash abaca-manage db current ``` To view available migration steps: ```bash abaca-manage db history ``` ### Worker fleet rotation Worker VMs are short-lived by design: the conductor boots them on demand and they register via cloud-init. Fleet rotation is therefore a rolling replacement — you update the image reference in the conductor's fleet configuration so that new workers boot with the new image, then terminate existing workers after their in-progress jobs complete. **Step 1 — Stage the new worker image** Ensure the new RHEL worker image (with the updated `abaca-worker-agent` and Kopia ≥ 0.23.1) is registered in your OpenStack Glance and accessible to the service project. **Step 2 — Update the worker image reference in conductor configuration** Update the conductor's worker image ID to point to the new image. The conductor will use the new image for all subsequently provisioned workers. **Step 3 — Drain existing workers** Existing workers continue to run jobs against the old image until they are reaped. You can use `abaca-dev` to probe fleet health and confirm all workers are running the new image version after rotation: ```bash abaca-dev fleet status ``` **Step 4 — Verify worker registration** New workers register themselves with the conductor on cloud-init boot by casting `worker_registered` with their Nova instance UUID, hostname, and capacity slot count. You can observe registered workers via the API: ```bash openstack share protection worker list ``` > **Note for reviewer:** The exact `openstack share protection worker list` sub-command shape is not confirmed in the source material. Verify the correct sub-command name against the installed `python-abacaclient` version. **Step 5 — Confirm FIPS KDF status after rotation** If the new Kopia build adds PBKDF2 support, check your backup targets for the `fips_kdf_gap` flag. Targets enrolled before the upgrade will retain their original KDF; new enrollments will use PBKDF2. ```bash openstack share protection target list openstack share protection target show ``` A target with `fips_kdf_gap: true` is still protected — it is using scrypt, which is cryptographically strong — but it is not using PBKDF2 as preferred by strict FIPS 140 interpretations. Re-enrolling the target (which re-initializes the Kopia repository) is the only way to change the KDF for an existing repository. ### Post-upgrade maintenance check After the control plane is running, confirm that the reconciliation loop is scheduling maintenance jobs for all enrolled targets. Overdue maintenance means retention policies are not being enforced: ```bash # List recent maintenance jobs openstack share protection job list --type maintenance ``` If maintenance is overdue, you can trigger it by allowing the reconciliation loop to run (it fires every `reconciliation_interval` seconds, default 300) or by waiting for the next scheduled tick. ## Examples ### Example 1 — Check migration status before upgrading Before starting the upgrade, confirm the current database revision to have a known baseline. ```bash abaca-manage db current ``` Expected output: ``` INFO [abaca.manage] Current revision: a1b2c3d4e5f6 (head) ``` > **Note for reviewer:** The actual revision hash is release-specific and not available in the source material. The `(head)` suffix indicates the database is at the most recent migration for the currently installed version. --- ### Example 2 — Full control plane upgrade sequence This example shows the complete sequence for a patch-level control plane upgrade with no Kopia repository format change. ```bash # 1. Confirm no active jobs openstack share protection job list --state transferring # (expect empty output before proceeding) # 2. Scale down control plane oc scale deployment/abaca-conductor --replicas=0 oc scale deployment/abaca-api --replicas=0 oc wait pod --for=delete -l 'app in (abaca-api,abaca-conductor)' --timeout=120s # 3. Run migrations oc run abaca-migrate \ --image=/abaca-api: \ --restart=Never \ --env-file= \ -- abaca-manage db upgrade oc wait pod/abaca-migrate --for=condition=Succeeded --timeout=300s oc logs abaca-migrate oc delete pod abaca-migrate # 4. Apply new images and scale up oc set image deployment/abaca-api abaca-api=/abaca-api: oc set image deployment/abaca-conductor abaca-conductor=/abaca-conductor: oc scale deployment/abaca-api --replicas=2 oc scale deployment/abaca-conductor --replicas=1 oc rollout status deployment/abaca-api oc rollout status deployment/abaca-conductor # 5. Verify API health openstack share protection job list ``` Expected final output (step 5): ``` +--------------------------------------+--------+-----------+---------------------+ | ID | Type | State | Created At | +--------------------------------------+--------+-----------+---------------------+ | ... | backup | available | ... | +--------------------------------------+--------+-----------+---------------------+ ``` --- ### Example 3 — Verify fleet image version after worker rotation After updating the worker image reference and allowing the conductor to cycle workers, confirm all active workers are running the new image. ```bash abaca-dev fleet status ``` Expected output (illustrative): ``` Worker Fleet Status =================== Total workers : 4 Active : 4 Dead : 0 Draining : 0 Worker ID AZ Image Version Slots Running Jobs -------------------------------------- ---- -------------- ----- ------------ az1 4 0 az1 4 1 az2 4 0 az2 4 0 ``` > **Note for reviewer:** The exact output format of `abaca-dev fleet status` is not confirmed in the source material; the above is illustrative. --- ### Example 4 — Inspect a backup target for FIPS KDF gap after fleet rotation ```bash openstack share protection target show 3fa85f64-5717-4562-b3fc-2c963f66afa6 ``` Expected output (illustrative, showing a target with a KDF gap): ``` +------------------------------+--------------------------------------------------+ | Field | Value | +------------------------------+--------------------------------------------------+ | id | 3fa85f64-5717-4562-b3fc-2c963f66afa6 | | name | production-target | | status | available | | fips_kdf_gap | true | | enrollment_status | enrolled | | ... | ... | +------------------------------+--------------------------------------------------+ ``` A `fips_kdf_gap: true` value means this repository was initialized with scrypt KDF because the Kopia binary at enrollment time did not support PBKDF2. The repository remains fully protected; this flag is informational for FIPS compliance auditing. ## Troubleshooting ### Migration fails with a database connection error **Symptom:** `abaca-manage db upgrade` exits immediately with a connection refused or authentication error. **Likely cause:** The migration pod or bastion cannot reach the database, or the database credentials in the environment are incorrect or missing. **Fix:** Confirm that the environment variables or ConfigMap injected into the migration pod include the correct database host, port, database name, and credentials. Verify network reachability from the migration pod to the MariaDB service within the cluster. Check that the database user has `ALTER TABLE` and `CREATE TABLE` privileges on the Abacá schema. --- ### API pods crash-loop after upgrade **Symptom:** `abaca-api` pods enter `CrashLoopBackOff` immediately after the upgraded Deployment is applied. **Likely cause:** The database migration was not run before scaling the services back up, causing the new code to encounter an older schema. **Fix:** Scale `abaca-api` back down to zero, run `abaca-manage db upgrade` successfully, then scale back up. Check pod logs for explicit migration-related errors: ```bash oc scale deployment/abaca-api --replicas=0 # run migration as shown in Installation Step 3 oc scale deployment/abaca-api --replicas= oc logs -l app=abaca-api --previous ``` --- ### Jobs remain stuck in a non-terminal state after conductor restart **Symptom:** After upgrading and restarting `abaca-conductor`, some jobs are stuck in `provisioning_network`, `transferring`, or another mid-flight state and never progress. **Likely cause:** The conductor was restarted while those jobs were in-flight. The reconciliation loop will reap workers whose heartbeat has lapsed and transition their jobs to `error`. **Fix:** Wait for one reconciliation interval (default 300 seconds). The reconciliation loop will call `reap_dead_workers` followed by `reap_stuck_jobs`, transitioning orphaned jobs to `error` with `error_code=abaca.worker_unavailable`. Inspect the failed jobs to determine whether they require tenant or operator action: ```bash openstack share protection job list --state error openstack share protection job show # Check the error_category field: # tenant_action_required — tenant must fix a configuration issue # operator_action_required — infrastructure or service needs attention ``` --- ### Worker VMs continue to boot with the old image after fleet rotation **Symptom:** `abaca-dev fleet status` shows workers still running the previous image version after the conductor configuration was updated. **Likely cause:** The conductor is still assigning jobs to existing workers from the old image because they are healthy and have available capacity slots. New workers are only provisioned when the pool needs to grow. **Fix:** To force rotation, reduce the desired fleet size to zero (draining existing workers) and then restore it. Alternatively, use `abaca-dev` to trigger explicit worker decommissioning. Workers in `DRAINING` state will not accept new jobs; they will be removed once their current jobs complete. --- ### Maintenance jobs are not running after upgrade **Symptom:** Backup targets show overdue maintenance; the `openstack share protection job list --type maintenance` output shows no recent successful maintenance jobs. **Likely cause:** The conductor's reconciliation loop may not have run yet after restart, or the `maintenance_interval_seconds` threshold has not elapsed since the last successful run. **Fix:** Wait for one reconciliation interval (default 300 seconds). If maintenance jobs remain absent, check conductor logs for errors in the `_sweep_target_maintenance` loop. Confirm that at least one healthy worker is registered and has available capacity slots, because maintenance jobs are dispatched to workers just like backup jobs. ```bash oc logs -l app=abaca-conductor | grep maintenance ``` --- ### `fips_kdf_gap` appears on all targets after upgrading Kopia **Symptom:** After rotating the worker fleet to a new Kopia build, existing backup targets still show `fips_kdf_gap: true`. **Likely cause:** The `fips_kdf_gap` flag reflects the KDF used when the Kopia repository was **initialized at enrollment**, not the current worker image. Existing repositories cannot change their KDF without re-initialization. **Fix:** This is expected behavior. New enrollments created after the fleet rotation will use PBKDF2 and will not show this flag. To remediate existing targets, you must delete and re-enroll them (which destroys the existing Kopia repository and all backups stored in it). Weigh this against your data retention requirements before proceeding.