Upgrading TrilioVault

This guide walks you through upgrading TrilioVault for OpenStack, covering the TVM appliance nodes, the wlm-api, wlm-scheduler, and wlm-workloads services, the datamover (tvault-contego), and the horizon-tvault-plugin. Keeping TrilioVault current ensures you benefit from new backup target support, network topology capture improvements, and bug fixes without disrupting your existing workloads or snapshot schedules. Because TrilioVault nodes run as QCOW2-based KVM appliances outside your OpenStack cloud, the upgrade path combines appliance image replacement with in-place package updates and configuration management scripts from triliovault-cfg-scripts. Following this procedure in the correct order protects snapshot continuity and prevents service interruptions across your cluster.

Before you begin the upgrade, confirm that every item below is satisfied:

• Current TrilioVault version: You know the installed version. Consult the TrilioVault release notes for the target version to identify any mandatory intermediate upgrade steps.
• OpenStack version: OpenStack Stein or later is installed and healthy.
• Cluster topology known: You know whether you are running a single-node or three-node HA cluster and which node is the current Pacemaker master.
• VIP confirmed: The cluster virtual IP (VIP) is reachable and registered as the Keystone service endpoint. This VIP is required even on single-node deployments.
• Backup target accessible: Your NFS, S3-compatible, or Swift backup target is mounted and reachable from all TrilioVault nodes.
• nova user UID/GID consistent: The POSIX UID and GID of the nova user are identical across all TrilioVault cluster nodes and all compute nodes, because all backup-target I/O runs as nova:nova.
• Cinder quota headroom: Each disk being backed up requires two Cinder snapshots and one temporary Cinder volume. Confirm quotas can absorb this during any snapshot jobs that may run during the maintenance window.
• TrilioVault Trustee RBAC role intact: The Trustee role must remain assigned throughout the upgrade; do not remove it.
• triliovault-cfg-scripts repository available: Ansible playbooks, Salt states, or Juju charms from triliovault-cfg-scripts are accessible on your upgrade host.
• Ansible installed (if using the Ansible upgrade path): Ansible is installed on the host from which you run playbooks.
• Maintenance window scheduled: Notify tenant users. wlm-api and wlm-scheduler are Pacemaker-controlled and will be briefly unavailable on non-master nodes during the upgrade.
• Snapshots not actively running: Verify no snapshot or restore jobs are in progress before starting. Check via the workloadmgr CLI or the Horizon Workloads panel.

The upgrade is performed in four stages: (1) upgrade the TVM appliance nodes, (2) upgrade the datamover on compute nodes, (3) upgrade the horizon-tvault-plugin on the Horizon host, and (4) verify the cluster. Perform stages sequentially; do not proceed to the next stage until the current one is confirmed healthy.

Stage 1 — Upgrade the TVM appliance nodes

TrilioVault ships as a QCOW2 image and runs on standalone KVM hosts outside the OpenStack cloud. Upgrading the appliance involves replacing the running image and re-applying configuration.

1. Place the cluster in maintenance mode

Log in to the TrilioVault Dashboard (default credentials: admin / password; change immediately if still default) and enable maintenance mode to pause wlm-cron from firing new scheduled snapshot jobs.

2. Identify the Pacemaker master node

ssh tvault-node-1 "crm_mon -1 --simple-status"

Note which node currently holds the master resource. wlm-api and wlm-scheduler run only on the master node; wlm-workloads runs on every node.

3. Stop TrilioVault services on a non-master node first

Upgrade non-master nodes first to minimize impact. On each non-master node:

ssh tvault-node-2
sudo systemctl stop wlm-workloads

4. Replace the QCOW2 appliance image

Shut down the KVM guest, swap in the new QCOW2 image on the KVM host, and boot the new image. The exact KVM host commands depend on your hypervisor management tooling. After boot, confirm the node is reachable:

ssh tvault-node-2 "workloadmgr --version"

5. Re-apply configuration using triliovault-cfg-scripts

Run the Ansible upgrade playbook from the triliovault-cfg-scripts repository against the newly booted node. Because TrilioVault configuration is idempotent, re-running the playbook is safe:

cd /path/to/triliovault-cfg-scripts
ansible-playbook -i inventory/hosts upgrade.yml --limit tvault-node-2

Replace upgrade.yml with the equivalent Salt state or Juju charm action if you use those configuration managers instead.

6. Verify the upgraded non-master node

ssh tvault-node-2 "sudo systemctl status wlm-workloads"

Expected: active (running).

7. Repeat for remaining non-master nodes

Repeat steps 3–6 for each remaining non-master node before touching the master.

8. Migrate the Pacemaker master resource

Once all non-master nodes are upgraded, move the master role to an already-upgraded node:

ssh tvault-node-1 "crm resource migrate wlm-master tvault-node-2"

Wait for wlm-api and wlm-scheduler to start on the new master:

ssh tvault-node-2 "sudo systemctl status wlm-api wlm-scheduler"

9. Upgrade the former master node

With the master role migrated, upgrade the original master node (now a non-master) by repeating steps 3–6 for it.

10. Remove the migration constraint

Allow Pacemaker to rebalance freely again:

ssh tvault-node-2 "crm resource unmigrate wlm-master"

Stage 2 — Upgrade the datamover on compute nodes

The datamover (tvault-contego) runs on each OpenStack compute node and performs the actual volume read operations during snapshot capture.

11. Run the datamover upgrade playbook

From the triliovault-cfg-scripts repository, target all compute nodes:

ansible-playbook -i inventory/hosts upgrade-datamover.yml

12. Verify the datamover on each compute node

ansible compute_nodes -i inventory/hosts -m shell \
  -a "systemctl status tvault-contego"

All nodes should report active (running). Check logs at /var/log/nova/tvault-contego.log if any node fails.

Stage 3 — Upgrade the Horizon plugin

The horizon-tvault-plugin adds the Workloads panel to OpenStack Horizon. It is installed on the Horizon host, not on TrilioVault nodes.

13. Run the Horizon plugin upgrade playbook

ansible-playbook -i inventory/hosts upgrade-horizon-plugin.yml \
  --limit horizon_hosts

14. Restart the Horizon web server

ssh horizon-host "sudo systemctl restart apache2"

Adjust the web server unit name (nginx, httpd, etc.) to match your Horizon deployment.

15. Confirm the Workloads panel loads

Log in to the Horizon dashboard and navigate to Project → Workloads. The panel should load without errors.

Stage 4 — Post-upgrade verification

16. Check all cluster services

ansible triliovault_nodes -i inventory/hosts -m shell \
  -a "systemctl is-active wlm-workloads"

On the master node only, also verify:

ssh tvault-master "systemctl is-active wlm-api wlm-scheduler wlm-cron"

17. Take a test snapshot

Using the workloadmgr CLI, trigger a manual snapshot of a non-critical workload to confirm end-to-end function:

workloadmgr snapshot-create <workload-id>

18. Disable maintenance mode

Return to the TrilioVault Dashboard and disable maintenance mode to resume scheduled snapshot jobs.

Most upgrade-time configuration is re-applied automatically by the triliovault-cfg-scripts Ansible playbooks (or equivalent Salt/Juju scripts). The following options are the ones most likely to require review or adjustment when upgrading between major versions.

Because the Ansible-based configuration is idempotent, you can safely re-run the playbook after changing any of these values to push updates to the cluster without side effects.

After upgrading, the day-to-day workloadmgr CLI usage is identical to the pre-upgrade experience. The following patterns cover the actions most operators perform immediately after an upgrade.

Verify the installed version

Confirm that wlm-api is reporting the expected version:

workloadmgr api-version

List all workloads to confirm continuity

Existing workloads and their associated snapshot policies are preserved across upgrades. Verify they are visible:

workloadmgr workload-list

List snapshots for a specific workload

Check that historical snapshots taken before the upgrade are still accessible:

workloadmgr snapshot-list --workload_id <workload-id>

Trigger a manual snapshot after upgrade

Before re-enabling scheduled policies, take a manual snapshot to establish a clean post-upgrade baseline:

workloadmgr snapshot-create <workload-id>

The wlm-scheduler assigns the job to the appropriate node, and wlm-workloads executes the data-movement. Monitor progress:

workloadmgr snapshot-show <snapshot-id>

Resume scheduled snapshot policies

If you paused wlm-cron via maintenance mode during the upgrade, disable maintenance mode through the TrilioVault Dashboard. Scheduled jobs defined in each workload's snapshot policy will resume automatically on their next trigger interval.

Re-enable the Horizon Workloads panel for users

Inform tenant users that the horizon-tvault-plugin is updated. Users can log in to Horizon, navigate to Project → Workloads, and verify their workloads, snapshots, and restore points are intact.

Validate file-level restore capability

If you use file-level restores, confirm the File Recovery Manager Glance image is present and tagged correctly:

openstack image show <file-recovery-manager-image-id> | grep tvault_recovery_manager

Expected output:

| tvault_recovery_manager | yes |

Example 1 — Check service health on all nodes after upgrade

Run from your Ansible control host to confirm wlm-workloads is active on every TrilioVault node:

ansible triliovault_nodes -i inventory/hosts -m shell \
  -a "systemctl is-active wlm-workloads"

Expected output (one line per node):

tvault-node-1 | CHANGED | rc=0 >>
active
tvault-node-2 | CHANGED | rc=0 >>
active
tvault-node-3 | CHANGED | rc=0 >>
active

Example 2 — Re-register the Keystone service endpoint after appliance replacement

If the Keystone endpoint is lost during node replacement, re-run the configuration playbook. The playbook is idempotent and will register the endpoint without duplicating it:

cd /path/to/triliovault-cfg-scripts
ansible-playbook -i inventory/hosts configure-keystone.yml

Verify registration:

openstack endpoint list --service workloadmgr

Expected output includes a row with the VIP address:

+------------------+-----------+--------------+--------------+---------+-----------+----------------------------------+
| ID               | Region    | Service Name | Service Type | Enabled | Interface | URL                              |
+------------------+-----------+--------------+--------------+---------+-----------+----------------------------------+
| <endpoint-id>    | RegionOne | workloadmgr  | workloadmgr  | True    | public    | http://<VIP>:8780/v1/%(tenant_id)s |
+------------------+-----------+--------------+--------------+---------+-----------+----------------------------------+

Example 3 — Trigger a post-upgrade baseline snapshot and monitor it

# Trigger the snapshot
workloadmgr snapshot-create 4a2b9c1d-0000-0000-0000-aabbccddeeff

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | 7f3e2a0c-1111-2222-3333-445566778899 |
| status      | available                            |
| workload_id | 4a2b9c1d-0000-0000-0000-aabbccddeeff |
+-------------+--------------------------------------+

Poll until the snapshot completes:

workloadmgr snapshot-show 7f3e2a0c-1111-2222-3333-445566778899

When the status field reads available, the snapshot is complete and stored on the backup target.

Example 4 — Confirm nova user UID/GID consistency after image replacement

Run from the Ansible control host to verify that nova has the same UID and GID across all TrilioVault and compute nodes:

ansible all -i inventory/hosts -m shell -a "id nova"

Expected output (all nodes must show identical UID and GID values):

tvault-node-1 | CHANGED | rc=0 >>
uid=162(nova) gid=162(nova) groups=162(nova)
tvault-node-2 | CHANGED | rc=0 >>
uid=162(nova) gid=162(nova) groups=162(nova)
compute-node-1 | CHANGED | rc=0 >>
uid=162(nova) gid=162(nova) groups=162(nova)

If any node shows a different UID or GID, correct it before allowing backup jobs to run, as backup-target I/O runs as nova:nova and mismatched IDs will cause permission errors.

Use the log locations below as your first reference for any issue:

• wlm-api: /var/log/workloadmgr/workloadmgr-api.log
• wlm-workloads: /var/log/workloadmgr/workloadmgr-workloads.log
• datamover (tvault-contego): /var/log/nova/tvault-contego.log
• Nova API extension: /var/log/nova/nova-api.log

Issue 1 — wlm-api or wlm-scheduler does not start after upgrade

Symptom: systemctl status wlm-api or systemctl status wlm-scheduler shows failed or inactive on the master node after appliance replacement.

Likely cause: These services are Pacemaker-controlled and run only on the master node. If Pacemaker has not yet elected the upgraded node as master, or if the Pacemaker resource configuration was not updated by the playbook, the services will not start via systemctl directly.

Fix: Check Pacemaker resource status and, if needed, re-run the configuration playbook:

crm_mon -1 --simple-status
ansible-playbook -i inventory/hosts configure-pacemaker.yml

Issue 2 — Snapshot jobs fail immediately after upgrade with permission errors on the backup target

Symptom: Snapshots transition to error status. /var/log/workloadmgr/workloadmgr-workloads.log contains Permission denied errors referencing the backup target path.

Likely cause: The new QCOW2 appliance image shipped with a different default UID or GID for the nova user, causing a mismatch between TrilioVault nodes and compute nodes. All backup-target I/O runs as nova:nova.

Fix: Audit and align the nova UID/GID across all nodes (see Example 4), then restart wlm-workloads:

sudo systemctl restart wlm-workloads

Issue 3 — Horizon Workloads panel is missing or blank after upgrading the plugin

Symptom: After upgrading horizon-tvault-plugin and restarting the web server, the Workloads panel does not appear in Horizon, or it appears but shows a blank page.

Likely cause: The web server was not fully restarted, or the static assets were not collected, leaving Horizon serving stale plugin files.

Fix: Re-collect static assets and restart the web server:

ssh horizon-host "sudo python manage.py collectstatic --noinput"
ssh horizon-host "sudo systemctl restart apache2"

Adjust the web server unit and manage.py path to match your Horizon installation.

Issue 4 — File-level restore mount target does not appear in Horizon after upgrade

Symptom: When initiating a file-level restore, the File Recovery Manager image is not listed as a valid mount target in the Horizon UI.

Likely cause: The upgraded or replaced File Recovery Manager Glance image is missing the required tvault_recovery_manager=yes property.

Fix: Apply the property to the image:

openstack image set --property tvault_recovery_manager=yes <file-recovery-manager-image-id>

Issue 5 — Workload or snapshot operations return authorization errors after upgrade

Symptom: CLI or Horizon operations on workloads, snapshots, or restores return HTTP 403 or an authorization error message.

Likely cause: The TrilioVault Trustee RBAC role was inadvertently removed from affected users during the upgrade, or the Keystone service endpoint was re-registered with an incorrect policy file.

Fix: Confirm the Trustee role is assigned:

openstack role assignment list --user <username> --project <project>

If the TrilioVaultTrustee role is absent, re-assign it. Remember: the OpenStack admin role cannot substitute for the Trustee role.

openstack role add --user <username> --project <project> TrilioVaultTrustee

Issue 6 — Datamover (tvault-contego) inactive on one or more compute nodes after upgrade

Symptom: ansible compute_nodes -m shell -a "systemctl is-active tvault-contego" returns inactive or failed for some nodes. Snapshot jobs targeting volumes on those compute nodes fail.

Likely cause: The datamover upgrade playbook did not complete successfully on those nodes, or a dependency package conflict occurred.

Fix: Re-run the datamover upgrade playbook limited to the affected nodes, then check the log:

ansible-playbook -i inventory/hosts upgrade-datamover.yml --limit <compute-node-hostname>
tail -100 /var/log/nova/tvault-contego.log