Release Notes

This page documents the release history of TrilioVault for OpenStack, with detailed coverage of the changes introduced in each version. Version 3.0 is the primary focus: it delivers S3-compatible object storage as a backup target, automatic tenant network topology capture and restore, and a redesigned high-availability cluster architecture managed by Pacemaker and HAProxy. Reviewing these notes helps you understand what changed between versions, assess upgrade impact, and take advantage of new capabilities in your data protection workflows.

Before reviewing or applying these release notes, confirm that your environment meets the following requirements:

• OpenStack version: Stein or later
• Required OpenStack services: Keystone (identity), Nova (compute), Cinder (block storage)
• Optional OpenStack services: Horizon dashboard (required only for the Horizon plugin and File Recovery Manager UI)
• Python: 3.6 or later (required on nodes running the workloadmgr CLI or workloadmgrclient library)
• Host operating system: Linux with systemd
• TrilioVault deployment model: QCOW2 image instantiated on standalone KVM hosts that are outside the OpenStack cloud being protected — TrilioVault nodes must never run as VMs under the cloud they protect
• Backup target (at least one):
  • NFS share (supported since initial release)
  • S3-compatible object storage endpoint (new in 3.0)
  • OpenStack Swift
• Cluster IP / VIP: One additional cluster IP is required even for single-node deployments; Pacemaker and HAProxy bind to this VIP, and it is registered as the Keystone service endpoint
• Cinder quota headroom: Two Cinder snapshots plus one temporary Cinder volume per disk being backed up
• nova user consistency: The POSIX UID and GID of the nova user must be identical across all TrilioVault cluster nodes and all OpenStack compute nodes
• TrilioVault Trustee RBAC role: Must be assigned to any user performing workload, snapshot, or restore operations; the OpenStack admin role is not a substitute
• Configuration management (optional): Ansible, Salt, or Juju if using triliovault-cfg-scripts playbooks/states/charms for deployment

These release notes do not replace the installation guide, but the steps below summarize how to bring up a TrilioVault 3.0 cluster so you can exercise the new features described in this release.

Step 1 — Obtain the TrilioVault QCOW2 image

Download the TrilioVault 3.0 QCOW2 appliance image from your Trilio distribution channel and copy it to each KVM host that will run a TrilioVault node.

# Example: copy the image to the first KVM host
scp triliovault-3.0.qcow2 kvmhost1:/var/lib/libvirt/images/

Step 2 — Instantiate TrilioVault nodes on standalone KVM hosts

Instantiate the QCOW2 image as a VM on each KVM host. For a three-node HA cluster, repeat this on three separate KVM hosts. Do not run these VMs inside the OpenStack cloud you are protecting.

# On each KVM host — adjust memory, vCPUs, and disk to your sizing requirements
virt-install \
  --name triliovault-node1 \
  --ram 8192 \
  --vcpus 4 \
  --disk path=/var/lib/libvirt/images/triliovault-3.0.qcow2,format=qcow2 \
  --import \
  --os-variant rhel8.0 \
  --network bridge=br0

Step 3 — Clone the configuration scripts repository

Use triliovault-cfg-scripts to drive all subsequent configuration. The Ansible playbooks in this repository are idempotent and can be re-run at any time to adjust settings.

git clone https://github.com/triliodata/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts

Step 4 — Edit the inventory and variable files

Open the Ansible inventory file and the relevant group variables file. At minimum, set:

• IP addresses of all three TrilioVault nodes (or one node for a single-node deployment)
• The cluster VIP address (required even for single-node)
• OpenStack Keystone endpoint and credentials
• Backup target type and connection details

# inventory/hosts (example for a three-node cluster)
[tvault_nodes]
tvault-node1 ansible_host=192.168.10.11
tvault-node2 ansible_host=192.168.10.12
tvault-node3 ansible_host=192.168.10.13

[tvault_nodes:vars]
cluster_vip=192.168.10.10

Step 5 — Run the deployment playbook

ansible-playbook -i inventory/hosts site.yml

The playbook installs and starts all TrilioVault services, configures Pacemaker and HAProxy, and registers the wlm-api endpoint in Keystone under the cluster VIP.

Step 6 — Verify service status

After the playbook completes, confirm that cluster-controlled services are running on the master node and that wlm-workloads is running on every node.

# Run on the Pacemaker master node
sudo pcs status

# Expect wlm-api and wlm-scheduler to show as Started on the master node
# Expect wlm-workloads to show as Started on all nodes

Step 7 — Change the default dashboard password

Open the TrilioVault Dashboard in a browser, log in with the default credentials (admin / password), and change the password immediately.

Step 8 — Install the Horizon plugin (optional)

If you want graphical access through OpenStack Horizon, deploy the horizon-tvault-plugin on your Horizon nodes. Refer to the Horizon plugin installation guide for the full procedure.

Backup Target

The backup target is the storage location where all snapshot data is written. All read and write operations are performed as the nova user (nova:nova); the POSIX UID and GID of nova must match across every TrilioVault node and every OpenStack compute node.

S3 note (3.0): The S3 backup target uses a FUSE-based driver that supports incremental-forever backups, configurable retention policies, and mountable snapshots. Configure the S3 endpoint, credentials, and bucket in the Ansible group variables before running the configurator playbook.

High-Availability Cluster

TrilioVault 3.0 introduces a Pacemaker/HAProxy-managed HA cluster. Key architectural rules:

• wlm-api and wlm-scheduler are cluster-controlled resources; Pacemaker runs them on the master node only.
• wlm-workloads runs on every node in the cluster to distribute data-movement work.
• wlm-cron fires scheduled snapshot jobs according to the policies defined for each workload.
• The cluster VIP is mandatory in all deployment sizes. It is the address registered as the Keystone workloadmgr service endpoint so that clients always reach the active master regardless of which physical node currently holds that role.

Snapshot Policy

Snapshot policies are attached per workload and control automatic backup frequency and retention.

Cinder Quota Requirements

During each backup TrilioVault creates Cinder snapshots and a temporary Cinder volume. Your OpenStack project quotas must allow two Cinder snapshots and one temporary Cinder volume per disk being backed up at any given time. Insufficient quota causes backup jobs to fail.

TrilioVault Trustee RBAC Role

The TrilioVault Trustee role must be assigned to any user or service account that creates workloads, triggers snapshots, or initiates restores. The OpenStack admin role does not grant these permissions and cannot be used as a substitute.

Email Notifications

Email alerts are configured per tenant. Requirements:

• The OpenStack user account must have a valid email address.
• An SMTP server must be configured by the backup administrator. Both conditions must be met; partial configuration produces no alerts.

File Recovery Manager Glance Image

For file-level restores through the Horizon UI, the File Recovery Manager Glance image must have the custom property tvault_recovery_manager=yes. Without this property the image does not appear as a valid mount target in the interface.

# Set the required property on the File Recovery Manager image
openstack image set --property tvault_recovery_manager=yes <image-id>

Creating a workload and taking your first snapshot

A workload is the fundamental unit of protection in TrilioVault — it groups the VMs that belong to a single application so they are snapshotted and restored consistently. Each VM may belong to only one workload at a time.

Use the workloadmgr CLI (provided by the workloadmgr-client package) to create a workload:

workloadmgr workload-create \
  --name "my-app-workload" \
  --instances <vm-uuid-1>,<vm-uuid-2> \
  --description "Production web tier"

Once the workload exists, take an on-demand snapshot:

workloadmgr snapshot-create <workload-id> --name "pre-upgrade-snapshot"

List snapshots to confirm success:

workloadmgr snapshot-list --workload_id <workload-id>

Attaching a snapshot policy

Attach a policy to automate recurring snapshots. wlm-cron reads the schedule and fires jobs at the defined interval; older snapshots beyond the retention count are automatically deleted.

workloadmgr workload-modify <workload-id> \
  --fullbackup_interval 7 \
  --retention_policy_value 30

Restoring a full workload

To restore all VMs and volumes from a snapshot:

workloadmgr workload-restore <snapshot-id> \
  --restore_type restore_to_new \
  --description "Restore after failed upgrade"

Performing a file-level restore

When you only need to recover one or a few files, a file-level restore avoids recreating entire VMs. Initiate the restore via the Horizon dashboard (Workloads → Snapshots → File Recovery) or through the API. The File Recovery Manager Glance image must have tvault_recovery_manager=yes set before you begin.

Restoring tenant network topology (3.0)

Starting in version 3.0, every snapshot automatically captures the full tenant network topology — networks, subnets, routers, static routes, ports, and floating IPs — without any action on your part. When restoring:

1. Ensure the tenant's network space is completely empty — no existing networks, subnets, routers, or ports. The restore fails on any conflict.
2. Initiate the restore normally; TrilioVault recreates the captured network topology.
3. Manually reconnect the restored private networks to the public network after the restore completes. TrilioVault does not perform this final step automatically.

Using an S3 backup target (3.0)

After configuring an S3 backup target (see Configuration), your workloads automatically use incremental-forever backups to S3. Snapshots stored on S3 are mountable, meaning you can attach them as block devices to perform file-level restores without a full workload restore.

Monitoring through the Horizon plugin

If horizon-tvault-plugin is installed, a Workloads panel appears in the Horizon dashboard. From there you can create workloads, view snapshot history, monitor job status, and launch restores without using the CLI.

Example 1 — Create a workload with two VMs

Create a workload that protects a two-VM application (web server and database).

workloadmgr workload-create \
  --name "webapp-production" \
  --instances d4e5f6a7-0001-0001-0001-000000000001,d4e5f6a7-0002-0002-0002-000000000002 \
  --description "Web + DB tier"

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | a1b2c3d4-aaaa-bbbb-cccc-ddddeeeeffffgg|
| name        | webapp-production                    |
| status      | available                            |
| description | Web + DB tier                        |
+-------------+--------------------------------------+

Example 2 — Take an on-demand snapshot

Trigger an immediate snapshot of the workload created above, for example before a risky change.

workloadmgr snapshot-create a1b2c3d4-aaaa-bbbb-cccc-ddddeeeeffffgg \
  --name "pre-patch-20240601"

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | 55667788-1111-2222-3333-444455556666 |
| name        | pre-patch-20240601                   |
| status      | executing                            |
| workload_id | a1b2c3d4-aaaa-bbbb-cccc-ddddeeeeffffgg|
+-------------+--------------------------------------+

Poll until status changes to available:

workloadmgr snapshot-show 55667788-1111-2222-3333-444455556666

Example 3 — List snapshots for a workload

workloadmgr snapshot-list --workload_id a1b2c3d4-aaaa-bbbb-cccc-ddddeeeeffffgg

Expected output:

+--------------------------------------+--------------------+-----------+---------------------+
| id                                   | name               | status    | created_at          |
+--------------------------------------+--------------------+-----------+---------------------+
| 55667788-1111-2222-3333-444455556666 | pre-patch-20240601 | available | 2024-06-01T10:30:00 |
+--------------------------------------+--------------------+-----------+---------------------+

Example 4 — Restore a full workload to a new environment

Restore the snapshot to a new OpenStack project after a failure.

workloadmgr workload-restore 55667788-1111-2222-3333-444455556666 \
  --restore_type restore_to_new \
  --description "DR restore 2024-06-01"

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | aabbccdd-9999-8888-7777-666655554444 |
| snapshot_id | 55667788-1111-2222-3333-444455556666 |
| status      | executing                            |
| type        | restore_to_new                       |
+-------------+--------------------------------------+

Example 5 — Tag the File Recovery Manager image for Horizon file restores

Before performing file-level restores through the Horizon UI, ensure the Glance image has the required property.

openstack image set \
  --property tvault_recovery_manager=yes \
  <file-recovery-manager-image-id>

Verify:

openstack image show <file-recovery-manager-image-id> | grep tvault_recovery_manager

Expected output:

| properties | tvault_recovery_manager='yes', ... |

Example 6 — Verify HA cluster status after deployment

After running the triliovault-cfg-scripts Ansible playbook, confirm that Pacemaker has started wlm-api and wlm-scheduler on the master node and wlm-workloads on all nodes.

sudo pcs status

Expected output (three-node cluster):

Cluster name: triliovault
Stack: corosync
Current DC: tvault-node1 (version ...) - partition with quorum
Last updated: ...

3 nodes configured
5 resources configured

Online: [ tvault-node1 tvault-node2 tvault-node3 ]

Full list of resources:

 wlm-api        (systemd:wlm-api):        Started tvault-node1
 wlm-scheduler  (systemd:wlm-scheduler):  Started tvault-node1
 wlm-workloads  (systemd:wlm-workloads):  Started tvault-node1 tvault-node2 tvault-node3
 cluster_vip    (ocf::heartbeat:IPaddr2):  Started tvault-node1

Use the following log files when diagnosing issues:

Issue 1 — Snapshot job fails immediately with a Cinder quota error

Symptom: A snapshot transitions to error status shortly after creation. The wlm-workloads log contains a message referencing Cinder quota exceeded.

Likely cause: The OpenStack project does not have enough Cinder quota. TrilioVault requires two Cinder snapshots and one temporary Cinder volume per disk being backed up during a single job. If other snapshots or volumes already exist, the quota may be exhausted.

Fix:

# Check current quota usage
openstack quota show --detail <project-id>

# Raise the snapshots and volumes quota as needed
openstack quota set --snapshots <new-limit> --volumes <new-limit> <project-id>

After adjusting quotas, re-trigger the snapshot.

Issue 2 — Network topology restore fails with a conflict error

Symptom: A restore that includes network topology fails with an error indicating a networking component already exists.

Likely cause: Version 3.0 automatically captures and restores tenant network topology. The restore engine requires the tenant's network space to be completely empty — any pre-existing network, subnet, router, port, or floating IP causes the restore to abort.

Fix:

1. List and delete all networking components in the target tenant:

openstack router list --project <project-id>
openstack router delete <router-id>
openstack network list --project <project-id>
openstack network delete <network-id>

2. Confirm the network space is empty, then re-initiate the restore.
3. After the restore completes, manually reconnect the restored private networks to the public network — TrilioVault does not perform this step automatically.

Issue 3 — File Recovery Manager image does not appear in Horizon restore UI

Symptom: When attempting a file-level restore through the Horizon Workloads panel, the File Recovery Manager image is not listed as a valid mount target.

Likely cause: The Glance image is missing the tvault_recovery_manager=yes property.

Fix:

openstack image set --property tvault_recovery_manager=yes <image-id>

Refresh the Horizon page; the image should now appear.

Issue 4 — wlm-api or wlm-scheduler is not running after node failover

Symptom: After a node failure or reboot, API calls return connection errors. pcs status shows wlm-api or wlm-scheduler in a stopped or failed state.

Likely cause: wlm-api and wlm-scheduler are cluster-controlled by Pacemaker and run only on the master node. If Pacemaker has not yet elected a new master, or if a resource agent is in a failed state, these services will be offline.

Fix:

# Check cluster and resource status
sudo pcs status

# Clear a failed resource state and allow Pacemaker to restart it
sudo pcs resource cleanup wlm-api
sudo pcs resource cleanup wlm-scheduler

# Verify the cluster VIP has migrated to the new master
sudo pcs status | grep cluster_vip

If the cluster has lost quorum (two or more nodes down in a three-node cluster), restore the failed nodes before attempting resource recovery.

Issue 5 — Backup or restore operations fail with permission errors on the backup target

Symptom: The wlm-workloads log reports permission denied errors when reading from or writing to the NFS share or S3 bucket.

Likely cause: All backup target I/O runs as the nova user (nova:nova). If the POSIX UID or GID of nova differs between TrilioVault nodes and OpenStack compute nodes, writes succeed on some nodes and fail on others.

Fix:

1. Check the nova UID/GID on each node:

id nova

2. Reconcile any mismatches using your configuration management tooling (for example, re-run the triliovault-cfg-scripts Ansible playbook with corrected nova_uid and nova_gid variables).
3. For NFS targets, also verify that the NFS export options grant write access to the nova UID.

Issue 6 — Workload operations return a 403 Forbidden error

Symptom: CLI or API calls for workload creation, snapshot, or restore return HTTP 403 even though the user has the OpenStack admin role.

Likely cause: The TrilioVault Trustee RBAC role is required for all workload, snapshot, and restore operations. The OpenStack admin role does not substitute for it.

Fix:

# Assign the TrilioVault Trustee role to the user
openstack role add --user <username> --project <project-name> "TrilioVault Trustee"

Retry the operation after the role assignment takes effect.

Issue 7 — Scheduled snapshots are not firing

Symptom: Workloads with a snapshot policy attached do not produce automatic snapshots at the expected interval.

Likely cause: wlm-cron is the service responsible for firing scheduled snapshot jobs. If it is stopped or failed, no scheduled backups execute.

Fix:

# Check wlm-cron service status on the master node
sudo systemctl status wlm-cron

# If stopped, start it
sudo systemctl start wlm-cron

# Review logs for scheduling errors
tail -n 100 /var/log/workloadmgr/workloadmgr-api.log

Also confirm that the workload's snapshot policy has a valid schedule and retention configuration by running:

workloadmgr workload-show <workload-id>