Deployment Guide

This guide walks you through the complete deployment lifecycle for TrilioVault for OpenStack, from environment preparation through component installation and post-install validation. Following these steps ensures that the Workload Manager service (workloadmgr) and its associated components are correctly registered with Keystone, connected to your backup target, and ready to protect your OpenStack workloads. Whether you are deploying on a generic OpenStack distribution or on Red Hat OpenStack Platform (RHOSP) via Red Hat Director, this guide provides the exact sequence of operations you need.

Before you begin, confirm that your environment satisfies every requirement listed below. Attempting to install TrilioVault without meeting these prerequisites is the most common cause of failed deployments.

OpenStack platform requirements

• OpenStack Stein or later (upstream or a supported distribution; see the support matrix in the Installation section)
• OpenStack Keystone — required for authentication and service endpoint registration
• OpenStack Nova — required for VM and compute integration
• OpenStack Cinder — required for volume snapshot and temporary-volume operations
• OpenStack Horizon (optional) — required only if you want the horizon-tvault-plugin graphical interface

Infrastructure requirements

• One or more dedicated KVM hosts (bare-metal or VM hosts that are not managed by the OpenStack cloud you intend to protect) — TrilioVault nodes must run outside the cloud they protect
• A cluster VIP (virtual IP address) even for single-node deployments; Pacemaker and HAProxy bind to this address, and it is registered as the Keystone service endpoint
• Three-node cluster recommended for production; single-node is supported for non-production use
• All TrilioVault cluster nodes and all OpenStack compute nodes must share the same POSIX UID and GID for the nova user (nova:nova)

Backup target (choose one)

• NFS share reachable from all TrilioVault nodes and compute nodes
• S3-compatible object storage endpoint (available from TrilioVault 3.0 onward)
• OpenStack Swift endpoint

Quota headroom in Cinder

• For every disk being backed up, Cinder quotas must allow for two additional Cinder snapshots and one additional temporary Cinder volume

Software and credentials

• Python ≥ 3.6 on any host used to run the workloadmgr CLI or the workloadmgrclient Python library
• OpenStack admin credentials (Keystone admin project) for service and endpoint registration
• The TrilioVault Trustee RBAC role — this role is mandatory and cannot be replaced by the OpenStack admin role for workload, snapshot, and restore operations
• The TrilioVault QCOW2 appliance image (obtain from Trilio)
• (Optional) Ansible, Salt, or Juju if you plan to use the triliovault-cfg-scripts automation path

Connectivity

• TrilioVault nodes must have IP connectivity to: the OpenStack API endpoints (Keystone, Nova, Cinder, Glance, Neutron), the backup target, and the OpenStack compute node management network

Support matrix

Before you install, verify that your OpenStack distribution and storage backend are supported.

Step 1 — Prepare your KVM hosts

Provision one or three dedicated KVM hosts. These hosts must not be virtual machines running under the OpenStack cloud you intend to protect.

1. Install a supported Linux distribution with KVM and libvirt on each host.
2. Ensure each host can reach the OpenStack API endpoints and the backup target.
3. Confirm that the nova user has the same POSIX UID and GID on every host you will use as a TrilioVault node and on every OpenStack compute node:

# Run on each TrilioVault node and each compute node
id nova
# Expected output (UIDs/GIDs must match across all nodes):
# uid=162(nova) gid=162(nova) groups=162(nova)

If the UIDs or GIDs differ, align them before continuing. All read and write operations on the backup target run as nova:nova; mismatched IDs cause permission errors during snapshot and restore operations.

4. Reserve a cluster VIP (virtual IP address) on the management network. Even for a single-node deployment, this VIP is required — Pacemaker and HAProxy bind to it, and it will be registered as the Keystone wlm-api endpoint.

Step 2 — Deploy the TrilioVault appliance from the QCOW2 image

TrilioVault ships as a QCOW2 disk image. Repeat this step on each node in your cluster.

# Copy the QCOW2 image to the KVM host
scp triliovault-<version>.qcow2 root@<kvm-host>:/var/lib/libvirt/images/

# Create the VM (adjust vCPU, RAM, and disk to your sizing requirements)
virt-install \
  --name triliovault-node1 \
  --ram 16384 \
  --vcpus 8 \
  --disk path=/var/lib/libvirt/images/triliovault-<version>.qcow2,format=qcow2,bus=virtio \
  --import \
  --os-variant rhel7 \
  --network bridge=br-mgmt,model=virtio \
  --network bridge=br-data,model=virtio \
  --noautoconsole

Once the VM boots, access the TrilioVault Dashboard at http://<node-ip>:8080. The default credentials are admin / password. You must change the password on first login.

Step 3 — Configure the cluster (Ansible path — recommended)

The triliovault-cfg-scripts component provides Ansible playbooks that automate cluster formation, Pacemaker/HAProxy setup, and OpenStack integration. Using Ansible is the recommended path because the playbooks are idempotent — you can re-run them at any time to change or reconfigure the cluster.

# Clone the configuration scripts on your Ansible control node
git clone https://github.com/triliodata/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts

# Copy and edit the inventory file
cp inventory/sample.ini inventory/production.ini
$EDITOR inventory/production.ini

At minimum, set the following in your inventory:

[tvault_nodes]
triliovault-node1 ansible_host=<node1-ip>
triliovault-node2 ansible_host=<node2-ip>
triliovault-node3 ansible_host=<node3-ip>

[tvault_nodes:vars]
vip=<cluster-vip>
openstack_auth_url=http://<keystone-host>:5000/v3
openstack_admin_user=admin
openstack_admin_password=<admin-password>
openstack_admin_project=admin
backup_target_type=nfs          # nfs | s3 | swift
nfs_export=<nfs-host>:<export-path>  # if backup_target_type=nfs

Run the deployment playbook:

ansible-playbook -i inventory/production.ini site.yml

The playbook will:

• Configure Pacemaker to manage the VIP and the wlm-api and wlm-scheduler services on the elected master node
• Start wlm-workloads on every node
• Start wlm-cron on the cluster
• Register the workloadmgr service and its endpoints in Keystone

Step 4 — Install TrilioVault on OpenStack controller nodes (generic path)

If you are not using Ansible, or if you need to manually install the Nova compute extension and the API components on your OpenStack controller and compute nodes, follow this path.

4a — Install on controller nodes

# On each OpenStack controller node
pip3 install workloadmgrclient

# Register the WorkloadMgr service in Keystone
openstack service create \
  --name workloadmgr \
  --description "TrilioVault Workload Manager" \
  workloadmgr

# Create endpoints (replace <vip> with your cluster VIP)
openstack endpoint create \
  --region RegionOne \
  workloadmgr public http://<vip>:8780/v1/%(tenant_id)s

openstack endpoint create \
  --region RegionOne \
  workloadmgr internal http://<vip>:8780/v1/%(tenant_id)s

openstack endpoint create \
  --region RegionOne \
  workloadmgr admin http://<vip>:8780/v1/%(tenant_id)s

4b — Create the Trustee role and assign it

The Trustee RBAC role is mandatory. Do not attempt to substitute the OpenStack admin role.

# Create the role
openstack role create Trustee

# Assign it to the user who will manage workloads
openstack role add \
  --project <tenant-project> \
  --user <tenant-user> \
  Trustee

4c — Install the compute extension on each compute node

# On each OpenStack compute node
pip3 install tvault-contego

# Restart nova-compute to load the extension
systemctl restart openstack-nova-compute

Step 5 — Install on RHOSP10 / Red Hat Director path

For Red Hat OpenStack Platform 10 deployments managed by Red Hat Director (TripleO), use the Heat environment files and puppet modules included in triliovault-cfg-scripts rather than the generic pip path.

# On the Director (undercloud) node
cd triliovault-cfg-scripts/rhosp

# Copy the TrilioVault environment file to your templates directory
cp environments/tvault-environment.yaml /home/stack/templates/

# Edit the environment file to match your deployment
$EDITOR /home/stack/templates/tvault-environment.yaml

Key parameters to set in tvault-environment.yaml:

parameter_defaults:
  TvaultVip: <cluster-vip>
  TvaultBackupTargetType: nfs      # nfs | s3 | swift
  TvaultNfsExport: <nfs-host>:<export-path>
  TvaultKeystoneAdminPassword: <admin-password>

Re-run the overcloud deployment including the TrilioVault environment:

openstack overcloud deploy \
  --templates \
  -e /home/stack/templates/tvault-environment.yaml \
  [... your other -e flags ...]

Red Hat Director will distribute the TrilioVault compute extension to all compute nodes and register the workloadmgr service in Keystone automatically.

Step 6 — Install the Horizon dashboard plugin (optional)

If your OpenStack deployment includes Horizon and you want the graphical Workloads panel, install the horizon-tvault-plugin on each Horizon node.

# On each Horizon (controller) node
pip3 install horizon-tvault-plugin

# Restart the web server to load the plugin
systemctl restart httpd        # RHEL/CentOS
# or
systemctl restart apache2      # Ubuntu/Debian

After restarting, log in to Horizon and verify that a Workloads panel appears in the Project menu.

Step 7 — Upload the File Recovery Manager image to Glance

File-level restores require the File Recovery Manager (FRM) image to be available in Glance with a specific property. Without this property, the image will not appear as a valid mount target in the Horizon UI.

# Upload the FRM image
openstack image create \
  --file tvault-file-recovery-manager-<version>.qcow2 \
  --disk-format qcow2 \
  --container-format bare \
  --property tvault_recovery_manager=yes \
  --public \
  "TrilioVault File Recovery Manager"

# Verify the property is set
openstack image show "TrilioVault File Recovery Manager" \
  -c properties

Expected output:

+------------+-------------------------------------+
| Field      | Value                               |
+------------+-------------------------------------+
| properties | tvault_recovery_manager='yes', ...  |
+------------+-------------------------------------+

Step 8 — Verify the installation

# Source your OpenStack credentials
source openrc

# Confirm the workloadmgr service endpoint is registered
openstack endpoint list --service workloadmgr

# List workloads (should return an empty list on a fresh install)
workloadmgr workload-list

# Check service status on the TrilioVault master node
systemctl status wlm-api
systemctl status wlm-scheduler
systemctl status wlm-workloads
systemctl status wlm-cron

All four services should be active (running). Note that wlm-api and wlm-scheduler are Pacemaker-controlled and run only on the master node; wlm-workloads runs on every cluster node.

Uninstall procedure

To remove TrilioVault from your environment:

# 1. Remove Keystone endpoints and service
openstack endpoint delete $(openstack endpoint list \
  --service workloadmgr -f value -c ID)
openstack service delete workloadmgr

# 2. Remove the Trustee role assignments (adjust project/user as needed)
openstack role remove \
  --project <tenant-project> \
  --user <tenant-user> \
  Trustee

# 3. Uninstall the compute extension from each compute node
pip3 uninstall tvault-contego
systemctl restart openstack-nova-compute

# 4. Uninstall the CLI/library from controller nodes
pip3 uninstall workloadmgrclient

# 5. Uninstall the Horizon plugin (if installed)
pip3 uninstall horizon-tvault-plugin
systemctl restart httpd   # or apache2

# 6. Power off and delete the TrilioVault VMs from your KVM hosts
virsh destroy triliovault-node1
virsh undefine triliovault-node1 --remove-all-storage
# Repeat for node2, node3 as applicable

# 7. Release the cluster VIP from Pacemaker / network configuration

If you used Ansible for deployment, you can also run the uninstall playbook:

ansible-playbook -i inventory/production.ini uninstall.yml

Backup target configuration

The backup target is set during initial deployment via the Ansible inventory or Heat environment file. You can reconfigure it by re-running the Ansible playbook (the configurator is idempotent). The three supported target types and their key parameters are:

Note: S3 backup target support was introduced in TrilioVault 3.0. If you are on an earlier version, only NFS and Swift are available.

All backup I/O runs as nova:nova. Ensure the NFS export grants read/write access to the UID/GID of the nova user, and that S3 or Swift credentials have read/write/delete permissions on the target bucket or container.

Cluster and high-availability configuration

Keystone and RBAC configuration

Cinder quota requirements

For every disk in a workload being backed up, TrilioVault creates two Cinder snapshots and one temporary Cinder volume during the backup process. Verify and adjust quotas before workloads are scheduled:

# Check current Cinder quotas for a project
openstack quota show <project-id>

# Increase snapshots and volumes if needed
openstack quota set \
  --snapshots <current + 2 * max_disks_per_backup_job> \
  --volumes <current + max_disks_per_backup_job> \
  <project-id>

Email notification configuration

Email notifications are per-tenant. To enable them:

1. The tenant user must have a valid email address on their OpenStack Keystone account.
2. The backup administrator must configure an SMTP server in the TrilioVault Dashboard under Settings → Email Notifications.

Both conditions must be true; notifications will not be sent if either is missing.

File Recovery Manager image property

The FRM Glance image must carry the property tvault_recovery_manager=yes. Without it, the image does not appear in the Horizon file-level restore mount-target selector. Set this at upload time (see Step 7 of the Installation section) or add it to an existing image:

openstack image set \
  --property tvault_recovery_manager=yes \
  <frm-image-id>

Once TrilioVault is deployed and configured, your day-to-day operations center on three workflows: creating workloads, taking and scheduling snapshots, and restoring workloads or individual files.

Working with workloads

A workload is the unit of protection in TrilioVault — it groups one or more VMs and their attached Cinder volumes so they are snapshotted and restored together. Each VM may belong to only one workload at a time.

Create a workload via the CLI:

# Source your tenant credentials (with the Trustee role)
source tenant-openrc

# List available VMs to find their IDs
openstack server list

# Create a workload containing two VMs
workloadmgr workload-create \
  --name "my-app-workload" \
  --description "Production web application" \
  --instance instance_id=<vm1-id> \
  --instance instance_id=<vm2-id>

Create a workload via Horizon:

1. Log in to Horizon and navigate to Project → Workloads → Workloads.
2. Click Create Workload, enter a name and description, and select the VMs to include.
3. Click Create to save.

Taking and scheduling snapshots

You can trigger a snapshot manually or attach a snapshot policy to automate recurring backups.

Trigger a manual (on-demand) snapshot:

workloadmgr snapshot-create \
  --workload_id <workload-id> \
  --name "pre-upgrade-snapshot"

Attach a snapshot policy when creating a workload:

workloadmgr workload-create \
  --name "my-app-workload" \
  --instance instance_id=<vm-id> \
  --jobschedule '{"start_date": "06/21/2024", "end_date": "12/31/2025", \
                  "interval": "24hrs", "start_time": "02:00 AM", \
                  "retention_policy_type": "Number of Snapshots", \
                  "retention_policy_value": "30"}'

The wlm-cron service reads these policies and fires snapshot jobs at the defined interval. Incremental snapshots capture only changed blocks since the last backup; full snapshots capture all data committed to storage volumes.

Restoring a workload

You can restore a full workload from any available snapshot, either in-place or into a different OpenStack project.

# List snapshots for a workload
workloadmgr snapshot-list --workload_id <workload-id>

# Restore the full workload from a snapshot
workloadmgr restore-create \
  --snapshot_id <snapshot-id> \
  --restore_type openstack \
  --name "restore-20240621"

Important: If the snapshot includes network topology data (all snapshots taken with TrilioVault 3.0 or later capture this automatically), the target tenant's network space must be completely empty of networking components before you initiate the restore. Any existing network object in the tenant will cause the restore to fail. After the restore completes, you must manually connect the restored private networks to the public network.

Performing a file-level restore

When you only need to recover individual files rather than recreating entire VMs, use file-level restore. The File Recovery Manager image must already be uploaded to Glance (see Step 7 of the Installation section).

1. In Horizon, navigate to Project → Workloads → Snapshots.
2. Select the snapshot containing the file you need.
3. Click File Recovery and choose the TrilioVault File Recovery Manager image as the mount target.
4. Mount the snapshot volume, browse to the file, and download it.

File-level restore is also available via the workloadmgr CLI and the wlm-api REST API for scripted recovery scenarios.

Example 1 — Create a workload and verify it

source tenant-openrc

workloadmgr workload-create \
  --name "ecommerce-app" \
  --description "Frontend + database VMs" \
  --instance instance_id=aaa11111-bbbb-cccc-dddd-eeeeeeeeeeee \
  --instance instance_id=fff22222-bbbb-cccc-dddd-eeeeeeeeeeee

Expected output:

+-------------------+--------------------------------------+
| Property          | Value                                |
+-------------------+--------------------------------------+
| id                | 12345678-0000-0000-0000-abcdefabcdef |
| name              | ecommerce-app                        |
| description       | Frontend + database VMs              |
| status            | available                            |
| instances         | 2                                    |
+-------------------+--------------------------------------+

Example 2 — Take an on-demand snapshot and monitor its status

workloadmgr snapshot-create \
  --workload_id 12345678-0000-0000-0000-abcdefabcdef \
  --name "pre-patching-20240621"

Expected output (snapshot transitions from executing to available):

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | snap-9999-aaaa-bbbb-cccc-dddddddddddd|
| name        | pre-patching-20240621                |
| status      | executing                            |
| workload_id | 12345678-0000-0000-0000-abcdefabcdef |
+-------------+--------------------------------------+

# Poll until status is 'available'
workloadmgr snapshot-show snap-9999-aaaa-bbbb-cccc-dddddddddddd

Expected final output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | snap-9999-aaaa-bbbb-cccc-dddddddddddd|
| name        | pre-patching-20240621                |
| status      | available                            |
| size        | 42 GB                                |
+-------------+--------------------------------------+

Example 3 — List snapshots and restore a workload

# List all snapshots for the workload
workloadmgr snapshot-list \
  --workload_id 12345678-0000-0000-0000-abcdefabcdef

Expected output:

+--------------------------------------+------------------------+-----------+
| ID                                   | Name                   | Status    |
+--------------------------------------+------------------------+-----------+
| snap-9999-aaaa-bbbb-cccc-dddddddddddd| pre-patching-20240621  | available |
| snap-8888-aaaa-bbbb-cccc-dddddddddddd| scheduled-20240620     | available |
+--------------------------------------+------------------------+-----------+

# Restore from the most recent snapshot
workloadmgr restore-create \
  --snapshot_id snap-9999-aaaa-bbbb-cccc-dddddddddddd \
  --restore_type openstack \
  --name "restore-post-incident"

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | rest-7777-aaaa-bbbb-cccc-dddddddddddd|
| name        | restore-post-incident                |
| status      | executing                            |
+-------------+--------------------------------------+

Example 4 — Verify service health after deployment

# On the TrilioVault master node
for svc in wlm-api wlm-scheduler wlm-workloads wlm-cron; do
  echo "=== $svc ==="
  systemctl is-active $svc
done

Expected output:

=== wlm-api ===
active
=== wlm-scheduler ===
active
=== wlm-workloads ===
active
=== wlm-cron ===
active

Example 5 — Confirm the FRM Glance image property

openstack image show "TrilioVault File Recovery Manager" \
  -c name -c properties

Expected output:

+------------+-------------------------------------------+
| Field      | Value                                     |
+------------+-------------------------------------------+
| name       | TrilioVault File Recovery Manager         |
| properties | tvault_recovery_manager='yes', ...        |
+------------+-------------------------------------------+

Use the following log paths as your first diagnostic step for any issue:

Issue: workloadmgr workload-list returns a 403 Forbidden error

Symptom: The CLI or API returns HTTP 403 when a user attempts any workload, snapshot, or restore operation.

Likely cause: The user has not been assigned the Trustee RBAC role. The OpenStack admin role does not substitute for Trustee.

Fix:

# Confirm the user's current roles
openstack role assignment list --user <username> --project <project>

# Assign the Trustee role
openstack role add \
  --project <project> \
  --user <username> \
  Trustee

Issue: Snapshots fail with a Cinder quota error

Symptom: Snapshot status transitions to error; the wlm-workloads log shows a Cinder quota exceeded message.

Likely cause: TrilioVault requires two Cinder snapshots and one temporary Cinder volume per disk being backed up. The project quota does not have enough headroom.

Fix:

# Check current usage vs. quotas
openstack quota show <project-id>

# Increase snapshots and volumes
openstack quota set \
  --snapshots <new-value> \
  --volumes <new-value> \
  <project-id>

Then retry the snapshot.

Issue: Backup I/O fails with permission denied on the NFS target

Symptom: Snapshot jobs fail; the wlm-workloads log shows Permission denied errors when writing to the NFS export path.

Likely cause: The nova user's POSIX UID or GID differs between TrilioVault nodes and/or compute nodes, or the NFS export does not grant write access to that UID/GID.

Fix:

# Check the nova UID/GID on the affected node
id nova

# Compare against a known-good node; align UIDs/GIDs if they differ
# Then verify the NFS export options on the NFS server include the correct uid/gid or no_root_squash as required
showmount -e <nfs-server>

After aligning UIDs/GIDs or correcting export options, retry the snapshot.

Issue: wlm-api or wlm-scheduler is not running after a node reboot

Symptom: After rebooting a TrilioVault node, one or both of these services are not active, even though wlm-workloads is running.

Likely cause: wlm-api and wlm-scheduler are Pacemaker-controlled and run only on the elected master node. If the rebooted node was not the master, these services are intentionally not started on it. If the rebooted node was the master, Pacemaker should have failed over — check Pacemaker status.

Fix:

# Check which node Pacemaker has elected as master
pcs status

# If a failover did not occur, check for Pacemaker fencing events
pcs status fencing

# Do NOT manually start wlm-api or wlm-scheduler outside of Pacemaker
# Let Pacemaker manage the placement

Issue: Network topology restore fails immediately

Symptom: A restore that should recreate network topology from the snapshot fails with a conflict error.

Likely cause: TrilioVault 3.0 and later automatically capture the full tenant network topology in every snapshot. The restore requires the target tenant's network space to be completely empty. Any existing network, subnet, router, port, or floating IP causes the restore to fail.

Fix:

1. Remove all networking components from the target tenant:

# List and delete floating IPs, ports, routers, subnets, and networks in the target tenant
openstack floating ip list --project <target-project>
openstack router list --project <target-project>
openstack network list --project <target-project>
# Delete each resource before retrying the restore

2. Retry the restore.
3. After the restore completes, manually connect the restored private networks to the public network — TrilioVault does not do this automatically.

Issue: File Recovery Manager image does not appear in the Horizon restore UI

Symptom: When initiating a file-level restore in Horizon, the FRM Glance image is not listed as a valid mount target.

Likely cause: The image is missing the required tvault_recovery_manager=yes property.

Fix:

# Find the image ID
openstack image list | grep -i recovery

# Set the required property
openstack image set \
  --property tvault_recovery_manager=yes \
  <frm-image-id>

# Verify
openstack image show <frm-image-id> -c properties

Issue: Ansible deployment playbook fails on a second run

Symptom: Re-running ansible-playbook site.yml to reconfigure the cluster produces errors about duplicate Keystone endpoints or existing resources.

Likely cause: This should not happen — the triliovault-cfg-scripts Ansible playbooks are designed to be idempotent. The most common cause of non-idempotent failures is manual changes to Keystone or Pacemaker resources made outside of Ansible.

Fix: Review what was changed manually and reconcile it, then re-run the playbook. If Keystone endpoints were manually duplicated, remove the duplicates:

openstack endpoint list --service workloadmgr
# Delete duplicates, keeping only one set of public/internal/admin endpoints
openstack endpoint delete <duplicate-endpoint-id>

Then re-run:

ansible-playbook -i inventory/production.ini site.yml