Installing on RHOSP10 with Red Hat Director

This guide walks you through deploying TrilioVault for OpenStack on Red Hat OpenStack Platform 10 (RHOSP10) using Red Hat Director (TripleO). Because TrilioVault ships as a QCOW2 image and its nodes must run on standalone KVM hosts outside the OpenStack cloud being protected, the Director workflow focuses on registering those KVM hosts, preparing Heat environment files, and integrating TrilioVault's configuration with your existing TripleO overcloud. Completing this guide gives you a working Workload Manager cluster — including wlm-api, wlm-scheduler, wlm-workloads, and wlm-cron — registered as an OpenStack service endpoint and ready to protect your RHOSP10 workloads.

Before you begin, confirm the following requirements are in place:

Platform

• Red Hat OpenStack Platform 10 (Newton) deployed and healthy via Red Hat Director
• OpenStack services operational: Keystone, Nova, Cinder, and (optionally) Horizon
• Python ≥ 3.6 on any management host where you run CLI commands
• Linux (systemd-based) on all TrilioVault KVM hosts

TrilioVault infrastructure

• At least one (recommended: three) standalone KVM hypervisor hosts that are not part of the RHOSP10 overcloud — TrilioVault nodes cannot run as VMs inside the cloud they protect
• The TrilioVault QCOW2 appliance image, obtained from your Trilio support portal
• An additional cluster VIP (virtual IP address) reserved and routable to Keystone — required even for single-node deployments; Pacemaker and HAProxy bind to this VIP
• A configured backup target: an NFS share, an S3-compatible endpoint, or OpenStack Swift

Accounts and roles

• Red Hat Director (undercloud) access with stack user privileges
• OpenStack admin credentials for the overcloud (sourced via overcloudrc)
• The TrilioVault Trustee RBAC role must be assignable — this role is mandatory for all workload, snapshot, and restore operations and cannot be substituted by the OpenStack admin role

Networking

• The POSIX UID and GID of the nova user must be identical on all TrilioVault cluster nodes and all Nova compute nodes — all backup target I/O runs as nova:nova
• Cinder quota headroom: at least two Cinder snapshots and one temporary Cinder volume per disk being protected, per concurrent backup job

Configuration management (optional)

• Ansible installed on your management host if you intend to use the triliovault-cfg-scripts Ansible playbooks for automated configuration
• Salt or Juju if you prefer those orchestration tools (playbooks and charms are provided by triliovault-cfg-scripts)

Step 1 — Prepare the TrilioVault KVM hosts

On each standalone KVM host (minimum one; three recommended for HA), install the required hypervisor packages and ensure libvirt is running.

# Run as root on each KVM host
yum install -y qemu-kvm libvirt virt-install
systemctl enable --now libvirtd

Verify that the host is not an overcloud node and has network access to the overcloud's API and storage networks.

Step 2 — Deploy the TrilioVault QCOW2 appliance

Copy the TrilioVault QCOW2 image to each KVM host and instantiate a VM for each TrilioVault cluster node. Adjust CPU, RAM, and disk to meet your sizing requirements.

# On each KVM host — replace <node-name> and <ip> with your values
cp triliovault-appliance.qcow2 /var/lib/libvirt/images/<node-name>.qcow2

virt-install \
  --name <node-name> \
  --ram 16384 \
  --vcpus 8 \
  --disk path=/var/lib/libvirt/images/<node-name>.qcow2,format=qcow2 \
  --import \
  --network bridge=br-ex,model=virtio \
  --os-type linux \
  --os-variant rhel7 \
  --noautoconsole

Repeat this step on each of the three KVM hosts for a full HA cluster.

Step 3 — Log in and change the default password

The default TrilioVault Dashboard credentials are admin / password. You must change the password on first login.

# Open a browser to the management IP of node 1
https://<node1-mgmt-ip>/
# Log in with admin / password and set a new password when prompted.

Step 4 — Clone the triliovault-cfg-scripts repository

The triliovault-cfg-scripts component provides the Ansible playbooks used to configure the cluster. Clone or extract it on your management host.

git clone https://github.com/triliodata/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts

Step 5 — Edit the Ansible inventory

Create an inventory file describing your three TrilioVault nodes and the cluster VIP. The VIP is mandatory — Pacemaker and HAProxy bind to it and it is registered as the Keystone endpoint.

# inventory/hosts
[triliovault]
tvault-node1 ansible_host=<node1-ip>
tvault-node2 ansible_host=<node2-ip>
tvault-node3 ansible_host=<node3-ip>

[triliovault:vars]
cluster_vip=<vip-ip>

Step 6 — Configure the Ansible group variables

Edit group_vars/all.yml (or the equivalent variables file in triliovault-cfg-scripts) to match your RHOSP10 environment.

# group_vars/all.yml — key variables (see Configuration section for full details)
openstack_auth_url: "http://<keystone-vip>:5000/v3"
openstack_admin_user: "admin"
openstack_admin_password: "<admin-password>"
openstack_admin_project: "admin"

backup_target_type: "nfs"          # nfs | s3 | swift
nfs_export: "<nfs-host>:<export-path>"

nova_uid: <uid>                     # Must match nova UID on all compute nodes
nova_gid: <gid>                     # Must match nova GID on all compute nodes

cluster_vip: "<vip-ip>"

Step 7 — Run the configuration playbook

The configurator is idempotent — you can re-run it at any time to apply changes.

ansible-playbook \
  -i inventory/hosts \
  site.yml \
  --ask-become-pass

The playbook installs and configures wlm-api, wlm-scheduler, wlm-workloads, and wlm-cron, registers the services with Keystone, configures Pacemaker/HAProxy on the VIP, and mounts the backup target.

Step 8 — Register the TrilioVault service endpoint in Keystone

Source your overcloud admin credentials, then verify that the Workload Manager endpoint is registered.

source ~/overcloudrc

openstack endpoint list | grep workloadmgr

You should see entries for the workloadmgr service pointing to the cluster VIP.

Step 9 — Assign the TrilioVault Trustee role

For each tenant user who needs to create workloads, snapshots, or restores, assign the TrilioVault_Trustee role. The OpenStack admin role cannot substitute for this role.

source ~/overcloudrc

# Replace <user> and <project> with the target tenant values
openstack role add \
  --user <user> \
  --project <project> \
  TrilioVault_Trustee

Step 10 — (Optional) Install the Horizon dashboard plugin

If your RHOSP10 deployment includes Horizon, install the horizon-tvault-plugin on each Horizon node to add the Workloads panel.

# Run on each Horizon / httpd node
pip install horizon-tvault-plugin

# Restart the web server to load the plugin
systemctl restart httpd

After restarting, log in to the Horizon dashboard and confirm that a Workloads panel appears under the Project menu.

Step 11 — Install the workloadmgr CLI client

Install the workloadmanager-client package on any host from which you want to manage TrilioVault using the command line.

pip install workloadmanager-client

# Verify the installation
workloadmgr --version

Step 12 — Verify all services are running

On the TrilioVault master node, confirm that all four services are active. Note that wlm-api and wlm-scheduler are cluster-controlled by Pacemaker and run only on the master node; wlm-workloads runs on every node.

# On the master node
systemctl status wlm-api
systemctl status wlm-scheduler
systemctl status wlm-workloads
systemctl status wlm-cron

# On each non-master node — only wlm-workloads should be active
systemctl status wlm-workloads

After the Ansible playbook has run, the primary configuration for TrilioVault lives in the variables file consumed by triliovault-cfg-scripts. The configurator is idempotent — re-run the Ansible playbook to apply changes at any time.

Keystone / Identity settings

Cluster and HA settings

Backup target settings

nova user identity

Email notification settings

Email notifications are per-tenant and require both a user email address on the OpenStack account and an SMTP server configured by the backup administrator.

Service placement

You do not need to configure service placement manually. Pacemaker enforces the following topology automatically:

• wlm-api — master node only (cluster-controlled)
• wlm-scheduler — master node only (cluster-controlled)
• wlm-workloads — every TrilioVault node
• wlm-cron — master node only (cluster-controlled)

This means wlm-api and wlm-scheduler automatically fail over to a new master if the current master node goes down, while wlm-workloads continues running on all surviving nodes.

Once TrilioVault is installed and all services are healthy, your day-to-day workflow follows four main patterns: creating workloads, taking snapshots, restoring workloads or files, and managing snapshot policies.

Source your credentials

All CLI commands require your overcloud credentials and the TrilioVault endpoint to be set in your environment.

source ~/overcloudrc
export OS_AUTH_URL=http://<keystone-vip>:5000/v3

Creating a workload

A workload is a named group of VMs — and their attached Cinder volumes — that belong to the same application. Each VM may belong to only one workload at a time.

workloadmgr workload-create \
  --name "my-app-workload" \
  --instance instance-id=<nova-instance-uuid-1> \
  --instance instance-id=<nova-instance-uuid-2>

The Workload Manager registers the group and returns a workload ID. Use this ID in all subsequent snapshot and restore commands.

Taking an on-demand snapshot

You can trigger an immediate snapshot at any time. Incremental snapshots capture only changed blocks since the last backup; full snapshots capture all data committed to storage volumes.

workloadmgr snapshot-create \
  --workload_id <workload-id> \
  --name "pre-maintenance-snapshot"

Defining a snapshot policy (schedule and retention)

A snapshot policy tells wlm-cron how often to take automatic snapshots and how many to retain.

workloadmgr workload-modify \
  --workload_id <workload-id> \
  --jobschedule '{"start_date": "06/17/2024", "end_date": "12/31/2024", \
                  "interval": "24hrs", "retention_policy_type": "Number of Snapshots", \
                  "retention_policy_value": "7"}'

wlm-cron evaluates this policy and fires snapshot jobs at the specified interval, automatically deleting older snapshots when the retention count is exceeded.

Restoring a full workload

Restore a workload from a snapshot either in-place (overwriting existing VMs) or into a new project.

workloadmgr restore-create \
  --snapshot_id <snapshot-id> \
  --restore_type openstack \
  --name "restore-my-app"

Network topology note: TrilioVault captures full tenant network topology (networks, subnets, routers, ports, floating IPs) in every snapshot automatically. If you are restoring the network topology, the tenant's network space must be completely empty before the restore begins — any existing networking component will cause the restore to fail. After the restore completes, you must manually reconnect the restored private networks to the public network.

Performing a file-level restore

When you need to recover only a small number of files, use file-level restore to retrieve individual files from inside a snapshot without recreating entire VMs.

1. In the Horizon dashboard (if the horizon-tvault-plugin is installed), navigate to Project → Workloads → Snapshots, select the snapshot, and choose File Recovery.
2. Select the VM and volume, then browse to the file you want to recover.
3. Download or restore the file to its original or an alternate path.

For the File Recovery Manager to appear as a valid mount target in the Horizon UI, the Glance image used for file recovery must have the property tvault_recovery_manager=yes set.

Monitoring snapshots

# List all snapshots for a workload
workloadmgr snapshot-list --workload_id <workload-id>

# Show details for a specific snapshot
workloadmgr snapshot-show <snapshot-id>

Example 1 — List available workloads

Verify that your workload was created successfully and note its ID for use in subsequent commands.

source ~/overcloudrc
workloadmgr workload-list

Expected output:

+--------------------------------------+------------------+---------+
| ID                                   | Name             | Status  |
+--------------------------------------+------------------+---------+
| a1b2c3d4-e5f6-7890-abcd-ef1234567890 | my-app-workload  | available|
+--------------------------------------+------------------+---------+

Example 2 — Create a workload with two VMs

Group two Nova instances into a single workload so they are snapshotted and restored as a consistent unit. Remember: each VM may belong to only one workload at a time.

workloadmgr workload-create \
  --name "web-tier-workload" \
  --instance instance-id=11111111-aaaa-bbbb-cccc-000000000001 \
  --instance instance-id=11111111-aaaa-bbbb-cccc-000000000002

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | f9e8d7c6-b5a4-3210-fedc-ba9876543210 |
| name        | web-tier-workload                    |
| status      | available                            |
+-------------+--------------------------------------+

Example 3 — Take an on-demand snapshot and verify it

# Trigger the snapshot
workloadmgr snapshot-create \
  --workload_id f9e8d7c6-b5a4-3210-fedc-ba9876543210 \
  --name "snap-before-upgrade"

Expected output (snapshot initiated):

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | cccc1111-2222-3333-4444-555566667777 |
| name        | snap-before-upgrade                  |
| status      | executing                            |
+-------------+--------------------------------------+

# Poll until the status transitions to 'available'
workloadmgr snapshot-show cccc1111-2222-3333-4444-555566667777

Expected output (completed):

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | cccc1111-2222-3333-4444-555566667777 |
| name        | snap-before-upgrade                  |
| status      | available                            |
| size        | 42949672960                          |
+-------------+--------------------------------------+

Example 4 — Restore a workload from a snapshot

workloadmgr restore-create \
  --snapshot_id cccc1111-2222-3333-4444-555566667777 \
  --restore_type openstack \
  --name "restore-web-tier"

Expected output:

+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| id          | aaaa9999-8888-7777-6666-555544443333 |
| name        | restore-web-tier                     |
| status      | executing                            |
+-------------+--------------------------------------+

Monitor progress with:

workloadmgr restore-show aaaa9999-8888-7777-6666-555544443333

Example 5 — Assign the Trustee role to a new tenant user

The TrilioVault_Trustee role is required before a tenant user can create or manage any workloads, snapshots, or restores.

source ~/overcloudrc

openstack role add \
  --user jsmith \
  --project acme-project \
  TrilioVault_Trustee

# Verify the assignment
openstack role assignment list \
  --user jsmith \
  --project acme-project \
  --names

Expected output:

+---------------------+---------+-------+---------------+--------+
| Role                | User    | Group | Project       | Domain |
+---------------------+---------+-------+---------------+--------+
| TrilioVault_Trustee | jsmith  |       | acme-project  |        |
+---------------------+---------+-------+---------------+--------+

Example 6 — Re-run the Ansible configurator to update backup target settings

Because the configurator is idempotent, you can safely re-run it after changing variables.

cd triliovault-cfg-scripts

# Update group_vars/all.yml to change backup_target_type to s3, then:
ansible-playbook \
  -i inventory/hosts \
  site.yml \
  --ask-become-pass

The playbook applies only the changed settings without disrupting running services.

Use the following log files as your first resource when diagnosing issues:

• wlm-api: /var/log/workloadmgr/workloadmgr-api.log
• wlm-workloads: /var/log/workloadmgr/workloadmgr-workloads.log
• datamover: /var/log/nova/tvault-contego.log
• nova-api extension: /var/log/nova/nova-api.log

Issue: wlm-api is not reachable at the cluster VIP

Symptom: workloadmgr workload-list returns a connection error or times out pointing at the VIP address.

Likely cause: Pacemaker has not promoted a master node, or HAProxy is not running on the VIP. This can happen if the VIP is not routable or if fewer than the required Pacemaker quorum nodes are up.

Fix:

# On any TrilioVault node
pcs status

# Check which node is master and whether the VIP resource is started
pcs resource show

# If the VIP resource is stopped, start it
pcs resource enable <vip-resource-name>

# Confirm wlm-api is listening on the master node
systemctl status wlm-api

Issue: Permission denied errors writing to the NFS backup target

Symptom: Snapshots fail with permission denied errors; /var/log/workloadmgr/workloadmgr-workloads.log shows EACCES on the NFS mount path.

Likely cause: The nova user's UID or GID on the TrilioVault node does not match the nova UID/GID on one or more compute nodes. All backup target I/O runs as nova:nova, and a mismatch causes NFS squash or UID mapping failures.

Fix:

# Check nova UID/GID on a TrilioVault node
id nova

# Check nova UID/GID on a compute node
ssh <compute-node> id nova

# If they differ, correct them on the affected hosts and re-run the Ansible playbook
ansible-playbook -i inventory/hosts site.yml --ask-become-pass

Issue: Snapshot fails with Cinder quota exceeded

Symptom: Snapshot job transitions to error state; wlm-workloads log shows quota errors referencing Cinder snapshots or volumes.

Likely cause: TrilioVault creates two Cinder snapshots and one temporary Cinder volume per disk during backup. If your Cinder quotas do not accommodate these, the job fails.

Fix:

source ~/overcloudrc

# Check current Cinder quota for the tenant project
openstack quota show <project-id>

# Increase snapshots and volumes quota as needed
# (multiply number-of-disks by 2 for snapshots, by 1 for volumes, add headroom)
openstack quota set \
  --snapshots <new-limit> \
  --volumes <new-limit> \
  <project-id>

Issue: Network topology restore fails immediately

Symptom: A restore that includes network topology reports an error such as "network conflict" or "resource already exists" as soon as it starts.

Likely cause: The tenant's network space is not empty. TrilioVault requires that all networking components (networks, subnets, routers, ports, floating IPs) are removed from the target project before a network topology restore can succeed.

Fix:

source ~/overcloudrc

# Delete all networking resources in the target project before retrying the restore
openstack floating ip list --project <project-id> -f value -c ID | xargs -r openstack floating ip delete
openstack router list --project <project-id> -f value -c ID | xargs -r openstack router delete
openstack network list --project <project-id> -f value -c ID | xargs -r openstack network delete

# Then retry the restore
workloadmgr restore-create \
  --snapshot_id <snapshot-id> \
  --restore_type openstack \
  --name "retry-restore"

After the restore completes, manually reconnect the restored private networks to the public network — TrilioVault does not perform this final step automatically.

Issue: File Recovery Manager image does not appear in Horizon UI

Symptom: When attempting a file-level restore in the Horizon Workloads panel, the File Recovery Manager image is absent from the mount target list.

Likely cause: The Glance image for the File Recovery Manager is missing the required metadata property.

Fix:

source ~/overcloudrc

# Identify the File Recovery Manager image ID
openstack image list | grep -i recovery

# Set the required property
openstack image set \
  --property tvault_recovery_manager=yes \
  <image-id>

Refresh the Horizon UI — the image should now appear as a valid mount target.

Issue: TrilioVault_Trustee role not found when assigning to a user

Symptom: openstack role add returns No role with name TrilioVault_Trustee or similar.

Likely cause: The Ansible configurator playbook did not complete successfully, or the Keystone role was not created during service registration.

Fix:

# Re-run the Ansible playbook to recreate the Keystone role
cd triliovault-cfg-scripts
ansible-playbook -i inventory/hosts site.yml --ask-become-pass

# Verify the role exists after the playbook completes
source ~/overcloudrc
openstack role list | grep TrilioVault_Trustee

Issue: wlm-cron is not firing scheduled snapshots

Symptom: Automatic snapshots defined in a workload's snapshot policy are not being created at the expected intervals.

Likely cause: wlm-cron is not running on the master node, or the workload's snapshot policy was not saved correctly.

Fix:

# On the master node
systemctl status wlm-cron
systemctl restart wlm-cron

# Verify the policy is attached to the workload
workloadmgr workload-show <workload-id>

# Check wlm-api log for scheduling errors
tail -n 100 /var/log/workloadmgr/workloadmgr-api.log