Troubleshooting

This runbook guides you through systematic diagnosis and resolution of common TrilioVault for OpenStack failures — spanning service health, backup target availability, identity and quota misconfigurations, and log analysis — so you can restore a degraded or non-functional TrilioVault cluster to normal operation.

In scope:

• Checking and recovering the health of wlm-api, wlm-scheduler, wlm-workloads, and wlm-cron services
• Verifying Pacemaker cluster status and the cluster VIP
• Diagnosing backup target mount failures (NFS, S3, Swift)
• Identifying and correcting nova:nova POSIX UID/GID misalignment across TrilioVault nodes and compute nodes
• Diagnosing missing or incorrect TrilioVault Trustee RBAC role assignments
• Diagnosing OpenStack Cinder quota shortfalls that block snapshot creation
• Locating and interpreting key log files for each TrilioVault component

Out of scope:

• Initial installation or fresh deployment of TrilioVault (see the Install and Configure guide)
• Horizon dashboard plugin installation or reconfiguration
• Network topology restore conflicts (covered in the Restore guide)
• TrilioVault version upgrades

Before working through this runbook, confirm the following:

• TrilioVault is already installed on standalone KVM hosts outside the OpenStack cloud being protected
• OpenStack ≥ Stein with Keystone, Nova, and Cinder services operational
• You have SSH access to all TrilioVault cluster nodes (the three-node HA cluster or single node)
• You hold OpenStack admin credentials and can run openstack CLI commands against the cloud
• You know the cluster VIP (the IP registered as the Keystone workloadmgr service endpoint)
• You have access to a shell on the master Pacemaker node (where wlm-api and wlm-scheduler run)
• Python ≥ 3.6 is available if you need to run workloadmgr CLI commands
• The TrilioVault Trustee RBAC role is assigned to the account you are using, or you are using the OpenStack admin account for diagnostic steps only — note that the admin role cannot substitute for the Trustee role for workload operations

Work through these steps in order. Each step identifies the responsible component, the command to run, and what a healthy result looks like. Skip steps that are not relevant to the symptom you are diagnosing.

Phase 1 — Establish which TrilioVault phase is failing

TrilioVault backup and restore operations pass through three distinct phases:

Before diving into individual services, identify which phase your failure belongs to:

• Workload creation or snapshot requests rejected immediately → likely wlm-api or an identity/quota issue
• Snapshot jobs queued but never start → likely wlm-scheduler or backup target issue
• Snapshot jobs start but fail mid-way → likely wlm-workloads or UID/GID issue
• Scheduled snapshots never fire → likely wlm-cron

Phase 2 — Check Pacemaker cluster health

Because wlm-api and wlm-scheduler are Pacemaker-controlled and run only on the current master node, a cluster fault can take both services offline simultaneously.

Step 1 — Check overall cluster status

sudo crm_mon --one-shot -r

A healthy cluster shows all nodes Online, all configured resources Started, and no FAILED resources. Confirm that the cluster VIP (your configured virtual IP) is Started on exactly one node — this is the master.

If any resource is in a FAILED or Stopped state, clean it before proceeding:

sudo crm resource cleanup <resource-name>

Step 2 — Verify the cluster VIP is reachable

From a machine outside the TrilioVault cluster:

ping -c 4 <cluster-vip>

If the VIP is unreachable, the Keystone endpoint is unavailable and all API calls will fail. Verify that only the Pacemaker master node holds the VIP:

ip addr show | grep <cluster-vip>

Run this on each node; exactly one should return a result.

Phase 3 — Check individual service health

Run the following on the master node for wlm-api, wlm-scheduler, and wlm-cron; run on every node for wlm-workloads.

Step 3 — Check wlm-api

sudo systemctl status wlm-api

Look for Active: active (running). If the service is failed or inactive, inspect recent log entries:

sudo journalctl -u wlm-api -n 100 --no-pager

Also check the dedicated log file:

sudo tail -100 /var/log/workloadmgr/workloadmgr-api.log

Common causes of wlm-api failure: misconfigured Keystone endpoint, missing configuration file, or port conflict. A successful start produces a line similar to Starting wlm-api on <cluster-vip>:8780.

Step 4 — Check wlm-scheduler

sudo systemctl status wlm-scheduler

If unhealthy:

sudo journalctl -u wlm-scheduler -n 100 --no-pager

The scheduler depends on wlm-api being healthy. If wlm-api is down, fix it first.

Step 5 — Check wlm-workloads on every node

On each TrilioVault node:

sudo systemctl status wlm-workloads

If unhealthy:

sudo tail -200 /var/log/workloadmgr/workloadmgr-workloads.log

wlm-workloads performs the actual data movement; if it is down on all nodes, no snapshots can complete even if scheduling succeeds.

Step 6 — Check wlm-cron

sudo systemctl status wlm-cron

If wlm-cron is stopped, scheduled snapshots defined in your snapshot policies will not fire. Restart it:

sudo systemctl restart wlm-cron

Phase 4 — Verify backup target mount availability

All snapshot data is written to the configured backup target. If the mount is unavailable, wlm-workloads will fail during data movement.

Step 7 — Identify the configured backup target type

Check your TrilioVault configuration to determine whether the backup target is NFS, S3-compatible object storage (available from TrilioVault 3.0), or OpenStack Swift.

Step 8 — NFS: verify the mount on every TrilioVault node

mount | grep <nfs-backup-share>

If the share is not mounted, attempt to mount it manually to isolate whether the issue is network, NFS server, or permissions:

sudo mount -t nfs <nfs-server>:<export-path> /mnt/test-mount

Confirm that the share is accessible as the nova user (see Phase 5 for UID/GID alignment). The nova user must be able to read and write to the mount point.

Step 9 — S3: verify endpoint reachability and credentials

Confirm that the S3 endpoint is reachable from each TrilioVault node:

curl -v https://<s3-endpoint-hostname>/

Check your TrilioVault configuration for correct bucket name, access key, and secret key. Authentication failures appear in /var/log/workloadmgr/workloadmgr-workloads.log as HTTP 403 or connection-refused errors.

Step 10 — Swift: verify Swift service availability

openstack object store account show

A successful response confirms Swift is operational. Verify that the credentials configured in TrilioVault have access to the designated Swift container.

Phase 5 — Diagnose nova:nova POSIX UID/GID misalignment

All read and write operations on the backup target are performed as the nova user (nova:nova). If the POSIX UID or GID for nova differs between any TrilioVault node and any compute node, backup target writes will fail with permission errors.

Step 11 — Compare the nova UID and GID on every node

On each TrilioVault node and on each OpenStack compute node:

id nova

Record the uid= and gid= values. Every node in both groups must return identical values.

Step 12 — Identify mismatched nodes

If any node returns a different UID or GID, that node is misaligned. Misalignment typically appears in logs as:

Permission denied
Operation not permitted
chown failed

in /var/log/workloadmgr/workloadmgr-workloads.log or /var/log/nova/tvault-contego.log.

Step 13 — Correct the UID/GID on the misaligned node

Warning: Changing a user's UID/GID on a live system requires care. Coordinate this change during a maintenance window.

To realign a node to match the canonical nova UID/GID (for example, uid=162 gid=162):

sudo usermod -u <canonical-uid> nova
sudo groupmod -g <canonical-gid> nova
sudo find / -user <old-uid> -exec chown -h nova {} \;
sudo find / -group <old-gid> -exec chgrp -h nova {} \;

After the change, restart wlm-workloads on that node:

sudo systemctl restart wlm-workloads

Phase 6 — Diagnose TrilioVault Trustee RBAC role issues

The TrilioVault Trustee RBAC role is mandatory for all workload, snapshot, and restore operations. The OpenStack admin role cannot substitute for it. Missing role assignment causes 403 Forbidden errors at the API layer.

Step 14 — Verify the Trustee role exists

openstack role list | grep -i trustee

If no Trustee role appears, it was not created during installation. Re-run the TrilioVault configurator (Ansible playbook from triliovault-cfg-scripts) to create it:

ansible-playbook <triliovault-cfg-scripts-path>/site.yml --tags keystone

Step 15 — Verify the Trustee role is assigned to the affected user

openstack role assignment list --user <username> --project <project-name>

Confirm that the Trustee role appears in the output for the correct project.

Step 16 — Assign the Trustee role if missing

openstack role add --user <username> --project <project-name> TrilioVault_Trustee

Replace TrilioVault_Trustee with the exact role name returned in Step 14. After assignment, retry the failing workload or snapshot operation — no service restart is required.

Phase 7 — Diagnose Cinder quota shortfalls

During each backup, TrilioVault creates two Cinder snapshots and one temporary Cinder volume per disk being backed up. If the OpenStack project's Cinder quota is exhausted, the snapshot job fails.

Step 17 — Check current Cinder quota usage for the affected project

openstack quota show --detail <project-name>

Examine the snapshots and volumes rows. Compare in_use against limit. If in_use is at or near limit, the project will be unable to accommodate the temporary resources TrilioVault needs.

Step 18 — Calculate the required headroom

For a workload with N disks being backed up simultaneously, the project needs at least:

• 2 × N additional Cinder snapshot slots
• 1 × N additional Cinder volume slots

Step 19 — Increase the Cinder quota

As an OpenStack admin:

openstack quota set --snapshots <new-limit> --volumes <new-limit> <project-name>

Set <new-limit> to the current in_use value plus the headroom calculated in Step 18, plus a safety margin for concurrent workloads.

Step 20 — Check for stale temporary volumes

If a previous backup failed mid-way, TrilioVault may not have cleaned up temporary Cinder volumes. List volumes that may be orphaned:

openstack volume list --project <project-name> --status error
openstack volume list --project <project-name> --status creating

Safe-delete any orphaned temporary volumes after confirming they are not part of an active operation.

Phase 8 — Review key log files

Use the following log files to gather detailed error context for any phase:

Step 21 — Grep for ERROR or CRITICAL entries around the failure time

sudo grep -E 'ERROR|CRITICAL' /var/log/workloadmgr/workloadmgr-workloads.log | tail -50
sudo grep -E 'ERROR|CRITICAL' /var/log/workloadmgr/workloadmgr-api.log | tail -50
sudo grep -E 'ERROR|CRITICAL' /var/log/nova/tvault-contego.log | tail -50

Match the timestamps in these logs against the time your snapshot or restore operation was reported as failed. The error message and the originating component will direct you back to the appropriate phase in this runbook.

After completing the relevant steps above, confirm that TrilioVault has returned to normal operation by checking all of the following:

1. Pacemaker cluster is fully healthy:

   Copy

   sudo crm_mon --one-shot -r
   

   All nodes are Online, all resources are Started, and the cluster VIP is active on exactly one node.

2. All services are running:

   Copy

   sudo systemctl is-active wlm-api wlm-scheduler wlm-cron
   # On every node:
   sudo systemctl is-active wlm-workloads
   

   Each command returns active.

3. The wlm-api endpoint responds:

   Copy

   curl -s http://<cluster-vip>:8780/
   

   A valid HTTP response (even an authentication challenge) confirms the API is accepting connections.

4. nova UID/GID is consistent across all nodes:

   Copy

   id nova
   

   Returns the same uid and gid on every TrilioVault node and every compute node.

5. Trustee role is assigned:

   Copy

   openstack role assignment list --user <username> --project <project-name>
   

   The TrilioVault Trustee role appears in the output.

6. Cinder quota has sufficient headroom:

   Copy

   openstack quota show --detail <project-name>
   

   The snapshots and volumes limits exceed current in_use by at least 2 × (disks per workload) and 1 × (disks per workload) respectively.

7. A test snapshot completes successfully: Trigger a manual snapshot on a small workload using the workloadmgr CLI or the Horizon Workloads panel and confirm it reaches available status without errors.

The steps in this runbook are primarily diagnostic and corrective; most do not introduce new configuration changes that require reversal. However, if a specific corrective action caused a regression, use the following guidance:

• Service restart caused unexpected behaviour: If restarting any service (wlm-api, wlm-scheduler, wlm-workloads, wlm-cron) made things worse, check the service log for the new failure mode before restarting again. Use journalctl -u <service> -n 200 to review what changed.

• UID/GID change caused filesystem permission issues: If changing the nova UID/GID on a node broke access to other files, reverse the change:

  Copy

  sudo usermod -u <original-uid> nova
  sudo groupmod -g <original-gid> nova
  sudo find / -user <new-uid> -exec chown -h nova {} \;
  sudo find / -group <new-gid> -exec chgrp -h nova {} \;
  

  Then re-evaluate all nodes to determine the canonical UID/GID before retrying.

• Cinder quota increase caused resource overcommit concerns: Reducing a Cinder quota below current usage is blocked by OpenStack. If you need to reduce a quota, first ensure in-use resources are freed, then:

  Copy

  openstack quota set --snapshots <lower-limit> --volumes <lower-limit> <project-name>
  

• Trustee role accidentally removed from a user: Re-add it:

  Copy

  openstack role add --user <username> --project <project-name> TrilioVault_Trustee
  

• Pacemaker resource cleanup caused a failover: If crm resource cleanup triggered an unintended master failover, confirm that the cluster VIP has migrated successfully and that services are running on the new master. The TrilioVault configuration is idempotent — you can re-run the Ansible playbooks from triliovault-cfg-scripts to restore any configuration that diverged:

  Copy

  ansible-playbook <triliovault-cfg-scripts-path>/site.yml
  

If you have completed all applicable steps in this runbook and TrilioVault remains in a degraded or non-functional state, escalate to Trilio support with the following information:

Information to gather before escalating:

1. TrilioVault version — available from the TrilioVault Dashboard or from the Workload Manager service configuration.

2. OpenStack version and distribution.

3. Pacemaker cluster status output:

   Copy

   sudo crm_mon --one-shot -r > /tmp/crm_status.txt
   

4. Service status for all four services on all nodes:

   Copy

   sudo systemctl status wlm-api wlm-scheduler wlm-workloads wlm-cron > /tmp/service_status.txt
   

5. Full log bundles from the failure window:

   • /var/log/workloadmgr/workloadmgr-workloads.log
   • /var/log/workloadmgr/workloadmgr-api.log
   • /var/log/nova/tvault-contego.log
   • /var/log/nova/nova-api.log

6. nova UID/GID output from all nodes:

   Copy

   id nova
   

7. Cinder quota report for the affected project:

   Copy

   openstack quota show --detail <project-name>
   

8. The exact error message seen by the user or in logs, including timestamp and any request/workload/snapshot IDs (idempotency keys).

9. Backup target type (NFS, S3, or Swift) and any relevant mount or connectivity errors.

Provide all of the above when opening a support case so that Trilio engineers can reproduce and diagnose the issue without requiring a second round of information gathering.