Disaster Recovery

This page explains how TrilioVault for OpenStack delivers disaster recovery (DR) capabilities — from cross-cloud workload restoration to automatic network topology reconstruction. Because TrilioVault snapshots capture not only VM state and volume data but also the full tenant network topology (networks, subnets, routers, static routes, ports, and floating IPs), you can bring a complete multi-VM application back to life in a target OpenStack cloud with minimal manual intervention. Use this guide to understand the DR workflow, its prerequisites, and how to set realistic Recovery Point Objective (RPO) and Recovery Time Objective (RTO) targets for your environment.

Before executing a disaster recovery operation, confirm the following:

Environment

• Source and target OpenStack clouds both running OpenStack Stein or later
• TrilioVault deployed on standalone KVM hosts (QCOW2 image) outside the OpenStack cloud being protected; TVM nodes must not run as VMs under the cloud they protect
• A three-node TrilioVault HA cluster is the recommended deployment for production DR scenarios; single-node is supported but not recommended for fault tolerance
• The TrilioVault cluster VIP (cluster IP) registered as the Keystone service endpoint in the target cloud

Credentials and roles

• The TrilioVault Trustee RBAC role assigned to every user who will create workloads, trigger snapshots, or perform restores — the OpenStack admin role cannot substitute for this role
• Valid OpenStack Keystone credentials for both source and target projects

Backup target

• A reachable backup target (NFS share, S3-compatible endpoint, or OpenStack Swift) that is accessible from the target cloud
• S3 backup target support requires TrilioVault version 3.0 or later
• All read/write operations on the backup target run as the nova user (nova:nova); the POSIX UID and GID of nova must be identical across all TrilioVault cluster nodes and all compute nodes in both the source and target clouds

Cinder quotas

• OpenStack Cinder quotas in the target project must accommodate two Cinder snapshots and one temporary volume per disk being restored

Network topology restore

• The tenant's network space in the target project must be completely empty (no networks, subnets, routers, ports, or floating IPs) before a network topology restore; any pre-existing networking components will cause the restore to fail
• After topology restore, you must manually connect restored private networks to the public network — TrilioVault does not perform this final step

Glance image (file-level restore only)

• If you intend to use File Recovery Manager during DR, the File Recovery Manager Glance image must have the property tvault_recovery_manager=yes set

Snapshot policy

• At least one snapshot exists for each workload you intend to restore; scheduled snapshots are managed by wlm-cron

TrilioVault itself is deployed once as part of your environment setup. The steps below assume TrilioVault is already installed in or accessible from your target cloud and focus on the actions required to prepare for and execute disaster recovery operations.

Step 1 — Verify TrilioVault services are running on the target cluster

SSH into the master TrilioVault node and confirm that the cluster-managed services are active:

# Check Pacemaker-controlled services (run on master node only)
sudo pcs status

Expected output includes wlm-api and wlm-scheduler shown as Started on the master node, and wlm-workloads shown as Started on every node.

# Confirm wlm-workloads is running on the current node
sudo systemctl status wlm-workloads

Step 2 — Register the backup target in the target cloud

If the backup target is not yet registered in the target TrilioVault instance, use the workloadmgr CLI to add it:

# Example: register an NFS backup target
workloadmgr backup-target-create \
  --backup-target-type nfs \
  --backup-target-name dr-nfs-target \
  --nfs-share <NFS_SERVER_IP>:<EXPORT_PATH>

# Example: register an S3-compatible backup target (requires TrilioVault 3.0+)
workloadmgr backup-target-create \
  --backup-target-type s3 \
  --backup-target-name dr-s3-target \
  --s3-endpoint <S3_ENDPOINT_URL> \
  --s3-bucket <BUCKET_NAME> \
  --s3-access-key <ACCESS_KEY> \
  --s3-secret-key <SECRET_KEY>

Step 3 — Assign the Trustee role to target-cloud users

In the target OpenStack cloud, assign the TrilioVault_Trustee role to every user who will perform restores. Run the following against the target Keystone:

openstack role add \
  --user <USERNAME> \
  --project <TARGET_PROJECT> \
  TrilioVault_Trustee

Step 4 — Verify Cinder quotas in the target project

Ensure the target project has sufficient Cinder quota before attempting a restore. For a workload with N total disks across all VMs, you need headroom for at least 2 × N snapshots and N volumes beyond current usage:

openstack quota show <TARGET_PROJECT_ID>

Increase quotas if needed:

openstack quota set \
  --snapshots <NEW_SNAPSHOT_LIMIT> \
  --volumes <NEW_VOLUME_LIMIT> \
  <TARGET_PROJECT_ID>

Step 5 — Clear the target tenant's network space (network topology restore)

If you intend to restore the full network topology, the target project must have no existing networking components. List and remove them before proceeding:

# List existing networks in the target project
openstack network list --project <TARGET_PROJECT_ID>

# Remove each router, subnet, port, and network as needed
openstack router delete <ROUTER_ID>
openstack subnet delete <SUBNET_ID>
openstack network delete <NETWORK_ID>

Warning: Clearing the network space is irreversible. Confirm you are operating in the correct target project before deleting any resources.

Step 6 — Confirm snapshot availability

List available snapshots for the workload you plan to restore:

workloadmgr snapshot-list --workload-id <WORKLOAD_ID>

Note the snapshot-id of the point-in-time you want to recover to — this becomes your recovery point.

The following configuration options directly affect disaster recovery behavior. These settings are applied to workloads and snapshot policies and determine your achievable RPO and RTO.

Snapshot policy (RPO driver)

A snapshot policy attached to a workload controls how often wlm-cron fires automatic snapshots and how many are retained. The interval between snapshots defines your maximum RPO — if your policy runs daily, the worst-case data loss is 24 hours.

Edit or attach a snapshot policy through the workloadmgr CLI:

# Create a daily snapshot policy with 7-snapshot retention
workloadmgr schedulepolicy-create \
  --name daily-dr-policy \
  --interval 1 \
  --interval-unit days \
  --retention-policy-value 7

# Attach the policy to a workload
workloadmgr workload-modify \
  --workload-id <WORKLOAD_ID> \
  --schedulepolicy-id <POLICY_ID>

Network topology capture (automatic, version 3.0+)

As of TrilioVault 3.0, every snapshot automatically captures the full tenant network topology — networks, subnets, routers, static routes, ports, and floating IPs — with no additional user configuration required. There is no option to disable this capture; it is always included.

Cross-cloud restore targeting

When performing a restore into a different OpenStack project or cloud, you specify the target project credentials at restore time. No separate "DR mode" configuration flag exists; the same restore workflow used for in-place recovery applies, with target-cloud credentials supplied:

# The workloadmgr client reads credentials from environment variables
export OS_AUTH_URL=<TARGET_CLOUD_KEYSTONE_URL>
export OS_USERNAME=<TARGET_USERNAME>
export OS_PASSWORD=<TARGET_PASSWORD>
export OS_PROJECT_NAME=<TARGET_PROJECT_NAME>
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default

Email notifications

To receive alerts when a scheduled DR snapshot succeeds or fails, configure per-tenant email notifications. This requires:

1. A valid email address on the user's OpenStack account
2. An SMTP server configured by the backup administrator

Notifications are configured in the TrilioVault Dashboard (Horizon plugin) under Workloads → Settings → Notifications, or via the WorkloadMgr API.

Planning your DR strategy with RPO and RTO

Before running a disaster recovery, define your targets:

• RPO (Recovery Point Objective): How much data loss is acceptable? Set your snapshot policy interval to match. For example, a 1-hour interval means the maximum data loss is one hour.
• RTO (Recovery Time Objective): How quickly must the application be back online? RTO is influenced by workload size, number of VMs, volume sizes, network bandwidth to the backup target, and whether you are restoring a full snapshot or chaining incremental snapshots.

To minimize RTO, prefer restoring from the most recent full snapshot when possible, as this avoids chaining incremental deltas. When storage efficiency is more important than restore speed, incremental snapshots are appropriate.

Triggering a manual (on-demand) snapshot before a planned DR event

Before a planned maintenance window or failover, take an on-demand snapshot to minimize RPO:

workloadmgr snapshot-create \
  --workload-id <WORKLOAD_ID> \
  --name pre-dr-snapshot

Monitor the snapshot until it reaches available status:

workloadmgr snapshot-show <SNAPSHOT_ID>

Restoring a workload to a target cloud (cross-cloud DR)

This is the primary DR workflow. It restores the entire workload — all VMs and volumes — and reconstructs the network topology from snapshot metadata.

1. Source the target cloud credentials:

source <target-cloud-openrc.sh>

2. Initiate the restore:

workloadmgr restore-create \
  --workload-id <WORKLOAD_ID> \
  --snapshot-id <SNAPSHOT_ID> \
  --restore-name dr-restore-$(date +%Y%m%d) \
  --restore-type restore

3. Monitor restore progress:

workloadmgr restore-show <RESTORE_ID>

Wait until status shows available.

4. Reconnect private networks to the public network (mandatory manual step):

TrilioVault recreates the tenant's private networks, subnets, and routers but does not connect them to the external/public network. After the restore completes, you must add the external gateway manually:

# Identify the restored router
openstack router list --project <TARGET_PROJECT_ID>

# Attach the external gateway
openstack router set \
  --external-gateway <PUBLIC_NETWORK_ID> \
  <RESTORED_ROUTER_ID>

Your application is now accessible through the restored floating IPs.

Using the Horizon dashboard for DR restores

If you have the horizon-tvault-plugin installed in the target cloud's Horizon:

1. Log in to the Horizon dashboard with your target-cloud credentials.
2. Navigate to Project → Workloads → Snapshots.
3. Locate the snapshot you want to restore and click Restore.
4. Follow the restore wizard, selecting Restore Type: Restore for a full workload recovery.
5. After the restore completes, manually reconnect the restored router to the public network as described above.

File-level restore during DR

If only specific files are lost and you do not need to rebuild entire VMs, use file-level restore to retrieve individual files from a snapshot without recreating the workload:

1. Ensure the File Recovery Manager Glance image has the property tvault_recovery_manager=yes.
2. In the Horizon Workloads panel, select the snapshot and choose File Restore.
3. Mount the snapshot volume and browse to the required files.

File-level restore is faster than a full workload restore and is suitable when RTO for individual files is more important than full-environment recovery.

Example 1 — End-to-end DR restore of a two-VM application

Scenario: A web application consisting of a frontend VM and a database VM has been backed up daily. You need to restore it to a secondary OpenStack cloud after a failure in the primary cloud.

Step 1: Source target-cloud credentials and list available snapshots

source /etc/openstack/target-cloud-openrc.sh

workloadmgr snapshot-list --workload-id a1b2c3d4-e5f6-7890-abcd-ef1234567890

Expected output:

+--------------------------------------+---------------------+-----------+---------------------+
| ID                                   | Name                | Status    | Created At          |
+--------------------------------------+---------------------+-----------+---------------------+
| 9f8e7d6c-5b4a-3210-fedc-ba9876543210 | daily-2024-01-15    | available | 2024-01-15T02:00:11 |
| 8e7d6c5b-4a32-10fe-dcba-987654321098 | daily-2024-01-14    | available | 2024-01-14T02:00:09 |
+--------------------------------------+---------------------+-----------+---------------------+

Step 2: Confirm the target project network space is empty

openstack network list --project dr-target-project

Expected output:

(no output — network list is empty)

Step 3: Initiate the restore

workloadmgr restore-create \
  --workload-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --snapshot-id 9f8e7d6c-5b4a-3210-fedc-ba9876543210 \
  --restore-name webapp-dr-2024-01-15 \
  --restore-type restore

Expected output:

+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| id          | 7c6b5a49-3827-4f1e-90ab-cd1234567890 |
| name        | webapp-dr-2024-01-15                 |
| status      | in-progress                          |
| workload_id | a1b2c3d4-e5f6-7890-abcd-ef1234567890 |
| snapshot_id | 9f8e7d6c-5b4a-3210-fedc-ba9876543210 |
+-------------+--------------------------------------+

Step 4: Poll until restore is complete

workloadmgr restore-show 7c6b5a49-3827-4f1e-90ab-cd1234567890

Expected output (when complete):

+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| id          | 7c6b5a49-3827-4f1e-90ab-cd1234567890 |
| name        | webapp-dr-2024-01-15                 |
| status      | available                            |
+-------------+--------------------------------------+

Step 5: Reconnect the restored router to the public network

# Find the restored router
openstack router list --project dr-target-project

+--------------------------------------+----------------+--------+
| ID                                   | Name           | Status |
+--------------------------------------+----------------+--------+
| 3d2c1b0a-9876-5432-10fe-dcba98765432 | webapp-router  | ACTIVE |
+--------------------------------------+----------------+--------+

openstack router set \
  --external-gateway <PUBLIC_NETWORK_ID> \
  3d2c1b0a-9876-5432-10fe-dcba98765432

Expected output: No output on success. Verify connectivity by pinging a restored floating IP.

Example 2 — Take an on-demand snapshot before a planned failover

workloadmgr snapshot-create \
  --workload-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --name pre-failover-snapshot

Expected output:

+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| id          | 1a2b3c4d-5e6f-7081-92a3-b4c5d6e7f809 |
| name        | pre-failover-snapshot                |
| status      | in-progress                          |
+-------------+--------------------------------------+

Wait for status available before proceeding with the failover.

Example 3 — Create a snapshot policy for a 4-hour RPO

workloadmgr schedulepolicy-create \
  --name every-4h-policy \
  --interval 4 \
  --interval-unit hours \
  --retention-policy-value 14

workloadmgr workload-modify \
  --workload-id a1b2c3d4-e5f6-7890-abcd-ef1234567890 \
  --schedulepolicy-id <RETURNED_POLICY_ID>

Expected output (policy create):

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| id                     | 4f3e2d1c-0b9a-8765-4321-fedcba098765 |
| name                   | every-4h-policy                      |
| interval               | 4                                    |
| interval_unit          | hours                                |
| retention_policy_value | 14                                   |
+------------------------+--------------------------------------+

With this policy, wlm-cron fires a snapshot every 4 hours and retains the 14 most recent snapshots (~56 hours of recovery points).

Issue 1 — Restore fails immediately with a network conflict error

Symptom: The restore operation transitions to error status shortly after starting, and the wlm-workloads log shows a message about existing network components.

Log to check:

tail -f /var/log/workloadmgr/workloadmgr-workloads.log

Likely cause: The target project's network space is not empty. TrilioVault's network topology restore requires the tenant to have no pre-existing networks, subnets, routers, ports, or floating IPs. Any conflict causes the restore to abort.

Fix:

1. List and remove all networking resources in the target project:

openstack router list --project <TARGET_PROJECT_ID>
openstack network list --project <TARGET_PROJECT_ID>
# Delete routers first, then subnets, then networks
openstack router delete <ROUTER_ID>
openstack subnet delete <SUBNET_ID>
openstack network delete <NETWORK_ID>

2. Re-run the restore once the project network space is completely empty.

Issue 2 — Restored VMs have no external connectivity after restore completes

Symptom: The restore finishes with available status, VMs are running, but they cannot be reached from outside the tenant network and floating IPs are unreachable.

Likely cause: TrilioVault restores private networks, subnets, routers, and floating IPs but does not attach the restored router to the external/public network. This final step is always manual.

Fix:

# Identify the restored router
openstack router list --project <TARGET_PROJECT_ID>

# Attach the external gateway
openstack router set \
  --external-gateway <PUBLIC_NETWORK_ID> \
  <RESTORED_ROUTER_ID>

Issue 3 — workloadmgr CLI returns a 403 Forbidden error

Symptom: Any workloadmgr command returns HTTP 403 or a message indicating insufficient permissions.

Likely cause: The user does not have the TrilioVault_Trustee RBAC role in the target project. The OpenStack admin role is not a substitute.

Fix:

openstack role add \
  --user <USERNAME> \
  --project <TARGET_PROJECT_NAME> \
  TrilioVault_Trustee

Re-run the workloadmgr command after the role is assigned.

Issue 4 — Restore fails with a Cinder quota exceeded error

Symptom: The restore enters error state and the wlm-api log shows a quota-related error from Cinder.

Log to check:

tail -f /var/log/workloadmgr/workloadmgr-api.log

Likely cause: During restore, TrilioVault creates Cinder snapshots and temporary volumes. If the target project does not have quota for at least two Cinder snapshots and one temporary volume per disk being restored, the operation will fail.

Fix:

# Check current quota and usage
openstack quota show <TARGET_PROJECT_ID>

# Increase snapshot and volume quota
openstack quota set \
  --snapshots <NEW_LIMIT> \
  --volumes <NEW_LIMIT> \
  <TARGET_PROJECT_ID>

Then retry the restore.

Issue 5 — Scheduled DR snapshots are not being taken

Symptom: The workload has a snapshot policy attached, but no new snapshots appear at the expected interval.

Likely cause: The wlm-cron service is not running, or the policy is not correctly attached to the workload.

Fix:

1. Verify wlm-cron is running:

sudo systemctl status wlm-cron

If it is stopped, start it:

sudo systemctl start wlm-cron

2. Confirm the policy is attached to the workload:

workloadmgr workload-show <WORKLOAD_ID>

Check that schedulepolicy_id is populated. If not, attach the policy:

workloadmgr workload-modify \
  --workload-id <WORKLOAD_ID> \
  --schedulepolicy-id <POLICY_ID>

Issue 6 — Backup target is unreachable from the target cloud

Symptom: Restore operations fail immediately with a storage or I/O error, or the backup target shows as unavailable.

Likely cause: The backup target (NFS share, S3 endpoint, or Swift) is not accessible from the target TrilioVault cluster, or the nova user UID/GID on the target cluster nodes does not match that on the source cluster nodes.

Fix:

1. Verify network reachability from the TrilioVault node to the backup target.
2. Confirm the nova user POSIX UID and GID are identical on all TrilioVault nodes and all compute nodes in the target cloud:

id nova

UIDs and GIDs must match across every node. Correct any mismatch using usermod before retrying. 3. For NFS targets, confirm the export is mounted and writable as the nova user:

su -s /bin/bash nova -c "ls <NFS_MOUNT_PATH>"