TrilioVault Architecture

This page describes how TrilioVault for OpenStack is structured — its physical deployment model, core software components, high-availability cluster topology, and how it integrates with your OpenStack control plane. Understanding the architecture helps you make informed decisions about placement, network design, and cluster sizing before you deploy, and gives you the mental model you need to diagnose problems and plan upgrades confidently.

TrilioVault is composed of eight distinct software components spread across three deployment boundaries: the TrilioVault cluster nodes, your OpenStack controller nodes, and your OpenStack compute nodes.

workloadmgr (Workload Manager — anchor service)

Type: Core service
Runs on: TrilioVault cluster nodes

The Workload Manager is the central coordination engine for all backup, scheduling, and restore operations. It exposes the REST API and orchestrates every other internal service. From your perspective, workloadmgr is the service you register in Keystone, the process whose health determines cluster availability, and the target of every workloadmgr CLI call or Horizon action. All other TrilioVault services exist to support this core.

wlm-api

Type: API gateway
Runs on: Master TrilioVault node (Pacemaker-controlled)

wlm-api accepts and validates every incoming REST request — whether it originates from the workloadmgr CLI, the Horizon plugin, or a direct API call — and returns results to the caller. It is the single entry point into the TrilioVault control plane. Because wlm-api is cluster-controlled by Pacemaker, it runs only on the current master node; all traffic is routed to it through the cluster VIP and HAProxy. This design ensures that API availability follows the master election and does not split across nodes.

Log location: /var/log/workloadmgr/workloadmgr-api.log

wlm-scheduler

Type: Scheduling service
Runs on: Master TrilioVault node (Pacemaker-controlled)

wlm-scheduler receives job requests from wlm-api and decides which TrilioVault node (and, by extension, which compute node's DataMover) should handle each backup or restore task. It evaluates capacity, availability zone alignment, and current load. Like wlm-api, it is Pacemaker-controlled and runs only on the master node, ensuring a single authoritative scheduler at all times.

wlm-workloads

Type: Worker agent
Runs on: Every TrilioVault node

wlm-workloads is the data-movement executor. Once wlm-scheduler assigns a job, the wlm-workloads instance on the designated node communicates with the DataMover agent on the relevant compute node, reads volume data, and writes snapshots to the configured backup target. Because wlm-workloads runs on every node in the cluster (not just the master), backup and restore I/O is distributed across all TrilioVault nodes — giving you horizontal throughput scaling.

Log location: /var/log/workloadmgr/workloadmgr-workloads.log

wlm-cron

Type: Scheduler/timer service
Runs on: TrilioVault cluster

wlm-cron fires scheduled snapshot jobs according to the snapshot policies you define for each workload. It acts as the internal alarm clock: at the configured interval, it submits a snapshot job to wlm-api exactly as if you had triggered it manually. Retention enforcement — deleting snapshots that exceed your policy's keep count — is also coordinated through wlm-cron.

DataMover (tvault-contego)

Type: Compute-node agent
Runs on: Every OpenStack compute node
Log location: /var/log/nova/tvault-contego.log

The DataMover is a Nova extension deployed on every compute node. It runs in the Nova service context and is responsible for the low-level work of reading VM disk data and streaming it to (or from) the backup target. All backup target I/O is performed as the nova user (nova:nova), and the POSIX UID/GID of nova must be identical across all TrilioVault cluster nodes and all compute nodes — a mismatch will cause permission errors on the backup target.

The DataMover also exposes the wlm-api Nova extension, which appears on controller nodes alongside the standard Nova API and handles workload-specific API surface within the compute service namespace.

Nova API extension log: /var/log/nova/nova-api.log

horizon-tvault-plugin

Type: Horizon dashboard plugin
Runs on: OpenStack controller nodes (Horizon service)

The horizon-tvault-plugin adds a Workloads panel to the OpenStack Horizon web dashboard. From this panel, tenant users and administrators can create workloads, trigger on-demand snapshots, configure snapshot policies, browse available snapshots, and initiate full workload or file-level restores — all without touching the CLI. The plugin communicates with TrilioVault exclusively through the wlm-api endpoint registered in Keystone.

workloadmanager-client

Type: CLI / Python library
Installed on: Any host with network access to the wlm-api endpoint

The workloadmanager-client package provides the workloadmgr command-line tool and the workloadmgrclient Python library. Operators and administrators use the CLI to automate workload management from scripts or the terminal. Every operation available through Horizon is also available through the workloadmgr CLI.

triliovault-cfg-scripts

Type: Configuration management toolkit
Used by: Operators during deployment and upgrade

triliovault-cfg-scripts hosts the Ansible playbooks, Salt states, Juju charms, and other configuration management artifacts used to deploy, configure, and upgrade TrilioVault across your OpenStack environment. TrilioVault configuration via these scripts is idempotent — you can re-run them any number of times to apply changes or reconfigure the cluster without side effects.

The following trace walks through a scheduled snapshot from policy trigger to backup target, illustrating how control and data move through the full architecture.

Step 1 — Policy trigger (wlm-cron)

At the interval defined in a workload's snapshot policy, wlm-cron fires and submits a snapshot job request to wlm-api over the cluster's internal network. This is functionally identical to a user clicking Take Snapshot in Horizon or running workloadmgr snapshot-create — wlm-cron acts as an automated caller of the same API.

Step 2 — Request acceptance and authentication (wlm-api)

wlm-api receives the request at the cluster VIP (the same public URL registered as the Keystone service endpoint). It authenticates the request against Keystone — verifying that the calling project has the TrilioVault Trustee RBAC role — and validates the job parameters. wlm-api then persists the job and forwards it to wlm-scheduler.

Why the VIP? Even on a single-node deployment, all traffic enters through the cluster VIP managed by HAProxy and Pacemaker. This means the Keystone endpoint registration never changes when the master node shifts, and clients never need reconfiguration.

Step 3 — Scheduling (wlm-scheduler)

wlm-scheduler evaluates the workload's VM inventory, the availability zones of those VMs, and the current load across TrilioVault nodes. It selects the optimal wlm-workloads instance to handle the job and dispatches the task to it. If a workload spans multiple VMs in different availability zones, wlm-scheduler coordinates across the appropriate wlm-workloads workers to capture all VMs consistently.

Step 4 — Volume quiescing and Cinder snapshot creation (wlm-workloads + Nova/Cinder)

The assigned wlm-workloads instance instructs the DataMover (tvault-contego) on each relevant compute node to quiesce the VM's disks. wlm-workloads then calls Cinder to create point-in-time Cinder snapshots and, if needed, temporary Cinder volumes for reading volume data. OpenStack Cinder quotas must accommodate two Cinder snapshots and one temporary volume per disk being backed up — ensure your quota headroom is sufficient before defining large workloads.

For workloads including network topology, TrilioVault automatically captures the full tenant network topology (networks, subnets, routers, static routes, ports, and floating IPs) at this stage, with no additional user action required.

Step 5 — Data movement to backup target (DataMover)

The DataMover on each compute node reads block data from the Cinder snapshots or temporary volumes and streams it across the dedicated backup/internal network to the configured backup target (NFS share, S3-compatible endpoint, or OpenStack Swift). All I/O is performed as the nova user. For incremental snapshots, only changed blocks since the previous snapshot are transferred; for full snapshots, only blocks that contain committed data (non-zero blocks) are written — reducing transfer time and storage consumption.

Step 6 — Completion and retention enforcement

Once data movement is complete, wlm-workloads reports success to wlm-api, which updates the snapshot record and notifies the tenant (if email notifications are configured). wlm-cron then evaluates the workload's retention policy; if the number of stored snapshots exceeds the configured keep count, it submits deletion jobs for the oldest snapshots.

Restore data flow (summary)

A restore request follows the same path in reverse: the user or operator submits a restore request through Horizon or the workloadmgr CLI → wlm-api validates and authenticates → wlm-scheduler selects the appropriate wlm-workloads worker → wlm-workloads coordinates with the DataMover to stream snapshot data from the backup target back into new or existing Cinder volumes → Nova boots the restored VMs. For a network topology restore, TrilioVault recreates all captured networking components, but the final step of connecting restored private networks to the public network must be performed manually by the tenant.

Deploy TrilioVault as a QCOW2 appliance on standalone KVM hosts

TrilioVault nodes are instantiated from a QCOW2 image on KVM hosts that exist outside the OpenStack cloud being protected. TVM nodes cannot run as VMs under the same OpenStack cloud they protect.

Rationale: If TrilioVault were deployed as VMs inside the protected cloud, a failure of that cloud (hypervisor crash, control plane outage, storage failure) could take down the backup service at precisely the moment it is most needed. Running TrilioVault on independent KVM hosts guarantees fault domain separation between the protected workload and the protection service itself.

Three-node HA cluster with Pacemaker, HAProxy, and a mandatory VIP

The recommended deployment is a three-node TrilioVault cluster, even though single-node is supported. All deployments — including single-node — require an additional cluster IP (VIP) managed by Pacemaker and HAProxy.

Rationale: A three-node cluster tolerates the loss of any single node without interrupting backup or restore operations. The VIP serves two purposes: it provides a stable, single address that HAProxy load-balances across healthy nodes, and it is the address registered as the Keystone service endpoint. Because the endpoint registration never changes, clients — including Horizon and the workloadmgr CLI — never require reconfiguration when a node fails or when the master election changes. Requiring the VIP even in single-node deployments ensures that a future scale-out to three nodes does not require re-registering the Keystone endpoint.

wlm-api and wlm-scheduler run only on the master node; wlm-workloads runs on every node

Pacemaker designates one TrilioVault node as master at any time. Only the master runs wlm-api and wlm-scheduler. Every node — including the master — runs wlm-workloads.

Rationale: Having a single authoritative API and scheduler eliminates split-brain conditions where two nodes could independently accept the same job, create duplicate snapshots, or conflict over Cinder resources. Distributing wlm-workloads across all nodes, by contrast, is desirable: it allows backup and restore data movement to be parallelized across the cluster, increasing aggregate throughput without requiring a single node to handle all I/O.

All backup target I/O runs as the nova user (nova:nova)

Every read and write to the backup target is performed under the nova POSIX user identity. The UID and GID of nova must be identical across all TrilioVault cluster nodes and all compute nodes.

Rationale: The DataMover runs in the Nova service context on compute nodes and must have permission to access Cinder volume data on those nodes. Tying backup target I/O to the same user avoids the need for a separate service account and simplifies permission management. The identity consistency requirement exists because NFS and similar POSIX-permission file systems use numeric UID/GID — a mismatch produces access-denied errors that are difficult to diagnose post-deployment.

The TrilioVault Trustee RBAC role is mandatory and cannot be substituted by the admin role

TrilioVault introduces a dedicated Keystone role (Trustee) that must be granted to any user or service account performing workload, snapshot, or restore operations.

Rationale: The OpenStack admin role grants broad, cloud-wide privileges that violate the principle of least privilege when delegated to individual tenant users for backup purposes. The Trustee role scopes TrilioVault permissions appropriately per tenant, enabling multi-tenant backup environments where each tenant's data and policies are isolated from other tenants — without requiring elevated cloud-admin rights.

Workload-centric protection model (groups of VMs, not individual VMs)

Protection is defined at the workload (application) level: a named group of VMs and their attached volumes that are snapshotted and restored together. Each VM may belong to only one workload at a time.

Rationale: Multi-VM applications (e.g., a web tier, an application tier, and a database tier) have internal consistency requirements. Backing up individual VMs at different times creates snapshots that are inconsistent with each other at the application level. Capturing all VMs in a workload as a single atomic snapshot ensures that restore produces a coherent application state, not a mix of VMs from different points in time.

Automatic full network topology capture in every snapshot

Beginning with version 3.0, TrilioVault automatically captures the full tenant network topology — networks, subnets, routers, static routes, ports, and floating IPs — in every snapshot, without any user configuration.

Rationale: A VM restore without its associated network configuration is rarely useful in practice; operators would need to manually reconstruct networking before the workload could communicate. Making network capture automatic and unconditional removes a common source of incomplete restores and reduces the operational burden on tenants.

Single-node deployment: simplicity at the cost of resilience

A single-node TrilioVault deployment is supported and easier to operate, but it introduces a single point of failure. If that node becomes unavailable, all backup and restore operations halt until the node recovers — including scheduled snapshots, which will be missed for the outage duration. For production environments, the three-node cluster is best practice. Reserve single-node deployments for development, testing, or resource-constrained environments where downtime of the backup service itself is acceptable.

Cluster VIP adds network infrastructure dependency

The mandatory cluster VIP requires coordination with your network team: a free IP address in the appropriate subnet, correct routing so that OpenStack controller nodes and compute nodes can reach it, and no conflicts with existing Keepalived or VRRP deployments in your environment. The trade-off is operational complexity at setup time in exchange for stable Keystone endpoint registration and transparent failover at runtime.

nova:nova UID/GID consistency is a hard operational constraint

Requiring identical nova UID/GID across all nodes eliminates the need for a dedicated service account but means you must enforce UID/GID consistency every time you add a compute node or TrilioVault node. In large, heterogeneous OpenStack environments where compute nodes were provisioned at different times or by different teams, UID drift is a real risk. If you detect backup target permission errors, UID/GID mismatch is the first thing to check.

Cinder quota consumption during backup

TrilioVault creates Cinder snapshots and temporary Cinder volumes during backup — two Cinder snapshots and one temporary volume per disk being backed up. In environments with tight Cinder quotas or many large workloads running concurrently, this can exhaust quota and cause backup failures. The alternative — streaming directly from live volumes — would risk data inconsistency. Plan your Cinder quotas with backup headroom in mind, and consider staggering large workload schedules to avoid simultaneous quota peaks.

Network topology restore requires a clean network namespace

When you restore network topology from a snapshot, the tenant's network space must be completely empty of networking components before the restore begins. Any existing networks, subnets, routers, ports, or floating IPs will cause the restore to fail. This all-or-nothing requirement prevents partial conflicts but means you cannot restore a network topology into a partially provisioned project. Plan tenant network restores as a full replacement operation, not an additive one.

Post-restore public network reconnection is manual

After a network topology restore, TrilioVault does not reconnect the restored private networks to the public (external) network. This final attachment step must be performed manually by the tenant. This is a deliberate boundary: TrilioVault does not assume that the public network available at restore time is the same as at snapshot time, and automatic reconnection could expose VMs to an unintended public network. Document this step clearly in your runbooks so that operators are not surprised when restored VMs are unreachable immediately after restore.

S3 backup target support requires version 3.0 or later

If you are running a TrilioVault version earlier than 3.0, S3-compatible object storage is not available as a backup target — only NFS is supported. If your backup storage strategy relies on S3, ensure your TrilioVault version meets this minimum requirement before configuring the target.

Horizon plugin is optional but strongly recommended for tenant self-service

All TrilioVault operations are available through the workloadmgr CLI without the Horizon plugin. However, the horizon-tvault-plugin significantly lowers the barrier for tenant users who are not comfortable with the CLI. If your user base is primarily self-service tenants rather than operators, installing the plugin will reduce support burden. If your environment does not deploy Horizon at all, the CLI and direct REST API are fully capable alternatives.