Workload Policies

Workload policies let OpenStack administrators define reusable backup schedules and retention rules that can be assigned to workloads by tenants. Instead of requiring every tenant to configure snapshot frequency, full-backup intervals, and retention counts individually, an administrator creates named policies in the Horizon admin area and makes them available cloud-wide. This page explains how to create and manage workload policies as an administrator, what each policy parameter controls, and how tenants assign a policy to a workload — and how that differs from configuring a per-workload schedule directly.

Before creating or managing workload policies, ensure the following are in place:

• TrilioVault for OpenStack is installed and all core services — wlm-api, wlm-scheduler, wlm-workloads, and wlm-cron — are running.
• OpenStack ≥ Stein is deployed and operational.
• OpenStack Keystone is configured and the TrilioVault service endpoints are registered.
• TrilioVault Trustee RBAC role is assigned to any user who will create or manage workloads. The standard OpenStack admin role is not a substitute for the Trustee role.
• OpenStack Horizon with the horizon-tvault-plugin installed, if you intend to manage policies through the dashboard. CLI-based management requires the workloadmgr command-line tool from the workloadmanager-client package.
• Python ≥ 3.6 on any host running the workloadmgr CLI.
• A configured and accessible backup target (NFS, S3-compatible endpoint, or Swift) — policies that trigger snapshots require a functional backup target to store the resulting data.
• Administrator-level access to the Horizon dashboard or a valid Keystone token with admin privileges to create and publish policies.

Workload policies are a built-in feature of the Workload Manager service and do not require a separate installation step. However, the following setup must be completed before policies can be created and used.

Step 1 — Verify that all WLM services are running

On each TrilioVault node, confirm that the workload manager services are active:

systemctl status wlm-api
systemctl status wlm-scheduler
systemctl status wlm-workloads
systemctl status wlm-cron

All four services must show active (running). If any service is stopped, start it:

systemctl start wlm-api
systemctl start wlm-scheduler
systemctl start wlm-workloads
systemctl start wlm-cron

Note: wlm-api and wlm-scheduler are cluster-controlled by Pacemaker and run on the master node only. wlm-workloads runs on every TrilioVault node. Do not attempt to manually start wlm-api or wlm-scheduler on non-master nodes.

Step 2 — Verify that the Horizon plugin is installed (dashboard workflow only)

If you will manage policies through the Horizon dashboard, confirm that the horizon-tvault-plugin is installed and the Workloads panel is visible under the Project menu. If the panel is absent, install the plugin by following your site's triliovault-cfg-scripts Ansible playbook procedure before continuing.

Step 3 — Verify workloadmgr CLI access (CLI workflow only)

Confirm that the workloadmgr CLI is available and can reach the WLM API endpoint:

workloadmgr workload-policy-list

A successful response returns an (empty or populated) list of policies. An authentication error indicates that your OpenStack credentials or the Keystone endpoint registration need attention.

Each workload policy is defined by a set of parameters that control how often snapshots are taken, how often full snapshots are forced, and how many snapshots are retained. The table below describes every configurable parameter.

Key behavioral notes

• Retention vs. storage: Retention count governs the number of snapshots kept, not the storage size. Plan your backup target capacity so that retention count × average snapshot size fits comfortably within available storage.
• Incremental vs. full snapshots: Incremental snapshots reduce storage consumption and backup time by capturing only changed blocks since the last backup. Full snapshots capture all data committed to volumes (empty/zeroed blocks are skipped). The full-backup interval lets you periodically consolidate the chain.
• Policy scope: Policies created in the Horizon Admin area are available to all tenants. Policies created within a tenant project scope are available only to that tenant.
• Cinder quota impact: Every snapshot job — whether triggered by a policy or manually — causes TrilioVault to create Cinder snapshots and a temporary Cinder volume. Ensure that OpenStack Cinder quotas accommodate two Cinder snapshots and one temporary volume per disk in each workload that the policy will protect.

Administrator workflow — creating and publishing a policy

As an administrator, you create workload policies from the Horizon Admin panel so that tenants can assign them without defining their own schedules.

1. Log in to Horizon with administrator credentials.
2. Navigate to Admin → TrilioVault → Workload Policies.
3. Click Create Policy.
4. Fill in the policy name, optional description, snapshot frequency, full-backup interval, and retention count.
5. Click Create. The policy is immediately visible to tenants in the Project → Workloads panel.

To update a policy, select it from the list and click Edit Policy. Changes take effect on the next scheduled snapshot run managed by wlm-cron.

To delete a policy, select it and click Delete Policy. Deleting a policy does not delete existing snapshots, but workloads that were assigned the deleted policy will no longer have automated snapshots scheduled until a new policy is assigned.

Tenant workflow — assigning a policy to a workload

Tenants assign an administrator-defined policy when creating a new workload, or by editing an existing workload.

When creating a workload:

1. In Horizon, go to Project → TrilioVault → Workloads and click Create Workload.
2. Complete the workload name and VM selection steps.
3. On the Schedule tab, choose Use Workload Policy and select the desired policy from the dropdown.
4. Click Create Workload. wlm-cron will begin firing snapshots according to the selected policy.

When editing an existing workload:

1. Navigate to Project → TrilioVault → Workloads, find the workload, and select Edit Workload.
2. On the Schedule tab, switch from a per-workload schedule to Use Workload Policy (or change to a different policy).
3. Save the workload. The updated schedule takes effect on the next wlm-cron evaluation cycle.

Policy assignment via CLI

You can also assign a policy when creating a workload using the workloadmgr CLI:

workloadmgr workload-create \
  --name "my-app-workload" \
  --workload-type-id <workload_type_id> \
  --instance-id <vm_uuid_1> \
  --instance-id <vm_uuid_2> \
  --workload-policy-id <policy_uuid>

To list available policies so you can find the correct <policy_uuid>:

workloadmgr workload-policy-list

Distinguishing policies from per-workload schedules

TrilioVault supports two mutually exclusive approaches to scheduling snapshots for a workload:

A workload can use either a policy or a per-workload schedule, not both simultaneously. When a policy is assigned, the per-workload schedule fields are ignored. This distinction keeps administrative governance separate from tenant-level customization.

Example 1 — List all workload policies

Retrieve all currently defined workload policies visible to your credentials:

workloadmgr workload-policy-list

Expected output:

+--------------------------------------+--------------------+-------------+-----------------+----------+
| ID                                   | Name               | Fullbackup  | Retention Count | Status   |
|                                      |                    | Days        |                 |          |
+--------------------------------------+--------------------+-------------+-----------------+----------+
| a1b2c3d4-e5f6-7890-abcd-ef1234567890 | daily-7day         | 7           | 7               | available|
| b2c3d4e5-f6a7-8901-bcde-f12345678901 | hourly-48hr        | 1           | 48              | available|
+--------------------------------------+--------------------+-------------+-----------------+----------+

Example 2 — Create a workload policy via CLI

Create a policy named weekly-gold that takes a full snapshot every 7 days and retains the last 4 snapshots:

workloadmgr workload-policy-create \
  --name "weekly-gold" \
  --description "Weekly full backup, 4-snapshot retention for gold-tier workloads" \
  --fullbackup-days 7 \
  --retention-policy-value 4

Expected output:

+-------------------------+--------------------------------------+
| Property                | Value                                |
+-------------------------+--------------------------------------+
| id                      | c3d4e5f6-a7b8-9012-cdef-012345678902 |
| name                    | weekly-gold                          |
| description             | Weekly full backup, 4-snapshot       |
|                         | retention for gold-tier workloads    |
| fullbackup_days         | 7                                    |
| retention_policy_value  | 4                                    |
| status                  | available                            |
+-------------------------+--------------------------------------+

Example 3 — Assign a policy to a new workload

Create a workload that protects two VMs and immediately attaches the weekly-gold policy:

workloadmgr workload-create \
  --name "erp-production" \
  --workload-type-id 4d7c8a1b-0e2f-3456-789a-bcdef0123456 \
  --instance-id 11111111-2222-3333-4444-555555555555 \
  --instance-id 66666666-7777-8888-9999-000000000000 \
  --workload-policy-id c3d4e5f6-a7b8-9012-cdef-012345678902

Expected output:

+--------------------+--------------------------------------+
| Property           | Value                                |
+--------------------+--------------------------------------+
| id                 | d4e5f6a7-b8c9-0123-def0-123456789012 |
| name               | erp-production                       |
| status             | available                            |
| workload_policy_id | c3d4e5f6-a7b8-9012-cdef-012345678902 |
+--------------------+--------------------------------------+

wlm-cron will now schedule snapshots for erp-production according to the weekly-gold policy.

Example 4 — Show details of a single policy

workloadmgr workload-policy-show c3d4e5f6-a7b8-9012-cdef-012345678902

Expected output:

+-------------------------+--------------------------------------+
| Property                | Value                                |
+-------------------------+--------------------------------------+
| id                      | c3d4e5f6-a7b8-9012-cdef-012345678902 |
| name                    | weekly-gold                          |
| description             | Weekly full backup, 4-snapshot       |
|                         | retention for gold-tier workloads    |
| fullbackup_days         | 7                                    |
| retention_policy_value  | 4                                    |
| status                  | available                            |
+-------------------------+--------------------------------------+

Example 5 — Delete a policy

workloadmgr workload-policy-delete c3d4e5f6-a7b8-9012-cdef-012345678902

Expected output:

Request to delete workload policy c3d4e5f6-a7b8-9012-cdef-012345678902 has been accepted.

Important: Workloads previously assigned this policy will not be automatically reassigned. Review affected workloads and assign a replacement policy or configure a per-workload schedule to avoid unprotected workloads.

Issue 1 — Policy does not appear in the tenant workload creation form

Symptom: A tenant opens Project → TrilioVault → Workloads → Create Workload but the policy dropdown is empty, even though an administrator created policies.

Likely cause: The wlm-api service is not running or is not reachable, so the Horizon plugin cannot fetch the policy list. Alternatively, the tenant user does not have the TrilioVault Trustee RBAC role assigned.

Fix:

1. Verify wlm-api is running on the master node: systemctl status wlm-api.
2. Check the API log for errors: tail -n 100 /var/log/workloadmgr/workloadmgr-api.log.
3. Confirm that the Keystone service endpoint for WLM is correctly registered.
4. Confirm that the tenant user has the Trustee role: the OpenStack admin role is not a substitute.

Issue 2 — Scheduled snapshots are not firing for a policy-assigned workload

Symptom: A workload has a policy assigned with a defined snapshot frequency, but no automatic snapshots appear after the expected interval.

Likely cause: The wlm-cron service is stopped or has encountered an error. Alternatively, the workload may have been left in an error state from a previous failed snapshot, which prevents new jobs from being queued.

Fix:

1. Confirm wlm-cron is running: systemctl status wlm-cron. Restart if stopped: systemctl start wlm-cron.
2. Check the wlm-workloads log for prior job failures: tail -n 200 /var/log/workloadmgr/workloadmgr-workloads.log.
3. Check the workload status via CLI: workloadmgr workload-show <workload_id>. If the status is error, resolve the underlying error and reset the workload before wlm-cron can schedule new jobs.

Issue 3 — Policy deletion fails with a conflict error

Symptom: Running workloadmgr workload-policy-delete <policy_id> returns an error indicating the policy is still in use.

Likely cause: One or more workloads are still actively assigned to the policy. The API refuses deletion to prevent workloads from being left in an undefined scheduling state.

Fix:

1. List workloads assigned to the policy and reassign them to a different policy or configure per-workload schedules.
2. Once no workloads reference the policy, retry the deletion.

Issue 4 — Retention count is not being honored; old snapshots persist

Symptom: Snapshots accumulate beyond the retention count defined in the policy.

Likely cause: The wlm-cron service handles retention enforcement at the time a new snapshot is created. If wlm-cron was stopped or erroring during recent snapshot runs, the purge step was skipped.

Fix:

1. Review wlm-cron logs for errors around the expected snapshot times.
2. Ensure wlm-cron is running continuously: systemctl status wlm-cron.
3. Manually delete excess snapshots using workloadmgr snapshot-delete <snapshot_id> if immediate cleanup is required.
4. Verify that Cinder quotas are not exhausted — a quota limit can cause snapshot creation (and therefore the retention purge cycle) to fail silently.

Issue 5 — Error: "Trustee role not found" when creating a policy via CLI

Symptom: Running workloadmgr workload-policy-create returns an authorization error referencing a missing Trustee role.

Likely cause: The user account running the CLI command does not have the TrilioVault Trustee RBAC role assigned in Keystone. The standard admin role does not grant access to TrilioVault workload and policy operations.

Fix:

1. As an OpenStack administrator, assign the Trustee role to the user: use the Keystone CLI or Horizon Identity panel to add the role.
2. Re-source your OpenStack credentials and retry the command.

Issue 6 — wlm-api log shows 500 errors on policy endpoints

Symptom: Policy list, create, or show operations fail with HTTP 500 errors. The workloadmgr-api.log contains database or connectivity errors.

Likely cause: The Workload Manager database is unreachable or the wlm-api service lost its connection to the backend database after a restart.

Fix:

1. Check /var/log/workloadmgr/workloadmgr-api.log for the specific error.
2. Verify database connectivity from the TrilioVault master node.
3. Restart wlm-api via Pacemaker (do not restart the systemd unit directly on a clustered deployment): use your Pacemaker management tooling to relocate or restart the wlm-api resource.