Worker VM

This page explains how Abacá's worker VMs are deployed, how the conductor service manages their lifecycle, and what happens inside a worker during a backup or restore job. Worker VMs are the data-plane of Abacá: they are the only component that ever touches plaintext share data or encryption keys, and they do so entirely within the tenant's own network scope, keeping the trust boundary tight. Understanding how workers are provisioned, assigned jobs, and reaped is essential for operators tuning fleet capacity, diagnosing stuck jobs, and reasoning about the security model.

Before working with worker VMs, ensure the following are in place:

• OpenStack ≥ 2023.1 (Antelope) / RHOSO 18 with the following services active:
  • Keystone — for tenant authentication and trust-scoped token issuance
  • Barbican — for tenant-owned storage of S3 credentials and the Kopia repository password
  • Manila — for share access rules and, in DHSS=true deployments, share network metadata
  • Nova and Neutron — required when using the AttachToShareNetwork network attachment strategy (DHSS=true); optional for StaticReachability (DHSS=false)
• OpenShift ≥ 4.12 running the Abacá control plane (abaca-api and abaca-conductor pods)
• RHEL worker VM image with Kopia ≥ 0.23.1 baked in — the abacа-worker-agent and Kopia binary must be present in the image before it is registered with the conductor
• S3-compatible object storage (AWS S3, MinIO, Ceph RGW, ODF, or Wasabi) with a properly configured tenant bucket (versioning + object lock enabled; see Tenant S3 Bucket Requirements)
• RabbitMQ — the conductor dispatches job commands to workers over oslo.messaging; the abaca-conductor topic must be reachable
• MariaDB ≥ 10.x (Kolla 2024.2+) — stores job state, worker registration records, and capacity slot counts
• A registered backup target for the tenant (enrollment must be in available status before backup or restore jobs can be dispatched to a worker)

Worker VMs are not installed manually by end users. The conductor provisions and manages them automatically. What you, as an operator, must do is build and register the worker VM image that the conductor will boot.

Step 1 — Pin the Kopia version in the build environment

The worker image build reads the Kopia version from build/kopia.env. Verify the pin matches the supported version before building:

cat build/kopia.env
# KOPIA_VERSION=0.23.1

Do not change this value unless you have validated the new version with abaca-dev (see Step 4). The version pinned here must match the Kopia binary used during enrollment so that FIPS key-derivation algorithm support (--key-derivation-algorithm) is consistent across all workers.

Step 2 — Build the worker VM image

Use the image build tooling provided in the repository. The resulting RHEL image must contain:

• The abaca-worker-agent binary (the in-VM agent that receives RPC commands from the conductor)
• The Kopia binary at the version pinned in build/kopia.env
• RHEL FIPS mode enabled (required for FIPS-validated deployments)

# Build the worker image (exact tooling path confirmed by operator)
bash build/build-worker-image.sh

Step 3 — Upload the image to Glance

Once built, upload the worker image to OpenStack Image service (Glance) so the conductor can reference it when booting worker VMs via Nova:

openstack image create \
  --file worker-vm.qcow2 \
  --disk-format qcow2 \
  --container-format bare \
  --property abaca_worker_image=true \
  abaca-worker-rhel

Record the image ID; you will reference it in the conductor configuration.

Step 4 — Validate the image with abaca-dev

Before registering the image for production use, run the abaca-dev CLI to probe the data path and confirm fidelity and capability validation against your real OpenStack environment:

export OS_CLOUD=abaca-tenant
abaca-dev enroll

Review the findings report that abaca-dev generates. Pay particular attention to the fips_kdf_gap field in the target's capabilities after enrollment — if it is true, the pinned Kopia binary does not support --key-derivation-algorithm and the repository was initialized with scrypt instead of pbkdf2, which is not FIPS-140 compliant. Rebuild the image with a newer Kopia binary and re-enroll.

Step 5 — Configure the conductor to use the image

Update the conductor configuration (see the Configuration section) with the Glance image ID and the desired capacity_slots per worker. Restart the abaca-conductor pod to apply changes:

# On OpenShift
oc rollout restart deployment/abaca-conductor -n <namespace>

Worker VM behavior is controlled by settings in the conductor's configuration. The table below covers the options that govern fleet sizing, job dispatch, and the Kopia executor.

Fleet and capacity settings

Kopia executor

The [kopia] executor setting controls how the conductor dispatches Kopia commands:

FIPS mode

When FIPS mode is required, the worker VM image must be built from RHEL with FIPS mode enabled at the OS level. Kopia is configured by the worker agent to use only FIPS-approved algorithms:

If the pinned Kopia binary does not support --key-derivation-algorithm, the conductor records fips_kdf_gap: true on the target's capabilities and emits a WARNING. The repository is still created but uses scrypt, which is not FIPS-140 compliant for key derivation. Rebuild the image with Kopia ≥ 0.23.1 and re-enroll the target.

Network attachment strategy

The network attachment strategy determines how the conductor connects a booted worker VM to the tenant's share network:

Choose the strategy that matches your Manila deployment's DHSS configuration. The strategy is set per-target during enrollment and cannot be changed without re-enrolling.

Subprocess timeout

The [kopia] subprocess_timeout setting bounds how long the conductor waits for a Kopia command in a worker to complete before treating the job as failed. If a repository create times out (for example, on a slow or distant S3 endpoint), the job moves to error with error_category: operator_action_required. Increase this value for high-latency S3 endpoints.

As a tenant or operator, you do not interact with worker VMs directly — the conductor manages their entire lifecycle. Your primary interactions are through the abaca-api REST API (or openstack share protection … CLI commands via python-abacaclient) to submit backup and restore jobs, and then to monitor those jobs as they move through the worker.

How the conductor assigns jobs to workers

When you create a backup or restore, the API records a job in the queued state and casts it to the conductor. The conductor runs the fleet picker, which examines every registered worker and counts how many non-terminal jobs are currently assigned to it. A worker is eligible only if its current running job count is strictly less than its capacity_slots. The conductor picks an eligible worker and dispatches the job to the abaca-worker-agent running inside that VM via oslo.messaging.

If no worker is currently below capacity, the job remains in queued until capacity frees up.

Polling job progress

Because all data-plane operations are asynchronous, you follow progress by polling the job resource:

# Poll a job by ID
openstack share protection job show <job-id>

Or directly via the REST API:

GET /v1/jobs/<job-id>
X-Auth-Token: <keystone-token>

The job moves through the following states as the worker executes it:

1. queued — waiting for a worker with available capacity
2. provisioning_network — the conductor is attaching the worker to the tenant's share network
3. provisioning_source — the snapshot strategy is preparing the share source (creating a Manila snapshot, clone, or access rule)
4. connecting_repository — the worker is connecting to the Kopia repository in the tenant's S3 bucket
5. transferring — Kopia is running the snapshot (backup) or restore operation
6. finalizing — the worker parses the Kopia manifest and persists statistics to the backup or restore record
7. releasing — cleanup runs in reverse order: repository disconnect → source unmount → network detach
8. available or error — terminal states

Understanding worker lifecycle from an operator perspective

Workers are long-lived across many jobs — they are not discarded after each backup. A worker continues to pick up new jobs until one of two things happens:

• It goes DEAD: the worker misses heartbeats beyond the reap threshold. The conductor marks it DEAD, walks its non-terminal jobs to error, and may boot a replacement.
• An operator drains it: you signal the conductor to stop assigning new jobs to a specific worker, let its current jobs complete, and then decommission it. Use this before updating the worker image.

Secrets handling during a job

When the conductor dispatches a job to a worker, S3 credentials and the Kopia repository password are fetched from Barbican using the tenant's Keystone trust and passed to the worker process as environment variables (KOPIA_PASSWORD, AWS_*). They are never written to disk, never passed as command-line arguments, and never logged. When the RPC returns, they are gone from process memory. You do not need to rotate secrets between jobs — each job fetches fresh credentials at dispatch time.

Example 1 — Monitor a running backup job through worker states

After triggering a backup, retrieve the job ID from the API response and poll until the job reaches a terminal state:

# Trigger a backup (returns 202 Accepted with job_id)
JOB_ID=$(openstack share protection backup create \
  --policy <policy-id> \
  -f value -c job_id)

echo "Tracking job: $JOB_ID"

# Poll until terminal
while true; do
  STATUS=$(openstack share protection job show "$JOB_ID" -f value -c status)
  echo "$(date -u +%H:%M:%S) state=$STATUS"
  [[ "$STATUS" == "available" || "$STATUS" == "error" ]] && break
  sleep 10
done

Expected output as the job moves through worker states:

09:00:01 state=queued
09:00:11 state=provisioning_network
09:00:21 state=provisioning_source
09:00:31 state=connecting_repository
09:00:41 state=transferring
09:02:15 state=finalizing
09:02:25 state=releasing
09:02:35 state=available

Example 2 — Inspect a failed job and identify who needs to act

If a job lands in error, check the error_category field to determine whether the tenant or the operator needs to act:

openstack share protection job show <job-id> -f json

Example response excerpt for a bucket misconfiguration caught during connecting_repository:

{
  "id": "a1b2c3d4-...",
  "status": "error",
  "state": "connecting_repository",
  "error_category": "tenant_action_required",
  "error_message": "Bucket has lifecycle expiration rules that would corrupt the Kopia repository."
}

A tenant_action_required category means the tenant must fix the S3 bucket configuration (remove expiration lifecycle rules, in this case) and re-enroll the target. An operator_action_required category means there is an infrastructure problem — for example, the conductor cannot reach the object store, or a Kopia subprocess timed out.

Example 3 — Verify FIPS key-derivation capability after enrollment

After enrolling a backup target, check the target's capabilities to confirm the worker's Kopia binary supports FIPS-compliant key derivation:

openstack share protection target show <target-id> -f json | \
  python3 -c "import sys,json; c=json.load(sys.stdin)['capabilities']; \
  print('fips_kdf_gap:', c.get('fips_kdf_gap')); \
  print('kdf:', c.get('kopia_key_derivation'))"

Expected output when the worker image contains Kopia ≥ 0.14 with --key-derivation-algorithm support:

fips_kdf_gap: False
kdf: pbkdf2

If fips_kdf_gap is True and kopia_key_derivation is scrypt, the repository is not FIPS-compliant. Rebuild the worker image with Kopia ≥ 0.23.1 and re-enroll the target.

Example 4 — Set up host-side NAT egress for worker VMs on VMware ESXi

On VMware ESXi standard vSwitches, workers on the tenant network cannot reach external S3 endpoints through a standard Neutron router because the vSwitch drops unicast replies not addressed to the vNIC's own MAC. Use the provided script to plumb a host-side NAT gateway:

# Plumb NAT egress for the tenant network named 'dr-net'
TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh

With explicit overrides:

TENANT_NET_NAME=dr-net \
  GW_IP=192.168.100.254 \
  NAT_EGRESS_IFACE=ens160 \
  VETH_HOST_NAME=hnat0 \
  bash deploy/kolla/setup-host-nat-egress.sh

After running the script, existing worker VMs must renew their DHCP leases to pick up the new gateway. SSH into each affected worker and run:

dhclient -r && dhclient

Or reboot the worker VM. This NAT configuration is not persistent across controller reboots; re-run the script after any host reboot.

Job stuck in queued

Symptom: A backup or restore job remains in queued for an extended period without advancing.

Likely cause: All registered worker VMs are at full capacity — their current running job count equals their capacity_slots. No eligible worker is available for the fleet picker to select.

Fix:

1. Check current job assignments across the fleet. If all workers are busy, either wait for running jobs to complete or increase the capacity_slots value in the conductor configuration and restart the conductor pod.
2. If workers appear idle but jobs are still stuck, check whether any workers are in a DEAD state. Dead workers are not eligible for job assignment. Look for missed heartbeat warnings in the abaca-conductor pod logs.
3. Verify RabbitMQ connectivity. If the conductor cannot publish to the abaca-conductor topic, jobs will queue indefinitely.

Job fails in provisioning_network with operator_action_required

Symptom: A backup job advances from queued to provisioning_network and then moves to error with error_category: operator_action_required.

Likely cause: The conductor could not attach the worker VM to the tenant's share network. For AttachToShareNetwork (DHSS=true), this typically means a Neutron or Nova API failure. For StaticReachability (DHSS=false), the pre-existing route may be broken.

Fix:

1. Check the abaca-conductor pod logs for the specific Neutron or Nova error.
2. Verify that the Nova and Neutron APIs are healthy: openstack network list and openstack server list.
3. If you are on VMware ESXi and workers cannot reach external endpoints, set up host-side NAT egress (see Example 4 in the Examples section).
4. For StaticReachability, verify that the route from the worker's network to the share network is active.

Job fails in connecting_repository with tenant_action_required

Symptom: A job reaches connecting_repository and fails with error_category: tenant_action_required.

Likely cause: The S3 bucket does not pass preflight conformance checks. Common causes include:

• The bucket has lifecycle expiration rules that would delete backup objects.
• The bucket contains unrecognized (non-Kopia) objects and cannot be adopted.
• The S3 credentials stored in Barbican cannot be fetched via the Keystone trust (for example, the trust has expired or been revoked).
• The bucket is not reachable from the worker's network scope.

Fix:

1. Inspect the error_message field on the job for the specific check that failed.
2. For lifecycle expiration rules: remove all expiration and delete-transition lifecycle rules from the bucket. Object lock and versioning rules are required and must remain.
3. For mixed-use buckets: S3 object lock prevents deletion of existing objects within the retention window. You must create a new, empty bucket with versioning and object lock enabled, and re-enroll with the new bucket.
4. For trust or credential failures: re-run abaca-dev enroll as the tenant to refresh the Keystone trust and re-store credentials in Barbican.
5. For reachability failures on development environments: verify the worker can reach the S3 endpoint. On Kolla with VMware, the Neutron router may not provide egress — see the NAT egress setup in the Examples section.

Job fails in connecting_repository with operator_action_required (subprocess timeout)

Symptom: A target enrollment or backup job fails at connecting_repository or transferring with error_category: operator_action_required and an error message referencing a timeout.

Likely cause: The Kopia subprocess running in the worker was killed after exceeding the [kopia] subprocess_timeout value. This typically happens with slow or high-latency S3 endpoints during initial repository creation.

Fix:

1. Increase the [kopia] subprocess_timeout value in the conductor configuration.
2. Restart the abaca-conductor pod and retry the enrollment or backup.
3. If the issue persists, verify S3 endpoint latency from the worker's network scope.

fips_kdf_gap: true on a newly enrolled target

Symptom: After enrolling a backup target, openstack share protection target show reports fips_kdf_gap: true and kopia_key_derivation: scrypt in the target's capabilities.

Likely cause: The Kopia binary in the worker VM image does not support the --key-derivation-algorithm flag (present in Kopia ≥ 0.14). The repository was initialized with scrypt, which is not FIPS-140 compliant for key derivation.

Fix:

1. Rebuild the worker VM image with Kopia ≥ 0.23.1 (the supported version pinned in build/kopia.env).
2. Upload the new image to Glance and update the conductor configuration to reference the new image ID.
3. Drain existing workers running the old image.
4. Re-enroll the affected target. Re-enrollment creates a new Kopia repository with pbkdf2 key derivation. Note that any backups stored under the old scrypt-keyed repository are still accessible as long as the old credentials are retained, but a new repository will be initialized.

Worker goes DEAD and jobs move to error

Symptom: One or more jobs that were in a non-terminal state (for example, transferring) suddenly move to error. The error_message references a dead worker.

Likely cause: The worker VM stopped sending heartbeats to the conductor beyond the reap threshold. This can be caused by the VM being stopped, a network partition between the worker and the conductor's RabbitMQ endpoint, or the abaca-worker-agent process crashing inside the VM.

Fix:

1. Check the worker VM's status in Nova: openstack server list.
2. If the VM is running, SSH into it and check the abaca-worker-agent process. Review its logs for panic or OOM errors.
3. Check connectivity from the worker VM to the RabbitMQ endpoint used by the conductor.
4. Failed jobs must be retried by the tenant (create a new backup or restore). The reconciliation loop in abaca-conductor will walk orphaned stuck jobs to error automatically, but does not automatically retry them.
5. If the worker VM is permanently lost, the conductor will eventually boot a replacement according to its fleet management policy. Verify that the worker VM image is still available in Glance.