--- title: Worker VM product: trilio-share-protection doc_type: guide version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/worker-vm --- # Worker VM _How the worker VM is deployed and is managed by the conductor service_ ## Overview This page explains how Abacá's worker VMs are deployed, how the conductor service manages their lifecycle, and what happens inside a worker during a backup or restore job. Worker VMs are the data-plane of Abacá: they are the only component that ever touches plaintext share data or encryption keys, and they do so entirely within the tenant's own network scope, keeping the trust boundary tight. Understanding how workers are provisioned, assigned jobs, and reaped is essential for operators tuning fleet capacity, diagnosing stuck jobs, and reasoning about the security model. ## Prerequisites Before working with worker VMs, ensure the following are in place: - **OpenStack ≥ 2023.1 (Antelope) / RHOSO 18** with the following services active: - **Keystone** — for tenant authentication and trust-scoped token issuance - **Barbican** — for tenant-owned storage of S3 credentials and the Kopia repository password - **Manila** — for share access rules and, in DHSS=true deployments, share network metadata - **Nova and Neutron** — required when using the `AttachToShareNetwork` network attachment strategy (DHSS=true); optional for `StaticReachability` (DHSS=false) - **OpenShift ≥ 4.12** running the Abacá control plane (`abaca-api` and `abaca-conductor` pods) - **RHEL worker VM image with Kopia ≥ 0.23.1 baked in** — the `abacа-worker-agent` and Kopia binary must be present in the image before it is registered with the conductor - **S3-compatible object storage** (AWS S3, MinIO, Ceph RGW, ODF, or Wasabi) with a properly configured tenant bucket (versioning + object lock enabled; see Tenant S3 Bucket Requirements) - **RabbitMQ** — the conductor dispatches job commands to workers over oslo.messaging; the `abaca-conductor` topic must be reachable - **MariaDB ≥ 10.x (Kolla 2024.2+)** — stores job state, worker registration records, and capacity slot counts - A registered **backup target** for the tenant (enrollment must be in `available` status before backup or restore jobs can be dispatched to a worker) ## Installation Worker VMs are not installed manually by end users. The conductor provisions and manages them automatically. What you, as an operator, must do is build and register the worker VM image that the conductor will boot. **Step 1 — Pin the Kopia version in the build environment** The worker image build reads the Kopia version from `build/kopia.env`. Verify the pin matches the supported version before building: ```bash cat build/kopia.env # KOPIA_VERSION=0.23.1 ``` Do not change this value unless you have validated the new version with `abaca-dev` (see Step 4). The version pinned here must match the Kopia binary used during enrollment so that FIPS key-derivation algorithm support (`--key-derivation-algorithm`) is consistent across all workers. **Step 2 — Build the worker VM image** Use the image build tooling provided in the repository. The resulting RHEL image must contain: - The `abaca-worker-agent` binary (the in-VM agent that receives RPC commands from the conductor) - The Kopia binary at the version pinned in `build/kopia.env` - RHEL FIPS mode enabled (required for FIPS-validated deployments) ```bash # Build the worker image (exact tooling path confirmed by operator) bash build/build-worker-image.sh ``` > **Note for reviewer:** The exact image build script name and path should be confirmed from source. The above reflects the documented `build/kopia.env` location; the build entry point script name is not specified in the available source material. **Step 3 — Upload the image to Glance** Once built, upload the worker image to OpenStack Image service (Glance) so the conductor can reference it when booting worker VMs via Nova: ```bash openstack image create \ --file worker-vm.qcow2 \ --disk-format qcow2 \ --container-format bare \ --property abaca_worker_image=true \ abaca-worker-rhel ``` Record the image ID; you will reference it in the conductor configuration. **Step 4 — Validate the image with `abaca-dev`** Before registering the image for production use, run the `abaca-dev` CLI to probe the data path and confirm fidelity and capability validation against your real OpenStack environment: ```bash export OS_CLOUD=abaca-tenant abaca-dev enroll ``` Review the findings report that `abaca-dev` generates. Pay particular attention to the `fips_kdf_gap` field in the target's `capabilities` after enrollment — if it is `true`, the pinned Kopia binary does not support `--key-derivation-algorithm` and the repository was initialized with scrypt instead of pbkdf2, which is not FIPS-140 compliant. Rebuild the image with a newer Kopia binary and re-enroll. **Step 5 — Configure the conductor to use the image** Update the conductor configuration (see the Configuration section) with the Glance image ID and the desired `capacity_slots` per worker. Restart the `abaca-conductor` pod to apply changes: ```bash # On OpenShift oc rollout restart deployment/abaca-conductor -n ``` ## Configuration Worker VM behavior is controlled by settings in the conductor's configuration. The table below covers the options that govern fleet sizing, job dispatch, and the Kopia executor. ### Fleet and capacity settings | Option | Description | Effect | |---|---|---| | `capacity_slots` | Number of concurrent jobs a single worker VM can run | The conductor's fleet picker (`WorkerFleet.pick_worker_for_job`) only assigns a job to a worker whose current running job count is below this value. Increase to improve throughput per VM; decrease to reduce memory pressure per worker. | | Worker VM image (Glance image ID) | The RHEL image containing `abaca-worker-agent` and Kopia | Must be set before the conductor can boot workers. Changing the image does not affect already-running workers until they are drained and rebooted. | | Heartbeat cadence | How often each worker VM reports liveness to the conductor | Workers that miss heartbeats beyond the reap threshold are marked DEAD. The conductor then walks their non-terminal jobs to `error` and may boot a replacement. | | Reap threshold | Number of missed heartbeats before a worker is marked DEAD | Tune conservatively in environments with high network latency between the worker and the control plane. | ### Kopia executor The `[kopia] executor` setting controls how the conductor dispatches Kopia commands: | Value | Behaviour | When to use | |---|---|---| | `worker_rpc` | Production default. Each Kopia command is cast to a fleet-picked worker VM via oslo.messaging. The conductor never holds tenant S3 credentials or the repository password — those exist only in the worker process's environment for the duration of the RPC. | All production deployments (R2+). | > **Note for reviewer:** Additional executor values (for example, a local/dev executor) are referenced in the source architecture notes but their configuration names are not fully specified in the available source material. Confirm from `abaca-conductor` configuration schema before documenting them. ### FIPS mode When FIPS mode is required, the worker VM image must be built from RHEL with FIPS mode enabled at the OS level. Kopia is configured by the worker agent to use only FIPS-approved algorithms: | Algorithm role | FIPS-approved value | |---|---| | Encryption | `AES256-GCM-HMAC-SHA256` | | Block hash | `HMAC-SHA256-128` | | Key derivation | `pbkdf2` (requires Kopia ≥ 0.14 with `--key-derivation-algorithm` support) | If the pinned Kopia binary does not support `--key-derivation-algorithm`, the conductor records `fips_kdf_gap: true` on the target's `capabilities` and emits a WARNING. The repository is still created but uses scrypt, which is not FIPS-140 compliant for key derivation. Rebuild the image with Kopia ≥ 0.23.1 and re-enroll the target. ### Network attachment strategy The network attachment strategy determines how the conductor connects a booted worker VM to the tenant's share network: | Strategy | DHSS mode | Mechanism | |---|---|---| | `AttachToShareNetwork` | DHSS=true | The conductor hot-plugs a Neutron port into the tenant's share network. Requires Nova and Neutron. | | `StaticReachability` | DHSS=false | The conductor relies on a pre-existing routable path to the share network. No Neutron port manipulation. | Choose the strategy that matches your Manila deployment's DHSS configuration. The strategy is set per-target during enrollment and cannot be changed without re-enrolling. ### Subprocess timeout The `[kopia] subprocess_timeout` setting bounds how long the conductor waits for a Kopia command in a worker to complete before treating the job as failed. If a repository create times out (for example, on a slow or distant S3 endpoint), the job moves to `error` with `error_category: operator_action_required`. Increase this value for high-latency S3 endpoints. ## Usage As a tenant or operator, you do not interact with worker VMs directly — the conductor manages their entire lifecycle. Your primary interactions are through the `abaca-api` REST API (or `openstack share protection …` CLI commands via `python-abacaclient`) to submit backup and restore jobs, and then to monitor those jobs as they move through the worker. ### How the conductor assigns jobs to workers When you create a backup or restore, the API records a job in the `queued` state and casts it to the conductor. The conductor runs the fleet picker, which examines every registered worker and counts how many non-terminal jobs are currently assigned to it. A worker is eligible only if its current running job count is strictly less than its `capacity_slots`. The conductor picks an eligible worker and dispatches the job to the `abaca-worker-agent` running inside that VM via oslo.messaging. If no worker is currently below capacity, the job remains in `queued` until capacity frees up. ### Polling job progress Because all data-plane operations are asynchronous, you follow progress by polling the job resource: ```bash # Poll a job by ID openstack share protection job show ``` Or directly via the REST API: ```http GET /v1/jobs/ X-Auth-Token: ``` The job moves through the following states as the worker executes it: 1. `queued` — waiting for a worker with available capacity 2. `provisioning_network` — the conductor is attaching the worker to the tenant's share network 3. `provisioning_source` — the snapshot strategy is preparing the share source (creating a Manila snapshot, clone, or access rule) 4. `connecting_repository` — the worker is connecting to the Kopia repository in the tenant's S3 bucket 5. `transferring` — Kopia is running the snapshot (backup) or restore operation 6. `finalizing` — the worker parses the Kopia manifest and persists statistics to the backup or restore record 7. `releasing` — cleanup runs in reverse order: repository disconnect → source unmount → network detach 8. `available` or `error` — terminal states ### Understanding worker lifecycle from an operator perspective Workers are long-lived across many jobs — they are not discarded after each backup. A worker continues to pick up new jobs until one of two things happens: - **It goes DEAD**: the worker misses heartbeats beyond the reap threshold. The conductor marks it DEAD, walks its non-terminal jobs to `error`, and may boot a replacement. - **An operator drains it**: you signal the conductor to stop assigning new jobs to a specific worker, let its current jobs complete, and then decommission it. Use this before updating the worker image. ### Secrets handling during a job When the conductor dispatches a job to a worker, S3 credentials and the Kopia repository password are fetched from Barbican using the tenant's Keystone trust and passed to the worker process as environment variables (`KOPIA_PASSWORD`, `AWS_*`). They are never written to disk, never passed as command-line arguments, and never logged. When the RPC returns, they are gone from process memory. You do not need to rotate secrets between jobs — each job fetches fresh credentials at dispatch time. ## Examples ### Example 1 — Monitor a running backup job through worker states After triggering a backup, retrieve the job ID from the API response and poll until the job reaches a terminal state: ```bash # Trigger a backup (returns 202 Accepted with job_id) JOB_ID=$(openstack share protection backup create \ --policy \ -f value -c job_id) echo "Tracking job: $JOB_ID" # Poll until terminal while true; do STATUS=$(openstack share protection job show "$JOB_ID" -f value -c status) echo "$(date -u +%H:%M:%S) state=$STATUS" [[ "$STATUS" == "available" || "$STATUS" == "error" ]] && break sleep 10 done ``` Expected output as the job moves through worker states: ``` 09:00:01 state=queued 09:00:11 state=provisioning_network 09:00:21 state=provisioning_source 09:00:31 state=connecting_repository 09:00:41 state=transferring 09:02:15 state=finalizing 09:02:25 state=releasing 09:02:35 state=available ``` --- ### Example 2 — Inspect a failed job and identify who needs to act If a job lands in `error`, check the `error_category` field to determine whether the tenant or the operator needs to act: ```bash openstack share protection job show -f json ``` Example response excerpt for a bucket misconfiguration caught during `connecting_repository`: ```json { "id": "a1b2c3d4-...", "status": "error", "state": "connecting_repository", "error_category": "tenant_action_required", "error_message": "Bucket has lifecycle expiration rules that would corrupt the Kopia repository." } ``` A `tenant_action_required` category means the tenant must fix the S3 bucket configuration (remove expiration lifecycle rules, in this case) and re-enroll the target. An `operator_action_required` category means there is an infrastructure problem — for example, the conductor cannot reach the object store, or a Kopia subprocess timed out. --- ### Example 3 — Verify FIPS key-derivation capability after enrollment After enrolling a backup target, check the target's `capabilities` to confirm the worker's Kopia binary supports FIPS-compliant key derivation: ```bash openstack share protection target show -f json | \ python3 -c "import sys,json; c=json.load(sys.stdin)['capabilities']; \ print('fips_kdf_gap:', c.get('fips_kdf_gap')); \ print('kdf:', c.get('kopia_key_derivation'))" ``` Expected output when the worker image contains Kopia ≥ 0.14 with `--key-derivation-algorithm` support: ``` fips_kdf_gap: False kdf: pbkdf2 ``` If `fips_kdf_gap` is `True` and `kopia_key_derivation` is `scrypt`, the repository is not FIPS-compliant. Rebuild the worker image with Kopia ≥ 0.23.1 and re-enroll the target. --- ### Example 4 — Set up host-side NAT egress for worker VMs on VMware ESXi On VMware ESXi standard vSwitches, workers on the tenant network cannot reach external S3 endpoints through a standard Neutron router because the vSwitch drops unicast replies not addressed to the vNIC's own MAC. Use the provided script to plumb a host-side NAT gateway: ```bash # Plumb NAT egress for the tenant network named 'dr-net' TENANT_NET_NAME=dr-net bash deploy/kolla/setup-host-nat-egress.sh ``` With explicit overrides: ```bash TENANT_NET_NAME=dr-net \ GW_IP=192.168.100.254 \ NAT_EGRESS_IFACE=ens160 \ VETH_HOST_NAME=hnat0 \ bash deploy/kolla/setup-host-nat-egress.sh ``` After running the script, existing worker VMs must renew their DHCP leases to pick up the new gateway. SSH into each affected worker and run: ```bash dhclient -r && dhclient ``` Or reboot the worker VM. This NAT configuration is not persistent across controller reboots; re-run the script after any host reboot. ## Troubleshooting ### Job stuck in `queued` **Symptom:** A backup or restore job remains in `queued` for an extended period without advancing. **Likely cause:** All registered worker VMs are at full capacity — their current running job count equals their `capacity_slots`. No eligible worker is available for the fleet picker to select. **Fix:** 1. Check current job assignments across the fleet. If all workers are busy, either wait for running jobs to complete or increase the `capacity_slots` value in the conductor configuration and restart the conductor pod. 2. If workers appear idle but jobs are still stuck, check whether any workers are in a DEAD state. Dead workers are not eligible for job assignment. Look for missed heartbeat warnings in the `abaca-conductor` pod logs. 3. Verify RabbitMQ connectivity. If the conductor cannot publish to the `abaca-conductor` topic, jobs will queue indefinitely. --- ### Job fails in `provisioning_network` with `operator_action_required` **Symptom:** A backup job advances from `queued` to `provisioning_network` and then moves to `error` with `error_category: operator_action_required`. **Likely cause:** The conductor could not attach the worker VM to the tenant's share network. For `AttachToShareNetwork` (DHSS=true), this typically means a Neutron or Nova API failure. For `StaticReachability` (DHSS=false), the pre-existing route may be broken. **Fix:** 1. Check the `abaca-conductor` pod logs for the specific Neutron or Nova error. 2. Verify that the Nova and Neutron APIs are healthy: `openstack network list` and `openstack server list`. 3. If you are on VMware ESXi and workers cannot reach external endpoints, set up host-side NAT egress (see Example 4 in the Examples section). 4. For `StaticReachability`, verify that the route from the worker's network to the share network is active. --- ### Job fails in `connecting_repository` with `tenant_action_required` **Symptom:** A job reaches `connecting_repository` and fails with `error_category: tenant_action_required`. **Likely cause:** The S3 bucket does not pass preflight conformance checks. Common causes include: - The bucket has lifecycle expiration rules that would delete backup objects. - The bucket contains unrecognized (non-Kopia) objects and cannot be adopted. - The S3 credentials stored in Barbican cannot be fetched via the Keystone trust (for example, the trust has expired or been revoked). - The bucket is not reachable from the worker's network scope. **Fix:** 1. Inspect the `error_message` field on the job for the specific check that failed. 2. For lifecycle expiration rules: remove all expiration and delete-transition lifecycle rules from the bucket. Object lock and versioning rules are required and must remain. 3. For mixed-use buckets: S3 object lock prevents deletion of existing objects within the retention window. You must create a new, empty bucket with versioning and object lock enabled, and re-enroll with the new bucket. 4. For trust or credential failures: re-run `abaca-dev enroll` as the tenant to refresh the Keystone trust and re-store credentials in Barbican. 5. For reachability failures on development environments: verify the worker can reach the S3 endpoint. On Kolla with VMware, the Neutron router may not provide egress — see the NAT egress setup in the Examples section. --- ### Job fails in `connecting_repository` with `operator_action_required` (subprocess timeout) **Symptom:** A target enrollment or backup job fails at `connecting_repository` or `transferring` with `error_category: operator_action_required` and an error message referencing a timeout. **Likely cause:** The Kopia subprocess running in the worker was killed after exceeding the `[kopia] subprocess_timeout` value. This typically happens with slow or high-latency S3 endpoints during initial repository creation. **Fix:** 1. Increase the `[kopia] subprocess_timeout` value in the conductor configuration. 2. Restart the `abaca-conductor` pod and retry the enrollment or backup. 3. If the issue persists, verify S3 endpoint latency from the worker's network scope. --- ### `fips_kdf_gap: true` on a newly enrolled target **Symptom:** After enrolling a backup target, `openstack share protection target show` reports `fips_kdf_gap: true` and `kopia_key_derivation: scrypt` in the target's `capabilities`. **Likely cause:** The Kopia binary in the worker VM image does not support the `--key-derivation-algorithm` flag (present in Kopia ≥ 0.14). The repository was initialized with scrypt, which is not FIPS-140 compliant for key derivation. **Fix:** 1. Rebuild the worker VM image with Kopia ≥ 0.23.1 (the supported version pinned in `build/kopia.env`). 2. Upload the new image to Glance and update the conductor configuration to reference the new image ID. 3. Drain existing workers running the old image. 4. Re-enroll the affected target. Re-enrollment creates a new Kopia repository with pbkdf2 key derivation. Note that any backups stored under the old scrypt-keyed repository are still accessible as long as the old credentials are retained, but a new repository will be initialized. --- ### Worker goes DEAD and jobs move to `error` **Symptom:** One or more jobs that were in a non-terminal state (for example, `transferring`) suddenly move to `error`. The `error_message` references a dead worker. **Likely cause:** The worker VM stopped sending heartbeats to the conductor beyond the reap threshold. This can be caused by the VM being stopped, a network partition between the worker and the conductor's RabbitMQ endpoint, or the `abaca-worker-agent` process crashing inside the VM. **Fix:** 1. Check the worker VM's status in Nova: `openstack server list`. 2. If the VM is running, SSH into it and check the `abaca-worker-agent` process. Review its logs for panic or OOM errors. 3. Check connectivity from the worker VM to the RabbitMQ endpoint used by the conductor. 4. Failed jobs must be retried by the tenant (create a new backup or restore). The reconciliation loop in `abaca-conductor` will walk orphaned stuck jobs to `error` automatically, but does not automatically retry them. 5. If the worker VM is permanently lost, the conductor will eventually boot a replacement according to its fleet management policy. Verify that the worker VM image is still available in Glance.