Runbook

Troubleshooting

Common errors, failure modes, and debugging procedures

Objective

This runbook helps you diagnose and resolve common failure modes in Trilio Share Protection (Abacá), covering preflight check failures, job state machine errors, enrollment problems, restore issues, and data-fidelity gaps — use it when a backup target, protection policy, backup job, or restore job enters an error state or produces unexpected results.

Scope

Covered:

Preflight check failures during backup target enrollment
Job state machine errors (all states from queued through available or error)
Enrollment failures and Kopia repository initialization problems
Restore failures and known data-fidelity limitations
S3 object lock and lifecycle rule conflicts
Worker VM connectivity and network attachment failures
error_category interpretation and escalation routing
Recovering bookkeeping from repositories after database loss (summary guidance)

Not covered:

Initial deployment and infrastructure provisioning (see the deployment runbooks)
OpenShift control-plane installation
Worker VM image builds
Fleet health monitoring and coverage reporting
Internal Kopia repository format upgrades
Billing and chargeback integration

Prerequisites

Before using this runbook, ensure you have:

Access to the abaca-api service endpoint and a valid Keystone token scoped to the relevant project
python-abacaclient installed (openstack share protection … commands available)
abaca-manage available on the control-plane node (for operator-level diagnostics)
abaca-dev available if you need to run live data-path or worker-boot probes
Read access to job records (poll GET /v1/jobs/{job_id}) and target records (GET /v1/targets/{target_id})
S3 bucket credentials (access key, secret key, endpoint URL) and the corresponding Barbican secret hrefs
OpenStack CLI (openstack) authenticated with admin credentials for Neutron/Nova diagnostics
Trilio Share Protection ≥ version 4.12
Kopia ≥ 0.23.1 baked into the worker VM image
Python ≥ 3.11 if running any SDK-based diagnostic scripts

Steps

1. Identify the failing resource and retrieve its job record

Every asynchronous operation in Abacá — backup, restore, enrollment, or maintenance — is tracked as a job. Start here.

openstack share protection job show <job_id>

Or via the REST API:

GET /v1/jobs/{job_id}

Note the following fields in the response:

status — the current state in the job state machine
error_category — either tenant_action_required or operator_action_required
error_message — a human-readable description of the failure
preflight_detail — present on enrollment jobs; contains per-check results

Why: The error_category field tells you immediately who needs to act. tenant_action_required means the problem is in tenant-controlled configuration (bucket policy, credentials, lifecycle rules). operator_action_required means infrastructure or service-level attention is needed.

2. Interpret the job state machine position

Every job passes through this fixed sequence:

queued → provisioning_network → provisioning_source → connecting_repository
       → transferring → finalizing → releasing → available
                                                         ↘ error

Match the status at failure to the likely cause:

State at failure	Likely cause
`provisioning_network`	Worker VM failed to boot or attach to the tenant share network; check Nova/Neutron logs and network attachment strategy
`provisioning_source`	Manila share snapshot could not be created or accessed; check share status and snapshot access strategy
`connecting_repository`	Kopia could not connect to the S3 bucket; check S3 credentials in Barbican and target enrollment status
`transferring`	Data transfer error — network interruption, S3 write failure, or Kopia internal error
`finalizing`	Post-transfer metadata recording failed; check conductor logs
`releasing`	Worker VM teardown failed; the conductor reconciliation loop will eventually clean up orphaned workers

3. Diagnose preflight check failures on enrollment

When a backup target enrollment fails, the job record's preflight_detail contains a structured audit trail. Each check is one of: ok, warn, fail, or adoptable.

The checks run in this order, and a fail on endpoint_reach short-circuits all subsequent checks:

Check name	What it tests	Common fix
`endpoint_reach`	S3 endpoint reachable, DNS, TCP, TLS, and IAM	Verify the endpoint URL, CA certificate, and that the bucket exists
`rw_roundtrip`	PUT → GET → DELETE sentinel object	Check bucket policy grants read, write, and delete to the provided credentials
`addressing_mode`	Virtual-hosted vs. path-style addressing	For MinIO, use path-style; for AWS S3 and Ceph RGW, virtual-hosted is preferred
`object_lock`	Whether S3 object lock is enabled	Not a failure — only a capability flag; enable object lock at bucket creation for immutability
`lifecycle_rules`	Expiration lifecycle rules present	Remove all expiration lifecycle rules from the bucket — they silently delete Kopia objects and corrupt the repository
`bucket_contents`	Empty, adoptable Kopia repo, or mixed-use	Use a dedicated bucket; if an existing Kopia repository is present it will be flagged `adoptable`

Retrieving preflight detail:

openstack share protection target show <target_id>

Look for the preflight_detail field. Each entry has name, status, detail, and (on failure) category.

Lifecycle rule failure example:

If lifecycle_rules is fail, the detail.rule_ids field lists the offending rules. Remove them from your bucket using your S3 provider's console or CLI before retrying enrollment:

# AWS CLI example — adapt for your provider
aws s3api delete-bucket-lifecycle --bucket <your-bucket-name>

Then re-trigger enrollment:

openstack share protection target enroll <target_id>

Object lock warning:

If object_lock returns warn, Abacá could not read the lock configuration (access denied). Enrollment can proceed, but you should verify object lock status out-of-band. Object lock must be enabled at bucket creation time and cannot be added to an existing bucket.

Mixed-use bucket failure:

If bucket_contents is fail with reason mixed use, move your existing non-Kopia objects to a different bucket. Abacá requires a dedicated bucket per backup target.

4. Diagnose S3 credential and Barbican reference problems

Abacá never stores credentials directly — it holds references (hrefs) to secrets in Barbican. If the Barbican secret has been deleted or its ACL changed, jobs will fail at connecting_repository.

Check that the Barbican secret hrefs on the target are valid:

openstack secret show <secret_href>

If the secret is missing or inaccessible, you must create a new Barbican secret with the correct credentials and update the target record.

Check the Keystone trust is still valid:

Abacá uses a Keystone trust to act on the tenant's behalf. If the trust has expired or been revoked, enrollment and backup jobs will fail. Re-create the trust by re-running the enrollment workflow or by updating the target with a fresh trust.

5. Diagnose worker VM network attachment failures

If a job fails at provisioning_network, the worker VM could not be booted or could not attach to the tenant's share network.

Check whether the service project has permission to create/attach ports on tenant share networks (DHSS=true):

openstack share protection target show <target_id>
# Look at the network_attachment_strategy field

# Probe empirically using abaca-dev
abaca-dev worker-boot --target <target_id>

The findings are recorded in the abaca-dev report. Port privilege is the only known potential blocker for DHSS=true environments. DHSS=false (StaticReachability) is unaffected by port privileges.

Check Nova and Neutron for boot errors:

openstack server list --project <worker_project>
openstack server show <worker_vm_id>

If the worker VM is in ERROR state, retrieve its fault:

openstack server show <worker_vm_id> -f json | python3 -c \
  "import json,sys; d=json.load(sys.stdin); print(d.get('fault',{}))"

Check that the worker flavor has the required CPU crypto instructions (required for FIPS mode; see the FIPS section below).

6. Diagnose Manila share access and snapshot failures

If a job fails at provisioning_source, Abacá could not get a consistent read-only view of the share.

Check Manila share and snapshot status:

openstack share show <share_id>
openstack share snapshot list --share-id <share_id>

Export policy root-squash: If the Manila backend export policy squashes root, the worker agent cannot perform the root-level read and ownership operations needed for a faithful backup. This is detected at runtime as a read+chown probe failure and surfaces as operator_action_required. Ask your storage administrator to allow root access from the worker's IP on this share's export policy.

ONTAP snapdir availability: If your Manila backend is NetApp ONTAP and .snapshot directories are not enabled on provisioned volumes, Abacá falls back to an alternative snapshot access strategy automatically. The runtime probe records which strategy was used in the job detail. No action is required unless all fallbacks are exhausted.

7. Diagnose restore failures and understand data-fidelity limitations

Restore job failing at connecting_repository or transferring: Follow the same S3 credential and Barbican diagnosis in step 4.

Restore completes but files have wrong ownership or missing setuid/setgid bits:

This is a known phase-1 limitation of the Kopia-based restore path. Empirical validation on Kopia 0.17.0 and 0.23.1 confirms that Kopia does not preserve numeric UID/GID or setuid/setgid bits during filesystem-mode restore — files come back owned by the invoking user and special mode bits are stripped.

Affected workloads: Shares storing files with system-level ownership (files owned by root, postgres, mysql, or other service UIDs), or setuid executables restored to a target that respects those identities.

Unaffected workloads: Manila shares accessed by a single application UID with no setuid binaries — the restored share is functionally identical to the original.

What is preserved correctly:

File contents
Modification times (mtime)
Sparse file layout (restored sparse)
Symlinks (absolute and relative)
Standard POSIX rwx and sticky bits
Directory structure
Hardlinks restored as independent identical copies (expected behavior, not a bug)

What is NOT preserved (phase 1):

Numeric UID/GID (known Kopia limitation — planned compensation: stat-manifest sidecar)
setuid/setgid bits (same root cause)
NFSv4/NTFS-style ACLs and extended attributes (xattrs) — explicitly out of scope for phase 1
Sockets and device nodes — not captured

Crash consistency note: Backups using the LiveBackup strategy are crash-consistent but not point-in-time consistent. If your application had unflushed in-flight writes at backup time, those writes may not appear in the restore. This is equivalent to crash recovery — not data corruption.

Hardlinks restored as copies: Multiple directory entries sharing an inode on the source are restored as independent copies. Total disk usage after restore may be higher than on the original share. This is expected behavior.

8. Diagnose FIPS-related failures

Abacá configures Kopia to use only FIPS-approved encryption and hash algorithms and runs worker VMs in RHEL FIPS mode. If your environment has a FIPS compliance requirement:

Verify the worker VM image was built with RHEL FIPS mode enabled.
Verify the worker VM flavor's compute host has CPU crypto instructions (AES-NI or equivalent); without them, FIPS-mode cryptographic performance degrades significantly and jobs may time out.
The fips_kdf_gap flag in the target's capabilities indicates whether the pinned Kopia build lacks PBKDF2 key derivation. If this flag is set, contact your operator — a patched Kopia build is required before the target can be used in a validated FIPS configuration.

openstack share protection target show <target_id>
# Check capabilities.fips_kdf_gap

9. Handle stuck or orphaned jobs

If a job remains in a non-terminal state for longer than expected, the abaca-conductor reconciliation loop will eventually detect and remediate it. You can check whether reconciliation has run recently by examining conductor logs.

For operator-level intervention (forcing a job to error, releasing a stuck worker, or running a manual reconcile pass), use abaca-manage:

abaca-manage job show <job_id>
abaca-manage reconcile --force

10. Recover bookkeeping from repositories after database loss

The Kopia repository stored in your S3 bucket is the authoritative record of your backup data. The Abacá database holds only a summary index. If the database is lost:

Do not delete or modify the S3 bucket or its contents.
Use abaca-manage to re-initialize the database schema:
```
abaca-manage db_sync
```
Re-enroll each backup target. Because the bucket already contains a Kopia repository, the bucket_contents preflight check will return adoptable and the enrollment path will adopt the existing repository rather than creating a new one.
After adoption, run a reconciliation pass to re-index existing backups from the repository:
```
abaca-manage reconcile --force
```

Verify that backup records reappear under the target:

openstack share protection backup list --target <target_id>

Verification

After resolving a failure, confirm recovery using the following checks:

1. Job reached available status:

openstack share protection job show <job_id>
# status should be: available

2. Target enrollment is complete and preflight shows no failures:

openstack share protection target show <target_id>
# preflight_detail.overall should be: ok or adoptable
# enrollment_status should be: enrolled

3. Backup record exists and carries expected statistics:

openstack share protection backup list --target <target_id>
openstack share protection backup show <backup_id>
# Check: status=available, size_bytes > 0, consistency_flag present

4. Restore completed and share is accessible:

openstack share protection restore show <restore_id>
# status should be: available
openstack share show <restored_share_id>
# status should be: available

5. For ownership/permission fidelity validation (phase-1 known gap):

Run abaca-dev fidelity probes against the restored share to get an empirical fidelity matrix:

abaca-dev fidelity-probe --share <restored_share_id>

The probe emits a findings report documenting which attributes round-tripped correctly. Compare against the known-gap description in step 7 to confirm the outcome matches expectations.

6. Preflight checks can be re-run independently against a target to confirm S3 bucket configuration is now valid without requiring a full re-enrollment.

Rollback

Most Abacá operations are non-destructive by design — backups write to your S3 bucket but do not modify the source share, and restores write to a new share by default. However:

Rolling back a failed enrollment:

If enrollment created partial state, delete the target and re-create it after fixing the underlying issue:

openstack share protection target delete <target_id>

The S3 bucket is not modified by a failed enrollment unless a Kopia repository was partially initialized. If bucket_contents returns anything other than empty after deletion, inspect the bucket and remove any .abaca-preflight/* sentinel objects left by the rw_roundtrip check. These are automatically cleaned up by a successful check but may linger after an interrupted run.

Rolling back a restore:

By default, restores target a new share. Delete the new share if the restore result is not usable:

openstack share delete <restored_share_id>

If you performed an in-place restore to the original share and the result is incorrect, restore again from an earlier backup point.

Rolling back a protection policy:

Detach the policy from the share to stop future scheduled backups:

openstack share protection policy delete <policy_id>

Existing backups are not affected by policy deletion — they remain in the repository until pruned by the retention maintenance job.

Full environment teardown (Kolla-based deployments):

If you need to completely reverse an Abacá deployment, the teardown script at deploy/kolla/99-teardown.sh stops and removes the abaca-api and abaca-conductor containers, removes the Horizon plugin, drops the MariaDB schema and RabbitMQ vhost, and cleans up Keystone catalog entries. This is destructive and irreversible for the control-plane state — S3 bucket contents are not touched.

bash deploy/kolla/99-teardown.sh

Warning: Teardown does not delete S3 bucket contents. Your backup data remains in place and can be adopted by a future re-enrollment.

Escalation

If this runbook does not resolve the issue, escalate with the following information:

1. Job record (full JSON):

openstack share protection job show <job_id> -f json

Include status, error_category, error_message, and preflight_detail (for enrollment jobs).

2. Target record (full JSON):

openstack share protection target show <target_id> -f json

Do not include raw S3 credentials. Include only the Barbican secret hrefs.

3. abaca-dev findings report:

If the failure involves worker provisioning, network attachment, or data-path fidelity, run:

abaca-dev worker-boot --target <target_id>

Attach the findings JSON report. This is especially important for open item #1 (port privilege) and FIPS KDF gap issues.

4. Conductor and API logs from the relevant time window. Request these from your operator if you do not have direct log access.

5. Environment details:

OpenStack release (e.g., RHOSO 18, Kolla-Ansible 2024.2)
Trilio Share Protection version (minimum 4.12)
Manila backend type and DHSS setting
S3-compatible storage provider (AWS S3, MinIO, Ceph RGW, ODF, Wasabi)
Whether FIPS mode is enabled

Routing by error_category:

tenant_action_required — the tenant or application team must fix bucket configuration, credentials, or lifecycle rules before re-opening with support.
operator_action_required — escalate to your infrastructure or platform operations team, and include Nova, Neutron, and Manila logs for the affected share network.