--- title: Troubleshooting product: trilio-share-protection doc_type: runbook version: master source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-share-protection/trilio-share-protection-/troubleshooting --- # Troubleshooting _Common errors, failure modes, and debugging procedures_ ## Objective This runbook helps you diagnose and resolve common failure modes in Trilio Share Protection (Abacá), covering preflight check failures, job state machine errors, enrollment problems, restore issues, and data-fidelity gaps — use it when a backup target, protection policy, backup job, or restore job enters an error state or produces unexpected results. ## Scope **Covered:** - Preflight check failures during backup target enrollment - Job state machine errors (all states from `queued` through `available` or `error`) - Enrollment failures and Kopia repository initialization problems - Restore failures and known data-fidelity limitations - S3 object lock and lifecycle rule conflicts - Worker VM connectivity and network attachment failures - `error_category` interpretation and escalation routing - Recovering bookkeeping from repositories after database loss (summary guidance) **Not covered:** - Initial deployment and infrastructure provisioning (see the deployment runbooks) - OpenShift control-plane installation - Worker VM image builds - Fleet health monitoring and coverage reporting - Internal Kopia repository format upgrades - Billing and chargeback integration ## Prerequisites Before using this runbook, ensure you have: - Access to the `abaca-api` service endpoint and a valid Keystone token scoped to the relevant project - `python-abacaclient` installed (`openstack share protection …` commands available) - `abaca-manage` available on the control-plane node (for operator-level diagnostics) - `abaca-dev` available if you need to run live data-path or worker-boot probes - Read access to job records (poll `GET /v1/jobs/{job_id}`) and target records (`GET /v1/targets/{target_id}`) - S3 bucket credentials (access key, secret key, endpoint URL) and the corresponding Barbican secret hrefs - OpenStack CLI (`openstack`) authenticated with admin credentials for Neutron/Nova diagnostics - Trilio Share Protection ≥ version 4.12 - Kopia ≥ 0.23.1 baked into the worker VM image - Python ≥ 3.11 if running any SDK-based diagnostic scripts ## Steps ### 1. Identify the failing resource and retrieve its job record Every asynchronous operation in Abacá — backup, restore, enrollment, or maintenance — is tracked as a **job**. Start here. ```bash openstack share protection job show ``` Or via the REST API: ``` GET /v1/jobs/{job_id} ``` Note the following fields in the response: - `status` — the current state in the job state machine - `error_category` — either `tenant_action_required` or `operator_action_required` - `error_message` — a human-readable description of the failure - `preflight_detail` — present on enrollment jobs; contains per-check results **Why:** The `error_category` field tells you immediately who needs to act. `tenant_action_required` means the problem is in tenant-controlled configuration (bucket policy, credentials, lifecycle rules). `operator_action_required` means infrastructure or service-level attention is needed. --- ### 2. Interpret the job state machine position Every job passes through this fixed sequence: ``` queued → provisioning_network → provisioning_source → connecting_repository → transferring → finalizing → releasing → available ↘ error ``` Match the `status` at failure to the likely cause: | State at failure | Likely cause | |---|---| | `provisioning_network` | Worker VM failed to boot or attach to the tenant share network; check Nova/Neutron logs and network attachment strategy | | `provisioning_source` | Manila share snapshot could not be created or accessed; check share status and snapshot access strategy | | `connecting_repository` | Kopia could not connect to the S3 bucket; check S3 credentials in Barbican and target enrollment status | | `transferring` | Data transfer error — network interruption, S3 write failure, or Kopia internal error | | `finalizing` | Post-transfer metadata recording failed; check conductor logs | | `releasing` | Worker VM teardown failed; the conductor reconciliation loop will eventually clean up orphaned workers | --- ### 3. Diagnose preflight check failures on enrollment When a backup target enrollment fails, the job record's `preflight_detail` contains a structured audit trail. Each check is one of: `ok`, `warn`, `fail`, or `adoptable`. The checks run in this order, and a `fail` on `endpoint_reach` short-circuits all subsequent checks: | Check name | What it tests | Common fix | |---|---|---| | `endpoint_reach` | S3 endpoint reachable, DNS, TCP, TLS, and IAM | Verify the endpoint URL, CA certificate, and that the bucket exists | | `rw_roundtrip` | PUT → GET → DELETE sentinel object | Check bucket policy grants read, write, and delete to the provided credentials | | `addressing_mode` | Virtual-hosted vs. path-style addressing | For MinIO, use path-style; for AWS S3 and Ceph RGW, virtual-hosted is preferred | | `object_lock` | Whether S3 object lock is enabled | Not a failure — only a capability flag; enable object lock at bucket creation for immutability | | `lifecycle_rules` | Expiration lifecycle rules present | **Remove all expiration lifecycle rules from the bucket** — they silently delete Kopia objects and corrupt the repository | | `bucket_contents` | Empty, adoptable Kopia repo, or mixed-use | Use a dedicated bucket; if an existing Kopia repository is present it will be flagged `adoptable` | **Retrieving preflight detail:** ```bash openstack share protection target show ``` Look for the `preflight_detail` field. Each entry has `name`, `status`, `detail`, and (on failure) `category`. **Lifecycle rule failure example:** If `lifecycle_rules` is `fail`, the `detail.rule_ids` field lists the offending rules. Remove them from your bucket using your S3 provider's console or CLI before retrying enrollment: ```bash # AWS CLI example — adapt for your provider aws s3api delete-bucket-lifecycle --bucket ``` Then re-trigger enrollment: ```bash openstack share protection target enroll ``` **Object lock warning:** If `object_lock` returns `warn`, Abacá could not read the lock configuration (access denied). Enrollment can proceed, but you should verify object lock status out-of-band. Object lock must be enabled at bucket creation time and cannot be added to an existing bucket. **Mixed-use bucket failure:** If `bucket_contents` is `fail` with reason `mixed use`, move your existing non-Kopia objects to a different bucket. Abacá requires a dedicated bucket per backup target. --- ### 4. Diagnose S3 credential and Barbican reference problems Abacá never stores credentials directly — it holds references (hrefs) to secrets in Barbican. If the Barbican secret has been deleted or its ACL changed, jobs will fail at `connecting_repository`. **Check that the Barbican secret hrefs on the target are valid:** ```bash openstack secret show ``` If the secret is missing or inaccessible, you must create a new Barbican secret with the correct credentials and update the target record. **Check the Keystone trust is still valid:** Abacá uses a Keystone trust to act on the tenant's behalf. If the trust has expired or been revoked, enrollment and backup jobs will fail. Re-create the trust by re-running the enrollment workflow or by updating the target with a fresh trust. --- ### 5. Diagnose worker VM network attachment failures If a job fails at `provisioning_network`, the worker VM could not be booted or could not attach to the tenant's share network. **Check whether the service project has permission to create/attach ports on tenant share networks (DHSS=true):** ```bash openstack share protection target show # Look at the network_attachment_strategy field ``` ```bash # Probe empirically using abaca-dev abaca-dev worker-boot --target ``` The findings are recorded in the `abaca-dev` report. Port privilege is the only known potential blocker for DHSS=true environments. DHSS=false (StaticReachability) is unaffected by port privileges. **Check Nova and Neutron for boot errors:** ```bash openstack server list --project openstack server show ``` If the worker VM is in `ERROR` state, retrieve its fault: ```bash openstack server show -f json | python3 -c \ "import json,sys; d=json.load(sys.stdin); print(d.get('fault',{}))" ``` **Check that the worker flavor has the required CPU crypto instructions** (required for FIPS mode; see the FIPS section below). --- ### 6. Diagnose Manila share access and snapshot failures If a job fails at `provisioning_source`, Abacá could not get a consistent read-only view of the share. **Check Manila share and snapshot status:** ```bash openstack share show openstack share snapshot list --share-id ``` **Export policy root-squash:** If the Manila backend export policy squashes root, the worker agent cannot perform the root-level read and ownership operations needed for a faithful backup. This is detected at runtime as a read+chown probe failure and surfaces as `operator_action_required`. Ask your storage administrator to allow root access from the worker's IP on this share's export policy. **ONTAP snapdir availability:** If your Manila backend is NetApp ONTAP and `.snapshot` directories are not enabled on provisioned volumes, Abacá falls back to an alternative snapshot access strategy automatically. The runtime probe records which strategy was used in the job detail. No action is required unless all fallbacks are exhausted. --- ### 7. Diagnose restore failures and understand data-fidelity limitations **Restore job failing at `connecting_repository` or `transferring`:** Follow the same S3 credential and Barbican diagnosis in step 4. **Restore completes but files have wrong ownership or missing setuid/setgid bits:** This is a **known phase-1 limitation** of the Kopia-based restore path. Empirical validation on Kopia 0.17.0 and 0.23.1 confirms that Kopia does not preserve numeric UID/GID or setuid/setgid bits during filesystem-mode restore — files come back owned by the invoking user and special mode bits are stripped. **Affected workloads:** Shares storing files with system-level ownership (files owned by `root`, `postgres`, `mysql`, or other service UIDs), or setuid executables restored to a target that respects those identities. **Unaffected workloads:** Manila shares accessed by a single application UID with no setuid binaries — the restored share is functionally identical to the original. **What is preserved correctly:** - File contents - Modification times (`mtime`) - Sparse file layout (restored sparse) - Symlinks (absolute and relative) - Standard POSIX rwx and sticky bits - Directory structure - Hardlinks restored as independent identical copies (expected behavior, not a bug) **What is NOT preserved (phase 1):** - Numeric UID/GID (known Kopia limitation — planned compensation: stat-manifest sidecar) - setuid/setgid bits (same root cause) - NFSv4/NTFS-style ACLs and extended attributes (xattrs) — explicitly out of scope for phase 1 - Sockets and device nodes — not captured **Crash consistency note:** Backups using the LiveBackup strategy are crash-consistent but not point-in-time consistent. If your application had unflushed in-flight writes at backup time, those writes may not appear in the restore. This is equivalent to crash recovery — not data corruption. **Hardlinks restored as copies:** Multiple directory entries sharing an inode on the source are restored as independent copies. Total disk usage after restore may be higher than on the original share. This is expected behavior. --- ### 8. Diagnose FIPS-related failures Abacá configures Kopia to use only FIPS-approved encryption and hash algorithms and runs worker VMs in RHEL FIPS mode. If your environment has a FIPS compliance requirement: - Verify the worker VM image was built with RHEL FIPS mode enabled. - Verify the worker VM flavor's compute host has CPU crypto instructions (AES-NI or equivalent); without them, FIPS-mode cryptographic performance degrades significantly and jobs may time out. - The `fips_kdf_gap` flag in the target's capabilities indicates whether the pinned Kopia build lacks PBKDF2 key derivation. If this flag is set, contact your operator — a patched Kopia build is required before the target can be used in a validated FIPS configuration. ```bash openstack share protection target show # Check capabilities.fips_kdf_gap ``` --- ### 9. Handle stuck or orphaned jobs If a job remains in a non-terminal state for longer than expected, the `abaca-conductor` reconciliation loop will eventually detect and remediate it. You can check whether reconciliation has run recently by examining conductor logs. For operator-level intervention (forcing a job to `error`, releasing a stuck worker, or running a manual reconcile pass), use `abaca-manage`: ```bash abaca-manage job show abaca-manage reconcile --force ``` > **Note for reviewer:** The exact `abaca-manage` subcommand names for job inspection and forced reconciliation are not confirmed in the available source material. Replace with confirmed command shapes before publishing. --- ### 10. Recover bookkeeping from repositories after database loss The Kopia repository stored in your S3 bucket is the **authoritative record** of your backup data. The Abacá database holds only a summary index. If the database is lost: 1. Do not delete or modify the S3 bucket or its contents. 2. Use `abaca-manage` to re-initialize the database schema: ```bash abaca-manage db_sync ``` 3. Re-enroll each backup target. Because the bucket already contains a Kopia repository, the `bucket_contents` preflight check will return `adoptable` and the enrollment path will adopt the existing repository rather than creating a new one. 4. After adoption, run a reconciliation pass to re-index existing backups from the repository: ```bash abaca-manage reconcile --force ``` 5. Verify that backup records reappear under the target: ```bash openstack share protection backup list --target ``` > **Note for reviewer:** The exact `abaca-manage` adoption and reconciliation command shapes are not confirmed in the available source material. Validate against the actual CLI before publishing. ## Verification After resolving a failure, confirm recovery using the following checks: **1. Job reached `available` status:** ```bash openstack share protection job show # status should be: available ``` **2. Target enrollment is complete and preflight shows no failures:** ```bash openstack share protection target show # preflight_detail.overall should be: ok or adoptable # enrollment_status should be: enrolled ``` **3. Backup record exists and carries expected statistics:** ```bash openstack share protection backup list --target openstack share protection backup show # Check: status=available, size_bytes > 0, consistency_flag present ``` **4. Restore completed and share is accessible:** ```bash openstack share protection restore show # status should be: available openstack share show # status should be: available ``` **5. For ownership/permission fidelity validation** (phase-1 known gap): Run `abaca-dev` fidelity probes against the restored share to get an empirical fidelity matrix: ```bash abaca-dev fidelity-probe --share ``` The probe emits a findings report documenting which attributes round-tripped correctly. Compare against the known-gap description in step 7 to confirm the outcome matches expectations. **6. Preflight checks can be re-run independently** against a target to confirm S3 bucket configuration is now valid without requiring a full re-enrollment. ## Rollback Most Abacá operations are non-destructive by design — backups write to your S3 bucket but do not modify the source share, and restores write to a new share by default. However: **Rolling back a failed enrollment:** If enrollment created partial state, delete the target and re-create it after fixing the underlying issue: ```bash openstack share protection target delete ``` The S3 bucket is not modified by a failed enrollment unless a Kopia repository was partially initialized. If `bucket_contents` returns anything other than empty after deletion, inspect the bucket and remove any `.abaca-preflight/*` sentinel objects left by the `rw_roundtrip` check. These are automatically cleaned up by a successful check but may linger after an interrupted run. **Rolling back a restore:** By default, restores target a new share. Delete the new share if the restore result is not usable: ```bash openstack share delete ``` If you performed an in-place restore to the original share and the result is incorrect, restore again from an earlier backup point. **Rolling back a protection policy:** Detach the policy from the share to stop future scheduled backups: ```bash openstack share protection policy delete ``` Existing backups are not affected by policy deletion — they remain in the repository until pruned by the retention maintenance job. **Full environment teardown (Kolla-based deployments):** If you need to completely reverse an Abacá deployment, the teardown script at `deploy/kolla/99-teardown.sh` stops and removes the `abaca-api` and `abaca-conductor` containers, removes the Horizon plugin, drops the MariaDB schema and RabbitMQ vhost, and cleans up Keystone catalog entries. This is destructive and irreversible for the control-plane state — S3 bucket contents are not touched. ```bash bash deploy/kolla/99-teardown.sh ``` > **Warning:** Teardown does not delete S3 bucket contents. Your backup data remains in place and can be adopted by a future re-enrollment. ## Escalation If this runbook does not resolve the issue, escalate with the following information: **1. Job record (full JSON):** ```bash openstack share protection job show -f json ``` Include `status`, `error_category`, `error_message`, and `preflight_detail` (for enrollment jobs). **2. Target record (full JSON):** ```bash openstack share protection target show -f json ``` Do **not** include raw S3 credentials. Include only the Barbican secret hrefs. **3. `abaca-dev` findings report:** If the failure involves worker provisioning, network attachment, or data-path fidelity, run: ```bash abaca-dev worker-boot --target ``` Attach the findings JSON report. This is especially important for open item #1 (port privilege) and FIPS KDF gap issues. **4. Conductor and API logs** from the relevant time window. Request these from your operator if you do not have direct log access. **5. Environment details:** - OpenStack release (e.g., RHOSO 18, Kolla-Ansible 2024.2) - Trilio Share Protection version (minimum 4.12) - Manila backend type and DHSS setting - S3-compatible storage provider (AWS S3, MinIO, Ceph RGW, ODF, Wasabi) - Whether FIPS mode is enabled **Routing by `error_category`:** - `tenant_action_required` — the tenant or application team must fix bucket configuration, credentials, or lifecycle rules before re-opening with support. - `operator_action_required` — escalate to your infrastructure or platform operations team, and include Nova, Neutron, and Manila logs for the affected share network. > **Note for reviewer:** Add specific support contact information, internal ticket routing queues, and SLA references appropriate to your organization before publishing.