Troubleshooting

Diagnose and resolve the most common Abacá operational failures — authentication and endpoint discovery errors, misconfiguration, policy denials, RPC connectivity problems, and database connectivity issues — so that the service returns to a healthy, job-processing state.

This runbook covers:

• Keystone authentication failures (token acquisition, trust errors, catalog lookup)
• Service endpoint registration and discovery problems
• Configuration errors in the Abacá .conf file (oslo.config sections)
• oslo.policy access-denial errors
• RPC (RabbitMQ) connectivity failures between abaca-api and abaca-conductor
• Database (MySQL/MariaDB) connectivity and schema migration failures
• Worker VM heartbeat and fleet registration issues that prevent jobs from being claimed

This runbook does not cover:

• Diagnosing a specific failed or stuck backup/restore job (see the "Diagnose a degraded or stuck job" runbook)
• Rolling out a new worker image (see the "Roll out a new worker image" runbook)
• Restoring the control plane after database loss (see the "Restore the control plane from database loss" runbook)
• S3/Kopia data-path errors (object lock, bucket conformance, repository corruption)

Before you begin, ensure you have:

• Access to the OpenShift cluster hosting the abaca namespace with sufficient RBAC to view and exec into pods and read logs
• oc CLI authenticated to the OpenShift cluster
• openstack CLI configured with admin credentials (clouds.yaml or environment variables) pointing at the RHOSO 18 Keystone endpoint
• The python-abacaclient package installed so that openstack share protection subcommands are available
• Read access to the Abacá .conf file (mounted into the abaca-api and abaca-conductor pods)
• Access to the dedicated RabbitMQ management UI or rabbitmqctl on the Abacá broker (separate from the platform RabbitMQ)
• Access to the dedicated MySQL/MariaDB (Galera) instance used by Abacá
• Abacá version ≥ 0.23.1; RHOSO ≥ 18; OpenShift ≥ 4.14

1. Confirm which pods are running

Before diving into specific failure categories, check that both control-plane pods are present and in a healthy state.

oc -n abaca get pods

Expect to see at least one abaca-api-* pod and one abaca-conductor-* pod in Running status. A db-sync-* Job pod in Completed state is normal. If a pod is in CrashLoopBackOff or Error, retrieve its logs before proceeding:

oc -n abaca logs deployment/abaca-api --previous
oc -n abaca logs deployment/abaca-conductor --previous

The log stream is the primary diagnostic surface for all failures below.

2. Diagnose authentication and endpoint failures

2a. Verify the Keystone service entry and catalog registration

Abacá registers itself in the Keystone service catalog under the share-protection service type (the value of catalog_type in [DEFAULT]). If catalog discovery fails, every client request will fail.

openstack service list --long | grep share-protection

Expect one entry of type share-protection. If absent:

# The install script registers the service — re-run the identity setup step:
bash deploy/rhoso/01-identity.sh

Then verify the three endpoints (public, internal, admin) exist:

openstack endpoint list --service share-protection

Confirm that the public URL resolves to the OpenShift Route for abaca-api and that the port matches bind_port (default 9797) in [api].

2b. Check that the service account exists and has the correct roles

openstack user show <abaca-service-user>
openstack role assignment list --user <abaca-service-user> --names

The service user identified by service_user_name in [abaca] must exist and hold the roles required by the configured oslo.policy rules (at minimum an admin-equivalent role on the service project).

2c. Validate Keystone trust functionality

Abacá relies on Keystone trusts (delegation tokens) so the service user can act on behalf of tenants for Barbican secret reads and Manila access-rule management. If trusts are disabled in Keystone, backup and restore jobs will fail with a 403 during the provisioning phase.

# Check Keystone trust feature flag
grep -i trust /etc/keystone/keystone.conf

Ensure [trust] enabled = true. If you cannot access the Keystone config directly, test by attempting to create a trust via the OpenStack API:

openstack trust create --help

In the abaca-conductor logs, a trust failure appears as:

Failed to impersonate tenant via trust: 403 Forbidden

Resolve by ensuring [trust] enabled = true in Keystone and restarting the Keystone pods.

2d. Validate token acquisition by the API pod

oc -n abaca exec deployment/abaca-api -- \
  openstack --os-cloud abaca-service token issue

A successful response confirms that the service user credentials in the pod's clouds.yaml or environment are valid. A 401 Unauthorized means the password or domain scoping is wrong — check the secret mounted into the pod and compare against the Keystone user record.

3. Diagnose configuration errors

Abacá uses oslo.config. Errors in the .conf file cause the pod to fail on startup with a ConfigFileValueError or NoSuchOptError in the log.

3a. Locate and inspect the running configuration

oc -n abaca exec deployment/abaca-api -- cat /etc/abaca/abaca.conf

Key sections and the options most likely to be misconfigured:

3b. Validate that oslo.config accepts the file

You can invoke abaca-manage inside the pod to perform a config check without starting the full service:

oc -n abaca exec deployment/abaca-conductor -- \
  abaca-manage db_sync --config-file /etc/abaca/abaca.conf --dry-run

A clean exit with no ERROR lines means oslo.config parsed the file successfully. Any NoSuchOptError or ConfigFileValueError lines identify the offending key and section.

3c. Apply corrected configuration

Abacá configuration is managed via the Kubernetes ConfigMap or Secret that the kustomize manifests mount into pods. Edit the appropriate object:

oc -n abaca edit configmap abaca-config

After saving, restart the affected deployments to pick up the change:

oc -n abaca rollout restart deployment/abaca-api
oc -n abaca rollout restart deployment/abaca-conductor

Wait for the rollout to complete:

oc -n abaca rollout status deployment/abaca-api
oc -n abaca rollout status deployment/abaca-conductor

4. Diagnose policy (oslo.policy) access-denial errors

Abacá uses oslo.policy for access control. A policy denial surfaces as an HTTP 403 Forbidden response from abaca-api, with a log line like:

policy.enforce: Rule denied: <rule-name> for token with roles [<roles>]

4a. Identify the failing rule

In the abaca-api pod log, find the 403 entry and note the rule name. Common rules involve abaca_domain_owner (required for target template and bucket registration operations) and service-admin rules (required for fleet and coverage operations).

4b. Check the user's role assignments

openstack role assignment list --user <username> --project <project> --names
openstack role assignment list --user <username> --domain <domain> --names

For Domain-level operations (creating target templates, registering buckets), the user must hold the abaca_domain_owner role scoped to the correct Keystone Domain — not just a project role.

openstack role add \
  --user <username> \
  --domain <domain-name> \
  abaca_domain_owner

4c. Override policy rules if needed

Default policy rules are embedded in the service. To override them, place a policy.yaml file at the path configured for oslo.policy (typically /etc/abaca/policy.yaml) and mount it into the pod. For example, to broaden the default rule for listing jobs:

# /etc/abaca/policy.yaml
"job:list": "role:reader"

After mounting the file, restart the abaca-api pod.

Warning: Do not set any rule to "" (always allow) in production. Abacá's rules are designed around the Keystone token context — weakening them bypasses the tenant isolation model.

5. Diagnose RPC (RabbitMQ) connectivity failures

Abacá uses a dedicated RabbitMQ broker (separate from the platform's shared broker) for oslo.messaging RPC between abaca-api and abaca-conductor. If the broker is unreachable, API calls that require conductor involvement (job creation, worker management) will time out.

5a. Identify RPC timeout symptoms

In abaca-api logs:

ostlo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply

In abaca-conductor logs, absence of any incoming RPC messages when the API is receiving requests.

5b. Verify the connection string

Check [DEFAULT] transport_url (an oslo.config standard option) or the equivalent oslo.messaging configuration in the .conf file. Confirm it points to the dedicated Abacá RabbitMQ instance, not the platform's shared broker.

oc -n abaca exec deployment/abaca-api -- \
  python3 -c "
import oslo_messaging as messaging
transport = messaging.get_transport(None, url='<transport_url>')
print('Transport OK')
"

Replace <transport_url> with the value from the .conf file.

5c. Check RabbitMQ broker health

On the Abacá RabbitMQ host:

rabbitmqctl status
rabbitmqctl list_queues name messages consumers

Expect to see queues for the Abacá service with at least one consumer (the conductor). A queue with 0 consumers and a growing messages count means the conductor is not connected.

5d. Check for credential or vhost errors

RabbitMQ connection errors due to wrong credentials or vhost appear in both the pod logs and the RabbitMQ broker log as ACCESS_REFUSED. Verify the username, password, and vhost in the transport_url match the broker configuration.

5e. Restart conductor after broker recovery

Once the broker is healthy:

oc -n abaca rollout restart deployment/abaca-conductor

The conductor reconnects automatically on startup via oslo.messaging retry logic, but a restart ensures a clean reconnection.

6. Diagnose database connectivity and schema failures

Abacá's dedicated MySQL/MariaDB (Galera) database stores all metadata. The database must be reachable from both abaca-api and abaca-conductor pods, and the schema must be current.

6a. Identify database connectivity symptoms

In pod logs:

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '<host>' ([Errno 111] Connection refused)")

or:

sqlalchemy.exc.OperationalError: Access denied for user 'abaca'@'%'

6b. Test connectivity from inside the pod

oc -n abaca exec deployment/abaca-api -- \
  python3 -c "
import sqlalchemy
e = sqlalchemy.create_engine('<connection-string-from-conf>')
with e.connect() as c:
    print(c.execute(sqlalchemy.text('SELECT 1')).scalar())
"

A result of 1 confirms the database is reachable and credentials are accepted. Replace <connection-string-from-conf> with the value of connection in [database].

6c. Run db_sync to repair schema drift

If the pod starts but immediately logs Alembic or column-not-found errors, the schema may not be current. Run the migration job:

oc -n abaca create job db-sync-manual --from=cronjob/db-sync 2>/dev/null || \
  oc -n abaca exec deployment/abaca-conductor -- \
    abaca-manage db_sync --config-file /etc/abaca/abaca.conf

Abacá migrations are additive-only (no destructive changes), so running db_sync is safe at any time.

Watch the job log:

oc -n abaca logs job/db-sync-manual -f

A successful run ends with no ERROR lines and the last log message confirming migration completion.

6d. Verify the Galera cluster is healthy

If using MariaDB Galera, check cluster state:

SHOW STATUS LIKE 'wsrep_cluster_size';
SHOW STATUS LIKE 'wsrep_local_state_comment';

wsrep_local_state_comment should be Synced on all nodes. A non-primary component or a node in Donor/Desynced state can cause intermittent write failures.

7. Diagnose worker VM registration and heartbeat failures

If jobs queue but never progress, the conductor may have no healthy workers. Worker VMs register with abaca-conductor over outbound HTTPS to worker_api_url.

7a. Check the conductor log for fleet state

oc -n abaca logs deployment/abaca-conductor | grep -E 'worker|heartbeat|fleet'

Look for:

• No healthy workers available — no worker is registered or all are past their heartbeat deadline
• Worker heartbeat deadline exceeded — a worker stopped phoning home; the conductor marks it dead after worker_heartbeat_deadline_seconds (default: 60 s)
• Worker boot failure — the conductor attempted to boot a VM but Nova returned an error

7b. Check worker VM boot failures

The conductor boots worker VMs using the Nova compute API into the project identified by worker_project_name/worker_project_id. Check Nova for failed instances:

openstack server list --project <worker-project-name> --status ERROR

Inspect the fault message of any ERROR server:

openstack server show <server-id> -f json | python3 -m json.tool | grep -A5 fault

Common causes:

• Flavor not found: worker_boot_flavor (default: m1.small) does not exist — create the flavor or update the config
• Image not found: worker_boot_image (default: abaca-worker-0.23.1) is not in Glance — re-run deploy/rhoso/worker/20-worker-image.sh
• Network not found: worker_boot_network is blank or wrong — set it to the network ID the conductor should attach workers to
• Port privilege denied: The service project is not allowed to create ports on the tenant share network (Open Item 1 in the design) — verify Neutron policy or use DHSS=false share types

7c. Verify worker_api_url is reachable from worker VMs

Worker VMs call back to worker_api_url over outbound HTTPS. If this URL is not routable from the tenant network, workers will never register.

# From a test VM on the same network as workers:
curl -k <worker_api_url>/healthz

If worker_api_url is blank in the config, worker registration is impossible — set it explicitly.

7d. Adjust heartbeat and boot parameters if needed

If workers are being declared dead due to transient network latency, consider increasing worker_heartbeat_deadline_seconds (default: 60) in [conductor]. If the conductor is boot-looping due to repeated VM failures, check worker_boot_max_failures (default: 3) and worker_boot_failure_window_seconds (default: 7200) — the conductor will stop attempting boots after reaching the failure threshold within the window.

After fixing the underlying cause, restart the conductor to reset the failure counter:

oc -n abaca rollout restart deployment/abaca-conductor

After applying fixes, confirm the service is healthy by working through each layer:

1. Pods are running:

oc -n abaca get pods

All abaca-api-* and abaca-conductor-* pods should be in Running state with no recent restarts.

2. Service catalog is correct:

openstack endpoint list --service share-protection

Three endpoints (public, internal, admin) should be present.

3. API responds and authenticates correctly:

openstack share protection --help

Subcommands should be listed without error. Then make a simple authenticated call:

openstack share protection target list

A successful (possibly empty) list response — not a 401, 403, or connection error — confirms auth, catalog, and the API pod are all healthy.

4. RPC is flowing:

oc -n abaca logs deployment/abaca-conductor | tail -20

The conductor log should show periodic scheduler and reconciliation loop activity (controlled by scheduler_interval and reconciliation_interval), not a wall of RPC timeout errors.

5. Database is accessible and schema is current:

oc -n abaca exec deployment/abaca-conductor -- \
  abaca-manage db_sync --config-file /etc/abaca/abaca.conf

A clean exit with no errors confirms connectivity and a current schema.

6. Workers are registered (if jobs were failing):

oc -n abaca logs deployment/abaca-conductor | grep 'worker registered\|healthy workers'

At least one worker should appear registered. Alternatively, issue an on-demand backup for a known share and confirm the job progresses past queued into provisioning_network.

Most actions in this runbook are diagnostic or corrective configuration changes rather than destructive operations. The following rollback guidance applies by category:

Configuration changes (abaca-config ConfigMap edits): Revert the ConfigMap to its previous state using the revision history:

oc -n abaca rollout undo deployment/abaca-api
oc -n abaca rollout undo deployment/abaca-conductor

Or re-edit the ConfigMap to restore the previous values and trigger a new rollout.

Database schema migrations (db_sync): Abacá migrations are additive-only — db_sync never drops columns or tables. There is no automated rollback of a migration; if a new migration caused a regression, contact the Abacá engineering team for a compensating migration.

Role assignments (openstack role add): Revert an unintended role grant:

openstack role remove \
  --user <username> \
  --domain <domain-name> \
  abaca_domain_owner

Policy overrides (policy.yaml): Remove or empty the policy.yaml file and restart abaca-api to revert to built-in default rules.

Worker VM boot parameter changes: If changing worker_boot_flavor, worker_boot_image, or worker_boot_network caused workers to fail to boot, revert the ConfigMap and restart the conductor. Any ERROR-state VMs from failed boots can be cleaned up:

openstack server delete <failed-worker-vm-id>

Keystone service or endpoint changes: If identity setup scripts were re-run and created duplicate entries, remove the duplicate service or endpoints:

openstack endpoint delete <duplicate-endpoint-id>

If this runbook does not resolve the issue, escalate with the following information:

Who to contact:

• Your Abacá support channel or the engineering team responsible for the Abacá service deployment.

Information to collect before escalating:

1. Pod logs (last 500 lines from each pod, including --previous if the pod has crashed):

oc -n abaca logs deployment/abaca-api --tail=500 > abaca-api.log
oc -n abaca logs deployment/abaca-conductor --tail=500 > abaca-conductor.log
oc -n abaca logs deployment/abaca-api --previous --tail=500 >> abaca-api.log 2>/dev/null

2. Pod describe output (for crash or scheduling problems):

oc -n abaca describe pods > abaca-pods-describe.txt

3. Sanitized configuration (remove passwords and secret values before sharing):

oc -n abaca exec deployment/abaca-api -- \
  cat /etc/abaca/abaca.conf | sed 's/password=.*/password=REDACTED/g' > abaca.conf.sanitized

4. Keystone service and endpoint list:

openstack service list --long > ks-services.txt
openstack endpoint list --service share-protection > ks-endpoints.txt

5. Database connectivity test result (pass/fail and error message if failed)

6. RabbitMQ queue state (rabbitmqctl list_queues output)

7. OpenStack and RHOSO versions:

openstack versions show
oc version

8. The exact error message and HTTP status code a client received, along with the X-Request-Id header value if available (this correlates to a specific log line in abaca-api)

9. Whether the failure is reproducible and, if so, the exact openstack share protection command or API call that triggers it

10. Any recent changes to the deployment (config updates, image upgrades, Keystone policy changes, RabbitMQ or database maintenance)