Service Management

This page explains how to start, stop, and restart the Abacá control-plane services and the worker agent, how to inspect service logs, and how to configure runtime quotas that govern API pagination and concurrency. Abacá runs as two OpenShift Deployment objects (abaca-api and abaca-conductor) in the abaca namespace on RHOSO 18, while the abaca-worker-agent runs as a systemd unit inside each worker VM. Understanding the operational shape of each component—container-based for the control plane, systemd-based for worker VMs—is essential before attempting any lifecycle action or quota change.

Before performing any service-management task on this page you need:

• RHOSO 18 control plane running on OpenShift ≥ 4.14
• oc CLI authenticated to the OpenShift cluster with at least edit rights on the abaca namespace
• kubectl or oc access to inspect and restart Deployments
• openstack CLI with the python-abacaclient plugin installed and a valid clouds.yaml or environment-variable auth configuration
• SSH access to worker VMs (for worker-agent lifecycle operations), or access to the Nova console
• The abaca OpenShift namespace already deployed (see the deployment guide)
• Familiarity with oc logs, oc rollout, and journalctl

The Abacá control-plane services are deployed as Kubernetes Deployment objects and do not require a separate installation step for service management. The abaca-worker-agent systemd unit is baked into the worker VM image during the image-build phase.

If you need to redeploy or update the control-plane manifests, apply the kustomize base from the repository:

# From the repository root, apply the full kustomize overlay for your environment
oc apply -k deploy/rhoso/kustomize/overlays/<your-overlay>

To verify that both control-plane Deployments are present and healthy after installation:

oc -n abaca get deployments

Expected output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
abaca-api          1/1     1            1           5m
abaca-conductor    1/1     1            1           5m

To verify the worker-agent systemd unit is present on a worker VM (run this on the VM via SSH or the Nova console):

systemctl status abaca-worker-agent

If the unit file is missing, the worker VM image was not built correctly. Rebuild the image using deploy/rhoso/worker/20-worker-image.sh and re-register it in Glance.

All Abacá configuration is managed by oslo.config. Options are grouped by section in an INI-style .conf file mounted into each pod as a Kubernetes Secret. The sections most relevant to service management and quotas are described below.

[api] — API server behaviour

Example [api] stanza:

[api]
bind_host = 0.0.0.0
bind_port = 9797
noauth = false
max_limit = 1000
default_limit = 100

[conductor] — Conductor scheduling and worker fleet

[worker] — Worker agent runtime

These options appear in the abaca.conf file written to each worker VM by cloud-init. They are not set in the control-plane secrets.

[database]

Applying configuration changes

Configuration is stored in the abaca-config-data Kubernetes Secret. To update it:

# Edit the Secret (base64-encoded values)
oc -n abaca edit secret abaca-config-data

After saving, restart the affected Deployment so the new config is picked up (see Usage below). The worker-agent config is re-written by cloud-init at VM boot; to change it on a running worker you must replace the VM.

Starting, stopping, and restarting the control-plane services

Both abaca-api and abaca-conductor run as Kubernetes Deployment objects in the abaca namespace. Use oc rollout to manage their lifecycle.

Restart the API service (e.g., after a config change):

oc -n abaca rollout restart deployment/abaca-api

Restart the conductor (e.g., after a config change):

oc -n abaca rollout restart deployment/abaca-conductor

The conductor Deployment uses strategy: Recreate. This means OpenShift terminates the existing pod before starting the replacement, ensuring that at most one conductor is running at any time. Do not change this strategy — running two conductors simultaneously would double every timer and produce duplicate job dispatches.

Stop the API service (scale to zero):

oc -n abaca scale deployment/abaca-api --replicas=0

Start the API service (restore to one replica):

oc -n abaca scale deployment/abaca-api --replicas=1

Stop the conductor:

oc -n abaca scale deployment/abaca-conductor --replicas=0

Start the conductor:

oc -n abaca scale deployment/abaca-conductor --replicas=1

Scaling the conductor to zero suspends all scheduled backups, job dispatching, and reconciliation sweeps. Any jobs that were in flight when the conductor stopped will be recovered by the reconciliation sweep when the conductor is restarted.

Check rollout status:

oc -n abaca rollout status deployment/abaca-api
oc -n abaca rollout status deployment/abaca-conductor

Viewing control-plane logs

Stream API logs:

oc -n abaca logs -f deployment/abaca-api

Stream conductor logs:

oc -n abaca logs -f deployment/abaca-conductor

To view logs from the previous pod incarnation (useful after a crash-restart):

oc -n abaca logs deployment/abaca-api --previous
oc -n abaca logs deployment/abaca-conductor --previous

Managing the worker agent on a worker VM

The abaca-worker-agent runs as a systemd unit inside each worker VM. Connect to the VM via SSH or the Nova console to manage it.

Check the agent status:

systemctl status abaca-worker-agent

Restart the agent (e.g., after a config change or transient failure):

systemctl restart abaca-worker-agent

Stop the agent:

systemctl stop abaca-worker-agent

Start the agent:

systemctl start abaca-worker-agent

Stream agent logs from the journal:

journalctl -u abaca-worker-agent -f

View recent agent logs:

journalctl -u abaca-worker-agent -n 200 --no-pager

The agent runs as root because it must mount NFS shares and hot-plug Neutron interfaces. This is expected behaviour, not a misconfiguration.

The worker agent config file is written by cloud-init at /etc/abaca/abaca.conf. If the file is absent at startup, ExecStartPre fails and systemd retries up to 30 times over ~5 minutes, waiting for cloud-init to complete. If the file is never written, the agent never starts — this indicates a cloud-init or user-data problem, not a service failure.

Adjusting API quotas

The max_limit and default_limit options in the [api] section control list-response pagination. These are the primary levers for protecting the database from oversized queries.

1. Edit the abaca-config-data Secret to update the relevant .conf key.
2. Restart the API Deployment to apply the change:

oc -n abaca rollout restart deployment/abaca-api

Callers that pass ?limit= values greater than max_limit will have their request silently capped at max_limit; they will not receive an error.

Adjusting worker fleet concurrency

The capacity_slots option in the [worker] section of each worker VM's /etc/abaca/abaca.conf controls how many concurrent jobs that VM accepts. Because this value is written into the VM at boot via cloud-init, it cannot be changed on a running VM — you must replace the VM with a new one booted from an updated cloud-init configuration or a new image. The conductor's min_workers option controls how many warm workers are kept available.

Example 1 — Verify both control-plane pods are running

oc -n abaca get pods -l app.kubernetes.io/part-of=abaca

Expected output:

NAME                               READY   STATUS    RESTARTS   AGE
abaca-api-7d9f8b6c4-xk9p2          1/1     Running   0          2d
abaca-conductor-5c7b9f8d6-lm3r1    1/1     Running   0          2d

If a pod is in CrashLoopBackOff, retrieve the last 50 log lines to identify the cause:

oc -n abaca logs deployment/abaca-api --previous --tail=50

Example 2 — Restart both services after a configuration update

This is the standard procedure after editing the abaca-config-data Secret.

# Restart API
oc -n abaca rollout restart deployment/abaca-api

# Wait for the API rollout to complete before restarting the conductor
oc -n abaca rollout status deployment/abaca-api --timeout=120s

# Restart conductor
oc -n abaca rollout restart deployment/abaca-conductor
oc -n abaca rollout status deployment/abaca-conductor --timeout=120s

Expected output for each rollout status call:

deployment "abaca-api" successfully rolled out
deployment "abaca-conductor" successfully rolled out

Example 3 — Temporarily stop the conductor for maintenance

Stop scheduled backups and job dispatching without affecting the API:

# Scale conductor to zero
oc -n abaca scale deployment/abaca-conductor --replicas=0

# Confirm the conductor pod is gone
oc -n abaca get pods -l app.kubernetes.io/component=conductor
# Expected: No resources found in abaca namespace.

# ... perform maintenance ...

# Restore the conductor
oc -n abaca scale deployment/abaca-conductor --replicas=1
oc -n abaca rollout status deployment/abaca-conductor --timeout=120s

Any jobs that were queued or in-flight when the conductor was stopped will be recovered by the reconciliation sweep (which runs within reconciliation_interval seconds, default 300) once the conductor is back.

Example 4 — View worker agent logs on a worker VM

SSH into a worker VM (obtain its IP from Nova):

ssh cloud-user@<worker-vm-ip>

# Check the unit status
systemctl status abaca-worker-agent

# Stream live logs
journalctl -u abaca-worker-agent -f

Expected output from a healthy, registered worker:

Jan 15 10:23:01 worker-vm-001 abaca-worker-agent[1234]: Registered with conductor, worker_id=a3f2...
Jan 15 10:23:11 worker-vm-001 abaca-worker-agent[1234]: Heartbeat sent, next in 10s
Jan 15 10:23:21 worker-vm-001 abaca-worker-agent[1234]: Polling for commands (long_poll=20s)

Example 5 — Lower the default API page size to reduce database load

Edit the abaca-config-data Secret. The relevant key is abaca-api.conf:

oc -n abaca edit secret abaca-config-data

In the decoded data, change (or add) to the [api] section:

[api]
default_limit = 50
max_limit = 500

Save and restart the API:

oc -n abaca rollout restart deployment/abaca-api
oc -n abaca rollout status deployment/abaca-api --timeout=120s

Verify the change is active by making a list call without a limit parameter and confirming the response contains at most 50 items.

Issue: abaca-api pod is in CrashLoopBackOff

Symptom: oc -n abaca get pods shows the abaca-api pod repeatedly crashing and restarting.

Likely causes:

• The abaca-config-data Secret is missing or contains an unparseable .conf file.
• The database DSN in [database] connection is unreachable or has wrong credentials.
• The TLS certificate Secret (cert-abaca-internal-svc) is missing or expired.

Fix:

1. Retrieve the crash log: oc -n abaca logs deployment/abaca-api --previous --tail=100
2. Look for oslo.config parse errors or database connection refused messages.
3. Verify the Secret exists: oc -n abaca get secret abaca-config-data
4. If the database is unreachable, check connectivity from inside the pod: oc -n abaca exec deployment/abaca-api -- /bin/sh -c 'curl -v <galera-host>:3306'
5. After fixing the underlying cause, restart: oc -n abaca rollout restart deployment/abaca-api

Issue: abaca-conductor pod starts and then immediately stops

Symptom: The conductor pod reaches Running briefly and then terminates; oc -n abaca logs deployment/abaca-conductor --previous shows it exits cleanly or with a config error.

Likely causes:

• worker_boot_network is not set in [conductor] — this required option has no default.
• worker_api_url is not set in [conductor].
• The clouds.yaml volume mount is missing or the abaca-service cloud entry is absent, so the conductor cannot authenticate to Nova.

Fix:

1. Check logs: oc -n abaca logs deployment/abaca-conductor --previous --tail=100
2. Confirm required options are present in the conductor's conf: oc -n abaca get secret abaca-config-data -o jsonpath='{.data.abaca-conductor\.conf}' | base64 -d | grep -E 'worker_boot_network|worker_api_url'
3. Verify the clouds.yaml subPath is mounted and contains the abaca-service cloud entry.
4. Fix the Secret and restart: oc -n abaca rollout restart deployment/abaca-conductor

Issue: Worker agent never starts after VM boot (unit loops in start)

Symptom: systemctl status abaca-worker-agent on the worker VM shows repeated ExecStartPre failures with exit-code 1.

Likely cause: Cloud-init has not yet written /etc/abaca/abaca.conf to the VM. The ExecStartPre test guards against this and intentionally fails until the file exists. This is expected behaviour during VM boot; the unit will retry up to 30 times over ~5 minutes.

Fix:

1. Wait for cloud-init to complete: cloud-init status --wait
2. Confirm the file was written: ls -la /etc/abaca/abaca.conf
3. If cloud-init completed but the file is absent, inspect the cloud-init log: journalctl -u cloud-init --no-pager | tail -50
4. If the user-data was not applied, the VM may have been booted without the expected cloud-init user-data payload. Terminate the VM and let the conductor boot a replacement, or manually write /etc/abaca/abaca.conf and run systemctl start abaca-worker-agent.

Issue: Worker agent starts but the worker never appears as registered in the conductor

Symptom: systemctl status abaca-worker-agent shows the unit active (running), but the conductor logs show no registration event and the worker VM shows no activity.

Likely causes:

• [worker] api_url in the worker's /etc/abaca/abaca.conf points to an unreachable or wrong URL.
• The CA bundle at [worker] api_ca_file does not match the certificate served by the API Route, causing TLS validation failure.
• The worker's [worker] token has expired (TTL controlled by [conductor] worker_token_ttl_seconds).

Fix:

1. On the worker VM, check agent logs: journalctl -u abaca-worker-agent -n 100 --no-pager
2. Look for TLS errors or HTTP 401/403 responses.
3. Verify the API URL is reachable from the worker: curl -v --cacert /etc/abaca/worker-api.pem <worker_api_url>/healthcheck
4. If TLS fails, the CA bundle written by cloud-init does not match the Route certificate. Update [conductor] worker_api_ca_file in the conductor config, restart the conductor, and replace the worker VM so it receives the corrected cloud-init payload.

Issue: Scheduled backups are not running

Symptom: Policies are configured with valid cron expressions but no backup jobs appear.

Likely causes:

• The conductor is scaled to zero or has not completed its rollout.
• [conductor] scheduler_interval is very large (check the current value).
• No worker VMs are available and min_workers = 0, so jobs queue but cannot be claimed.

Fix:

1. Confirm the conductor is running: oc -n abaca get pods -l app.kubernetes.io/component=conductor
2. Check conductor logs for scheduler activity: oc -n abaca logs deployment/abaca-conductor | grep -i scheduler
3. Check for queued jobs: openstack share protection job list --status queued
4. If jobs are stuck in queued beyond queued_job_deadline_seconds (default 300 s) the reconciliation sweep will fail them with operator_action_required. Check conductor logs for fleet-related errors.
5. If no workers are available, verify min_workers is set to at least 1 and that worker_boot_network, worker_boot_image, and worker_api_url are correctly configured.

Issue: API list responses return fewer items than expected

Symptom: Paginated list calls return a maximum of 100 items even when more exist, or a custom limit value is silently ignored.

Likely cause: The [api] default_limit (default 100) or [api] max_limit (default 1000) caps the response. Requests with limit above max_limit are silently capped.

Fix:

1. Confirm current values: oc -n abaca get secret abaca-config-data -o jsonpath='{.data.abaca-api\.conf}' | base64 -d | grep -E 'max_limit|default_limit'
2. If the values need raising, edit abaca-config-data and increase max_limit and/or default_limit.
3. Restart the API: oc -n abaca rollout restart deployment/abaca-api
4. In API calls, always pass an explicit limit parameter up to the new max_limit value; do not rely on the default.