Health Checks

This page describes the health check endpoints exposed by the abaca-api service and explains how to use them to verify that the service is alive and ready to accept traffic. Kubernetes (and OpenShift) use these probes to decide whether to route requests to a pod and whether to restart it — getting them right is essential for zero-downtime deployments and reliable operations. Use these endpoints in your liveness and readiness probe configurations, in integration smoke tests, and in monitoring scripts to confirm that abaca-api is healthy before running backup or restore workflows.

Before working with the health check endpoints you need:

• A running abaca-api deployment (containerised on OpenShift ≥ 4.12, or locally via Docker Compose)
• Network access to the abaca-api service on port 9797 (the port declared in deploy/kustomize/base/api-deployment.yaml)
• curl or any HTTP client capable of issuing GET requests
• (For Kubernetes/OpenShift probe configuration) kubectl or oc CLI access to the cluster with permission to edit Deployment specs

The health check endpoints are built into abaca-api — no additional installation steps are required. To reach them you only need the service running and exposed.

1. Identify the API service address.

On OpenShift, retrieve the service ClusterIP and port:

oc get svc abaca-api -o jsonpath='{.spec.clusterIP}:{.spec.ports[0].port}'

For a local Docker Compose stack, the default address is http://localhost:9797.

2. Verify basic reachability.

curl -sf http://<ABACA_API_HOST>:9797/health

A healthy service returns HTTP 200 with a JSON body. If curl times out or returns a connection-refused error, the pod is not yet running — check pod status:

oc get pods -l app=abaca-api

3. (Optional) Configure Kubernetes/OpenShift probes.

Add the following probe stanza to the abaca-api container spec in your Kustomize overlay or Helm values. These settings are safe starting points; tune initialDelaySeconds and periodSeconds for your environment:

livenessProbe:
  httpGet:
    path: /live
    port: 9797
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 9797
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

The health check endpoints themselves have no runtime configuration knobs — they are always enabled when abaca-api is running. The behaviour of the probes is controlled through your Kubernetes or OpenShift Deployment spec.

abaca-api is configured via /etc/abaca/abaca.conf (oslo.config) loaded automatically by the gunicorn entry point (abaca.api.wsgi:get_application()). Health check paths are not affected by that configuration file.

Use these three endpoints depending on what you need to verify:

Overall service health — GET /health

Call this endpoint from monitoring scripts or smoke tests to confirm the service is operational. The response body tells you the aggregate status at a glance.

curl -s http://<ABACA_API_HOST>:9797/health

Expected response (HTTP 200):

{"status": "ok"}

Kubernetes liveness probe — GET /live

This endpoint tells Kubernetes whether the process is alive. If it stops returning HTTP 200, Kubernetes restarts the container. Do not call it in application logic — it exists exclusively for the Kubernetes control plane.

curl -o /dev/null -w "%{http_code}" http://<ABACA_API_HOST>:9797/live

Expected output: 200

Kubernetes readiness probe — GET /ready

This endpoint tells Kubernetes whether the pod is ready to receive traffic. It returns HTTP 200 only when the service has completed initialisation and its dependencies are reachable. If you need to gate a workflow on service readiness (for example, in a CI pipeline after deploying a new version), poll this endpoint until it returns 200:

until curl -sf http://<ABACA_API_HOST>:9797/ready; do
  echo "Waiting for abaca-api to become ready..."
  sleep 5
done
echo "abaca-api is ready."

All three endpoints are unauthenticated — no Keystone token is required. This is intentional: probes must work before a tenant session exists, and none of the responses expose sensitive data.

Example 1 — Check overall health and parse the status field

Use jq to extract the status value for use in a shell conditional:

STATUS=$(curl -sf http://<ABACA_API_HOST>:9797/health | jq -r '.status')
if [ "$STATUS" = "ok" ]; then
  echo "abaca-api is healthy."
else
  echo "abaca-api reported status: $STATUS" >&2
  exit 1
fi

Expected output when healthy:

abaca-api is healthy.

Example 2 — Liveness check returning the HTTP status code

curl -o /dev/null -w "%{http_code}\n" http://<ABACA_API_HOST>:9797/live

Expected output:

200

Example 3 — Readiness poll in a CI/CD pipeline

Wait up to 60 seconds for abaca-api to become ready after a rollout, then fail the pipeline step if it does not:

MAX_ATTEMPTS=12
ATTEMPT=0
until curl -sf http://<ABACA_API_HOST>:9797/ready > /dev/null 2>&1; do
  ATTEMPT=$((ATTEMPT + 1))
  if [ "$ATTEMPT" -ge "$MAX_ATTEMPTS" ]; then
    echo "ERROR: abaca-api did not become ready after $((MAX_ATTEMPTS * 5))s" >&2
    exit 1
  fi
  echo "[${ATTEMPT}/${MAX_ATTEMPTS}] Not ready yet, retrying in 5s..."
  sleep 5
done
echo "abaca-api is ready after $((ATTEMPT * 5))s."

Expected output on success:

[1/12] Not ready yet, retrying in 5s...
[2/12] Not ready yet, retrying in 5s...
abaca-api is ready after 10s.

Example 4 — OpenShift Deployment patch adding both probes

Apply this patch to add liveness and readiness probes to the existing base deployment without modifying the base manifest directly:

# overlays/production/api-probe-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: abaca-api
spec:
  template:
    spec:
      containers:
        - name: abaca-api
          livenessProbe:
            httpGet:
              path: /live
              port: 9797
            initialDelaySeconds: 10
            periodSeconds: 15
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 9797
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3

Reference this patch in your kustomization.yaml:

patchesStrategicMerge:
  - api-probe-patch.yaml

Apply with:

oc apply -k overlays/production/

Symptom: curl returns Connection refused on port 9797

Likely cause: The abaca-api pod is not running or has not finished starting.

Fix:

1. Check pod status: oc get pods -l app=abaca-api
2. If the pod is in CrashLoopBackOff, inspect logs: oc logs -l app=abaca-api --previous
3. If the pod is Pending, check node resources and PVC bindings.
4. Ensure the Service object targets port 9797 and selects the app=abaca-api label.

Symptom: /health returns HTTP 200 but /ready returns a non-200 status

Likely cause: abaca-api is alive but has not yet established connections to its dependencies (database, RabbitMQ, or Keystone).

Fix:

1. Inspect abaca-api logs for connection errors: oc logs -l app=abaca-api
2. Confirm MariaDB and RabbitMQ pods are running and their Services are reachable from the abaca-api pod.
3. Verify /etc/abaca/abaca.conf contains correct connection strings for the database and message broker.
4. Increase initialDelaySeconds on the readiness probe if the service consistently takes longer to connect on startup.

Symptom: Kubernetes is restarting the abaca-api pod repeatedly

Likely cause: The liveness probe is timing out or returning a non-200 response, causing Kubernetes to treat the container as dead.

Fix:

1. Confirm the probe path and port match exactly: path /live, port 9797.
2. Increase initialDelaySeconds (try 30) if the process is still initialising when the first probe fires.
3. Increase timeoutSeconds if the service is under load and not responding within the default 1-second window.
4. Check whether a failureThreshold too low (for example, 1) is causing premature restarts under transient load.

Symptom: Health endpoint returns 404 Not Found

Likely cause: The path you are requesting does not match the registered Flask route.

Fix:

1. Confirm the exact paths from the Flask route definitions in the abaca-api source (see note in Prerequisites).
2. Ensure you are not including a /v1/ prefix — the health endpoints are not versioned API resources.
3. Check that no ingress or proxy is rewriting the path before it reaches the container.

Symptom: Health checks pass but backup jobs fail immediately

Likely cause: The health endpoints confirm process liveness but do not validate the full data path (Manila connectivity, S3 reachability, Barbican access). The service can be "healthy" while dependencies specific to a tenant's workflow are misconfigured.

Fix: Use abaca-dev to run a live data-path probe against your OpenStack environment to validate end-to-end fidelity and capability before relying solely on HTTP health checks for operational readiness.