Health Checks

Trilio Site Recovery for OpenStack exposes two health-check endpoints that let you verify the service is alive and correctly configured without performing any DR operation. The /healthz liveness endpoint returns the overall process health including database connectivity, and the version discovery endpoint at GET / confirms the API is reachable and advertises the supported microversion range. Use these endpoints in load-balancer health probes, monitoring scripts, and pre-flight checks before executing failover or failback operations.

Before using the health-check endpoints you need:

• A running protector-api process (deployed via the trilio-site-recovery-for-openstack Juju charm) listening on port 8788.
• Network access from your monitoring host or load balancer to port 8788 on each site's API node.
• curl or any HTTP client — no Keystone token is required for either endpoint.
• Both endpoints are unauthenticated and do not require the X-Auth-Token header.

The health endpoints are built into protector-api and require no separate installation step. They become available automatically once the charm has finished its install hook and the service is running.

1. Confirm the charm is active:

   Copy

   juju status trilio-site-recovery-for-openstack
   

   The unit status should show active/idle.

2. Identify the API address. The charm binds on 0.0.0.0:8788 by default. Use the unit's IP from juju status, or the VIP if you are behind a load balancer:

   Copy

   API_HOST=$(juju run trilio-site-recovery-for-openstack/0 -- \
       unit-get private-address)
   

3. Verify the endpoints are reachable:

   Copy

   curl -sf http://${API_HOST}:8788/healthz
   curl -sf http://${API_HOST}:8788/
   

   A 200 response from each command confirms the service is healthy and the API version is discoverable.

Neither health endpoint requires configuration to enable — they are always active when protector-api is running. The options below affect the address and port they listen on, and the database connection that /healthz checks.

Juju note: Do not edit /etc/protector/protector.conf directly — the charm overwrites it on every config-changed event. Set options via juju config using hyphenated names:

Copy

juju config trilio-site-recovery-for-openstack bind-port=8788

The Dockerfile configures the Docker container health check to poll /healthz every 30 seconds with a 5-second timeout and 3 retries. In Juju deployments the equivalent probe should be configured in your external monitoring tool pointing at http://<api-host>:8788/healthz.

Liveness check — GET /healthz

This endpoint returns HTTP 200 when the service is healthy, or HTTP 503 when the database is unavailable. It requires no authentication and no tenant ID prefix.

curl -si http://<api-host>:8788/healthz

Use this endpoint as:

• A load-balancer health probe to remove a node from rotation when its database connection fails.
• A Nagios/Prometheus check to alert when protector-api is down.
• A pre-flight check in automation before issuing failover or failback commands.

The JSON body indicates which subsystem failed, so a monitoring script can distinguish a total process failure (connection refused) from a partial failure (database unreachable but process alive).

Version discovery — GET /

This endpoint returns HTTP 200 with the supported API microversion range. It requires no authentication.

curl -si http://<api-host>:8788/

Use this endpoint to:

• Confirm the API is reachable before sending authenticated requests.
• Negotiate the microversion your client should use.
• Smoke-test a new deployment immediately after charm install.

The current maximum microversion is 1.1. When you include OpenStack-API-Version: protector 1.1 on versioned requests, you opt in to 1.1 behaviour. Omitting the header defaults to microversion 1.0. Values outside the 1.0–1.1 range return HTTP 400.

Checking both sites

Because each site runs an independent protector-api, you should poll both sites independently. Cross-site metadata sync requires bidirectional port 8788 connectivity, so a healthy response from both sites is a prerequisite before executing any DR operation:

for SITE_HOST in "${PRIMARY_API_HOST}" "${SECONDARY_API_HOST}"; do
  echo "Checking ${SITE_HOST}…"
  curl -sf http://${SITE_HOST}:8788/healthz || echo "  FAILED"
done

Example 1 — Healthy liveness response

A GET /healthz when both the process and database are up:

curl -si http://10.10.0.20:8788/healthz

HTTP/1.1 200 OK
Content-Type: application/json

{"status": "ok", "checks": {"database": "ok"}}

Example 2 — Unhealthy liveness response (database down)

When the MariaDB connection is broken, /healthz returns HTTP 503 and identifies the failing subsystem:

curl -si http://10.10.0.20:8788/healthz

HTTP/1.1 503 Service Unavailable
Content-Type: application/json

{"status": "unhealthy", "checks": {"database": "OperationalError"}}

The checks.database value is the exception class name raised when the probe query fails. Use this to distinguish a connection-refused scenario (process down, load balancer should see TCP timeout) from a database-down scenario (process alive, 503 body).

Example 3 — Version discovery

curl -si http://10.10.0.20:8788/

HTTP/1.1 200 OK
Content-Type: application/json

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://10.10.0.20:8788/v1/"
        }
      ]
    }
  ]
}

Key response fields:

Example 4 — Scripted pre-flight check before failover

This pattern is suitable for CI pipelines or runbook automation:

#!/usr/bin/env bash
set -euo pipefail

PRIMARY="http://10.10.0.20:8788"
SECONDARY="http://10.10.1.20:8788"

check_site() {
  local url="$1"
  local label="$2"
  local status
  status=$(curl -so /dev/null -w "%{http_code}" "${url}/healthz")
  if [[ "${status}" == "200" ]]; then
    echo "[OK]  ${label} (${url})"
  else
    echo "[FAIL] ${label} returned HTTP ${status}" >&2
    exit 1
  fi
}

check_site "${PRIMARY}"   "Primary site API"
check_site "${SECONDARY}" "Secondary site API"

echo "Both sites healthy — safe to proceed with DR operation."

Example 5 — Microversion negotiation

To use microversion 1.1 on a versioned API call, include the header alongside your Keystone token. The version discovery response confirms 1.1 is the maximum:

curl -si \
  -H "X-Auth-Token: ${OS_TOKEN}" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://10.10.0.20:8788/v1/${OS_PROJECT_ID}/protection-groups

Omitting OpenStack-API-Version is valid and defaults to microversion 1.0. Passing an out-of-range value such as protector 2.0 returns HTTP 400.

Issue: curl returns Connection refused on port 8788

Symptom: curl: (7) Failed to connect to <host> port 8788: Connection refused

Likely causes:

• protector-api is not running on that host.
• The charm install hook has not completed.
• The bind address in [api] bind_host is set to a specific IP that does not match the interface you are probing.

Fix:

1. Check the charm status: juju status trilio-site-recovery-for-openstack
2. Check the systemd unit: juju ssh trilio-site-recovery-for-openstack/0 -- systemctl status protector-api
3. Check /var/log/protector/ for startup errors.
4. If bind_host was customised, verify it matches the IP you are querying.

Issue: /healthz returns HTTP 503 with "database": "OperationalError"

Symptom: Process responds but reports database unavailable.

Likely causes:

• The MariaDB service is down or the protector database user credentials are wrong.
• The [database] connection DSN in protector.conf points to the wrong host or port.
• The shared-db Juju relation has not been satisfied (relation is missing or the related mysql-innodb-cluster unit is not ready).

Fix:

1. Verify the shared-db relation: juju relations trilio-site-recovery-for-openstack
2. Check the database service: juju status mysql-innodb-cluster
3. Do not edit protector.conf directly — the charm renders it. If the DSN is wrong, check which charm config options are set: juju config trilio-site-recovery-for-openstack
4. Run the Juju health-check action to get structured diagnostics: juju run trilio-site-recovery-for-openstack/0 health-check (Juju 3.x) or juju run-action trilio-site-recovery-for-openstack/0 health-check (Juju 2.9).

Issue: GET / returns HTTP 404 instead of the version document

Symptom: Requests to http://<host>:8788/ return 404.

Likely causes:

• You are hitting a path other than exactly / — for example, /v1 or /api. The version discovery endpoint only responds to the bare root path.
• A reverse proxy in front of the API is rewriting the path.

Fix:

• Confirm the exact path: curl -si http://<host>:8788/ (trailing slash or no trailing slash, both are accepted).
• Check your load-balancer or proxy configuration for path rewriting.
• Do not use /v1 for version discovery — that prefix requires a {tenant_id} segment and an X-Auth-Token.

Issue: /healthz returns 200 but versioned API calls return 404

Symptom: Health probe succeeds, but GET /v1/{tenant_id}/protection-groups returns 404.

Likely causes:

• The URL is missing the /v1/ segment. For example, /{tenant_id}/protection-groups is not a valid route.
• The {tenant_id} segment is absent or malformed.

Fix:

• Ensure every versioned request uses the full path: /v1/{tenant_id}/<resource>.
• Verify your OS_PROJECT_ID environment variable is set correctly.
• Use version discovery (GET /) to confirm the correct base URL: the links[].href field shows the exact versioned root.

Issue: Microversion header causes HTTP 400

Symptom: Adding OpenStack-API-Version: protector <version> returns HTTP 400 Bad Request.

Likely causes:

• The microversion value is outside the supported 1.0–1.1 range.
• The header value is malformed (for example, protector/1.1 instead of protector 1.1).

Fix:

• Use OpenStack-API-Version: protector 1.0 or OpenStack-API-Version: protector 1.1.
• Confirm the supported range by checking the version and min_version fields in the GET / response before sending versioned requests.
• Omitting the header entirely is valid and defaults to microversion 1.0.