---
title: Health Checks
product: trilio-site-recovery-for-openstack
doc_type: guide
version: master
source: git2docs (code-derived, validation-filtered)
canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-openstack/trilio-site-recovery-for-openstack/health-checks
---

# Health Checks

_Liveness and readiness endpoints_

## Overview

Trilio Site Recovery for OpenStack exposes two health-check endpoints that let you verify the service is alive and correctly configured without performing any DR operation. The `/healthz` liveness endpoint returns the overall process health including database connectivity, and the version discovery endpoint at `GET /` confirms the API is reachable and advertises the supported microversion range. Use these endpoints in load-balancer health probes, monitoring scripts, and pre-flight checks before executing failover or failback operations.

## Prerequisites

Before using the health-check endpoints you need:

- A running `protector-api` process (deployed via the `trilio-site-recovery-for-openstack` Juju charm) listening on port 8788.
- Network access from your monitoring host or load balancer to port 8788 on each site's API node.
- `curl` or any HTTP client — **no** Keystone token is required for either endpoint.
- Both endpoints are unauthenticated and do not require the `X-Auth-Token` header.

## Configuration

Neither health endpoint requires configuration to enable — they are always active when `protector-api` is running. The options below affect the address and port they listen on, and the database connection that `/healthz` checks.

| Section | Option | Default | Effect |
|---|---|---|---|
| `[api]` | `bind_host` | `0.0.0.0` | Address the API process binds to. Set to a specific IP to restrict which interface accepts health-probe traffic. |
| `[api]` | `bind_port` | `8788` | TCP port for all API traffic including `/healthz` and `GET /`. |
| `[api]` | `workers` | `4` (oslo default); `2` in Juju charm | Number of API worker processes. Does not affect the endpoints themselves but controls concurrency. |
| `[database]` | `connection` | `mysql+pymysql://protector:password@localhost/protector` | MariaDB/MySQL DSN. `/healthz` actively tests this connection on every request. A wrong DSN causes `/healthz` to return HTTP 503. |

> **Juju note:** Do not edit `/etc/protector/protector.conf` directly — the charm overwrites it on every `config-changed` event. Set options via `juju config` using hyphenated names:
>
> ```bash
> juju config trilio-site-recovery-for-openstack bind-port=8788
> ```

The Dockerfile configures the Docker container health check to poll `/healthz` every 30 seconds with a 5-second timeout and 3 retries. In Juju deployments the equivalent probe should be configured in your external monitoring tool pointing at `http://<api-host>:8788/healthz`.

## Usage

### Liveness check — `GET /healthz`

This endpoint returns HTTP 200 when the service is healthy, or HTTP 503 when the database is unavailable. It requires no authentication and no tenant ID prefix.

```bash
curl -si http://<api-host>:8788/healthz
```

Use this endpoint as:
- A **load-balancer health probe** to remove a node from rotation when its database connection fails.
- A **Nagios/Prometheus check** to alert when `protector-api` is down.
- A **pre-flight check** in automation before issuing failover or failback commands.

The JSON body indicates which subsystem failed, so a monitoring script can distinguish a total process failure (connection refused) from a partial failure (database unreachable but process alive).

---

### Version discovery — `GET /`

This endpoint returns HTTP 200 with the supported API microversion range. It requires no authentication.

```bash
curl -si http://<api-host>:8788/
```

Use this endpoint to:
- **Confirm the API is reachable** before sending authenticated requests.
- **Negotiate the microversion** your client should use.
- **Smoke-test a new deployment** immediately after charm install.

The current maximum microversion is `1.1`. When you include `OpenStack-API-Version: protector 1.1` on versioned requests, you opt in to 1.1 behaviour. Omitting the header defaults to microversion 1.0. Values outside the `1.0`–`1.1` range return HTTP 400.

---

### Checking both sites

Because each site runs an independent `protector-api`, you should poll both sites independently. Cross-site metadata sync requires bidirectional port 8788 connectivity, so a healthy response from both sites is a prerequisite before executing any DR operation:

```bash
for SITE_HOST in "${PRIMARY_API_HOST}" "${SECONDARY_API_HOST}"; do
  echo "Checking ${SITE_HOST}…"
  curl -sf http://${SITE_HOST}:8788/healthz || echo "  FAILED"
done
```

## Examples

### Example 1 — Healthy liveness response

A `GET /healthz` when both the process and database are up:

```bash
curl -si http://10.10.0.20:8788/healthz
```

```
HTTP/1.1 200 OK
Content-Type: application/json

{"status": "ok", "checks": {"database": "ok"}}
```

---

### Example 2 — Unhealthy liveness response (database down)

When the MariaDB connection is broken, `/healthz` returns HTTP 503 and identifies the failing subsystem:

```bash
curl -si http://10.10.0.20:8788/healthz
```

```
HTTP/1.1 503 Service Unavailable
Content-Type: application/json

{"status": "unhealthy", "checks": {"database": "OperationalError"}}
```

The `checks.database` value is the exception class name raised when the probe query fails. Use this to distinguish a connection-refused scenario (process down, load balancer should see TCP timeout) from a database-down scenario (process alive, 503 body).

---

### Example 3 — Version discovery

```bash
curl -si http://10.10.0.20:8788/
```

```
HTTP/1.1 200 OK
Content-Type: application/json

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://10.10.0.20:8788/v1/"
        }
      ]
    }
  ]
}
```

Key response fields:

| Field | Meaning |
|---|---|
| `id` | Version identifier — always `"v1.0"` |
| `status` | `"CURRENT"` for the active version |
| `version` | Maximum supported microversion (`"1.1"`) |
| `min_version` | Minimum supported microversion (`"1.0"`) |
| `links[].href` | Base URL for all versioned resource endpoints |

---

### Example 4 — Scripted pre-flight check before failover

This pattern is suitable for CI pipelines or runbook automation:

```bash
#!/usr/bin/env bash
set -euo pipefail

PRIMARY="http://10.10.0.20:8788"
SECONDARY="http://10.10.1.20:8788"

check_site() {
  local url="$1"
  local label="$2"
  local status
  status=$(curl -so /dev/null -w "%{http_code}" "${url}/healthz")
  if [[ "${status}" == "200" ]]; then
    echo "[OK]  ${label} (${url})"
  else
    echo "[FAIL] ${label} returned HTTP ${status}" >&2
    exit 1
  fi
}

check_site "${PRIMARY}"   "Primary site API"
check_site "${SECONDARY}" "Secondary site API"

echo "Both sites healthy — safe to proceed with DR operation."
```

---

### Example 5 — Microversion negotiation

To use microversion 1.1 on a versioned API call, include the header alongside your Keystone token. The version discovery response confirms `1.1` is the maximum:

```bash
curl -si \
  -H "X-Auth-Token: ${OS_TOKEN}" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://10.10.0.20:8788/v1/${OS_PROJECT_ID}/protection-groups
```

Omitting `OpenStack-API-Version` is valid and defaults to microversion 1.0. Passing an out-of-range value such as `protector 2.0` returns HTTP 400.

## Troubleshooting

### Issue: `curl` returns `Connection refused` on port 8788

**Symptom:** `curl: (7) Failed to connect to <host> port 8788: Connection refused`

**Likely causes:**
- `protector-api` is not running on that host.
- The charm install hook has not completed.
- The bind address in `[api] bind_host` is set to a specific IP that does not match the interface you are probing.

**Fix:**
1. Check the charm status: `juju status trilio-site-recovery-for-openstack`
2. Check the systemd unit: `juju ssh trilio-site-recovery-for-openstack/0 -- systemctl status protector-api`
3. Check `/var/log/protector/` for startup errors.
4. If `bind_host` was customised, verify it matches the IP you are querying.

---

### Issue: `/healthz` returns HTTP 503 with `"database": "OperationalError"`

**Symptom:** Process responds but reports database unavailable.

**Likely causes:**
- The MariaDB service is down or the `protector` database user credentials are wrong.
- The `[database] connection` DSN in `protector.conf` points to the wrong host or port.
- The `shared-db` Juju relation has not been satisfied (relation is missing or the related `mysql-innodb-cluster` unit is not ready).

**Fix:**
1. Verify the `shared-db` relation: `juju relations trilio-site-recovery-for-openstack`
2. Check the database service: `juju status mysql-innodb-cluster`
3. Do not edit `protector.conf` directly — the charm renders it. If the DSN is wrong, check which charm config options are set: `juju config trilio-site-recovery-for-openstack`
4. Run the Juju health-check action to get structured diagnostics: `juju run trilio-site-recovery-for-openstack/0 health-check` (Juju 3.x) or `juju run-action trilio-site-recovery-for-openstack/0 health-check` (Juju 2.9).

---

### Issue: `GET /` returns HTTP 404 instead of the version document

**Symptom:** Requests to `http://<host>:8788/` return 404.

**Likely causes:**
- You are hitting a path other than exactly `/` — for example, `/v1` or `/api`. The version discovery endpoint only responds to the bare root path.
- A reverse proxy in front of the API is rewriting the path.

**Fix:**
- Confirm the exact path: `curl -si http://<host>:8788/` (trailing slash or no trailing slash, both are accepted).
- Check your load-balancer or proxy configuration for path rewriting.
- Do not use `/v1` for version discovery — that prefix requires a `{tenant_id}` segment and an `X-Auth-Token`.

---

### Issue: `/healthz` returns 200 but versioned API calls return 404

**Symptom:** Health probe succeeds, but `GET /v1/{tenant_id}/protection-groups` returns 404.

**Likely causes:**
- The URL is missing the `/v1/` segment. For example, `/{tenant_id}/protection-groups` is not a valid route.
- The `{tenant_id}` segment is absent or malformed.

**Fix:**
- Ensure every versioned request uses the full path: `/v1/{tenant_id}/<resource>`.
- Verify your `OS_PROJECT_ID` environment variable is set correctly.
- Use version discovery (`GET /`) to confirm the correct base URL: the `links[].href` field shows the exact versioned root.

---

### Issue: Microversion header causes HTTP 400

**Symptom:** Adding `OpenStack-API-Version: protector <version>` returns `HTTP 400 Bad Request`.

**Likely causes:**
- The microversion value is outside the supported `1.0`–`1.1` range.
- The header value is malformed (for example, `protector/1.1` instead of `protector 1.1`).

**Fix:**
- Use `OpenStack-API-Version: protector 1.0` or `OpenStack-API-Version: protector 1.1`.
- Confirm the supported range by checking the `version` and `min_version` fields in the `GET /` response before sending versioned requests.
- Omitting the header entirely is valid and defaults to microversion 1.0.