Authentication

Trilio Site Recovery for OpenStack authenticates all API requests through OpenStack Keystone — the same identity service your OpenStack cloud already uses. Every call to the protector-api on port 8788 must carry a valid Keystone token in the X-Auth-Token header. The service never issues tokens itself and has no /auth endpoint. This page explains how to obtain tokens, structure authenticated requests, use API microversions, understand role requirements, and handle authentication errors.

Before making authenticated API calls, ensure you have:

• A working OpenStack cloud (≥ Victoria) with Keystone available on port 5000
• Trilio Site Recovery deployed via the Juju charm (trilio-site-recovery-for-openstack)
• The openstack CLI client installed via pip install python-openstackclient (not the snap — the snap is incompatible with the triliositerecovery plugin)
• The triliositerecovery CLI plugin installed separately: pip install -e ./client from the repository root, or apt-get install python3-triliositerecovery on charm-deployed nodes
• Your OpenStack credentials (RC file or environment variables: OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, OS_PROJECT_NAME, etc.)
• The Keystone roles dr_operator and dr_site_admin created on both sites before any cross-site operation (trust creation fails if these roles are absent)
• Your user granted the member + dr_operator roles on any project you want to protect; site-level operations additionally require dr_site_admin
• Network connectivity to protector-api on port 8788 — bidirectional between sites

The triliositerecovery CLI plugin is a separate Python package from the server components. Installing the server package does not install the CLI.

1. Install the CLI plugin (pip / development)

# From the repository root
pip install -e ./client

The package name is triliositerecovery. Do not use trilioclient — that name does not exist.

2. Install the CLI plugin (Debian package)

On charm-deployed nodes the Debian package is preferred:

apt-get install python3-triliositerecovery

3. Verify the plugin is recognised

openstack dr --help

You should see the dr command group listed. If it is missing, confirm that python-openstackclient was installed via pip, not the snap.

4. Source your OpenStack credentials

source ~/openrc          # or whichever RC file your cloud provides

The CLI reads the standard OS_* environment variables and exchanges them for a Keystone token automatically before each command.

Authentication-related configuration lives in two sections of /etc/protector/protector.conf. In Juju charm deployments, do not edit this file directly — the charm overwrites it on every config-changed event. Use juju config trilio-site-recovery <key>=<value> with hyphenated option names instead.

[keystone_authtoken] section

This section is consumed by keystonemiddleware and controls how protector-api validates incoming tokens against Keystone.

The Juju charm creates the protector Keystone service account automatically via the identity-credentials relation. You do not need to create it manually.

[service_credentials] section

These options control how protector-engine authenticates cross-site calls (to the remote site's Keystone, Nova, Cinder, and Neutron).

TLS verification: If your cross-site endpoints use self-signed certificates, set cafile to point at your CA bundle. Setting insecure = true disables all certificate checks and must not be used in production.

Keystone roles

The service never creates roles — it looks them up. Both dr_operator and dr_site_admin must exist on both sites before any cross-site operation. Create them with:

openstack role create dr_operator
openstack role create dr_site_admin

Grant them to your users:

# For DR operators (Protection Group operations)
openstack role add --project <your-project> --user <username> member
openstack role add --project <your-project> --user <username> dr_operator

# For site administrators (site registration, failover at site level)
openstack role add --project <your-project> --user <username> dr_site_admin

Grant member + dr_operator to the protector service account on every DR-enabled project on each site — not only on the service project.

Obtaining a Keystone token

All protector-api requests require an X-Auth-Token header containing a valid Keystone token scoped to your project. Obtain one with:

export OS_TOKEN=$(openstack token issue -f value -c id)
export OS_PROJECT_ID=$(openstack token issue -f value -c project_id)

Or source your RC file and let the openstack dr CLI handle token acquisition automatically.

Making authenticated REST requests

All versioned resource endpoints require the /v1/{tenant_id}/ prefix. A URL omitting /v1/ returns HTTP 404.

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  http://<protector-api-host>:8788/v1/${OS_PROJECT_ID}/protection-groups

API microversion header

The OpenStack-API-Version header is optional. When omitted, the middleware defaults to microversion 1.0 and processes the request normally. The supported range is 1.0 to 1.1.

# Explicitly request microversion 1.1
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://<protector-api-host>:8788/v1/${OS_PROJECT_ID}/protection-groups

A malformed value or a version outside 1.0–1.1 returns HTTP 400 Bad Request.

Version discovery (no authentication required)

The root endpoint GET / returns the supported API versions without requiring a token:

curl -s http://<protector-api-host>:8788/

Using the CLI

The openstack dr CLI handles token acquisition automatically from your environment variables or RC file:

source ~/openrc
openstack dr protection group list

Keystone trusts (cross-site operations)

When you trigger a DR operation (failover, failback, test failover), the engine needs to act on your behalf on the remote site. It does this through Keystone trusts — a delegation mechanism where you (the trustor) grant the protector service account (the trustee) the right to call remote APIs as your project.

Trusts are created automatically on your first DR operation. You do not configure them manually. The engine re-uses cached trust-scoped sessions for all subsequent calls within an operation, refreshing them before expiry.

The roles delegated in a trust default to member,dr_operator. Both must exist on the remote site or trust creation fails with an error naming the missing role.

Role requirements summary

Example 1 — Version discovery (unauthenticated)

Confirm the API is reachable and discover the supported microversion range:

curl -s http://10.0.0.10:8788/

Expected response (HTTP 200):

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://10.0.0.10:8788/v1/"
        }
      ]
    }
  ]
}

Example 2 — Obtain a token and list sites

# Source credentials
source ~/openrc

# Get a scoped token and project ID
OS_TOKEN=$(openstack token issue -f value -c id)
OS_PROJECT_ID=$(openstack token issue -f value -c project_id)

# List registered DR sites
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://10.0.0.10:8788/v1/${OS_PROJECT_ID}/sites

Expected response (HTTP 200):

{
  "sites": [
    {
      "id": "a1b2c3d4-...",
      "name": "site-primary",
      "site_type": "PRIMARY",
      "status": "ACTIVE",
      "auth_url": "http://keystone-site1:5000/v3"
    },
    {
      "id": "e5f6g7h8-...",
      "name": "site-secondary",
      "site_type": "SECONDARY",
      "status": "ACTIVE",
      "auth_url": "http://keystone-site2:5000/v3"
    }
  ]
}

Example 3 — Authenticated request with explicit microversion

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://10.0.0.10:8788/v1/${OS_PROJECT_ID}/protection-groups

Example 4 — Trigger a failover and poll the operation

# Trigger a planned failover
RESPONSE=$(curl -s -X POST \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"failover": {"failover_type": "planned"}}' \
  http://10.0.0.10:8788/v1/${OS_PROJECT_ID}/protection-groups/<pg-id>/action)

echo $RESPONSE

# Extract the operation ID
OPERATION_ID=$(echo $RESPONSE | python3 -c "import sys,json; print(json.load(sys.stdin)['operation_id'])")

# Poll operation status
curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://10.0.0.10:8788/v1/${OS_PROJECT_ID}/operations/${OPERATION_ID}

Initial action response (HTTP 202):

{
  "message": "Failover initiated",
  "status": "pending",
  "protection_group_id": "<pg-id>",
  "operation_id": "<op-id>",
  "operation": { "id": "<op-id>", "status": "pending", "progress": 0 }
}

Note: the initial status is "pending", not "running".

Example 5 — CLI equivalents

The openstack dr CLI wraps the same API calls and manages token acquisition automatically:

# List protection groups
openstack dr protection group list

# Show replication health
openstack dr replication health <pg-id>

# Planned failover with --wait to block until complete
openstack dr failover <pg-id> --failover-type planned --wait

# Poll an operation manually
openstack dr operation show <operation-id>

Example 6 — Handling an invalid microversion (HTTP 400)

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  -H "OpenStack-API-Version: protector 9.9" \
  http://10.0.0.10:8788/v1/${OS_PROJECT_ID}/protection-groups

Response (HTTP 400):

{
  "code": 400,
  "message": "Version 9.9 is not supported. Supported versions: 1.0 to 1.1"
}

Issue: HTTP 401 Unauthorized on all requests

Symptom: Every API call returns HTTP 401, even with a token that looks valid.

Likely causes:

• The token has expired (default Keystone token lifetime is 1 hour)
• The token was issued against a different Keystone than the one protector-api validates against
• keystonemiddleware is misconfigured in [keystone_authtoken] — wrong auth_url or bad service account credentials

Fix:

1. Re-issue the token: openstack token issue
2. Confirm [keystone_authtoken] auth_url in protector.conf points to the same Keystone your client uses
3. In Juju deployments, check the identity-credentials relation is established: juju status trilio-site-recovery

Issue: HTTP 403 Forbidden on Protection Group operations

Symptom: You can list Protection Groups but cannot create, modify, or trigger failover.

Likely cause: Your user has only the member role. Mutating operations require dr_operator.

Fix:

openstack role add --project <your-project> --user <username> dr_operator

Then re-issue a new token (role changes are not reflected in existing tokens).

Issue: HTTP 403 on site registration or tenant mapping creation

Symptom: POST /v1/{tenant_id}/sites or POST /v1/{tenant_id}/tenant-mappings returns 403.

Likely cause: These endpoints require dr_site_admin, not just dr_operator.

Fix:

openstack role add --project <your-project> --user <username> dr_site_admin

Issue: Trust creation fails with "Role does not exist on site"

Symptom: The first cross-site DR operation fails with an error like Role dr_operator does not exist on site site-secondary.

Likely cause: The dr_operator or dr_site_admin role was not created on both sites before the first operation. The service looks up roles — it does not create them.

Fix: Create the missing role on the remote site:

# Source the RC file for the remote site
source ~/openrc-site2
openstack role create dr_operator
openstack role create dr_site_admin

Then retry the operation.

Issue: HTTP 400 — bad microversion

Symptom: {"code": 400, "message": "Version X.Y is not supported"} even though you believe the version exists.

Likely cause: The supported range is 1.0 to 1.1. Any value outside this range, or a malformed header value, returns 400.

Fix: Omit the header entirely (defaults to 1.0) or use OpenStack-API-Version: protector 1.1.

Issue: HTTP 404 on all versioned resource paths

Symptom: Calls to /protection-groups or /{tenant_id}/protection-groups return 404.

Likely cause: The /v1/ prefix is missing. All versioned resource endpoints require the form /v1/{tenant_id}/resource.

Fix: Correct your URL: http://<host>:8788/v1/<project-id>/protection-groups

Issue: Cross-site operations fail with TLS certificate errors

Symptom: Failover or metadata sync fails with SSL verification errors in the engine log at /var/log/protector/.

Likely cause: The remote site's endpoints use a certificate signed by a private CA not trusted by default.

Fix: Set cafile in [service_credentials] to point at your CA bundle:

In protector.conf (or via juju config in charm deployments):

[service_credentials]
cafile = /etc/ssl/certs/my-ca-bundle.pem

Do not set insecure = true in production — this disables all TLS verification for cross-site calls.

Issue: openstack dr commands not found after installing the package

Symptom: openstack: 'dr' is not an openstack command after installing openstack-protector.

Likely cause: The CLI plugin (triliositerecovery) is a separate package from the server. Installing the server package does not install the CLI.

Fix:

# From the repository root
pip install -e ./client
# or via apt
apt-get install python3-triliositerecovery

Also verify you are using pip-installed python-openstackclient, not the openstack snap — the snap is incompatible with this plugin.