Error Codes

This page is the authoritative reference for every HTTP status code Trilio Site Recovery can return, the JSON error envelope format, and the exception types the type field carries. When a request fails, this page tells you what the code means, how to distinguish causes that share a code (two independent paths return 503, for example), and what corrective action to take. One important exception to the general envelope shape: microversion rejections come from the version middleware, which runs before the fault middleware, so their response body carries only code and message — no type field.

To interpret and act on error responses from protector-api you need:

• A working protector-api deployment (port 8788 reachable).
• A valid Keystone token obtained from the Keystone service on port 5000, passed as X-Auth-Token on every authenticated request.
• The openstack dr CLI plugin installed separately (pip install -e ./client) if you are issuing commands via the CLI rather than raw HTTP.
• Familiarity with standard OpenStack error conventions — responses use JSON bodies and HTTP semantics consistent with other OpenStack services.
• For microversion-related errors, awareness that the supported range is 1.0 – 1.1 and that the OpenStack-API-Version: protector <version> header is optional; omitting it defaults to 1.0.

No additional installation is required beyond a working protector-api deployment. Error responses are returned automatically by the service.

To install the CLI plugin so you can exercise the API interactively:

1. Install the server package (Juju charm deployment — the only supported customer-facing method):

   Copy

   juju deploy trilio-site-recovery-for-openstack --base ubuntu@22.04 --to <machine-id>
   

2. Install the CLI plugin separately on any machine where you want to run openstack dr commands:

   Copy

   pip install -e ./client
   

   Important: Running pip install -e . from the repository root installs only the server binaries (protector-api, protector-engine, protector-manage). The CLI is the separate triliositerecovery package and requires the explicit ./client path.

3. Verify the API is reachable with an unauthenticated request to the version discovery endpoint:

   Copy

   curl http://<protector-api-host>:8788/
   

   A healthy service returns HTTP 200:

   Copy

   {
     "versions": [
       {
         "id": "v1.0",
         "status": "CURRENT",
         "version": "1.1",
         "min_version": "1.0",
         "links": [{"rel": "self", "href": "http://<host>:8788/v1/"}]
       }
     ]
   }
   

4. Verify the liveness endpoint (no authentication required):

   Copy

   curl http://<protector-api-host>:8788/healthz
   

   Returns HTTP 200 with {"status": "ok", "checks": {"database": "ok"}} when healthy.

Error behaviour is controlled by a small set of configuration options. In a Juju charm deployment, set these via juju config trilio-site-recovery <key>=<value> using hyphenated option names — do not edit /etc/protector/protector.conf directly, as the charm overwrites it on every config-changed event.

API binding ([api] section)

Database ([database] section)

Microversion negotiation

The OpenStack-API-Version: protector <version> request header is optional on all requests:

• Omitted: The middleware defaults to microversion 1.0 and processes the request normally.
• Valid value (1.0 or 1.1): Processed at the requested version.
• Malformed or out-of-range: Returns HTTP 400 with {"error": {"code": 400, "message": "..."}}.

No configuration option controls the supported version range; it is fixed at 1.0–1.1.

TLS and cross-site connectivity ([service_credentials] section)

Error envelope format

All errors from protector-api — except microversion rejections — share this JSON envelope:

{
  "error": {
    "code": 422,
    "message": "Instance a1b2c3d4 is not a Cinder boot instance. All protected VMs must boot from a Cinder volume.",
    "type": "InstanceNotCinderBoot"
  }
}

• code — the HTTP status code, repeated in the body for clients that cannot inspect response headers
• message — a human-readable description of the problem
• type — a string you can branch on programmatically (see catalogue below)

Note: type is usually the exception class name from protector/common/exception.py, but two values are synthesised by the fault middleware and have no corresponding class: ResourceConflict (409) and ServiceUnavailable (503). Do not assume type is always a class name you can import.

Microversion rejections

Version middleware rejections use the same envelope shape ({"error": {"code": 400, "message": "..."}}), but the type field is absent because the version middleware runs outside the fault middleware. Any 400 whose body lacks a type field is a microversion rejection. The response also includes the supported range in headers:

OpenStack-API-Version: protector 1.1
OpenStack-API-Minimum-Version: protector 1.0

Reading the type field for automation

Use type as your primary branching key in scripts and client code:

import requests

resp = requests.post(
    f"http://{host}:8788/v1/{tenant_id}/protection-groups/{pg_id}/members",
    headers={"X-Auth-Token": token},
    json={"member": {"instance_id": instance_id}}
)

if not resp.ok:
    err = resp.json()["error"]
    if err.get("type") == "InstanceNotCinderBoot":
        print("VM must boot from a Cinder volume, not ephemeral storage.")
    elif err.get("type") == "VolumeTypeNotReplicated":
        print("All volumes must use a Cinder type with replication_enabled='<is> True'.")
    elif err.get("type") == "ServiceUnavailable":
        print("Database is temporarily unavailable — retry after a moment.")
    else:
        print(f"Unexpected error ({err['code']}): {err['message']}")

Distinguishing the two sources of 503

503 Service Unavailable has two unrelated causes with different remediation paths:

The distinguishing signal is the type field. ServiceUnavailable always comes from the local service — even purely local read endpoints (like GET /v1/{tenant_id}/operations) will return it when the database is unreachable. Do not debug firewall rules between sites if local read-only calls also fail with ServiceUnavailable.

Pre-validating member eligibility

Before adding a VM to a Protection Group, use the validate endpoint to get detailed eligibility feedback without side effects:

# CLI
openstack dr protection group member validate <protection-group> \
  --instance <instance-id>

# API
POST /v1/{tenant_id}/protection-groups/{pg_id}/members/validate

This returns HTTP 200 with a structured errors array even when the instance is ineligible, making it safe to call in a pre-flight check loop.

Example 1 — Member add rejected: non-Cinder boot disk

Attempting to add a VM that boots from ephemeral storage:

openstack dr protection group member add my-pg --instance vm-ephemeral-boot

Response (HTTP 422):

{
  "error": {
    "code": 422,
    "message": "Instance vm-ephemeral-boot is not a Cinder boot instance. All protected VMs must boot from a Cinder volume.",
    "type": "InstanceNotCinderBoot"
  }
}

Fix: Boot the instance from a Cinder volume (openstack server create --boot-from-volume ...) before adding it to the Protection Group.

Example 2 — Member add rejected: volume type not replicated

Attempting to add a VM whose attached volume uses a non-replicated Cinder type:

openstack dr protection group member add my-pg --instance vm-standard-volume

Response (HTTP 422):

{
  "error": {
    "code": 422,
    "message": "Volume type standard does not support replication",
    "type": "VolumeTypeNotReplicated"
  }
}

Fix: Retype the volume to a Cinder volume type that has replication_enabled='<is> True' set as an extra-spec, then retry.

Example 3 — Failover action on a Protection Group in a bad state

Triggering failover on a PG that is already failing over:

openstack dr failover my-pg --failover-type planned

Response (HTTP 409):

{
  "error": {
    "code": 409,
    "message": "protection_group is in invalid state failing_over: cannot initiate failover while another operation is in progress",
    "type": "InvalidState"
  }
}

Fix: Poll openstack dr operation show <id> until the in-progress operation completes, then retry. If the operation is stuck after an engine restart, use openstack dr protection group reconcile my-pg to recover.

Example 4 — Duplicate resource name conflict

Creating a Protection Group with the same name as a recently deleted one:

openstack dr protection group create web-tier \
  --primary-site site-a --secondary-site site-b \
  --replication-type async --volume-type replicated-ssd

Response (HTTP 409):

{
  "error": {
    "code": 409,
    "message": "The request conflicts with an existing resource. A name may still be held by a deleted resource.",
    "type": "ResourceConflict"
  }
}

Fix: Choose a different name, or wait for the soft-delete tombstone to clear. ResourceConflict has no class in protector.common.exception — it is synthesised by the fault middleware for database unique-constraint violations.

Example 5 — Database unavailable (local ServiceUnavailable)

Any request when the local MariaDB/MySQL is down:

openstack dr operation list

Response (HTTP 503):

{
  "error": {
    "code": 503,
    "message": "Database temporarily unavailable",
    "type": "ServiceUnavailable"
  }
}

Fix: Check the database service on the local host. In a Juju deployment, verify the mysql-innodb-cluster and trilio-site-recovery-mysql-router units are in active/idle state. This error is retryable once the database recovers.

Example 6 — Remote site unreachable

Triggering a failover when the secondary site's protector-api is unreachable:

openstack dr failover my-pg --failover-type unplanned

Response (HTTP 503):

{
  "error": {
    "code": 503,
    "message": "Remote site dr-site-b is unreachable: connection refused",
    "type": "RemoteSiteUnreachable"
  }
}

Fix: Verify bidirectional network connectivity on port 8788 between sites. Check firewall rules, routing, and that the remote protector-api service is running.

Example 7 — Microversion out of range

Sending a version header with an unsupported microversion:

curl -s -H "X-Auth-Token: $TOKEN" \
     -H "OpenStack-API-Version: protector 2.0" \
     http://protector-api:8788/v1/${TENANT_ID}/protection-groups

Response (HTTP 400):

{
  "error": {
    "code": 400,
    "message": "Invalid API version '2.0'. Supported versions: 1.0 to 1.1"
  }
}

Note the absence of a type field — this response comes from the version middleware, which runs before the fault middleware. The supported range is also returned in response headers:

OpenStack-API-Version: protector 1.1
OpenStack-API-Minimum-Version: protector 1.0

Example 8 — Version discovery

Checking the API's supported microversion range (no authentication required):

curl -s http://protector-api:8788/

Response (HTTP 200):

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://protector-api:8788/v1/"
        }
      ]
    }
  ]
}

HTTP 400 — Bad Request

Symptom: Request rejected immediately with InvalidInput, ValidationError, InvalidAction, or InvalidParameters.

Likely causes:

• A required field is missing from the request body
• An enum field (failover_type, replication_type, status) has an unrecognised value
• The action key in the /action request body is unknown (type will be InvalidAction)
• A boot_order value is negative or non-integer

Fix: Check the message field for the specific field name. Consult openstack dr <resource> <action> --help for required arguments and valid values. For the /action endpoint, valid action keys are failover, failback, test_failover, test_cleanup, and reconcile.

HTTP 400 — Microversion rejection (no type field)

Symptom: 400 Bad Request with a body that has no type key, only code and message.

Likely cause: The OpenStack-API-Version header contains a version outside the 1.0–1.1 range, or is malformed.

Fix: Check the response headers OpenStack-API-Version and OpenStack-API-Minimum-Version to see the supported range. Either correct the header or omit it entirely (the middleware defaults to 1.0 when the header is absent).

HTTP 400 — InvalidExecutionSite

Symptom: 400 Bad Request with type: InvalidExecutionSite when triggering a failover or other site-sensitive operation.

Likely cause: The operation is being driven from the wrong site. For example, a test failover must be executed from the secondary site.

Fix: Authenticate against the correct site's Keystone and reissue the request from that site.

HTTP 401 — Unauthorized

Symptom: 401 Unauthorized with type: Unauthorized or type: SiteAuthenticationFailed.

Likely causes:

• Expired or invalid X-Auth-Token
• The Keystone service credentials for a registered site are wrong (SiteAuthenticationFailed)
• Keystone trust creation failed because a required role (dr_operator or dr_site_admin) does not exist on the remote site

Fix: Re-authenticate with Keystone to get a fresh token. For SiteAuthenticationFailed, verify the service credentials with openstack dr site set <site> --service-username ... --service-password .... Ensure both dr_operator and dr_site_admin roles exist on both sites before any cross-site operation.

HTTP 403 — Forbidden

Symptom: 403 Forbidden with type: Forbidden or type: ProtectionGroupFailedOver.

Likely causes:

• ProtectionGroupFailedOver: you are attempting to modify a Protection Group that is currently failed over to the DR site. Modifications are blocked in this state.
• Forbidden: the authenticated user's role does not permit the operation (e.g., attempting a site-level operation without dr_site_admin).

Fix: For ProtectionGroupFailedOver, run openstack dr failback <pg-id> to return to the primary site before modifying the group. For policy errors, verify role assignments on the project.

HTTP 404 — Not Found

Symptom: 404 Not Found with a type such as ProtectionGroupNotFound, SiteNotFound, OperationNotFound, or MemberNotFound.

Likely causes:

• The UUID or name in the URL does not exist or belongs to a different tenant
• The resource was soft-deleted and is no longer visible
• The URL is missing the /v1/ prefix (URLs without /v1/ return 404 from the router)

Fix: Verify the resource ID with openstack dr <resource> list. Confirm the URL follows the pattern /v1/{tenant_id}/<resource>. If the resource was deleted and you are trying to recreate it with the same name, see HTTP 409 ResourceConflict below.

HTTP 409 — Conflict

Symptom: 409 Conflict with one of several type values.

Fix for InvalidState: Wait for the current operation to complete (poll openstack dr operation show <id>), then retry. If the operation is permanently stuck after an engine restart, use openstack dr protection group reconcile <pg> to recover.

Fix for ResourceConflict: Use a different name, or allow time for any tombstone constraints to clear.

HTTP 422 — Unprocessable Entity

Symptom: 422 Unprocessable Entity — a DR eligibility precondition failed.

For member-add eligibility, run openstack dr protection group member validate <pg> --instance <id> first to get a structured errors array without side effects.

HTTP 503 — Service Unavailable: database (local)

Symptom: Every request fails with 503 and type: ServiceUnavailable, including simple list operations like GET /v1/{tenant_id}/operations.

Distinguishing signal: type is ServiceUnavailable (not SiteUnreachable), and local-only endpoints fail too.

Fix: Check the local MariaDB/MySQL service. In a Juju deployment:

juju status mysql-innodb-cluster
juju status trilio-site-recovery-mysql-router

Verify the [database] connection URL in protector.conf (Juju: check juju config trilio-site-recovery). This error is retryable once the database recovers — do not investigate cross-site firewall rules for this error.

HTTP 503 — Service Unavailable: remote site unreachable

Symptom: Cross-site operations (failover, failback, tenant mapping, metadata sync) fail with 503 and type: SiteUnreachable or RemoteSiteUnreachable. Local read operations (listing PGs, listing operations) succeed.

Fix:

1. Verify bidirectional TCP connectivity on port 8788 between sites — one-way connectivity causes validation failures on remote operations
2. Confirm the remote protector-api service is running on the peer unit
3. In Juju deployments, check MAAS network space bindings are symmetric between sites — asymmetric routing can present as TLS certificate verification failures on cross-site calls
4. Verify [service_credentials] section is correctly configured, particularly cafile if TLS is in use, or set insecure = true only in test environments