Troubleshooting

This runbook helps you diagnose and resolve common API errors, service failures, and replication issues in Trilio Site Recovery for OpenStack, and describes when and how to collect a support bundle for escalation.

This runbook covers:

• Interpreting HTTP error codes returned by protector-api
• Diagnosing service health using systemd, logs, and the CLI
• Resolving common errors: authentication failures, member-add rejections, failover pre-check failures, and cross-site connectivity problems
• Collecting a support bundle from a Juju charm deployment
• Escalation guidance

This runbook does not cover:

• Initial deployment or charm configuration (see the Installation guide)
• Pure Storage FlashArray array-level troubleshooting
• OpenStack Nova, Cinder, or Neutron administration
• Non-Juju deployment paths (Kolla and OpenShift are internal development tools and are not supported customer-facing deployments)

Before using this runbook:

• The Trilio Site Recovery Juju charm (trilio-site-recovery) is deployed and related on both sites
• You have SSH access to the Juju controller and can run juju ssh
• The openstack CLI with the triliositerecovery plugin is installed via pip install python3-triliositerecovery (not the snap — the openstackclient snap is incompatible with this plugin)
• You have sourced a valid OpenRC file for the affected project
• You know the protection group ID or name for the affected workload
• Both sites are running Ubuntu Jammy or Noble with the trilio-site-recovery Debian package installed

Version note: The maximum supported API microversion is 1.1. Check OpenStack-API-Version headers in error responses to confirm the version negotiated.

Work through the sections below in order. Each section addresses a distinct failure class.

1. Confirm service health on both sites

On each site's Juju unit, check that both systemd services are running:

juju ssh trilio-site-recovery/leader \
  "sudo systemctl status protector-api protector-engine --no-pager"

Expected: both units show Active: active (running). If either is failed or inactive, restart it:

juju ssh trilio-site-recovery/leader \
  "sudo systemctl restart protector-api protector-engine"

To verify the API process is accepting connections (no authentication required for version discovery):

curl -s http://<site-ip>:8788/

A healthy response wraps version information in a versions array:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [{"rel": "self", "href": "http://<site-ip>:8788/v1/"}]
    }
  ]
}

If you receive a connection refused or timeout, the API process is not listening — proceed to step 2.

2. Read the service logs

Application logs are written to /var/log/protector/ on each unit. Retrieve them with:

juju ssh trilio-site-recovery/leader \
  "sudo journalctl -u protector-api --since '1h ago' --no-pager"

juju ssh trilio-site-recovery/leader \
  "sudo journalctl -u protector-engine --since '1h ago' --no-pager"

For older log history (up to 14 days of rotated files):

juju ssh trilio-site-recovery/leader \
  "sudo sh -c 'for f in /var/log/protector/*.log*; do [ -f \"\$f\" ] || continue; echo \"===== \$f ====="; zcat -f \"\$f\" 2>/dev/null; echo; done'"

Search for the Python exception class name from the error response (for example, ProtectionGroupNotFound, VolumeTypeNotReplicated) to jump directly to the relevant traceback.

3. Interpret HTTP error codes

Every error response from protector-api includes a type field (the exception class name), a code, and a message. Use the type field to identify the exact error:

{"type": "VolumeTypeNotReplicated", "code": 422, "message": "Volume type pure-local does not support replication"}

Exception: Microversion rejection errors from the version middleware return only code and message, without type.

Common errors and remediation

HTTP 400 — Bad request

HTTP 401 — Unauthorised

HTTP 403 — Forbidden

HTTP 404 — Not found

HTTP 409 — Conflict

HTTP 422 — Unprocessable entity

These errors indicate a DR eligibility failure. They are the most common errors when adding members or triggering failover.

Pre-checking member eligibility without side effects:

Before running openstack dr protection group member add, validate the instance:

openstack dr protection group member validate <pg-id> --instance <instance-id>

This returns HTTP 200 with a structured errors array listing any eligibility problems, with no state changes.

HTTP 500 — Internal server error

These indicate a failure inside the engine or storage layer. Collect logs (step 2) and look for the exception class name.

Common 500 errors:

HTTP 503 — Service unavailable

4. Diagnose cross-site connectivity failures

Cross-site connectivity failures are a common cause of SiteUnreachable, RemoteSiteUnreachable, MetadataSyncFailed, and trust authentication errors. Port 8788 must be open bidirectionally between both sites.

Verify connectivity from each site to the other:

# From site 1, reach site 2
curl -s http://<site-2-ip>:8788/

# From site 2, reach site 1
curl -s http://<site-1-ip>:8788/

Both must return the versions JSON shown in step 1. A one-way failure leaves tenant mappings created locally but unsynced to the remote site, causing validation failures on subsequent cross-site operations.

Also verify Keystone (port 5000), Nova (port 8774), and Cinder (port 8776) are reachable from each site to the other.

If your environment uses TLS and certificate verification fails, set cafile in protector.conf under [service_credentials] to a PEM-encoded CA bundle, or (development/test environments only) set insecure = true. In a Juju deployment, set these via juju config:

juju config trilio-site-recovery config-flags="service_credentials:cafile=/etc/ssl/certs/ca-certificates.crt"

In MAAS environments, asymmetric network space bindings present as broken TLS certificate verification on cross-site API calls — ensure Juju network space bindings are symmetric between sites.

5. Diagnose authentication and trust failures

If operations fail with TrustAuthFailed or SiteAuthenticationFailed:

1. Confirm the dr_operator and member Keystone roles exist on both sites:

   Copy

   openstack role list | grep -E 'dr_operator|member'
   

   If either role is missing, create it:

   Copy

   openstack role create dr_operator
   

   The service never creates roles — trust creation fails immediately, naming the missing role.

2. Confirm the protector service account has member and dr_operator on each DR-enabled project on each site:

   Copy

   openstack role assignment list --user protector --project <project-name>
   

3. Confirm tenant users have member and dr_operator granted on their project:

   Copy

   openstack role assignment list --user <user> --project <project-name>
   

4. Validate the site's service credentials against the remote Keystone:

   Copy

   openstack dr site set <site-id> \
     --service-username protector \
     --service-password <password> \
     --service-project-name service
   

   The API validates these credentials against the remote Keystone at save time and returns SiteAuthenticationFailed if they are incorrect.

6. Diagnose replication health failures

When failover is blocked by FailoverNotReady or ReplicationNotHealthy, inspect the replication health report:

openstack dr replication health <pg-id>

The response nests fields under replication_health. The programmatic readiness check is:

replication_health.failover_validation.can_failover

If can_failover is false, inspect failover_validation.blockers for the specific reason. Common blockers:

• Lag too high — the secondary is behind the RPO target. Wait for the next replication cycle and retry, or accept data loss with --failover-type unplanned.
• No recovery points — the replication has not completed a successful cycle yet. Wait for the Pure Storage Protection Group to complete a snapshot cycle.
• Replication link down — check FlashArray connectivity between arrays.

You can also query the replication health directly via the API:

curl -s \
  -H "X-Auth-Token: $OS_TOKEN" \
  http://<site-ip>:8788/v1/<tenant-id>/protection-groups/<pg-id>/replication-health

7. Recover a protection group stuck in an intermediate state

If an operation was interrupted (for example, by a service restart during failover), the protection group may be stuck in a non-terminal state such as failing_over or failing_back.

Check current status:

openstack dr protection group show <pg-id>

Check for any running operations:

openstack dr operation list --protection-group <pg-id> --status running

Cancel a stuck in-progress operation:

openstack dr operation cancel <operation-id>

If the PG remains inconsistent after cancellation, use the reconcile command to detect and repair the interrupted state:

openstack dr protection group reconcile <pg-id>

If the PG status is incorrect and you need to force a reset, use:

# Reset to active (primary is operational)
openstack dr protection group set <pg-id> --status active

# Reset to failed_over (secondary is operational)
openstack dr protection group set <pg-id> --status failed_over

# Reset to error to mark for investigation
openstack dr protection group set <pg-id> --status error

Only use manual status resets when you have confirmed the actual storage and VM state independently — setting an incorrect status can cause further inconsistency.

8. Diagnose database connectivity failures

If the API returns 500 on all requests and the logs show database connection errors:

1. In a Juju deployment the database and user are created automatically via the shared-db relation — no manual creation is required. Check that the relation is healthy:

   Copy

   juju status trilio-site-recovery --format yaml | grep -A5 'relations'
   

2. Confirm the charm-rendered configuration is correct. Do not edit /etc/protector/protector.conf directly — the charm overwrites it on every config-changed event. Use juju config to set options:

   Copy

   juju config trilio-site-recovery
   

3. Run the charm health-check action to confirm database reachability:

   Copy

   # Juju 3.x
   juju run trilio-site-recovery/leader health-check
   
   # Juju 2.9
   juju run-action trilio-site-recovery/leader health-check --wait
   

9. Diagnose storage driver misconfiguration

If failover or member-add operations fail with PureStorageError, PureConnectionError, or StorageDriverConfigError:

1. Confirm the storage driver is set to pure (not the default mock):

   Copy

   juju config trilio-site-recovery | grep storage-driver
   

   The default is mock, which is suitable for development only. Set it for production:

   Copy

   juju config trilio-site-recovery storage-driver=pure
   

2. Confirm the FlashArray management IP and API token are set. In a Juju deployment, use hyphenated config key names:

   Copy

   juju config trilio-site-recovery pure-primary-management-ip
   

3. Confirm verify_ssl is set correctly for your FlashArray certificate. If using a private CA, provide it via config-flags:

   Copy

   juju config trilio-site-recovery \
     config-flags="pure_storage:verify_ssl=true"
   

Config key format reminder: Charm configuration options use hyphens (e.g., pure-primary-management-ip). Config-flags overrides use colon separators: section:key=value. A dot separator in config-flags fails silently.

10. Collect a support bundle

If you cannot resolve the issue with the steps above, collect a support bundle before escalating. The bundle script collects logs, configuration (with credentials redacted), database state, and DR state from all units of the Juju application:

bash scripts/collect-support-bundle.sh \
  --app trilio-site-recovery \
  --since 8h \
  --pg <protection-group-id> \
  --openrc ~/openrc.sh \
  --output ~/support-bundle

Options:

The bundle includes (with credentials redacted from all output):

• Application logs from /var/log/protector/ including rotated history
• systemd journal for protector-api and protector-engine
• Charm hook logs
• Juju status (application and model)
• Charm configuration
• Database state: non-terminal protection groups, running operations, table row counts
• DR state: site list, operation list, protection group status, replication health
• Systemd service status and package versions
• Network connectivity checks (listening sockets, DNS, resolver)
• Disk, memory, and OOM evidence

After applying a fix, confirm resolution with these checks:

1. Service health: Both systemd units are active:

   Copy

   juju ssh trilio-site-recovery/leader \
     "sudo systemctl is-active protector-api protector-engine"
   

   Expected: active for both.

2. API responding: Version discovery returns HTTP 200:

   Copy

   curl -s http://<site-ip>:8788/
   

   Expected: JSON with a versions array containing "status": "CURRENT".

3. Protection group health: The PG is in a terminal state and replication is healthy:

   Copy

   openstack dr protection group show <pg-id>
   openstack dr replication health <pg-id>
   

   Expected: PG status is active or failed_over; replication_health.failover_validation.can_failover is true.

4. No stuck operations: No operations are in running state for more than the expected duration:

   Copy

   openstack dr operation list --protection-group <pg-id> --status running
   

   Expected: empty list, or operations that are actively progressing.

5. Cross-site connectivity: Both sites can reach each other on port 8788:

   Copy

   curl -s http://<site-1-ip>:8788/
   curl -s http://<site-2-ip>:8788/
   

   Both must return the versions JSON.

The troubleshooting steps in this runbook are primarily diagnostic and read-only. The actions that do change state are:

• Service restarts (systemctl restart protector-api protector-engine): Restarting the services is safe and idempotent. Any in-progress operation will be left in an interrupted state; use openstack dr operation cancel <id> followed by openstack dr protection group reconcile <pg-id> to recover.

• Manual protection group status reset (openstack dr protection group set <pg-id> --status active|failed_over|error): This is a low-level override. If you set an incorrect status, reset it to the correct value using the same command. The allowed values are active, failed_over, and error exactly.

• Site credential update (openstack dr site set ...): If you set incorrect credentials, update them again with the correct values. The API validates credentials against the remote Keystone at save time.

• Juju config changes (juju config trilio-site-recovery <key>=<value>): Revert any unintentional config change by setting the key back to its previous value. Do not edit /etc/protector/protector.conf directly — the charm overwrites it.

• Operation cancellation (openstack dr operation cancel <id>): Cancellation cannot be undone. After cancelling, use openstack dr protection group reconcile <pg-id> to restore consistency, then re-trigger the intended operation.

If this runbook does not resolve the issue, escalate to Trilio support at https://www.trilio.io with the following information:

1. Support bundle collected with collect-support-bundle.sh (see step 10). The bundle includes logs, configuration, and state with credentials automatically redacted.

2. Error details:

   • The exact error type, code, and message from the API response
   • The CLI command or API request that triggered the error
   • The protection group ID and both site IDs
   • The operation ID if an async operation failed (from openstack dr operation show <id>)

3. Environment details:

   • Output of juju status trilio-site-recovery --format yaml
   • Package versions from each unit: dpkg -l trilio-site-recovery python3-triliositerecovery
   • OpenStack version on each site
   • Pure Storage Purity//FA version on each array

4. Replication health report:

   Copy

   openstack dr replication health <pg-id>
   

5. Timeline: When the issue first appeared, what changed around that time (deployments, network changes, array maintenance), and whether the issue is reproducible.

Provide all of the above in your initial support request to avoid round-trips.