Development Guide

This guide covers setting up a local development environment for Trilio Site Recovery for OpenStack, iterating on code changes without full rebuilds, running the test suite against a live environment, and understanding the internal architecture well enough to add new features or debug failures. It targets developers who want to modify the service itself — the API, the engine workflows, the storage driver, or the CLI plugin — rather than operators deploying the product for end users. The validated customer-facing deployment method is the Juju charm; the local development workflow described here uses the bin/dev-launch.sh helper and a direct Python install, which is suitable only for development and testing.

Before starting local development you need:

• Python 3.10 or later — the codebase uses type annotations and match syntax available from 3.10+
• Two OpenStack environments ≥ Victoria with Nova, Cinder, Keystone, and Neutron on each — the test suite makes live API calls and cannot run against mocks alone
• MariaDB or MySQL ≥ 10.3 reachable from your development machine — the compiled-in default connection string is mysql+pymysql://protector:password@localhost/protector; override it in etc/protector/protector.conf
• RabbitMQ reachable from your machine — the [DEFAULT] transport_url option must point to it; without it protector-engine cannot receive RPC calls from protector-api
• python-openstackclient installed via pip, not the snap — the triliositerecovery CLI plugin is incompatible with the openstackclient snap
• Pure Storage FlashArray with async replication configured between the two lab arrays, or storage_driver = mock in [storage] to skip hardware entirely during development
• Keystone roles dr_operator and dr_site_admin pre-created on both sites — trust creation fails with a named error if either role is absent
• A dedicated test user (e.g. druser in dr-project) with member and dr_operator roles — never use admin credentials for DR operations; doing so masks permission bugs that tenant users will hit in production
• SSH access to both cluster nodes if you intend to deploy code to the lab containers
• Git clone of the repository with submodules initialised

1. Clone the repository

git clone <repository-url> openstack-protector
cd openstack-protector

2. Create and activate a virtual environment

python3 -m venv .venv
source .venv/bin/activate

3. Install the server package

Installing from the repository root installs the server binaries (protector-api, protector-engine, protector-manage) and the protector Python package:

pip install -e .

Important: This step does not install the CLI plugin. The CLI is a separate package.

4. Install the CLI plugin

The openstack dr commands live in the client/ subdirectory and must be installed separately:

pip install -e ./client

The package name is triliositerecovery. After installation, openstack dr --help should list all DR subcommands.

5. Install the Pure Storage SDK (optional, for Pure driver)

If you are developing against real FlashArray hardware:

pip install py-pure-client

For mock-only development, skip this step and set storage_driver = mock in your config.

6. Configure the service

Copy the example configuration and edit it for your environment:

cp etc/protector/protector.conf etc/protector/protector.conf.local

At minimum, set the database connection and RabbitMQ transport URL:

[DEFAULT]
transport_url = rabbit://openstack:<password>@<rabbitmq-host>:5672//

[database]
connection = mysql+pymysql://protector:<password>@<db-host>:3306/protector

For mock-only development (no FlashArray hardware):

[storage]
storage_driver = mock

For Pure Storage hardware:

[storage]
storage_driver = pure

[pure_storage]
primary_management_ip = <fa1-ip>
primary_api_token = <fa1-token>
secondary_management_ip = <fa2-ip>
secondary_api_token = <fa2-token>
primary_site_name = cluster1
secondary_site_name = cluster2
verify_ssl = false

7. Initialise the database schema

protector-manage db sync

This creates or migrates the protector schema. The canonical form is db sync; db upgrade is a historical alias for the same operation.

8. Start the services locally

# Start both API and engine in the background
bash bin/dev-launch.sh both

# Or start them individually with debug logging
bash bin/dev-launch.sh api --debug
bash bin/dev-launch.sh engine --debug

# Stop background services
bash bin/dev-launch.sh stop

Both services bind on port 8788 by default. Use a custom config file with:

CONFIG_FILE=/path/to/protector.conf bash bin/dev-launch.sh api

9. Verify the service is running

# Version discovery (no auth required)
curl http://localhost:8788/

# Health check (no auth required)
curl http://localhost:8788/healthz

A healthy service returns:

{"versions": [{"id": "v1.0", "status": "CURRENT", "version": "1.1", "min_version": "1.0", "links": [{"rel": "self", "href": "http://localhost:8788/v1"}]}]}

The /healthz endpoint returns {"status": "ok", "checks": {"database": "ok"}} when both the process and its database connection are healthy.

Configuration lives in etc/protector/protector.conf for the primary cluster and etc/protector-cluster2/protector.conf for the secondary. The charm renders this file from relation data — in production you must not edit it directly — but in local development you edit it freely.

[DEFAULT]

[api]

[database]

[engine]

[storage]

[pure_storage]

Required when storage_driver = pure:

Retention options in [pure_storage]:

[service_credentials]

[mock_storage]

Only relevant when storage_driver = mock:

Minimal development configuration

[DEFAULT]
debug = True
use_stderr = true
transport_url = rabbit://openstack:<password>@localhost:5672//

[api]
bind_host = 0.0.0.0
bind_port = 8788
workers = 2

[database]
connection = mysql+pymysql://protector:password@localhost/protector

[engine]
topic = protector-engine

[keystone_authtoken]
www_authenticate_uri = http://<keystone-host>:5000
auth_url = http://<keystone-host>:5000
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = protector
password = <service-password>

[service_credentials]
default_trust_roles = member,dr_operator

[storage]
storage_driver = mock

Running tests against a live environment

The test suite runs from your local machine — not on the cluster nodes — using the openstack CLI from your virtual environment. Source credentials before running:

source drtest-cluster1-openrc.sh   # druser credentials, not admin

Always test as a non-admin user. The test suite uses druser / dr-project with member and dr_operator roles. Running as admin masks permission bugs that tenants will hit in production.

Run the full regression suite:

python3 scripts/dr_test_suite.py --all

Run specific suites:

python3 scripts/dr_test_suite.py --suite basic,multi_volume

Run a stress test with repeated failover/failback cycles:

python3 scripts/dr_test_suite.py --suite stress --loops 10

Clean up stale test resources without running tests:

python3 scripts/dr_test_suite.py --cleanup-only

Keep resources after a test run for manual inspection:

python3 scripts/dr_test_suite.py --suite basic --no-cleanup

Deploying code changes to lab containers

The lab runs services as Docker containers (protector_api, protector_engine). Code is installed inside the image at /usr/local/lib/python3.12/dist-packages/. For rapid iteration without a full image rebuild, copy individual files:

# 1. Copy the modified file to the host
scp protector/engine/dr_operations.py pureuser@10.21.67.100:/tmp/dr_operations.py

# 2. Inject it into the running container
ssh pureuser@10.21.67.100 'echo pureuser | sudo -S docker cp \
  /tmp/dr_operations.py \
  protector_engine:/usr/local/lib/python3.12/dist-packages/protector/engine/dr_operations.py'

# 3. Clear the bytecode cache and restart
ssh pureuser@10.21.67.100 'echo pureuser | sudo -S docker exec protector_engine \
  find /usr/local/lib/python3.12/dist-packages/protector/engine \
  -name "__pycache__" -exec rm -rf {} + 2>/dev/null; \
  echo pureuser | sudo -S docker restart protector_engine'

To deploy Horizon plugin changes:

scp horizon-plugin/protector_dashboards/protection_groups/tables.py \
  pureuser@10.21.67.100:/tmp/tables.py

ssh pureuser@10.21.67.100 'echo pureuser | sudo -S docker cp /tmp/tables.py \
  horizon:/var/lib/kolla/venv/lib/python3.12/site-packages/protector_dashboards/protection_groups/tables.py;\
  echo pureuser | sudo -S docker restart horizon'

Interacting with the API during development

All authenticated API calls require an X-Auth-Token from Keystone. The protector-api has no authentication endpoint of its own.

Obtain a token:

TOKEN=$(openstack token issue -f value -c id)
PROJECT_ID=$(openstack token issue -f value -c project_id)

List sites:

curl -s -H "X-Auth-Token: $TOKEN" \
  http://localhost:8788/v1/${PROJECT_ID}/sites | python3 -m json.tool

Use an API microversion header (optional — when omitted, the service defaults to microversion 1.0):

curl -s \
  -H "X-Auth-Token: $TOKEN" \
  -H "OpenStack-API-Version: protector 1.1" \
  http://localhost:8788/v1/${PROJECT_ID}/protection-groups

Running database migrations during development

When you add new columns to existing models, protector-manage db sync creates new tables but does not add columns to existing ones. For new columns, run an ALTER TABLE directly:

python3 -c "
from sqlalchemy import create_engine, text
import configparser
config = configparser.ConfigParser()
config.read('etc/protector/protector.conf')
engine = create_engine(config['database']['connection'])
with engine.connect() as conn:
    conn.execute(text('ALTER TABLE protection_groups ADD COLUMN my_new_col VARCHAR(255)'))
    conn.commit()
"

CLI development workflow

CLI commands live in client/triliositerecovery/osc/v1/. After editing, reinstall the client package:

pip install -e ./client

The entry points are registered in client/setup.cfg. Changes to the argument parser or command class take effect immediately after reinstall without restarting any service.

Example 1 — Verify both services are reachable

# Version discovery (no authentication required)
curl -s http://localhost:8788/

Expected output:

{
  "versions": [
    {
      "id": "v1.0",
      "status": "CURRENT",
      "version": "1.1",
      "min_version": "1.0",
      "links": [
        {
          "rel": "self",
          "href": "http://localhost:8788/v1"
        }
      ]
    }
  ]
}

# Liveness check (no authentication required)
curl -s http://localhost:8788/healthz

Expected output when healthy:

{"status": "ok", "checks": {"database": "ok"}}

Expected output when the database is unreachable:

{"status": "unhealthy", "checks": {"database": "OperationalError"}}

Example 2 — Register sites and create a protection group (CLI)

source drtest-cluster1-openrc.sh

# Register the primary site
openstack dr site create cluster1 \
  --site-type PRIMARY \
  --auth-url http://10.21.67.100:5000 \
  --service-username protector \
  --service-password <password> \
  --service-project-name service \
  --service-user-id $(openstack user show protector -f value -c id)

# Register the secondary site
openstack dr site create cluster2 \
  --site-type SECONDARY \
  --auth-url http://10.21.67.101:5000 \
  --service-username protector \
  --service-password <password> \
  --service-project-name service \
  --service-user-id $(openstack user show protector -f value -c id)

# Capture site IDs
PRIMARY_SITE=$(openstack dr site list -f json | \
  python3 -c "import sys,json; \
  [print(s['ID']) for s in json.load(sys.stdin) if s['Name']=='cluster1']")
SECONDARY_SITE=$(openstack dr site list -f json | \
  python3 -c "import sys,json; \
  [print(s['ID']) for s in json.load(sys.stdin) if s['Name']=='cluster2']")

# Create a protection group
openstack dr protection group create my-pg \
  --primary-site $PRIMARY_SITE \
  --secondary-site $SECONDARY_SITE \
  --replication-type async \
  --volume-type replicated-ssd

Example 3 — Validate a member before adding it

Use member validate to check eligibility without side effects:

openstack dr protection group member validate my-pg \
  --instance <instance-uuid>

For the same check via the API:

curl -s -X POST \
  -H "X-Auth-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"member": {"instance_id": "<instance-uuid>"}}' \
  http://localhost:8788/v1/${PROJECT_ID}/protection-groups/<pg-id>/members/validate \
  | python3 -m json.tool

Example response for a valid instance:

{
  "validation": {
    "valid": true,
    "instance_id": "<instance-uuid>",
    "instance_name": "my-vm",
    "protection_group_id": "<pg-uuid>",
    "protection_group_name": "my-pg",
    "errors": [],
    "warnings": [],
    "details": {
      "volume_count": 2,
      "boot_volume_id": "<vol-uuid>",
      "backend_name": "pure"
    }
  }
}

Example response for an instance that boots from ephemeral storage:

{
  "validation": {
    "valid": false,
    "instance_id": "<instance-uuid>",
    "errors": [
      {
        "code": "InstanceNotCinderBoot",
        "message": "Instance <instance-uuid> is not a Cinder boot instance. All protected VMs must boot from a Cinder volume."
      }
    ]
  }
}

Example 4 — Trigger a test failover and poll for completion

# Trigger test failover (must be run from the secondary site context)
RESPONSE=$(curl -s -X POST \
  -H "X-Auth-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"test_failover": {"auto_network": true}}' \
  http://localhost:8788/v1/${PROJECT_ID}/protection-groups/<pg-id>/action)

echo $RESPONSE | python3 -m json.tool

OP_ID=$(echo $RESPONSE | python3 -c "import sys,json; print(json.load(sys.stdin)['operation_id'])")

# Poll the operation
while true; do
  STATUS=$(curl -s -H "X-Auth-Token: $TOKEN" \
    http://localhost:8788/v1/${PROJECT_ID}/operations/${OP_ID} | \
    python3 -c "import sys,json; op=json.load(sys.stdin)['operation']; \
    print(op['status'], op['progress'])")
  echo $STATUS
  echo $STATUS | grep -qE '^(completed|failed)' && break
  sleep 10
done

The initial 202 response body includes operation_id and status: "pending":

{
  "message": "Test failover initiated",
  "status": "pending",
  "protection_group_id": "<pg-uuid>",
  "operation_id": "<op-uuid>",
  "operation": { "id": "<op-uuid>", "status": "pending", "progress": 0 }
}

Clean up test failover resources when done:

openstack dr test failover cleanup <pg-id> --operation <op-id>

Example 5 — Check replication health programmatically

curl -s -H "X-Auth-Token: $TOKEN" \
  http://localhost:8788/v1/${PROJECT_ID}/protection-groups/<pg-id>/replication-health \
  | python3 -m json.tool

The response nests fields under replication_health:

{
  "replication_health": {
    "protection_group_id": "<pg-uuid>",
    "protection_group_name": "my-pg",
    "health_report": {
      "status": "healthy",
      "lag_seconds": 45,
      "in_sync": true,
      "recovery_points_available": 12
    },
    "failover_validation": {
      "can_failover": true,
      "warnings": [],
      "blockers": [],
      "message": "Protection group is ready for failover"
    },
    "storage_config": {}
  }
}

To check failover readiness in a script, use can_failover:

CAN_FAILOVER=$(curl -s -H "X-Auth-Token: $TOKEN" \
  http://localhost:8788/v1/${PROJECT_ID}/protection-groups/<pg-id>/replication-health | \
  python3 -c "import sys,json; h=json.load(sys.stdin)['replication_health']; \
  print(h['failover_validation']['can_failover'])")

if [ "$CAN_FAILOVER" = "True" ]; then
  echo "Ready to failover"
fi

Example 6 — Run the basic test suite with mock storage

For development without FlashArray hardware, configure the mock driver and run:

# etc/protector/protector.conf
[storage]
storage_driver = mock

bash bin/dev-launch.sh both
source drtest-cluster1-openrc.sh
python3 scripts/dr_test_suite.py --suite basic

Test output follows this pattern:

10:01:23 [PASS] basic: single_vm_failover (42.1s)
10:02:05 [PASS] basic: failback_after_failover (61.3s)
...
10:08:44 Results: 8/8 passed, 0 failed

Engine cannot start — RPC transport error

Symptom: protector-engine exits immediately with FAILED TO INITIALIZE RPC TRANSPORT.

Cause: [DEFAULT] transport_url is missing or points to an unreachable RabbitMQ host.

Fix: Set a valid transport URL in protector.conf:

[DEFAULT]
transport_url = rabbit://openstack:<password>@<host>:5672//

Verify RabbitMQ is reachable: nc -zv <host> 5672.

openstack dr commands not found after install

Symptom: openstack: 'dr' is not an openstack command. Running pip install -e . from the repo root did not install the CLI.

Cause: The CLI plugin (triliositerecovery) is a separate package in client/. Installing the server package does not install it.

Fix:

pip install -e ./client

Also check that you installed python-openstackclient via pip, not the snap. The snap is incompatible with the triliositerecovery plugin.

Member add fails with HTTP 422 InstanceNotCinderBoot

Symptom: openstack dr protection group member add <pg> --instance <id> returns HTTP 422 with error type InstanceNotCinderBoot.

Cause: The instance boots from ephemeral storage, not a Cinder volume. All VMs in a protection group must boot from a Cinder-backed boot volume.

Fix: Rebuild the instance from a Cinder volume (use --boot-from-volume at boot time, or create a volume from the instance's image and re-create the instance). Use openstack dr protection group member validate <pg> --instance <id> to pre-check before attempting the add.

Member add fails with HTTP 422 VolumeTypeNotReplicated

Symptom: Member add returns VolumeTypeNotReplicated for one or more volumes.

Cause: A volume attached to the instance uses a Cinder volume type that does not have replication_enabled='<is> True' as an extra-spec.

Fix: Create or update a volume type with the required extra-spec, then retype the volume:

# Create a replicated volume type
openstack volume type create replicated-ssd
openstack volume type set replicated-ssd \
  --property replication_enabled='<is> True' \
  --property replication_type='<in> async'

# Retype an existing volume
openstack volume retype <volume-id> replicated-ssd

Trust creation fails with Role <name> does not exist on site <site>

Symptom: The first DR operation (failover, member add, or metadata sync) fails with a message naming a missing Keystone role.

Cause: Either dr_operator or dr_site_admin was not created on one or both sites before the operation. The service looks up roles and fails immediately if they are absent; it does not fall back to a more privileged role.

Fix: Create the missing roles on both sites:

openstack role create dr_operator
openstack role create dr_site_admin

# Grant dr_operator to your test user on their project
openstack role add --user druser --project dr-project dr_operator

# Grant dr_operator to the service account on each DR-enabled project
openstack role add --user protector --project dr-project dr_operator

Replication health shows site unreachable

Symptom: openstack dr replication health <pg-id> returns a RemoteSiteUnreachable error or shows can_failover: false with a connectivity blocker.

Cause: Port 8788 is blocked in one or both directions between the sites. Cross-site metadata sync requires bidirectional connectivity on port 8788. One-way connectivity leaves the system inconsistent.

Fix: Verify bidirectional reachability:

# From site 1, reach site 2's protector-api
curl -s http://<site2-host>:8788/healthz

# From site 2, reach site 1's protector-api
curl -s http://<site1-host>:8788/healthz

Both should return {"status": "ok", ...}. Open port 8788 symmetrically in both firewalls.

Database schema out of date after a new column is added

Symptom: The engine logs OperationalError: (1054, "Unknown column '...' in 'field list'") after pulling a new commit.

Cause: protector-manage db sync creates new tables but does not add columns to existing tables. The Alembic migration was not applied.

Fix: Run the migration command:

protector-manage db sync

If the column was added outside a migration (e.g. a direct model change during development), apply it manually with an ALTER TABLE as shown in the Usage section.

Container health check fails — docker ps shows (unhealthy)

Symptom: docker ps --filter name=protector_api shows the container status as (unhealthy).

Cause: The Docker health check polls GET /healthz every 30 seconds. If the database connection is broken the endpoint returns HTTP 503.

Fix: Check the database connectivity from inside the container:

docker exec protector_api curl -s http://localhost:8788/healthz

If it returns {"status": "unhealthy", "checks": {"database": "..."}}:

1. Verify the [database] connection string in protector.conf is correct.
2. Verify the MariaDB/MySQL server is running and accepts connections from the container's network.
3. Restart the container after fixing the config: docker restart protector_api.

openstack dr test failover cleanup fails with operation not found

Symptom: Cleanup fails because you passed a test failover operation ID as the positional argument.

Cause: openstack dr test failover cleanup takes a protection group ID or name as its positional argument, not an operation ID. The operation is selected with the optional --operation <id> flag.

Fix:

# Correct form — positional is the PG, --operation selects the test run
openstack dr test failover cleanup <pg-id> --operation <op-id>

# Without --operation, defaults to the most recent test failover for the PG
openstack dr test failover cleanup <pg-id>

# Force cleanup even if some resources fail to delete
openstack dr test failover cleanup <pg-id> --force