Service Management
Starting/stopping the API and worker processes (systemd), quotas
This page explains how to manage the Trilio Site Recovery service processes — starting, stopping, and restarting the API and engine services — across all three supported deployment models: Juju (Charmed OpenStack), Kolla-Ansible, and OpenShift (RHOSO18). It also covers database schema management, log access, and health verification. Understanding these procedures is essential for routine operations, upgrades, and recovering from failures without disrupting active protection groups.
Before performing any service management operation, ensure:
- Trilio Site Recovery is installed and initially configured (see the Installation guide)
/etc/protector/protector.confexists and contains a validlocal_sitevalue in[DEFAULT]- The MariaDB/MySQL
protectordatabase is reachable and the schema is at the current head (025_add_member_removing) - RabbitMQ is running and the
protectorvhost and user are configured - Keystone is reachable and the
protectorservice account exists - For Juju deployments: Juju 2.9+ and the
trilio-site-recovery-for-openstackcharm is deployed - For Kolla deployments: the
protector:latestDocker image is available anddeploy/kolla/install.shhas been run - For OpenShift deployments:
ocorkubectlaccess to thetrilio-site-recoverynamespace - The
dr_operatoranddr_site_adminKeystone roles exist on both sites — trust creation fails if either role is absent
Service management does not require a separate installation step beyond the initial deployment. However, you must ensure the packages are present on each node that runs the CLI or service processes.
On Juju-managed units, the charm installs and maintains packages automatically. To ensure the latest packages are present on a unit:
juju run-action trilio-site-recovery/leader upgrade-packages --wait
Repeat for each unit that is not the leader (unit numbers from juju status):
juju run-action trilio-site-recovery/7 upgrade-packages --wait
Any machine running the openstack dr CLI that is not managed by the charm must be upgraded separately:
sudo apt-get install --only-upgrade python3-triliositerecovery
On Kolla nodes, the deploy/kolla/install.sh script manages the container lifecycle. To update containers to the latest image:
bash deploy/kolla/install.sh
On OpenShift, update the Deployment image tag by re-running the installer:
bash deploy/openshift/scripts/install.sh --image docker.io/xyzmd/trilio-site-recovery-openstack:dev
The installer applies all twelve manifests in dependency order (namespace → serviceaccount → configmap → secret → init Jobs → service → Deployments) and gates each init Job on oc wait --for=condition=complete. Do not apply individual manifests manually — the init Jobs (job-db-init, job-rabbitmq-init, job-keystone-init, job-db-sync) must complete before the Deployments are created.
The service reads /etc/protector/protector.conf on startup. On Juju deployments the charm owns this file and rewrites it on every hook; do not edit it directly. Use juju config for supported options and config-flags for anything else:
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
On Kolla and bare-metal deployments, edit /etc/protector/protector.conf directly and restart the services.
Critical options
[DEFAULT] section
| Option | Type | Default | Purpose |
|---|---|---|---|
local_site | string | (required) | Name of the DR site this instance belongs to. The service cannot participate in any DR operation until this is set. After a 1.0.20+ charm upgrade, the unit enters BLOCKED state with the message set local-site, pure-primary-site-name, pure-secondary-site-name until this is configured — this is expected behaviour, not a failed upgrade. |
use_mock_storage | boolean | false | Use the in-memory mock storage driver instead of Pure Storage. For evaluation only. |
use_mock_cinder | boolean | false | Simulate Cinder calls. For evaluation only. |
[api] section
| Option | Type | Default | Purpose |
|---|---|---|---|
bind_host | string | 0.0.0.0 | Address the gunicorn API process listens on. |
bind_port | integer | 8788 | Port the API listens on. Both sites must be reachable on this port from each other. |
workers | integer | 4 | Number of gunicorn worker processes. |
[engine] section
| Option | Type | Default | Purpose |
|---|---|---|---|
topic | string | protector-engine | RabbitMQ topic the engine consumes. Must match the API's dispatch target. |
cross_site_timeout | integer | 30 | Seconds to wait for cross-site API calls. Increase on high-latency links via config-flags. |
recovery_point_interval_minutes | integer | 0 | How often to create automatic recovery points. 0 disables automatic creation. |
recovery_point_max_count | integer | 20 | Maximum recovery points retained per protection group. |
recovery_point_ttl_hours | integer | 24 | Hours before a recovery point is eligible for pruning. |
auto_create_security_groups | boolean | false | Automatically create missing security groups on the recovery site during failover. |
[pure_storage] section (required when using the Pure storage driver)
| Option | Type | Default | Purpose |
|---|---|---|---|
primary_management_ip | string | (required) | Management IP of the primary FlashArray. |
primary_api_token | string | (required) | API token for the primary FlashArray. |
secondary_management_ip | string | (required) | Management IP of the secondary FlashArray. |
secondary_api_token | string | (required) | API token for the secondary FlashArray. |
primary_site_name | string | (required) | Name of the OpenStack site that uses primary_management_ip. Must exactly match the site name registered with openstack dr site create. |
secondary_site_name | string | (required) | Name of the OpenStack site that uses secondary_management_ip. |
default_rpo_seconds | integer | 900 | Recovery Point Objective in seconds for ActiveDR (continuous async) replication. |
snapshot_interval_seconds | integer | 300 | How often the engine triggers a snapshot for async replication. |
verify_ssl | boolean | false | Verify TLS certificates when connecting to FlashArray REST API. |
additional_arrays | list | (empty) | Names of additional [pure_array:<name>] sections for multi-array deployments (e.g. array-pair-2). Each named section carries its own management IPs, API tokens, primary_site_name, secondary_site_name, and cinder_backend_names. |
Important — Pure site name pairing:
primary_site_namemust name the site whose array is atprimary_management_ip. The driver resolves which array to use by comparing the protection group's primary site name against these values. An identical[pure_storage]section works on both nodes because the names travel with the IPs. Swapping the names breaks failover on the recovery site.
[service_credentials] section
| Option | Type | Default | Purpose |
|---|---|---|---|
default_trust_roles | string | member,dr_operator | Roles delegated by every per-tenant Keystone trust. Both roles must exist in Keystone before the service handles its first DR operation. Do not remove member — it provides Nova/Cinder access the trust needs. The default is correct; omit this option from your configuration rather than repeating the default. |
[storage] section
| Option | Type | Default | Purpose |
|---|---|---|---|
storage_driver | string | mock | Storage backend: mock (evaluation), pure (production Pure Storage), drbd (in development, not for production), or a full class path such as protector.engine.storage.drivers.pure_driver.PureStorageDriver. |
[database] section
| Option | Type | Default | Purpose |
|---|---|---|---|
connection | string | (required) | SQLAlchemy connection URL for the MariaDB/MySQL protector database. |
Starting and stopping services
The two services that make up Trilio Site Recovery are protector-api (the gunicorn WSGI API on port 8788) and protector-engine (the oslo.messaging RPC worker). How you manage them depends on your deployment model.
Juju
The charm manages both services as systemd units named protector-api and protector-engine. Use charm actions rather than restarting systemd units directly, so the charm's state machine stays consistent.
Restart both services on a unit:
juju run-action trilio-site-recovery/leader restart-services --wait
Pause both services on a unit (stops them and prevents hooks from restarting them):
juju run-action trilio-site-recovery/leader pause --wait
Resume paused services:
juju run-action trilio-site-recovery/leader resume --wait
Check service and API health:
juju run-action trilio-site-recovery/leader health-check --wait
View the rendered configuration (secrets redacted):
juju run-action trilio-site-recovery/leader show-config --wait
Kolla-Ansible
The protector_api and protector_engine Docker containers are managed by install.sh.
Stop both containers:
docker stop protector_api protector_engine
Start both containers:
bash deploy/kolla/install.sh
Restart a single container:
docker restart protector_api
docker restart protector_engine
OpenShift (RHOSO18)
Restart the API deployment by rolling it:
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-api
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-engine
Watch rollout status:
oc -n trilio-site-recovery rollout status deployment/trilio-site-recovery-api
oc -n trilio-site-recovery rollout status deployment/trilio-site-recovery-engine
Scale down (stop) a deployment:
oc -n trilio-site-recovery scale deployment/trilio-site-recovery-api --replicas=0
oc -n trilio-site-recovery scale deployment/trilio-site-recovery-engine --replicas=0
Scale back up:
oc -n trilio-site-recovery scale deployment/trilio-site-recovery-api --replicas=1
oc -n trilio-site-recovery scale deployment/trilio-site-recovery-engine --replicas=1
Viewing logs
Juju and Kolla
Logs are written to /var/log/protector/.
Stream API logs:
tail -f /var/log/protector/protector-api.log
Stream engine logs:
tail -f /var/log/protector/protector-engine.log
On Kolla, you can also read container logs directly:
docker logs -f protector_api
docker logs -f protector_engine
OpenShift
On OpenShift the service logs only to stdout (the ConfigMap sets use_stderr = true with no log_dir). kubectl logs / oc logs is the only path to service output — there is no /var/log/protector/ on OpenShift pods.
Both the app.kubernetes.io/name and app.kubernetes.io/component labels are required to target pods correctly:
oc -n trilio-site-recovery logs -l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api -f
oc -n trilio-site-recovery logs -l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine -f
Note: There is no bare
applabel on these pods. Using-l app=trilio-site-recovery-apior-l app=trilio-site-recovery-engineselects zero pods and prints "No resources found".
Database schema management
The current schema head is 025_add_member_removing. Migrations must run before the new service code starts — the models declare the columns, so reads against the affected tables fail until the schema is current.
The supported migration command is protector-manage db sync. It reads /etc/protector/protector.conf by default; pass --config-file for an alternate path.
Juju
The charm's leader unit runs db-sync automatically on install and after a package upgrade. To run it manually:
juju run-action trilio-site-recovery/leader db-sync --wait
Kolla and bare-metal
protector-manage db sync
With an alternate config file:
protector-manage db sync --config-file /path/to/protector.conf
Crossing migration 022 (ENUM preflight)
Migration 022_add_reconcile_operation_type modifies the native ENUM on dr_operations.operation_type. Before running db sync across this boundary, verify the existing ENUM tokens are safe:
protector-manage db preflight
A zero exit means the column is safe to migrate. A non-zero exit prints the unexpected tokens and you must not run db sync until the column is corrected.
Juju users: The charm's
db-syncandupgrade-packagesactions rundb preflightautomatically beforedb sync. Kolla and bare-metal operators must run it manually.
Check the current schema revision:
protector-manage db current
Stamp the schema at a specific revision without running migrations (useful after a manual schema creation):
protector-manage db stamp 025_add_member_removing
Service registration
After install or whenever the Keystone endpoints need to be (re-)created:
juju run-action trilio-site-recovery/leader register-service --wait
On Kolla and OpenShift deployments, service registration is handled by the job-keystone-init init Job run by install.sh.
Verifying service health
The API exposes a health endpoint that returns HTTP 200 when it can serve requests and HTTP 503 when it cannot. Use this endpoint for load balancer probes and peer-site reachability checks:
curl -s -o /dev/null -w "%{http_code}" http://<api-host>:8788/healthz
This endpoint is unauthenticated. The Kolla container HEALTHCHECK polls it every 30 seconds.
To check service health from the Juju charm:
juju run-action trilio-site-recovery/leader health-check --wait
Purging the database
Use this only for decommissioning or rebuilding a lab. This operation is irreversible — all site registrations, protection groups, members, mappings, and operation records are deleted. Stop before removing the application:
juju run-action trilio-site-recovery/leader purge-database confirm=trilio-site-recovery --wait
juju remove-application trilio-site-recovery
To also drop the database schema itself (use only when permanently removing the service):
juju run-action trilio-site-recovery/leader purge-database confirm=trilio-site-recovery drop-schema=true --wait
Check schema revision
protector-manage db current
Run preflight check then migrate
protector-manage db preflight && protector-manage db sync
Upgrade packages on all Juju units
Check status to get unit numbers, then run on each:
juju status trilio-site-recovery
juju run-action trilio-site-recovery/0 upgrade-packages --wait
juju run-action trilio-site-recovery/1 upgrade-packages --wait
Roll back to a known-good release:
juju run-action trilio-site-recovery/0 upgrade-packages version=1.0.0 --wait
Upgrade packages on Kolla nodes
sudo apt-get update
sudo apt-get install --only-upgrade trilio-site-recovery python3-triliositerecovery
protector-manage db sync
docker restart protector_api protector_engine
Stage a package upgrade without restarting (Juju)
Upgrade the schema first, then restart separately:
juju run-action trilio-site-recovery/leader upgrade-packages restart=false --wait
# ... verify schema is at head ...
juju run-action trilio-site-recovery/leader restart-services --wait
Restart services on OpenShift and watch rollout
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-api
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-engine
oc -n trilio-site-recovery rollout status deployment/trilio-site-recovery-api
oc -n trilio-site-recovery rollout status deployment/trilio-site-recovery-engine
Stream engine logs on OpenShift
oc -n trilio-site-recovery logs \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine \
-f
Set a config option not exposed by the Juju charm
The charm exposes 47 named options. Any config key the charm does not model must go through config-flags:
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Caution: Do not use
config-flagsfor a key the charm template already renders — doing so causes oslo.config to see the section twice, which breaks tools that use Python'sconfigparser.
Verify API is reachable after restart
curl -s -o /dev/null -w "%{http_code}" http://<api-host>:8788/healthz
Expect 200 when healthy, 503 when the API cannot reach its database or dependencies.
Unit enters BLOCKED state after charm upgrade with message about local-site
Symptom: After upgrading to charm revision 16 (1.0.24 service), juju status shows the unit BLOCKED with: set local-site, pure-primary-site-name, pure-secondary-site-name.
Cause: This is expected behaviour. The 1.0.20+ charm requires local_site to be explicitly configured; it cannot infer which site a node belongs to (the inference was wrong on the recovery site, where failover runs). When a Pure storage driver is configured, the site names must also be set.
Fix:
juju config trilio-site-recovery local-site=site-a
# If using Pure storage:
juju config trilio-site-recovery pure-primary-site-name=site-a pure-secondary-site-name=site-b
db preflight fails with unexpected ENUM tokens
Symptom: protector-manage db preflight exits non-zero and prints tokens the model does not define (e.g. lowercase values like failover instead of FAILOVER).
Cause: The dr_operations.operation_type ENUM column contains tokens that do not match the model's member names. Migration 022 would rewrite these tokens and corrupt existing operation history.
Fix: Do not run db sync until the ENUM tokens are corrected to uppercase member names (FAILOVER, FAILBACK, TEST_FAILOVER, TEST_CLEANUP, SYNC_VOLUMES, VALIDATE_REPLICATION, RECONCILE). Inspect the column:
SHOW COLUMNS FROM dr_operations LIKE 'operation_type';
Contact Trilio support if the column contains unrecognised values before modifying it.
db sync fails with "could not find alembic.ini"
Symptom: protector-manage db sync exits with a RuntimeError about not finding alembic.ini.
Cause: The Debian package installs alembic.ini at /usr/share/trilio-site-recovery/alembic.ini; the Kolla image puts it at /opt/openstack-protector/alembic.ini. Running protector-manage db sync from a source checkout without an installed package may not find either path.
Fix: Use protector-manage db sync from a node where the trilio-site-recovery Debian package is installed. On Juju, use the db-sync charm action. On Kolla and OpenShift, the init Job handles this automatically through install.sh.
Do not invoke Alembic directly with a hand-constructed
--configpath;protector-manage db syncis the only supported migration path.
OpenShift logs command returns "No resources found"
Symptom: oc logs -l app=trilio-site-recovery-api -n trilio-site-recovery prints "No resources found in trilio-site-recovery namespace".
Cause: There is no bare app label on the OpenShift pods. The pods use app.kubernetes.io/name and app.kubernetes.io/component.
Fix: Use the correct label selector:
oc -n trilio-site-recovery logs \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api -f
Failover fails with HTTP 403 volume_extension:volume_manage
Symptom: Failover starts successfully, completes storage work, then fails with: Policy doesn't allow volume_extension:volume_manage to be performed (HTTP 403).
Cause: Cinder's default policy restricts volume manage/unmanage to admins. The service runs as a tenant through a Keystone trust during failover. The required Cinder policy relaxations were not applied on the recovery site.
Fix: Apply all five Cinder policy relaxations on both sites:
"volume_extension:volume_manage": "rule:admin_or_owner"
"volume_extension:volume_unmanage": "rule:admin_or_owner"
"volume_extension:services:index": "rule:admin_or_owner"
"volume_extension:type_get": "rule:admin_or_owner"
"volume_extension:type_get_all": "rule:admin_or_owner"
On Juju:
juju attach-resource cinder policyd-override=cinder-policy-override.zip
Trust creation fails with "role not found"
Symptom: The first DR operation after installation (or after adding a tenant mapping) fails with an error naming a missing Keystone role.
Cause: The default_trust_roles option (default: member,dr_operator) names roles that must exist in Keystone before the service creates any trust. The charm creates the dr_operator and dr_site_admin roles during deployment, but if Keystone was not fully configured at that point, or the roles were deleted, trust creation fails.
Fix: Create the missing roles on both sites:
openstack role create dr_operator
openstack role create dr_site_admin
API health check returns 503
Symptom: curl http://<api-host>:8788/healthz returns HTTP 503.
Cause: The API process is running but cannot reach its database (MariaDB/MySQL). This is a local connectivity issue, unrelated to cross-site reachability.
Fix:
- Verify the
[database] connectionURL in/etc/protector/protector.confis correct. - Check that MariaDB/MySQL is running and the
protectoruser can connect:mysql -u protector -p -h <db-host> protector -e 'SELECT 1;' - Check service logs for database connection errors:
tail -50 /var/log/protector/protector-api.log - On Juju, run the health check action which also reports systemd state:
juju run-action trilio-site-recovery/leader health-check --wait
Pure storage driver fails with StorageSiteMappingUnknown
Symptom: Failover or replication health check fails with StorageSiteMappingUnknown.
Cause: The primary_site_name or secondary_site_name in [pure_storage] does not match the name of any registered site. The driver resolves which FlashArray to use by comparing the protection group's primary site name against these configured values; a mismatch causes the driver to refuse rather than guess.
Fix: Ensure primary_site_name names the site whose array is at primary_management_ip, and secondary_site_name names the site at secondary_management_ip. These values must exactly match the site names registered via openstack dr site create:
openstack dr site list
Then correct the charm config or protector.conf:
juju config trilio-site-recovery \
pure-primary-site-name=<exact-site-name> \
pure-secondary-site-name=<exact-site-name>