Operations
Running the service: process management, scaling, and HA
This page covers day-to-day operation of Trilio Site Recovery for OpenStack once the service is deployed: how to manage the protector-api and protector-engine processes under each deployment model (Juju, Kolla-Ansible, and OpenShift), how to scale the API tier, how to read service logs, and how to perform routine maintenance tasks such as database migration and package upgrades. Understanding process management is essential because DR operations — failover, failback, test failover — are dispatched by protector-api and executed by protector-engine over RabbitMQ RPC; if either service is degraded, operations queue silently or fail at dispatch. This page assumes the service is already deployed and registered with Keystone.
Before using this page, ensure:
- Trilio Site Recovery for OpenStack is deployed (Juju charm revision 14, Kolla-Ansible, or OpenShift manifests).
- Both
protector-apiandprotector-engineservices are installed from the Trilio apt repository athttps://apt.fury.io/trilio-os-site-recovery-master/(Juju and Kolla) or from thedocker.io/xyzmd/trilio-site-recovery-openstack:devimage (OpenShift). /etc/protector/protector.confis present andlocal_siteis set in[DEFAULT]; the service cannot participate in any DR operation without it.- The
dr_operatoranddr_site_adminKeystone roles exist on both sites. - You have shell access to the controller node (Juju/Kolla) or
oc/kubectlaccess to thetrilio-site-recoverynamespace (OpenShift). - For Juju deployments: Juju client configured with the correct controller and model.
Trilio Site Recovery is deployed as a service, not installed manually in isolation. Choose the path that matches your deployment model.
Juju (Charmed OpenStack)
The Juju charm installs and manages both protector-api and protector-engine automatically. To add the service to an existing Charmed OpenStack model:
1. Configure the Trilio apt source.
The charm reads this from its trilio-pkg-source config option. The default source line is:
deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /
2. Deploy the charm.
juju deploy trilio-site-recovery-for-openstack \
--config trilio-pkg-source="deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /" \
--config local-site="site-a"
Replace site-a with the name you will register for this OpenStack cluster.
3. Add relations.
juju relate trilio-site-recovery-for-openstack:shared-db mysql-innodb-cluster:shared-db
juju relate trilio-site-recovery-for-openstack:amqp rabbitmq-server:amqp
juju relate trilio-site-recovery-for-openstack:identity-credentials keystone:identity-credentials
Note the exact relation endpoint names: shared-db, amqp, and identity-credentials.
4. Run the database migration.
Once the charm reaches active/idle, run the db-sync action on the leader unit:
juju run trilio-site-recovery-for-openstack/leader db-sync
The current schema head is 021_add_member_status_reason. Migrations must complete before the new service code starts.
5. Register the service with Keystone.
juju run trilio-site-recovery-for-openstack/leader register-service
6. (Optional) Deploy the Horizon dashboard plugin.
juju deploy trilio-site-recovery-for-openstack-dashboard
juju relate trilio-site-recovery-for-openstack-dashboard:dashboard-plugin openstack-dashboard:dashboard-plugin
juju run trilio-site-recovery-for-openstack-dashboard/leader install-plugin
7. (Optional) Install the CLI on a client node.
The charm installs the CLI on the service unit. On any other machine:
echo 'deb [trusted=yes] https://apt.fury.io/trilio-os-site-recovery-master/ /' \
| sudo tee /etc/apt/sources.list.d/trilio-site-recovery.list
sudo apt-get update
sudo apt-get install -y python3-triliositerecovery
Kolla-Ansible
1. Run the Kolla install script.
From the deployment bundle:
bash deploy/kolla/install.sh
The script deploys the protector_api and protector_engine Docker containers using the protector:latest image and writes /etc/protector/protector.conf.
2. Run the database migration.
After the containers are running, execute the migration directly:
docker exec protector_api protector-manage db sync
3. Verify the API is reachable.
The API listens on port 8788. Confirm it returns version information:
curl -s http://127.0.0.1:8788/
OpenShift (RHOSO18)
Do not apply manifests individually. The installer script applies all twelve manifests in dependency order and gates each init Job on completion before proceeding:
bash deploy/openshift/scripts/install.sh \
--namespace trilio-site-recovery \
--image docker.io/xyzmd/trilio-site-recovery-openstack:dev
The script creates or updates trilio-openstack-secret (adding SrDatabasePassword and SrKeystonePassword keys) in the shared trilio-openstack namespace, applies the trilio-site-recovery-ca-bundle secret, runs the job-db-init, job-rabbitmq-init, job-keystone-init, and job-db-sync Jobs in sequence, and only then deploys the trilio-site-recovery-api and trilio-site-recovery-engine Deployments in the trilio-site-recovery namespace.
Applying only deployment-api.yaml and deployment-engine.yaml by hand leaves the Deployments without a database, RabbitMQ user, Keystone registration, or schema — the pods will start and immediately fail.
The service is configured through /etc/protector/protector.conf, an oslo.config INI file. On Juju deployments the charm owns this file and rewrites it on every hook — do not edit it by hand. Use juju config for supported options, or config-flags for any option the charm does not model directly:
juju config trilio-site-recovery config-flags="engine.cross_site_timeout=60"
On Kolla and OpenShift deployments, edit the file (or ConfigMap) before starting the service.
Mandatory settings
[DEFAULT] section
| Option | Type | Default | Purpose |
|---|---|---|---|
local_site | string | (none) | Required. The name of the DR site this instance belongs to. The service cannot participate in any DR operation until this is set. On Juju, set via juju config local-site=<name>. |
host | string | localhost | Hostname used to identify this service node in RPC routing. |
use_mock_storage | boolean | false | Use the in-memory mock storage driver instead of a real array. Set true for evaluation without hardware. |
use_mock_cinder | boolean | false | Use a mock Cinder backend. Set true only for testing. |
After a Juju upgrade to version 1.0.20, the charm enters BLOCKED with the message set local-site, pure-primary-site-name, pure-secondary-site-name until these are configured. This is expected; set the options to unblock.
[api] section
| Option | Type | Default | Purpose |
|---|---|---|---|
bind_host | string | 0.0.0.0 | Address the API binds to. |
bind_port | integer | 8788 | Port the API listens on. Inbound connections from clients and peer sites use this port. |
workers | integer | 4 | Number of gunicorn worker processes. Scale up on busy API nodes; each worker is a separate OS process. |
[database] section
| Option | Type | Default | Purpose |
|---|---|---|---|
connection | string | (none) | SQLAlchemy connection string for the protector MariaDB/MySQL database. Example: mysql+pymysql://protector:PASSWORD@db-host/protector |
[engine] section
| Option | Type | Default | Purpose |
|---|---|---|---|
topic | string | protector-engine | RabbitMQ RPC topic the engine listens on. Must match what the API uses to dispatch. |
recovery_point_interval_minutes | integer | 0 | Automatic recovery point interval. 0 disables automatic snapshots; set a positive value to enable scheduled recovery points. |
recovery_point_max_count | integer | 20 | Maximum recovery points retained per protection group. Older points are pruned when this limit is reached. |
recovery_point_ttl_hours | integer | 24 | Age in hours after which recovery points expire and are eligible for pruning. |
auto_create_security_groups | boolean | false | When true, the engine creates missing security groups on the recovery site during failover rather than failing. |
cross_site_timeout | integer | 30 | Timeout in seconds for cross-site API calls. Increase on high-latency inter-site links using config-flags on Juju. |
[service_credentials] section
| Option | Type | Default | Purpose |
|---|---|---|---|
username | string | (none) | Keystone username for the protector service account. |
project_name | string | (none) | Keystone project for the service account. |
default_trust_roles | string | member,dr_operator | Roles delegated by every per-tenant Keystone trust. Both member and dr_operator must already exist in Keystone; trust creation fails if either is absent. The order matters: member grants Nova/Cinder access; dr_operator satisfies Protector's own policy on cross-site sync endpoints. Do not override this value unless advised by support — dropping member causes failover to fail at the first Nova or Cinder call, after the storage work is complete. |
default_service_project | string | service | Default Keystone project for service credentials. |
default_service_user_domain | string | Default | Default Keystone domain for the service user. |
default_service_project_domain | string | Default | Default Keystone domain for the service project. |
cafile | string | (none) | CA certificate bundle for TLS verification of cross-site calls. |
insecure | boolean | false | Disable TLS verification for cross-site calls. Use only in lab environments. |
[pure_storage] section (production deployments)
| Option | Type | Default | Purpose |
|---|---|---|---|
primary_management_ip | string | (none) | Management IP of the FlashArray at the primary site. |
primary_api_token | string | (none) | API token for the primary FlashArray. |
secondary_management_ip | string | (none) | Management IP of the FlashArray at the secondary site. |
secondary_api_token | string | (none) | API token for the secondary FlashArray. |
primary_site_name | string | (none) | The DR site name that uses the primary FlashArray. Must match the site registered with openstack dr site create. The driver resolves which array to talk to by comparing this name to the protection group's primary site — the pairing is what matters, not which node the conf is on. |
secondary_site_name | string | (none) | The DR site name that uses the secondary FlashArray. |
default_rpo_seconds | integer | 900 | Maximum acceptable data age at failover (Recovery Point Objective). |
snapshot_interval_seconds | integer | 300 | Interval between automatic replication snapshots on the FlashArray. |
verify_ssl | boolean | false | Verify TLS certificates when connecting to FlashArray management APIs. |
The [pure_storage] section can be identical on both service nodes. The driver swaps primary/secondary based on site name, not node position. The invariant is the pairing: primary_site_name must name the site whose array is at primary_management_ip.
[storage] section
| Option | Type | Default | Purpose |
|---|---|---|---|
storage_driver | string | mock | Storage backend driver. Use mock for evaluation without hardware; use pure for Pure Storage FlashArray in production. |
Inspecting configuration safely
The [service_credentials], [keystone_authtoken], and [pure_storage] sections contain secrets. Use targeted grep rather than reading the file in full:
grep '^local_site' /etc/protector/protector.conf
grep '^storage_driver' /etc/protector/protector.conf
grep '^default_trust_roles' /etc/protector/protector.conf
Starting and stopping services
Juju
The charm manages protector-api and protector-engine as systemd services. Use charm actions rather than systemd directly on charm-managed units.
# Restart both services on all units
juju run trilio-site-recovery-for-openstack/leader restart-services
# Pause services (stops them without removing)
juju run trilio-site-recovery-for-openstack/leader pause
# Resume services
juju run trilio-site-recovery-for-openstack/leader resume
# Check service health
juju run trilio-site-recovery-for-openstack/leader health-check
Kolla-Ansible
The deploy/kolla/install.sh script manages container lifecycle. To restart the containers:
docker restart protector_api
docker restart protector_engine
To stop and start:
docker stop protector_api protector_engine
docker start protector_engine protector_api
Start protector_engine before protector_api so the RabbitMQ consumer is ready when the API begins dispatching.
OpenShift
Restart the Deployments by rolling them:
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-api
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-engine
Monitor rollout progress:
oc -n trilio-site-recovery rollout status deployment/trilio-site-recovery-api
oc -n trilio-site-recovery rollout status deployment/trilio-site-recovery-engine
Reading service logs
Juju and Kolla
Logs are written to /var/log/protector/:
tail -f /var/log/protector/protector-api.log
tail -f /var/log/protector/protector-engine.log
Filter for errors:
grep -i error /var/log/protector/protector-engine.log | tail -50
OpenShift
On OpenShift, the service writes to stdout only (the ConfigMap sets use_stderr = true with no log_dir). oc logs is the only path to service output — there is no /var/log/protector/ on OpenShift pods.
# API logs
oc -n trilio-site-recovery logs -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api \
--tail=100 -f
# Engine logs
oc -n trilio-site-recovery logs -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine \
--tail=100 -f
Both label selectors (app.kubernetes.io/name and app.kubernetes.io/component) are required to target a specific Deployment. There is no bare app label on any manifest in the OpenShift deployment.
Scaling the API tier
protector-api is stateless and horizontally scalable. protector-engine is a singleton RPC worker and must not be scaled beyond one active instance per site.
Increasing API workers (Juju)
juju config trilio-site-recovery api-workers=8
Increasing API workers (OpenShift)
oc -n trilio-site-recovery scale deployment/trilio-site-recovery-api --replicas=3
Running database migrations
The only supported migration command is protector-manage db sync. It reads /etc/protector/protector.conf by default.
Juju
juju run trilio-site-recovery-for-openstack/leader db-sync
Kolla
docker exec protector_api protector-manage db sync
OpenShift
The job-db-sync init Job runs automatically during install.sh. To re-run it manually:
oc -n trilio-site-recovery create job db-sync-manual \
--from=job/job-db-sync
oc -n trilio-site-recovery wait --for=condition=complete job/db-sync-manual --timeout=120s
The current schema head is 021_add_member_status_reason. Migrations must complete before the new service version starts.
Upgrading packages
Juju
The upgrade-packages action upgrades the service packages on the charm-managed unit:
juju run trilio-site-recovery-for-openstack/leader upgrade-packages
This upgrades only the unit running the charm. Any machine running the openstack dr CLI that is not managed by the charm requires a separate upgrade:
sudo apt-get install --only-upgrade python3-triliositerecovery
Kolla
Pull the new image and rerun the install script:
docker pull protector:latest
bash deploy/kolla/install.sh
OpenShift
Update the image tag in the Deployments and apply:
oc -n trilio-site-recovery set image \
deployment/trilio-site-recovery-api \
trilio-site-recovery-api=docker.io/xyzmd/trilio-site-recovery-openstack:NEW_TAG
oc -n trilio-site-recovery set image \
deployment/trilio-site-recovery-engine \
trilio-site-recovery-engine=docker.io/xyzmd/trilio-site-recovery-openstack:NEW_TAG
Run db-sync after the image update to apply any new schema migrations before the new code serves requests.
Checking service health
The health-check charm action performs an end-to-end validation of API reachability, database connectivity, and RabbitMQ RPC:
juju run trilio-site-recovery-for-openstack/leader health-check
For a quick reachability check from outside the charm, query the API version endpoint:
curl -s http://<api-host>:8788/
The version endpoint returns a JSON document describing available API versions when the API can serve requests.
Monitoring replication health
The service does not emit oslo.messaging notifications. Observability is poll-based:
# Check replication health for a protection group
openstack dr replication health <protection-group>
# Add --probe to verify that networks, security groups, and flavors resolve
# on the recovery site as the tenant it recovers into (slower — contacts remote site)
openstack dr replication health <protection-group> --probe
# List available recovery points
openstack dr recovery point list <protection-group>
# Audit replication across all protection groups
openstack dr replication audit
# List recent operations
openstack dr operation list
--probe asks the recovery site whether the group's networks, security groups, and flavors resolve there as the project it recovers into. It is not a storage or replication-lag check. Run it before a planned failover window to confirm resource resolution rather than discovering a missing network mid-failover.
Check unit status (Juju)
juju status trilio-site-recovery-for-openstack
View the last 100 engine log lines (Juju/Kolla)
tail -100 /var/log/protector/protector-engine.log
View live engine logs (OpenShift)
oc -n trilio-site-recovery logs -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine \
--tail=100 -f
Run a database migration (Kolla)
docker exec protector_api protector-manage db sync
Upgrade service packages on a Juju unit
juju run trilio-site-recovery-for-openstack/leader upgrade-packages
Upgrade the CLI on a non-charm machine
sudo apt-get install --only-upgrade python3-triliositerecovery
Check replication health with recovery-site probe
openstack dr replication health my-protection-group --probe
Check an operation's progress
openstack dr operation show <operation-id>
The output includes a progress field (0–100%) and the source_site and target_site by name. There is no estimated completion time — the service does not compute one.
Set a config option the charm does not model directly
juju config trilio-site-recovery config-flags="engine.cross_site_timeout=60"
Purge the service database (destructive — use with caution)
juju run trilio-site-recovery-for-openstack/leader purge-database
Verify API version response
curl -s http://127.0.0.1:8788/
Charm is BLOCKED after upgrade with message about local-site
Symptom: After upgrading to charm revision 14, the unit enters BLOCKED status with the message set local-site, pure-primary-site-name, pure-secondary-site-name.
Cause: This is expected behaviour. Version 1.0.20 requires local_site (and, when a Pure array is configured, both array site names) to be explicitly set. The charm does not infer them.
Fix:
juju config trilio-site-recovery local-site="<site-name>"
# If using Pure Storage:
juju config trilio-site-recovery \
pure-primary-site-name="<primary-site-name>" \
pure-secondary-site-name="<secondary-site-name>"
Engine logs show StorageSiteMappingUnknown
Symptom: The engine fails with StorageSiteMappingUnknown during a DR operation.
Cause: The primary_site_name or secondary_site_name in [pure_storage] does not match any registered site name. The driver resolves which array to use by comparing the protection group's primary site name against these values — a mismatch means it cannot determine which array is source and which is target.
Fix: Verify that pure_primary_site_name and pure_secondary_site_name in protector.conf exactly match the names used when registering sites:
openstack dr site list
grep 'primary_site_name\|secondary_site_name' /etc/protector/protector.conf
The invariant is the pairing: primary_site_name must name the site whose array is at primary_management_ip. Correct the names and restart the engine.
openstack dr CLI commands fail with 401 Unauthorized
Symptom: All openstack dr commands return HTTP 401.
Cause: The dr_operator role does not exist in Keystone, or the service user lacks it. Trust creation fails naming the missing role rather than skipping it, so the first DR operation stops the whole service for that tenant.
Fix: Verify the role exists on both sites:
openstack role show dr_operator
openstack role show dr_site_admin
Create missing roles:
openstack role create dr_operator
openstack role create dr_site_admin
Then re-run the charm's register-service action (Juju) or re-register manually.
Operations queue but never start (engine not processing)
Symptom: openstack dr operation list shows operations stuck in pending status indefinitely.
Cause: protector-engine is not running or is not connected to RabbitMQ.
Fix (Juju):
juju status trilio-site-recovery-for-openstack
juju run trilio-site-recovery-for-openstack/leader health-check
juju run trilio-site-recovery-for-openstack/leader restart-services
Fix (Kolla):
docker ps | grep protector_engine
docker logs protector_engine --tail=50
docker restart protector_engine
Fix (OpenShift):
oc -n trilio-site-recovery get pods -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine
oc -n trilio-site-recovery logs -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine \
--tail=50
oc -n trilio-site-recovery rollout restart deployment/trilio-site-recovery-engine
Failover fails with Policy doesn't allow volume_extension:volume_manage (HTTP 403)
Symptom: A failover fails partway through with a Cinder policy error. The service deployed and passed health checks without error.
Cause: Cinder restricts volume import operations (volume_manage, volume_unmanage, services:index, type_get, type_get_all) to admin by default. The engine operates as a tenant through a Keystone trust and needs these relaxed on both sites. The restriction is not detected at deployment time — it surfaces only on the first failover.
Fix: Apply the following Cinder policy overrides on both sites:
"volume_extension:volume_manage": "rule:admin_or_owner"
"volume_extension:volume_unmanage": "rule:admin_or_owner"
"volume_extension:services:index": "rule:admin_or_owner"
"volume_extension:type_get": "rule:admin_or_owner"
"volume_extension:type_get_all": "rule:admin_or_owner"
On Juju Cinder deployments, use policyd-override:
juju attach-resource cinder policyd-override=<path-to-policy-zip>
Database schema migration fails
Symptom: The service fails to start after an upgrade, or db-sync reports an error.
Cause: Either the migration has already been applied (no-op case) or the database is unreachable.
Fix: Run the migration directly and inspect output:
# Kolla
docker exec protector_api protector-manage db sync
# Juju
juju run trilio-site-recovery-for-openstack/leader db-sync
Do not use alembic directly or python3 -m protector.db.migration upgrade head — neither path is supported. The protector-manage db sync command is the only supported migration entry point.
OpenShift pods show No resources found when filtering with -l app=...
Symptom: oc logs -l app=trilio-site-recovery-api returns No resources found.
Cause: The OpenShift manifests use app.kubernetes.io/name and app.kubernetes.io/component labels. There is no bare app label on any manifest.
Fix: Use the correct label selectors:
# API
oc -n trilio-site-recovery logs -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api
# Engine
oc -n trilio-site-recovery logs -l \
app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine
config-flags causes DuplicateSectionError on startup
Symptom: After setting config-flags, the service fails to start and logs a DuplicateSectionError.
Cause: The key named in config-flags is already rendered by the charm template. Setting it via config-flags causes the charm to emit that [section] header twice, which configparser rejects (even though oslo.config tolerates it).
Fix: Check whether the option already has a dedicated charm config key (see the list of 47 supported options in the product facts). If it does, use that key directly via juju config. Only use config-flags for options the charm template does not already render — engine.cross_site_timeout is safe; service_credentials.default_trust_roles is not.