Trilio Site Recovery for OpenStackOpenStack native DR service
Guide

Sample Configuration File

An annotated, working example .conf


Overview

This page provides an annotated, working protector.conf for Trilio Site Recovery, the OpenStack-native disaster-recovery service. The file is an INI-style oslo.config file read by both the protector-api and protector-engine processes from /etc/protector/protector.conf. Understanding every section and option here is essential before you deploy, because several options — particularly local_site in [DEFAULT] and the storage driver pairing in [pure_storage] — have no safe default and cause silent misbehaviour or outright refusal when wrong. The page walks through each section in the order it appears in the file, explains what each option controls and why the default may need changing for your environment, and concludes with a complete, copy-ready sample for both the mock (evaluation) and Pure Storage (production) drivers.


Prerequisites

Before editing or deploying protector.conf, confirm you have:

  • Two independent OpenStack clusters or regions, each at Victoria or later
  • Keystone roles dr_operator and dr_site_admin created on both sites — trust creation fails naming any missing role, so the service will not complete its first DR operation until they exist
  • A MariaDB or MySQL database named protector and a user with full access to it, reachable from the node running protector-api and protector-engine
  • A RabbitMQ vhost named protector (default) and a user with read/write permissions on it
  • For production use: Pure Storage FlashArray arrays with async or ActiveDR replication configured between both sites, and the python3-pure-client package installed on every node running protector-engine
  • For Juju deployments: the charm writes /etc/protector/protector.conf automatically; use juju config trilio-site-recovery config-flags="section:key=value" to reach any option the charm does not model directly — do not hand-edit the file, because the charm overwrites it on every hook
  • For Kolla and bare-metal deployments: write the file manually, then run protector-manage db sync to apply schema migrations before starting the services

Installation

The configuration file is placed at /etc/protector/protector.conf. How it gets there depends on your deployment tool.

Juju (Charmed OpenStack)

The trilio-site-recovery-for-openstack charm generates the file from charm config. You do not write it by hand.

  1. Deploy or upgrade the charm as described in the deployment guide.
  2. Set mandatory options through juju config:
juju config trilio-site-recovery \
  local-site="site-a" \
  pure-primary-management-ip="192.168.10.10" \
  pure-primary-api-token="your-array-api-token" \
  pure-secondary-management-ip="192.168.20.10" \
  pure-secondary-api-token="your-peer-array-api-token" \
  pure-primary-site-name="site-a" \
  pure-secondary-site-name="site-b" \
  storage-driver="pure"
  1. For any protector.conf key the charm does not expose as a named option, use config-flags:
juju config trilio-site-recovery \
  config-flags="engine:cross_site_timeout=60"

Warning: A config-flags entry that names a key the charm template already renders causes that [section] to appear twice, which breaks oslo.config parsing. Only use config-flags for keys not already managed by the charm's 47 named options.

  1. After the charm settles, verify the file was written:
juju ssh trilio-site-recovery/0 -- cat /etc/protector/protector.conf

Kolla-Ansible

The deploy/kolla/install.sh script renders the file from a site worksheet:

bash deploy/kolla/install.sh all --worksheet deploy/site-worksheet.yaml

The script places the rendered file into the container at /etc/protector/protector.conf. To inspect it:

docker exec protector_api cat /etc/protector/protector.conf

OpenShift (RHOSO18)

On OpenShift, the configuration is stored in a ConfigMap and mounted into the containers. Run the installer script — it applies all twelve manifests in the correct order, including the ConfigMap:

bash deploy/openshift/scripts/install.sh

Do not apply the deployment manifests individually; the init Jobs (db-init, rabbitmq-init, keystone-init, db-sync) must complete in sequence before the API and engine Deployments start.

Manual / bare-metal

  1. Create the configuration directory:
sudo mkdir -p /etc/protector
  1. Copy the sample file from the package:
sudo cp /usr/share/trilio-site-recovery/protector.conf.sample \
        /etc/protector/protector.conf
  1. Edit the file, replacing every CHANGEME placeholder.
  2. Run the database migration before starting any service:
sudo protector-manage db sync

Note: If you are upgrading across migration 022_add_reconcile_operation_type, run the preflight check first:

sudo protector-manage db preflight
sudo protector-manage db sync

The preflight step runs automatically only through the Juju charm's db-sync and upgrade-packages actions. Kolla and bare-metal operators must run it manually.


Configuration

The file uses standard INI syntax as consumed by oslo.config. Options are grouped by [section]. Every option that has no default must be set before the service will start. Options marked secret are redacted from logs.


[DEFAULT]

Options in this section have no group prefix in code — they are registered directly on CONF.

OptionTypeDefaultPurpose
hoststringlocalhostHostname of this node. Used in RPC addressing.
local_sitestring(none)Required. The name this instance's DR site was registered under (openstack dr site create <name>). The service does not infer which site it is — an absent or wrong value causes operations to be refused or driven from the wrong side. Set it to exactly the same string shown by openstack dr site list.
rpc_conn_pool_sizeinteger30Size of the RabbitMQ RPC connection pool shared between API and engine. Raise this if you see pool-exhaustion warnings under high load.
use_mock_storagebooleanfalseSwitch the entire storage layer to the in-memory mock driver without touching [storage]. Provided for compatibility; prefer storage_driver = mock in [storage].
use_mock_cinderbooleanfalseSuppress real Cinder API calls. Intended for unit-test harnesses, not operator use.

The global transport URL also lives here (oslo.messaging convention):

transport_url = rabbit://protector:CHANGEME@rabbit-host:5672/protector

[api]

OptionTypeDefaultPurpose
bind_hoststring0.0.0.0Address the API WSGI server listens on.
bind_portinteger8788Port the API serves on. Both sites must be reachable on this port from each other.
workersinteger4Number of WSGI worker processes.

[database]

OptionTypeDefaultPurpose
connectionstringmysql+pymysql://protector:password@localhost/protectorSQLAlchemy connection URL. The default points at a local MariaDB/MySQL with the protector user, password, and database; override it with your real host, credentials, and database name. Format: mysql+pymysql://protector:PASSWORD@HOST:3306/protector

Keep this string out of tickets and chat — it contains the database password.


[engine]

OptionTypeDefaultPurpose
topicstringprotector-engineRabbitMQ topic the engine consumes. Must match what the API publishes to.
auto_create_security_groupsbooleanfalseWhen true, the engine creates missing security groups on the DR site during failover. Off by default because it writes to the DR cloud; a missing group may be intentional. The absence is always reported regardless of this setting.
cross_site_timeoutinteger30Seconds any single cross-site call (Keystone exchanges, remote API requests) may take. Without a timeout, an unreachable peer blocks the gunicorn worker until it is killed, leaving the client with a closed connection and no response — after the local half of the operation has already been committed. Raise this for slow inter-site links.
recovery_point_interval_minutesinteger0Interval between scheduled recovery-point snapshots for ActiveDR groups. 0 disables the scheduler.
recovery_point_max_countinteger20Maximum on-demand recovery-point snapshots retained per protection group.
recovery_point_ttl_hoursinteger24Recovery-point snapshots older than this are automatically deleted.
coverage_refresh_interval_minutesinteger15Background interval for re-verifying volume replication coverage. 0 disables the refresh.
floating_ip_refresh_interval_minutesinteger15Background interval for re-capturing floating IP assignments on protected instances. 0 disables.

[keystone_authtoken]

Consumed by keystonemiddleware to validate inbound Keystone tokens.

OptionTypeDefaultPurpose
www_authenticate_uristring(none)Keystone endpoint advertised to clients. Must be reachable by API callers.
auth_urlstring(none)Keystone endpoint this service uses internally to validate tokens. May be an internal address.

Additional keystonemiddleware options (not registered in protector/common/config.py but read by the middleware directly) must be added here: auth_type, project_domain_name, user_domain_name, memcached_servers, and so on. See the sample file below for a complete stanza.


[oslo_policy]

OptionTypeDefaultPurpose
policy_filestring(oslo default)Path to the YAML policy file. Set to /etc/protector/policy.yaml to load the shipped rules.

[service_credentials]

The service account the engine uses when it is not acting through a tenant trust — for example, when registering endpoints or performing site-level operations.

OptionTypeDefaultPurpose
usernamestring(none)Service account username. Typically protector.
project_namestring(none)Project the service account belongs to. Typically service.
default_service_projectstringserviceDefault project name for service accounts when not specified in site configuration.
default_service_user_domainstringDefaultDefault domain for service users.
default_service_project_domainstringDefaultDefault domain for service projects.
default_trust_roleslistmember,dr_operatorRoles delegated by every per-tenant Keystone trust. member carries Nova/Cinder access; dr_operator satisfies Protector's cross-site sync policy. Every role named here must exist on both sites — trust creation fails naming a missing role rather than silently dropping it. Do not remove member: dropping it removes the Nova/Cinder access the trust needs, so failover fails at the first instance or volume call after the storage work is done. Omit this option from your config to accept the default.
cafilestring(none)PEM-encoded CA bundle for verifying Keystone and OpenStack endpoint TLS certificates.
insecurebooleanfalseDisable TLS certificate verification. Not recommended for production.

[storage]

OptionTypeDefaultPurpose
storage_driverstringmockWhich storage backend to use. Accepted short names: mock (in-memory simulation, no array required), pure (Pure Storage FlashArray), drbd (in development — not for production). A full class path such as protector.engine.storage.drivers.pure_driver.PureStorageDriver is also accepted.

[mock_storage]

Read only when storage_driver = mock.

OptionTypeDefaultPurpose
storage_base_pathstring/var/lib/protector/mock_storageDirectory for the mock driver's SQLite state.
primary_array_namestringprimary-arrayDisplay name for the simulated primary array.
secondary_array_namestringsecondary-arrayDisplay name for the simulated secondary array.

[pure_storage]

Read only when storage_driver = pure. Delete this section entirely for a mock deployment.

OptionTypeDefaultPurpose
primary_management_ipstring(none)Management IP or hostname of the array that serves the primary site.
primary_api_tokenstring(none, secret)API token for the primary array.
secondary_management_ipstring(none)Management IP or hostname of the array that serves the secondary site.
secondary_api_tokenstring(none, secret)API token for the secondary array.
primary_site_namestring(empty)The OpenStack site name whose array is at primary_management_ip. Must match the name registered with openstack dr site create. The driver resolves which array to contact by comparing the protection group's primary site name against this value — if they are inverted, failover reads its answer from the wrong hardware, reporting all volumes missing while they sit on the other array. The same [pure_storage] section can be used unchanged on both nodes because the driver swaps the pair based on names, not node position.
secondary_site_namestring(empty)The OpenStack site name whose array is at secondary_management_ip.
verify_sslbooleanfalseVerify TLS certificates for array REST API calls.
default_rpo_secondsinteger900RPO target in seconds used for health-report warnings.
snapshot_interval_secondsinteger300Desired replication snapshot interval in seconds. The array may enforce a higher minimum; a warning is logged when the value is clamped.
additional_arrayslist(empty)Names of additional array-pair sections, each formatted as [pure_array:<name>]. Used for multi-array deployments.

Multi-array sections: [pure_array:<name>]

Each name listed in additional_arrays must have a corresponding section:

OptionTypePurpose
primary_management_ipstringManagement IP for this pair's primary array.
primary_api_tokenstring (secret)API token for this pair's primary array.
secondary_management_ipstringManagement IP for this pair's secondary array.
secondary_api_tokenstring (secret)API token for this pair's secondary array.
primary_site_namestringSite name whose array is at primary_management_ip for this pair.
secondary_site_namestringSite name whose array is at secondary_management_ip for this pair.
verify_sslbooleanVerify TLS for this pair.
cinder_backend_nameslistCinder backend names that map to this array pair.

Note: cinder_backend_names belongs in [pure_array:<name>] sections, not in [DEFAULT] or [pure_storage].


[oslo_messaging_rabbit]

Fine-tunes the RabbitMQ connection. The most commonly needed options:

[oslo_messaging_rabbit]
rabbit_durable_queues = true
amqp_durable_queues = true

Policy overrides: policy.yaml

Access control is enforced by oslo.policy, using rules declared in /etc/protector/policy.yaml. The shipped file defines four base rules:

  • context_is_admin — role:admin
  • context_is_dr_operator — role:dr_operator or role:dr_site_admin or role:admin
  • context_is_site_admin — role:dr_site_admin or role:admin
  • admin_or_owner — is_admin:True or is_system:True or project_id:%(project_id)s

Key things to note:

  • dr_site_admin is cloud-wide. The rules it gates (site create/update/delete, tenant mapping CRUD, site failover, replication audit) carry no project_id owner clause. Grant it on the admin project to cloud operators, not to per-tenant DR users.
  • protector:sites:storage_arrays requires role:admin, not dr_site_admin.
  • To relax a rule — for example, letting dr_operator list site operations — create or edit /etc/protector/policy.yaml and override only the rules you want to change:
# Allow dr_operator to list site-level operations
"protector:site_operations:index": "rule:context_is_dr_operator"

The service picks up policy file changes without a restart (oslo.policy re-reads on each request by default).

For Juju deployments, attach the override file as a charm resource:

juju attach-resource trilio-site-recovery-for-openstack \
  policyd-override=my-policy-overrides.zip

Usage

Choosing a storage driver

For initial deployment validation or hardware-free evaluation, leave storage_driver = mock. The mock driver stores its state under /var/lib/protector and simulates replication operations without touching any array. Switch to pure only when your FlashArray arrays are configured with async or ActiveDR replication between both sites.

Setting local_site correctly

Every node must have local_site set to the exact name of the DR site it belongs to. The value must match what openstack dr site list shows — the service compares strings, not UUIDs. Set it differently on each site. A Juju charm unit that is deployed or upgraded without local_site set will land in BLOCKED state with the message:

set local-site, pure-primary-site-name, pure-secondary-site-name

This is expected behaviour, not a failed deployment. Set the config and the unit will proceed.

Trust roles

Leave default_trust_roles at its default (member,dr_operator) unless you have a specific reason to change it. The two roles serve different purposes — member grants Nova and Cinder access, dr_operator satisfies the cross-site sync policy — and dropping either causes failover to fail after the storage work is done. Both roles must exist in Keystone on both sites before the service processes its first DR operation.

Cross-site timeout

The default cross_site_timeout = 30 seconds is adequate for sites on the same LAN. For sites connected over a WAN, raise it:

[engine]
cross_site_timeout = 60

For Juju deployments, use config-flags since there is no named charm option for this key:

juju config trilio-site-recovery-for-openstack \
  config-flags="engine:cross_site_timeout=60"

Auto-creating security groups

By default, if a protected instance uses a security group that does not exist on the DR site, the engine reports the gap but does not create the group. Set auto_create_security_groups = true in [engine] to let the engine create it automatically during failover. This writes to the DR cloud, so enable it only if you accept that behaviour.

Recovery points (ActiveDR)

For ActiveDR protection groups, you can enable scheduled recovery-point snapshots:

[engine]
recovery_point_interval_minutes = 60
recovery_point_max_count = 20
recovery_point_ttl_hours = 24

Leave recovery_point_interval_minutes = 0 (the default) to disable the scheduler and rely on on-demand snapshots only.


Examples

Example 1 — Mock driver (evaluation / CI)

A minimal configuration for a single-site evaluation without any storage array.

[DEFAULT]
debug = false
log_dir = /var/log/protector
use_stderr = true
state_path = /var/lib/protector
host = dr-node-01.example.com
local_site = site-a
rpc_conn_pool_size = 30
transport_url = rabbit://protector:rabbit-secret@rabbit.example.com:5672/protector

[oslo_messaging_rabbit]
rabbit_durable_queues = true
amqp_durable_queues = true

[api]
bind_host = 0.0.0.0
bind_port = 8788
workers = 2

[database]
connection = mysql+pymysql://protector:db-secret@db.example.com:3306/protector

[engine]
topic = protector-engine
auto_create_security_groups = false

[keystone_authtoken]
www_authenticate_uri = https://keystone.example.com:5000
auth_url = https://keystone.example.com:5000
auth_type = password
project_domain_name = Default
user_domain_name = Default
memcached_servers = memcache.example.com:11211

[oslo_policy]
policy_file = /etc/protector/policy.yaml

[service_credentials]
username = protector
password = keystone-service-secret
project_name = service
user_domain_name = Default
project_domain_name = Default
default_service_project = service
default_service_user_domain = Default
default_service_project_domain = Default
# default_trust_roles defaults to member,dr_operator — omit unless you need to change it

[storage]
storage_driver = mock

[mock_storage]
storage_base_path = /var/lib/protector/mock_storage
primary_array_name = primary-array
secondary_array_name = secondary-array

Example 2 — Pure Storage async replication (production)

A production configuration for a node at site-a, paired with site-b.

[DEFAULT]
debug = false
log_dir = /var/log/protector
use_stderr = true
state_path = /var/lib/protector
host = dr-primary-01.example.com
local_site = site-a
rpc_conn_pool_size = 30
transport_url = rabbit://protector:rabbit-secret@rabbit.example.com:5672/protector

[oslo_messaging_rabbit]
rabbit_durable_queues = true
amqp_durable_queues = true

[api]
bind_host = 0.0.0.0
bind_port = 8788
workers = 4

[database]
connection = mysql+pymysql://protector:db-secret@db.example.com:3306/protector

[engine]
topic = protector-engine
auto_create_security_groups = false
cross_site_timeout = 60
recovery_point_interval_minutes = 60
recovery_point_max_count = 20
recovery_point_ttl_hours = 48

[keystone_authtoken]
www_authenticate_uri = https://keystone-site-a.example.com:5000
auth_url = https://keystone-site-a.example.com:5000
auth_type = password
project_domain_name = Default
user_domain_name = Default
memcached_servers = memcache-site-a.example.com:11211

[oslo_policy]
policy_file = /etc/protector/policy.yaml

[service_credentials]
username = protector
password = keystone-service-secret
project_name = service
user_domain_name = Default
project_domain_name = Default
default_service_project = service
default_service_user_domain = Default
default_service_project_domain = Default
# Accepts the default: default_trust_roles = member,dr_operator

[storage]
storage_driver = pure

[pure_storage]
# The array at primary_management_ip serves site-a.
# The array at secondary_management_ip serves site-b.
# This same section can be deployed unchanged on the site-b node:
# the driver swaps the pair based on the site names, not node position.
primary_management_ip = 192.168.10.10
primary_api_token = your-site-a-array-api-token
secondary_management_ip = 192.168.20.10
secondary_api_token = your-site-b-array-api-token
primary_site_name = site-a
secondary_site_name = site-b
verify_ssl = false
default_rpo_seconds = 900
snapshot_interval_seconds = 900

Example 3 — Multi-array Pure Storage

For environments with more than one FlashArray pair, list additional array pairs using additional_arrays and define a section for each.

[pure_storage]
primary_management_ip = 192.168.10.10
primary_api_token = token-array-1-primary
secondary_management_ip = 192.168.20.10
secondary_api_token = token-array-1-secondary
primary_site_name = site-a
secondary_site_name = site-b
verify_ssl = false
default_rpo_seconds = 900
snapshot_interval_seconds = 900
additional_arrays = array-pair-2

[pure_array:array-pair-2]
primary_management_ip = 192.168.10.20
primary_api_token = token-array-2-primary
secondary_management_ip = 192.168.20.20
secondary_api_token = token-array-2-secondary
primary_site_name = site-a
secondary_site_name = site-b
verify_ssl = false
cinder_backend_names = cinder-backend-2a,cinder-backend-2b

Example 4 — Policy override for dr_operator listing site operations

Create /etc/protector/policy.yaml (or add to the existing file) to allow dr_operator to list and show site-level operations:

# /etc/protector/policy.yaml — partial override
"protector:site_operations:index": "rule:context_is_dr_operator"
"protector:site_operations:show": "rule:context_is_dr_operator"

For Juju, attach this as a zip archive via the policyd-override resource:

zip policy-overrides.zip policy.yaml
juju attach-resource trilio-site-recovery-for-openstack \
  policyd-override=policy-overrides.zip

Troubleshooting

Unit is BLOCKED after Juju upgrade with message "set local-site, pure-primary-site-name, pure-secondary-site-name"

Symptom: After upgrading the trilio-site-recovery-for-openstack charm to 1.0.20 or later, the unit enters BLOCKED state.

Cause: The charm now requires explicit configuration of local-site (and, when Pure Storage is configured, the two site names). It no longer infers these values, because inference was wrong on the recovery site — the site where failover actually runs.

Fix:

juju config trilio-site-recovery-for-openstack \
  local-site="site-a" \
  pure-primary-site-name="site-a" \
  pure-secondary-site-name="site-b"

This is expected behaviour, not a failed upgrade. The unit proceeds once these are set.


Failover fails with "Policy doesn't allow volume_extension:volume_manage to be performed (HTTP 403)"

Symptom: The service deploys and reports healthy, but the first failover fails partway through with a Cinder 403.

Cause: The service runs as a tenant through a Keystone trust and needs five Cinder policy relaxations on both sites. Stock Cinder restricts volume_manage to admin. The five rules that must be relaxed are:

"volume_extension:volume_manage": "rule:admin_or_owner"
"volume_extension:volume_unmanage": "rule:admin_or_owner"
"volume_extension:services:index": "rule:admin_or_owner"
"volume_extension:type_get": "rule:admin_or_owner"
"volume_extension:type_get_all": "rule:admin_or_owner"

Fix: Apply all five relaxations to the Cinder policy on both sites. The last two (type_get, type_get_all) are read-only lookups that some Cinder versions leave open by default, which is why a lab environment runs without them while a customer cloud does not.


Trust creation fails with "Role X not found"

Symptom: The first DR operation fails with an error indicating a Keystone role is missing.

Cause: Every role named in default_trust_roles must exist in Keystone on both sites before the service processes any DR operation. Trust creation fails naming the missing role rather than silently skipping it.

Fix: Create the missing roles on both sites:

openstack role create dr_operator
openstack role create dr_site_admin

If you have customised default_trust_roles, verify that every role you listed exists on both sites.


All volumes reported missing during failover (Pure Storage)

Symptom: Failover reports "2 of 2 volumes missing or destroyed" but the volumes exist on the array.

Cause: primary_site_name and secondary_site_name are inverted relative to their matching management IPs. The driver resolves which array to contact by comparing the protection group's primary site name against primary_site_name. If they are swapped, it reads the answer from the wrong array. The failure only manifests on the site you fail over to, because on the production site the wrong guess happens to land right.

Fix: Correct the pairing in [pure_storage] so that primary_site_name names the site whose array is at primary_management_ip, and secondary_site_name names the site whose array is at secondary_management_ip. The same section can be deployed unchanged on both nodes — the driver swaps the pair based on names when the protection group's primary site differs from primary_site_name.


Service fails to start with "DuplicateSectionError"

Symptom: protector-api or protector-engine exits immediately with a configparser DuplicateSectionError.

Cause (Juju): A config-flags entry names a configuration key that the charm template already renders, causing the same [section] to appear twice in the generated file.

Fix: Remove the conflicting key from config-flags. Only use config-flags for keys that are not already managed by one of the charm's 47 named options. For example, engine:cross_site_timeout is safe because the charm template does not render cross_site_timeout. Check whether a named charm option already controls the key you want to set.


Failback appears to hang on "unmanaging volumes"

Symptom: Failback is running but appears stuck; logs show repeated volume detach checks.

Cause: This is expected behaviour, not a hang. Failback waits up to 120 seconds per DR volume for Nova to release the Cinder volume attachment before unmanaging the volume. Nova releases the attachment asynchronously after the recovered instance is deleted, and unmanaging an attached volume fails. The wait is per-volume and serial, so a site whose detaches lag will show a visibly slower failback.

Fix: No action is required. If the wait expires, the volume is unmanaged anyway and the event is logged (not raised), so failback is not aborted. If this recurs frequently, check that Nova is processing instance deletions promptly on the DR site.


Operation list shows blank Source Site and Target Site columns

Symptom: openstack dr operation list or openstack dr operation show shows empty values for the site columns.

Cause: This was a bug (fixed in the current release) where the API emitted only source_site_id and target_site_id while the CLI read source_site and target_site. Both columns were structurally blank.

Fix: Upgrade to the current release. The API now emits the site name (falling back to its ID) alongside the raw IDs. If you are on the current release and still see blank columns, verify that your python3-triliositerecovery client package is also up to date — on machines not managed by the charm, upgrade it separately:

sudo apt-get install --only-upgrade python3-triliositerecovery

LUKS-encrypted volume rejected when adding a member

Symptom: openstack dr protection group member add fails with an InvalidInput error referencing encryption.

Cause: LUKS-encrypted volumes are not supported in protection groups. Cinder's manage API cannot associate Barbican keys on the DR site, so an encrypted volume cannot be recovered.

Fix: Use array-side encryption (Pure Storage data-at-rest encryption) instead of host-side LUKS. Remove the LUKS encryption from the volume before adding it to a protection group.