Observability

This page describes how to monitor Trilio Site Recovery using its built-in observability primitives: Custom Resource status fields, RPO and replication health CRDs, and structured controller logs. Because all DR operations are CRD-driven, the primary observability surface is Kubernetes itself—you can query replication health, RPO violations, and failover progress with standard kubectl commands, integrate them into GitOps pipelines, or scrape them for alerting. Understanding this surface lets you catch degraded replication before a disaster, verify RTO and RPO objectives are being met, and diagnose issues without requiring access to the underlying storage layer.

Before using the observability features described on this page:

• OpenShift ≥ 4.14 with oc or kubectl CLI access to all three clusters (quorum, primary, DR)
• Trilio Site Recovery deployed: site-recovery-quorum-control-plane chart running on the quorum cluster and site-recovery-workload-control-plane chart running on both primary and DR clusters
• Kubeconfig contexts configured for all clusters; the examples below use --context flags to distinguish them
• The tsr-gather diagnostic tool available if you need to collect a full support bundle
• Sufficient RBAC to read CRDs across namespaces: get, list, and watch on replicationgroupstatuses, rpoevents, protectiongroups, failoverrequests, testfailovers, and drbdresources in the DR namespace (dr-<name>) on the quorum cluster and on both workload clusters

Observability primitives are installed automatically as part of the standard Site Recovery deployment. No additional Helm charts or agents are required. Verify that the CRDs are present on the appropriate clusters after deploying the operator.

Step 1 – Confirm CRDs are registered on the quorum cluster

oc --context quorum get crd replicationgroupstatuses.siterecovery.trilio.io \
                            rpoevents.siterecovery.trilio.io

Expected output (both CRDs present):

NAME                                                   CREATED AT
replicationgroupstatuses.siterecovery.trilio.io        2024-06-01T10:00:00Z
rpoevents.siterecovery.trilio.io                       2024-06-01T10:00:00Z

Step 2 – Confirm CRDs are registered on the primary and DR clusters

oc --context primary get crd protectiongroups.siterecovery.trilio.io \
                            drbdresources.siterecovery.trilio.io

oc --context dr get crd protectiongroups.siterecovery.trilio.io \
                       drbdresources.siterecovery.trilio.io

Step 3 – Verify the quorum control plane is running

oc --context quorum get pods -n dr-<name> -l app.kubernetes.io/name=site-recovery-quorum-control-plane

All pods must be in Running state before replication health data is written to CRs.

Step 4 – Verify the workload control plane is running on both workload clusters

oc --context primary get pods -n dr-<name> -l app.kubernetes.io/name=site-recovery-workload-control-plane
oc --context dr     get pods -n dr-<name> -l app.kubernetes.io/name=site-recovery-workload-control-plane

Step 5 – (Optional) Configure the tsr-gather support bundle tool

tsr-gather is an oc adm must-gather compatible tool run against the quorum cluster. It collects logs, CRD state, and configuration from all clusters into a single diagnostic bundle.

oc adm must-gather --image=<tsr-gather-image> -- /usr/bin/tsr-gather

Replace <tsr-gather-image> with the registry path provided by your Trilio distribution.

Observability behavior is configured through spec fields on the ReplicationGroupStatus and RPOEvent CRDs, and through the ProtectionZone logging configuration. The fields below are the only configurable options—do not assume others exist.

ReplicationGroupStatus – polling and RPO objective

This CR is created in the DR namespace (dr-<name>) on the quorum cluster, one per Protection Group. It controls how often the replication monitor reconciler polls DRBD resource state and what lag threshold triggers a violation.

Example manifest:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ReplicationGroupStatus
metadata:
  name: prod-replication-status
  namespace: dr-production
spec:
  protectionGroupRef:
    name: prod-protection-group
    namespace: dr-production
  pollingIntervalSeconds: 30
  replicationProtocol: "C"
  rpoObjectiveSeconds: 0

With replicationProtocol: "C" and rpoObjectiveSeconds: 0, any observed lag records a critical RPOEvent, which is appropriate for synchronous (Protocol C) deployments where RPO=0 is required.

For asynchronous (Protocol A) deployments, set rpoObjectiveSeconds to your acceptable lag budget (for example, 30 for a 30-second RPO).

RPOEvent – event structure (read-only at runtime)

RPOEvent CRs are written by the replication-monitor reconciler; you do not create them manually. The fields below describe what the operator populates so you can filter and alert on them.

ProtectionZone – logging configuration

The ProtectionZone CR includes a spec.logging object that controls log verbosity for the quorum control plane components. Adjust this if you need more detailed output during troubleshooting.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: zone-production
  namespace: dr-production
spec:
  displayName: "Production DR Zone"
  storageBackendMode: "drbd-operator"
  clusters:
    - name: primary
    - name: dr
  logging:
    # Set log level here; consult your operator version docs for valid values
    level: info

The following patterns cover the most common day-to-day observability tasks.

Check aggregated replication health for a Protection Group

The ReplicationGroupStatus CR provides a single Healthy, Degraded, or Critical status for all volumes in a Protection Group. Query it on the quorum cluster.

oc --context quorum get replicationgroupstatus prod-replication-status \
  -n dr-production \
  -o jsonpath='{.status.health}: last sync {.status.lastSyncTimestamp}'

Watch for transitions in real time:

oc --context quorum get replicationgroupstatus prod-replication-status \
  -n dr-production -w

List all RPO violations for a Protection Group

RPOEvent CRs accumulate as an audit trail. Filter by the Protection Group label or name to see the full violation history.

oc --context quorum get rpoevents -n dr-production \
  -o custom-columns=\
'NAME:.metadata.name,TIMESTAMP:.spec.timestamp,LAG:.spec.rpoAtEvent,REASON:.spec.violationReason,STATUS:.spec.newRPOStatus'

To see the full detail of the most recent event:

oc --context quorum get rpoevents -n dr-production \
  --sort-by='.spec.timestamp' \
  -o yaml | tail -60

Inspect per-volume sync state on the DR cluster

DRBDResource CRs report per-volume sync progress and connection state for each VM. One CR exists per VM per cluster side.

oc --context dr get drbdresources -n dr-production

For detailed per-volume status:

oc --context dr get drbdresource prod-vm-1-drbd -n dr-production -o yaml

The status section of the CR shows sync percentage, connection state, and role (Primary/Secondary) for each volume.

Monitor an in-progress failover

FailoverRequest CRs track progress from Pending through InProgress to Completed or Failed. Watch the CR on the quorum cluster.

oc --context quorum get failoverrequest prod-failover -n dr-production -w

For full status including per-VM detail:

oc --context quorum get failoverrequest prod-failover -n dr-production -o yaml

Monitor a test failover

TestFailover CRs progress through CreatingSnapshots → VerifyingData → Succeeded (or Failed) → CleaningUp.

oc --context quorum get testfailover prod-test-failover -n dr-production -w

Read controller logs

All Site Recovery controllers run in the DR namespace on the quorum cluster. Use the following patterns to access their logs.

failover-controller logs:

oc --context quorum logs -n dr-production \
  -l app.kubernetes.io/component=failover-controller --tail=200 -f

protection-controller logs:

oc --context quorum logs -n dr-production \
  -l app.kubernetes.io/component=protection-controller --tail=200 -f

pg-sync-controller logs:

oc --context quorum logs -n dr-production \
  -l app.kubernetes.io/component=pg-sync-controller --tail=200 -f

test-failover-controller logs:

oc --context quorum logs -n dr-production \
  -l app.kubernetes.io/component=test-failover-controller --tail=200 -f

drbd-node-agent logs (on a specific worker node):

oc --context primary logs -n dr-production \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=worker-node-1 --tail=100 -f

Collect a full diagnostic bundle

Run tsr-gather against the quorum cluster to collect logs, CRD state, and configuration from all clusters:

oc adm must-gather --image=<tsr-gather-image> \
  --dest-dir=./tsr-bundle \
  -- /usr/bin/tsr-gather

The bundle is written to ./tsr-bundle and contains no secrets or sensitive credentials.

Example 1 – Confirm replication is healthy before a planned maintenance window

oc --context quorum get replicationgroupstatus prod-replication-status \
  -n dr-production \
  -o jsonpath='Health: {.status.health}\nLast sync: {.status.lastSyncTimestamp}\nPer-volume detail: {.status.volumes}\n'

Expected output when healthy:

Health: Healthy
Last sync: 2024-06-15T08:42:01Z
Per-volume detail: [{"name":"prod-vm-1-disk","syncState":"Consistent"},{"name":"prod-vm-2-disk","syncState":"Consistent"}]

If the health is Degraded or Critical, investigate RPOEvent CRs and DRBDResource status before proceeding with any planned failover.

Example 2 – List all RPO violations in the last 24 hours

oc --context quorum get rpoevents -n dr-production \
  -o custom-columns=\
'TIMESTAMP:.spec.timestamp,LAG_S:.spec.rpoAtEvent,BYTES_OUT:.spec.outOfSyncBytesAtEvent,REASON:.spec.violationReason'

Expected output:

TIMESTAMP                    LAG_S   BYTES_OUT   REASON
2024-06-15T06:12:04Z         8       204800      Network congestion on replication link
2024-06-15T06:13:34Z         0       0           Replication recovered

A LAG_S of 0 and BYTES_OUT of 0 in the most recent event indicates replication has recovered. An accumulating sequence of events with non-zero lag suggests a persistent connectivity or bandwidth problem.

Example 3 – Inspect per-volume DRBD sync state for a specific VM

oc --context dr get drbdresource prod-vm-1-drbd \
  -n dr-production \
  -o yaml

Expected status section (fully synced, secondary role):

status:
  role: Secondary
  connectionState: Connected
  volumes:
    - name: prod-vm-1-disk
      syncPercent: 100
      diskState: UpToDate
      replicationState: Established

Degraded example (sync in progress):

status:
  role: Secondary
  connectionState: Connected
  volumes:
    - name: prod-vm-1-disk
      syncPercent: 67
      diskState: Inconsistent
      replicationState: SyncTarget

A volume with diskState: Inconsistent and syncPercent below 100 is still synchronizing. Do not trigger a failover until diskState reaches UpToDate on the DR side.

Example 4 – Watch a failover request from start to completion

Create the FailoverRequest and immediately start watching it:

oc --context quorum apply -f - <<EOF
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: prod-planned-failover
  namespace: dr-production
spec:
  protectionGroupRef:
    name: prod-protection-group
    namespace: dr-production
  targetCluster: dr
  failoverType: planned
  drainTimeoutSeconds: 120
  batchBootTimeoutSeconds: 300
EOF

oc --context quorum get failoverrequest prod-planned-failover \
  -n dr-production -w

Expected status progression:

NAME                    STATUS    PHASE
prod-planned-failover   Pending   Initializing
prod-planned-failover   InProgress  StoppingVMs
prod-planned-failover   InProgress  PromotingVolumes
prod-planned-failover   InProgress  StartingVMs
prod-planned-failover   Completed   Done

Example 5 – Collect a diagnostic bundle after an incident

oc adm must-gather --image=<tsr-gather-image> \
  --dest-dir=./tsr-incident-bundle \
  -- /usr/bin/tsr-gather

ls -lh ./tsr-incident-bundle/

Expected output:

total 18M
drwxr-xr-x  quorum-cluster/
drwxr-xr-x  primary-cluster/
drwxr-xr-x  dr-cluster/
-rw-r--r--  tsr-gather-summary.txt

The bundle contains controller logs, CRD snapshots, and configuration from all clusters in the DR estate. Attach this directory when opening a support case with TrilioData.

Use a consistent diagnostic sequence: check the relevant CRD status first, then controller logs, then node-agent logs, then collect a tsr-gather bundle if the issue is not resolved.

ReplicationGroupStatus shows Degraded or Critical

Symptom: oc get replicationgroupstatus returns health: Degraded or health: Critical.

Likely causes:

• DRBD replication link between primary and DR worker nodes is interrupted (TCP port 7000–7999 blocked or network partition)
• A worker node running the drbd-node-agent DaemonSet has gone offline
• A DRBDResource volume is stuck in Inconsistent disk state and is not resynchronizing

Fix:

1. Check DRBDResource status for the affected VMs:
   Copy

   oc --context dr get drbdresources -n dr-production -o yaml
   

2. Look at drbd-node-agent logs on the affected nodes:
   Copy

   oc --context primary logs -n dr-production \
     -l app.kubernetes.io/component=drbd-node-agent --tail=100
   

3. Verify network connectivity on TCP ports 7000–7999 between primary and DR worker nodes.
4. If a node is offline, check node status:
   Copy

   oc --context primary get nodes
   

5. Once the link is restored, DRBD will begin resynchronizing automatically. Monitor sync progress:
   Copy

   oc --context dr get drbdresource <name> -n dr-production \
     -o jsonpath='{.status.volumes[*].syncPercent}'
   

RPOEvent records accumulating with non-zero rpoAtEvent

Symptom: oc get rpoevents shows repeated events with increasing spec.rpoAtEvent values and non-zero spec.outOfSyncBytesAtEvent.

Likely causes:

• Replication protocol is A (asynchronous) and network bandwidth between clusters is saturated
• Write workload on the primary has exceeded replication throughput
• Intermittent packet loss on the replication link causing retransmissions

Fix:

1. Check spec.violationReason on recent events for detail:
   Copy

   oc --context quorum get rpoevents -n dr-production \
     -o jsonpath='{range .items[-3:]}{.spec.timestamp}: {.spec.violationReason} ({.spec.rpoAtEvent}s lag)\n{end}'
   

2. Review the DRBDReplicationPolicy to confirm the protocol and endpoint configuration:
   Copy

   oc --context primary get drbdreplicationpolicy -n dr-production -o yaml
   

3. If lag is acceptable for Protocol A deployments but alerting is too aggressive, increase spec.rpoObjectiveSeconds in the ReplicationGroupStatus CR to match your actual RPO budget.
4. For sustained lag above your RPO objective, investigate network throughput between clusters and consider throttle settings in spec.resyncConfig on the DRBDReplicationPolicy.

FailoverRequest stuck in InProgress

Symptom: A FailoverRequest remains in InProgress for longer than the configured batchBootTimeoutSeconds.

Likely causes:

• VMs on the source cluster did not stop within drainTimeoutSeconds (unplanned failover scenario where source is partially reachable)
• DRBD volumes on the DR cluster failed to promote to Primary role
• VMs on the DR cluster are not scheduling (node taints, resource constraints, or missing PVCs)

Fix:

1. Inspect the full FailoverRequest status for error messages:
   Copy

   oc --context quorum get failoverrequest <name> -n dr-production -o yaml
   

2. Review the failover-controller logs:
   Copy

   oc --context quorum logs -n dr-production \
     -l app.kubernetes.io/component=failover-controller --tail=300
   

3. Check if VMs are in a Pending or Scheduling state on the DR cluster:
   Copy

   oc --context dr get vmi -n <vm-namespace>
   

4. Check for node taints preventing VM scheduling on DR worker nodes:
   Copy

   oc --context dr get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'
   

5. Verify that DRBDResource volumes on the DR side have reached role: Primary and diskState: UpToDate.

TestFailover stuck in CreatingSnapshots or CleaningUp

Symptom: A TestFailover CR does not advance past CreatingSnapshots or does not complete CleaningUp.

Likely causes:

• Volume snapshot infrastructure (VolumeSnapshotClass) is not configured on the DR cluster
• The test namespace already contains conflicting resources from a previous failed test
• The test-failover-controller pod has restarted and lost in-memory state

Fix:

1. Inspect the TestFailover status:
   Copy

   oc --context quorum get testfailover <name> -n dr-production -o yaml
   

2. Check test-failover-controller logs:
   Copy

   oc --context quorum logs -n dr-production \
     -l app.kubernetes.io/component=test-failover-controller --tail=200
   

3. Verify VolumeSnapshotClass is available on the DR cluster:
   Copy

   oc --context dr get volumesnapshotclass
   

4. If the CleaningUp phase is stuck, check for orphaned resources in the test namespace:
   Copy

   oc --context dr get all -n <test-namespace>
   

   Manually delete the test namespace if it is safe to do so, then delete and recreate the TestFailover CR.

Controller logs show repeated reconciliation errors

Symptom: Controller logs emit repeated error messages for the same resource without making progress.

Likely causes:

• RBAC permissions missing for the controller's service account
• The quorum cluster cannot reach the primary or DR cluster API server on TCP port 6443
• A CRD schema validation error is rejecting status patches

Fix:

1. Check the exact error in the logs:
   Copy

   oc --context quorum logs -n dr-production \
     -l app.kubernetes.io/name=site-recovery-quorum-control-plane --tail=100 | grep -i error
   

2. Verify API server reachability from the quorum cluster:
   Copy

   oc --context quorum exec -n dr-production deploy/site-recovery-quorum-control-plane \
     -- curl -sk https://<primary-api-server>:6443/healthz
   

3. Check service account RBAC:
   Copy

   oc --context quorum auth can-i get replicationgroupstatuses \
     --as=system:serviceaccount:dr-production:<controller-sa> -n dr-production
   

4. If RBAC is the issue, re-apply the Helm chart to restore the expected ClusterRoles and bindings:
   Copy

   helm upgrade --install site-recovery-quorum-control-plane \
     trilio/site-recovery-quorum-control-plane \
     -n dr-production
   

tsr-gather bundle is empty or missing cluster data

Symptom: The collected bundle directory is missing data from one or more clusters.

Likely cause: The kubeconfig contexts for the missing clusters are not accessible from the quorum cluster, or the tsr-gather image does not have network access to those API servers.

Fix:

1. Verify that the quorum cluster can reach each workload cluster API server on TCP port 6443 before running tsr-gather.
2. Confirm the kubeconfig used by tsr-gather includes valid contexts for all clusters.
3. Re-run with verbose output if the tool supports it:
   Copy

   oc adm must-gather --image=<tsr-gather-image> \
     --dest-dir=./tsr-bundle \
     -- /usr/bin/tsr-gather
   

4. If individual cluster data is still missing, collect logs manually from the affected cluster using the controller log commands in the Usage section and attach them alongside the bundle.