Observability

This page explains how to observe, monitor, and diagnose Trilio Site Recovery (TSR) deployments. It covers the Custom Resource fields that surface replication health, how to read and filter observability objects (ReplicationGroupStatus, RPOEvent), how to adjust log levels at runtime without restarting controllers, how to use the TSR web console (OpenShift only) and the tsr-gather support bundle tool, and which API endpoints expose health and status data. Understanding these signals lets you detect degraded replication before it affects your RPO, identify the root cause of a stuck or failed operation, and provide complete diagnostic data to support.

Before using the observability features described on this page:

• OpenShift 4.17 or later on the quorum cluster and both workload clusters (OpenShift is required; upstream Kubernetes is not supported).
• The TSR operator is installed via OLM on the quorum cluster and the ProtectionZone has reached Available (phase Available in status.phase).
• You have oc CLI access to the quorum cluster. oc is required for OpenShift-specific paths such as oc adm must-gather.
• Kubeconfig files for the quorum cluster and both workload clusters are available locally.
• To query workload-cluster objects (DRBDResource), your kubeconfig must point at the relevant workload cluster — those CRDs are not registered on the quorum cluster.
• To access the TSR web console, you must be on OpenShift: the oauth-proxy Route and console link are created automatically by the ProtectionZone controller. No browser access path exists on upstream Kubernetes.

No separate installation step is required for observability. All observability components are deployed automatically:

1. Replication monitor — the replication-monitor reconciler runs inside the unified quorum manager (site-recovery-quorum-control-plane) and automatically creates one ReplicationGroupStatus CR per ProtectionGroup, named <protection-group-name>-replication, in the zone's dr-<name> namespace.

2. RPO audit trail — the monitor writes RPOEvent CRs to the same namespace whenever replication state changes (sync completed, RPO violated, connection lost, etc.).

3. TSR web console (OpenShift only) — the operator installs site-recovery-frontend (Next.js/React behind an OpenShift oauth-proxy) and site-recovery-backend (Python Flask under gunicorn) with no feature flag and no opt-out. The ProtectionZone controller creates the Route and console link once the zone is Available. Navigate to the OpenShift web console and click the Site Recovery link in the application launcher.

4. tsr-gather support bundle — the site-recovery-must-gather image ships a gather binary at /usr/bin/gather. Run it with oc adm must-gather from the quorum cluster:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag>

On an OLM-installed cluster, --all-images picks the image from the CSV annotation automatically:

oc adm must-gather --all-images

5. Runtime log-level ConfigMap — the site-recovery-logging ConfigMap is created in the zone's release namespace by the ProtectionZone controller when spec.logging is set. You can also create or patch it manually (see Configuration).

Log levels

Log verbosity is controlled at two layers — startup flags (set by the chart) and a runtime ConfigMap that the manager polls every ~15 seconds without a restart.

Startup level

Set in ProtectionZone.spec.logging:

spec:
  logging:
    level: info           # global level: error | info | debug | trace
    controlPlaneLevel: debug   # quorum manager override
    nodeAgentLevel: info       # drbd-node-agent override

Valid values: error, info, debug, trace. There is no warn level (the logger uses a two-class model).

Runtime level (no restart needed)

The site-recovery-logging ConfigMap in the zone namespace (dr-<name>) is polled by the manager every ~15 seconds:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-<zone-name>
data:
  level: debug                    # global runtime level
  level.control-plane: debug      # quorum manager only
  level.drbd-node-agent: info     # node agent only
  level.protectionzone: info      # ProtectionZone controller only

Patch it to change levels live:

oc -n dr-<zone-name> patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"debug"}}'

Restore to info when done — debug and trace are verbose and intended for short diagnostic windows.

Replication polling interval

ReplicationGroupStatus.spec.pollingIntervalSeconds controls how often the replication monitor scrapes DRBD state. The valid range is 5–300 seconds (default: 15). Lower values reduce detection latency at the cost of higher API-server load:

spec:
  pollingIntervalSeconds: 15    # default; valid range 5–300

This field is immutable once set alongside replicationProtocol. Update it by deleting and recreating the ReplicationGroupStatus CR — but note that the monitor auto-creates exactly one per ProtectionGroup, so deleting it causes the monitor to recreate it on the next scrape with default values unless you patch the ProtectionGroup first.

Metrics endpoint

The quorum manager exposes a secure metrics endpoint on port 8443 (controlled by metrics.port and metrics.secure in the chart values). Authentication uses TokenReview/SubjectAccessReview. Access requires a bearer token bound to the metrics-reader ClusterRole. This endpoint also hosts the /debug/loglevel path for per-pod runtime level changes:

# Read current log level
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://<manager-service>:8443/debug/loglevel

# Set to debug (takes effect immediately, no restart)
curl -sk -X PUT -H "Authorization: Bearer $(oc whoami -t)" \
  -d '{"level":"debug"}' \
  https://<manager-service>:8443/debug/loglevel

Health probe

The manager health probe listens on :8081. You can use it to verify the manager is alive:

curl http://localhost:8081/healthz

This is not an externally routed endpoint — use it from within the pod or via oc port-forward.

Checking replication health

The ReplicationGroupStatus CR (short name rgs) aggregates DRBD replication health for a ProtectionGroup. The replication monitor creates exactly one per group, named <protection-group-name>-replication, in the zone's dr-<name> namespace:

# List all replication group statuses in a zone namespace
oc -n dr-<zone-name> get rgs

# Describe a specific one
oc -n dr-<zone-name> describe rgs <pg-name>-replication

Key fields to watch:

To filter by ProtectionGroup, use the label siterecovery.trilio.io/protection-group (field selectors on custom resource spec.*/status.* fields are not supported):

oc -n dr-<zone-name> get rgs \
  -l siterecovery.trilio.io/protection-group=<pg-name>

Monitoring RPO events

RPOEvent CRs (short name rpoe) record individual replication state changes with a timestamp, severity, and reason. They form an immutable audit trail — updates are rejected by the webhook:

# List all RPO events in a zone namespace, newest first
oc -n dr-<zone-name> get rpoe --sort-by=.spec.timestamp

# Filter by protection group
oc -n dr-<zone-name> get rpoe \
  -l siterecovery.trilio.io/protection-group=<pg-name>

# Show full detail for a specific event
oc -n dr-<zone-name> describe rpoe <event-name>

Important spec fields:

Checking per-volume replication state

ReplicationGroupStatus.status.volumeReplicationStatuses lists per-volume detail. Each entry includes:

• replicationState — one of Connected, Connecting, Disconnected, SyncSource, SyncTarget, PausedSyncS, PausedSyncT, Unknown
• connectionState — Connected, Degraded, or Disconnected
• outOfSyncBytes — bytes not yet replicated
• estimatedRPOSeconds — estimated RPO for this volume
• pvcName — the source PVC name
• vmName — the VM that owns this volume
• drbdResourceName — the corresponding DRBDResource CR name (format: drbd-<namespace>-<vm>-<hash8>)

oc -n dr-<zone-name> get rgs <pg-name>-replication \
  -o jsonpath='{.status.volumeReplicationStatuses[*]}' | python3 -m json.tool

Checking DRBDResource state on workload clusters

DRBDResource CRs (short name dres) live on the workload clusters (not on the quorum cluster — querying the quorum returns server doesn't have a resource type):

# On the primary workload cluster
kubectl --kubeconfig <primary-kubeconfig> -n <vm-namespace> get dres

# Watch sync progress for a specific resource
kubectl --kubeconfig <primary-kubeconfig> -n <vm-namespace> get dres <resource-name> \
  -o custom-columns='PHASE:.status.phase,STATE:.status.connectionState,ROLE:.status.role,SYNC:.status.syncProgress.percentage'

Healthy per-volume replicationState is Established (not Connected — that value reflects pre-TSR-90 naming and indicates a stale or misread object on a current stack).

Viewing logs

Quorum manager (failover, protection, pg-sync, replication-monitor, test-failover reconcilers)

The quorum Deployment runs a single container named manager:

# Find the manager pod
oc -n dr-<zone-name> get pods -l app.kubernetes.io/component=quorum-control-plane

# Stream logs
oc -n dr-<zone-name> logs -l app.kubernetes.io/component=quorum-control-plane -c manager -f

# Filter for a specific reconciler
oc -n dr-<zone-name> logs -l app.kubernetes.io/component=quorum-control-plane -c manager \
  | grep 'replication-monitor'

DRBD node agent (workload clusters)

The node agent runs as a DaemonSet named <release>-site-recovery-workload-control-plane-agent in trilio-site-recovery-system on each workload cluster. To get logs from a specific node:

# Step 1: find the agent pod on the target node
kubectl --kubeconfig <workload-kubeconfig> \
  -n trilio-site-recovery-system \
  get pods --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}'

# Step 2: stream logs from that pod
kubectl --kubeconfig <workload-kubeconfig> \
  -n trilio-site-recovery-system \
  logs <pod-name> -f

ProtectionZone controller

oc -n trilio-site-recovery-system get pods \
  -l app.kubernetes.io/component=protectionzone-controller

oc -n trilio-site-recovery-system logs \
  -l app.kubernetes.io/component=protectionzone-controller -f

Using the TSR web console (OpenShift only)

Once the ProtectionZone reaches Available, the controller creates an OpenShift Route and a console link. Click Site Recovery in the OpenShift web console application launcher. The console shows:

• Dashboard with protection zone health and VM counts
• Per-ProtectionGroup replication status and RPO compliance
• RPO event history and violation details
• Test failover results and verification checks

Zone-scoped pages derive the dr-<zoneId> namespace from the zone ID — a zone in a differently named namespace appears in the list but its detail pages will not load.

Using the backend REST API

The site-recovery-backend Flask API is accessible behind the oauth-proxy Route on OpenShift. Key observability endpoints:

# Overall quorum health
GET /api/quorum/health

# Zone-level health summary
GET /api/protection-zones/<zone_id>/health

# Run health checks
POST /api/protection-zones/<zone_id>/health/run

# Per-protection-group health
GET /api/protection-zones/<zone_id>/protection-groups/<pg_name>/health

# Replication status for the zone
GET /api/protection-zones/<zone_id>/replication-status

# Per-group replication status
GET /api/protection-zones/<zone_id>/protection-groups/<pg_name>/replication-status

# RPO events for the zone
GET /api/protection-zones/<zone_id>/rpo-events

# RPO history for a group
GET /api/protection-zones/<zone_id>/protection-groups/<pg_name>/rpo-history

# RPO compliance summary
GET /api/protection-zones/<zone_id>/rpo-compliance

# DRBD events
GET /api/protection-zones/<zone_id>/drbd-events

# Dashboard statistics
GET /api/protection-zones/<zone_id>/dashboard/stats

Collecting a support bundle

Run tsr-gather on the quorum cluster to collect logs, CRD state, and DRBD node status from all clusters in the estate:

# Full estate bundle
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag>

# Limit log window to the last 2 hours (faster, smaller)
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

# One zone only (multi-tenant quorum)
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --zone <zone-name>

# Quorum only, no reach-through to workload clusters
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --local-only

# Bound the run time
oc adm must-gather --timeout=10m \
  --image=quay.io/triliovault/site-recovery-must-gather:<tag> \
  -- /usr/bin/gather --since 2h

The bundle lands in ./must-gather.local.<hash>/. Read SUMMARY.md first for an estate-wide status overview, then manifest.yaml for per-cluster inventory counts.

Example 1 — Check whether replication is healthy for a protection group

oc -n dr-production get rgs pg-frontend-replication -o yaml

Expected output (healthy):

status:
  rpoStatus: Met
  aggregated:
    connectionState: Connected
    outOfSyncBytes: 0
    syncedVolumeCount: 3
    totalVolumeCount: 3
    avgWriteLatencyMilliseconds: 2
  currentRPOSeconds: 0
  lastFullySyncedAt: "2026-09-01T14:32:10Z"
  lastPolledAt: "2026-09-01T14:32:25Z"

rpoStatus: Met with outOfSyncBytes: 0 and currentRPOSeconds: 0 confirms Protocol C synchronous replication is operating normally.

Example 2 — List recent RPO events for a protection group

oc -n dr-production get rpoe \
  -l siterecovery.trilio.io/protection-group=pg-frontend \
  --sort-by=.spec.timestamp

Expected output:

NAME                         PROTECTION-GROUP   EVENT-TYPE          RPO-SECONDS   OUT-OF-SYNC   REASON            TIMESTAMP
pg-frontend-20260901-142200  pg-frontend        ConnectionLost      45            2097152       ConnectionLost    2026-09-01T14:22:00Z
pg-frontend-20260901-143100  pg-frontend        ConnectionRestored  0             0                               2026-09-01T14:31:00Z
pg-frontend-20260901-143210  pg-frontend        SyncCompleted       0             0                               2026-09-01T14:32:10Z

This shows a ~9-minute link outage that caused a temporary RPO violation, followed by successful resynchronisation.

Example 3 — Watch sync progress on a DRBDResource during initial protection

Run this on the workload cluster (not the quorum cluster):

kubectl --kubeconfig ~/.kube/config-primary \
  -n vm-production \
  get dres drbd-vm-production-web-vm-a1b2c3d4 -w \
  -o custom-columns=\
'PHASE:.status.phase,CONN:.status.connectionState,ROLE:.status.role,SYNC%:.status.syncProgress.percentage,REMAINING:.status.syncProgress.estimatedTimeRemaining'

Expected output during sync:

PHASE     CONN         ROLE      SYNC%   REMAINING
Syncing   Connected    Primary   42      14m30s
Syncing   Connected    Primary   68      8m12s
Syncing   Connected    Primary   95      1m04s
Consistent Connected   Primary   100     0s

Example 4 — Enable debug logging at runtime without restarting the manager

# Patch the logging ConfigMap in the zone namespace
oc -n dr-production patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"debug","level.control-plane":"debug"}}'

# Wait ~15 seconds, then confirm the manager picked up the change
oc -n dr-production logs -l app.kubernetes.io/component=quorum-control-plane \
  -c manager --since=30s | grep -i "log level"

# Restore to info when done
oc -n dr-production patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"info","level.control-plane":"info"}}'

Example 5 — Collect a focused support bundle for a recent failover incident

# Collect the last 4 hours of data for zone "production" only
oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:latest \
  --timeout=10m \
  -- /usr/bin/gather --zone production --since 4h

# Navigate the bundle
cd must-gather.local.*/*/
cat SUMMARY.md
ls quorum/namespaces/dr-production/crs/failoverrequest/

Example 6 — Identify all protected VMs across workload clusters

siterecovery.trilio.io/protected-vm=true is stamped on VirtualMachine objects on the workload cluster:

# On the primary workload cluster
kubectl --kubeconfig ~/.kube/config-primary \
  get vm -A -l siterecovery.trilio.io/protected-vm=true

Expected output:

NAMESPACE       NAME      AGE
vm-production   web-vm    2d
vm-production   db-vm     2d
vm-staging      app-vm    6h

Example 7 — Check per-volume replication detail from the ReplicationGroupStatus

oc -n dr-production get rgs pg-frontend-replication \
  -o jsonpath='{range .status.volumeReplicationStatuses[*]}{.vmName}\t{.pvcName}\t{.replicationState}\t{.outOfSyncBytes}\t{.estimatedRPOSeconds}\n{end}'

Expected output (healthy):

web-vm   web-data-pvc     Connected   0   0
db-vm    db-data-pvc      Connected   0   0
db-vm    db-log-pvc       Connected   0   0

Use a consistent approach for each issue: identify the symptom, find the likely cause using the CRs and logs described in the Usage section, then apply the fix.

Issue 1 — ReplicationGroupStatus shows Degraded or Critical

Symptom: oc get replicationgroupstatuses shows Degraded or Critical for one or more ProtectionGroups.

Likely causes:

• Network interruption between primary and DR worker nodes on TCP ports 7000–8000.
• The DRBD node agent on a workload node has not converged. Look for a siterecovery.trilio.io/not-ready:NoSchedule taint on the node.
• A workload node was rebooted and the DRBD kernel modules are reloading.

Fix:

1. Check the DRBDResource CRs on both clusters for per-volume connection state:
   Copy

   oc get drbdresource -n <workload-namespace> -o wide
   

2. Inspect node agent logs on the affected node:
   Copy

   oc logs -n <workload-namespace> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
     --follow
   

3. Verify network connectivity on TCP ports 7000–8000 between the primary and DR worker nodes.
4. Check whether the siterecovery.trilio.io/not-ready:NoSchedule taint is still applied to the node. If DRBD has converged, the node agent removes it automatically. If the taint persists after several minutes, the agent pod may itself be unhealthy.
5. Review RPOEvent CRs for a timeline of when the degradation started:
   Copy

   oc get rpoevents -n dr-<name> --sort-by=.spec.timestamp
   

Issue 2 — No site-recovery-logging ConfigMap; log level change has no effect

Symptom: After patching spec.logging.runtimeLevel on the ProtectionZone, no ConfigMap appears and log verbosity does not change.

Likely cause: The ProtectionZone has not reached Available, so the ProtectionZone controller has not reconciled the zone. The ConfigMap is only rendered when runtimeLevel is non-empty and the zone is healthy.

Fix:

1. Check the ProtectionZone status:
   Copy

   oc get protectionzone -n dr-<name>
   

2. If the zone is not Available, describe it for conditions:
   Copy

   oc describe protectionzone my-zone -n dr-<name>
   

3. Once the zone is Available, re-apply the patch. The ConfigMap should appear within one reconciliation cycle.

Issue 3 — Metrics endpoint returns a TLS error or connection refused

Symptom: curl https://localhost:8443/metrics from inside the manager pod fails with a certificate error or a connection refusal.

Likely causes:

• The webhook-server-cert Secret does not exist in the release namespace. The manager will not start without it.
• The metrics Service is not present (e.g., metrics.enabled was set to false internally).

Fix:

1. Verify the Secret exists:
   Copy

   oc get secret webhook-server-cert -n dr-<name>
   

   If absent, create it with your CA-signed certificate and key:
   Copy

   oc create secret tls webhook-server-cert \
     --cert=tls.crt --key=tls.key \
     -n dr-<name>
   

2. Check the manager pod is running (expected 1/1):
   Copy

   oc get pods -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
   

3. Use -k to skip TLS verification for ad-hoc curl requests against the self-signed cert, or supply your CA bundle.

Issue 4 — ProtectionRequest stuck in Validating or moves to Failed

Symptom: A ProtectionRequest does not progress past Validating, or immediately transitions to Failed.

Likely cause: The protection controller found a configuration violation. Common causes include a VM disk that does not use volumeMode: Block, the VM not being in a running state at protection time, or a VM being on a node without a published InternalIP.

Fix:

1. Describe the ProtectionRequest for conditions and events:
   Copy

   oc describe protectionrequest <name> -n <workload-namespace>
   

   Look for a Validated=False condition and the associated warning event message.
2. Correct the VM configuration (disk mode, VM state, node configuration).
3. The Failed phase is terminal. Create a new ProtectionRequest after fixing the underlying issue — you cannot retry the existing CR.

Issue 5 — TestFailover is stuck and the test namespace cannot be cleaned up

Symptom: A TestFailover CR is stuck in a phase (e.g., CreatingVMs) and you want to remove the test namespace.

Likely cause: The test-failover controller manages the lifecycle of the test namespace (status.testNamespace). If the controller is healthy, it will eventually transition to CleaningUp and then Cleaned.

Fix:

1. Check the TestFailover phase and test namespace:
   Copy

   oc get testfailover <name> -n dr-<name> \
     -o jsonpath='{.status.phase} {.status.testNamespace}'
   

2. Stream controller logs filtered to the test-failover reconciler to identify the blocking step:
   Copy

   oc logs -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
     -c manager | jq 'select(.controller == "testfailover")'
   

3. Do not attempt to manually delete the test namespace or patch finalizers — the test-failover controller owns the namespace lifecycle. Manual interference can leave the protection group in an inconsistent state.
4. If the controller pod itself is unhealthy, restart it by deleting the pod (the Deployment will recreate it):
   Copy

   oc delete pod -n dr-<name> \
     -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane
   

Issue 6 — oc adm drain does not complete on a node hosting a protected VM

Symptom: oc adm drain <node> hangs and never completes.

Likely cause: The node hosts a running protected VM. Every KubeVirt migration path — including drain-triggered evacuation — is denied for protected VMs: live migration of VM "<name>" is not allowed: it uses DRBD-protected volumes with node-pinned PVs. The anchor pod for stopped protected VMs also blocks the drain because it has no controller owner and is node-pinned, so --force does not help.

Fix:

• For in-place maintenance (reboot, kernel update) of the same node: proceed directly. DRBD modules reload on boot, the anchor pod returns to the same node, and DRBD reconnects and resynchronises the delta automatically. A drain is not needed.
• To permanently remove the node: unprotect all VMs pinned to that node, perform node removal, then re-protect the VMs on a new node. Re-protection requires a full initial resynchronisation.
• If the node was lost unexpectedly: follow the node-loss recovery runbook (docs/runbooks/drbdresource-node-loss.md). Remove only the agent finalizer as documented — removing the placement finalizer instead permanently leaks a PVC/PV pair.

Issue 7 — No metrics or log data in a third-party monitoring system

Symptom: Your Prometheus instance or log aggregator is not receiving data from TSR.

Likely cause: The metrics endpoint requires bearer-token authentication (TokenReview/SubjectAccessReview). An unauthenticated scrape job will be rejected. Log forwarding requires configuring your OpenShift log forwarding pipeline to target the dr-<name> namespace.

Fix:

1. Configure your Prometheus scrape job to include a bearer token bound to the metrics-reader ClusterRole (which grants get and put on /debug/loglevel and get on /metrics).
2. The metrics endpoint is on port 8443 with TLS. Supply the CA bundle or disable TLS verification in your scrape config.
3. For log aggregation, configure your OpenShift log forwarding pipeline (e.g., ClusterLogForwarder) to forward logs from the dr-<name> namespace. Logs are emitted as structured JSON from the manager container.