Trilio Site Recovery for Kubernetes
Docs/site-recovery/Troubleshooting/RBAC Issues
Guide

RBAC Issues

Permission denied errors, service account problems, cluster-scoped access issues

Overview

This page helps you diagnose and resolve RBAC-related failures in Trilio Site Recovery, including permission denied errors from controllers, service account misconfigurations, and cluster-scoped access problems. RBAC issues are among the most common causes of stalled protection operations and rejected admission webhook calls, because the product installs several service accounts with distinct, narrowly scoped permissions across multiple clusters. Understanding which service account operates on which cluster and namespace—and which admission webhooks gate which resources—lets you pinpoint and fix permission failures quickly without escalating to Trilio support.

Prerequisites

Before working through these steps you need:

  • oc CLI installed and configured (required for OpenShift-specific paths; kubectl-only operation is not supported)
  • Kubeconfig files for all three clusters: the TSR-Manager cluster and both workload clusters
  • Cluster-admin or equivalent access on all three clusters so you can inspect ClusterRoles, bindings, and SCCs
  • Trilio Site Recovery deployed via OLM in the trilio-site-recovery-system namespace on the TSR-Manager cluster, with at least one ProtectionZone CR created
  • OpenShift 4.17 or later on all clusters (the supported floor for both the TSR-Manager cluster and workload clusters)
Installation

No additional installation is required for RBAC support. The charts install all required service accounts, roles, and bindings automatically when the ProtectionZone controller deploys the workload and manager control planes. The steps below are diagnostic and remediation steps only.

Step 1 — Confirm the operator is installed and the OLM subscription is healthy

Run this against the TSR-Manager cluster:

oc get subscription -n trilio-site-recovery-system
oc get csv -n trilio-site-recovery-system

The CSV should be in Succeeded phase. If it is not, OLM has not finished installing the operator bundle and the protection-zone-controller service account may not exist yet.

Step 2 — Confirm the ProtectionZone has reached Available

The ProtectionZone controller deploys the workload control plane (including its RBAC) only after the zone reaches Available. Check zone phase on the TSR-Manager cluster:

oc get pz -n dr-<zone-name>

The STATUS column must show Available. If it shows Provisioning or Degraded, resolve the zone condition first before chasing RBAC errors on the workload clusters.

Step 3 — Confirm service accounts exist on workload clusters

Run the following against each workload cluster (substitute the correct kubeconfig):

oc --kubeconfig <workload-kubeconfig> \
  get serviceaccount -n trilio-site-recovery-system

Expect to see at least the service account created by the workload chart (named after the Helm release fullname, defaulting to the chart fullname when serviceAccount.name is empty). If the namespace trilio-site-recovery-system does not exist on the workload cluster, the workload control plane was never deployed—check the ProtectionZone status conditions.

Step 4 — Confirm the privileged SCC binding on workload clusters

On OpenShift workload clusters the workload chart binds the node-agent service account to the privileged SCC. Verify this:

oc --kubeconfig <workload-kubeconfig> \
  get clusterrolebinding \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

If the binding is missing, the DRBD node-agent DaemonSet pods will fail to start with SCC or permission denied errors in the pod events.

Step 5 — Confirm the webhook serving certificate exists

Admission webhooks on both the TSR-Manager cluster and workload clusters require a TLS serving certificate in a Secret named webhook-server-cert in the zone namespace (dr-<zone-name>) on the TSR-Manager cluster, and in trilio-site-recovery-system on each workload cluster. This Secret is not managed by the charts and must be created out of band:

# TSR-Manager cluster
oc get secret webhook-server-cert -n dr-<zone-name>

# Each workload cluster
oc --kubeconfig <workload-kubeconfig> \
  get secret webhook-server-cert -n trilio-site-recovery-system

A missing or expired webhook-server-cert causes all admission calls to the affected webhook server to fail with TLS errors, which surface as connection refused or x509 errors in the API server audit log.

Configuration

RBAC configuration is controlled by the Helm chart values applied automatically by the ProtectionZone controller. Customers do not run helm install directly; these values are internal defaults. The relevant keys are:

rbac.create (default: true)

Controls whether the chart creates Roles, ClusterRoles, and their bindings. Set to false only if you are providing your own RBAC manifests. Disabling this on a workload cluster prevents the node-agent from managing DRBD resources.

serviceAccount.create (default: true)

Controls whether the chart creates the controller's ServiceAccount. When true and serviceAccount.name is empty, the account is named after the chart fullname.

serviceAccount.name (default: "")

When set, the chart uses this name instead of the generated fullname. Must match the name used in any externally created RoleBindings.

serviceAccount.agentName (default: "")

The ServiceAccount name for the DRBD node-agent DaemonSet. When empty, the chart generates a name. On OpenShift this account is bound to the privileged SCC by the chart; changing the name requires that the SCC binding reference the new name.

webhook.certSecretName (default: "webhook-server-cert")

The name of the Secret holding the TLS certificate for the admission webhook server. This Secret is not created by the chart and must be provisioned out of band before the webhook server starts. There is no cert-manager integration.

webhook.port (default: 443) and webhook.containerPort (default: 9443)

The external port registered in the MutatingWebhookConfiguration/ValidatingWebhookConfiguration (443) and the port the webhook server container actually listens on (9443). These values are set by the chart and should not be changed.

monitorServiceAccountName (default: "")

When set, the RPOEvent and ReplicationGroupStatus admission webhooks enforce that only the named service account may create, update, or delete those objects. An empty value disables the check. The replication-monitor controller uses this account to write RPO audit records; setting it incorrectly causes all RPO event writes to be rejected with a Forbidden error.

Cluster-scoped access via kubeconfig Secrets

Cross-cluster access is not implemented through RBAC on the remote cluster. Instead, the TSR-Manager controller mounts kubeconfig Secrets for each workload cluster. These are referenced by ProtectionZone.spec.clusters[].kubeconfigSecretRef and must exist in the zone namespace (dr-<zone-name>) on the TSR-Manager cluster before the zone reaches Available:

spec:
  clusters:
    - name: cluster1
      clusterID: cluster1
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

The webhook validates that both name and key are non-empty at admission time. Whether the Secret content is a valid kubeconfig that can reach the target cluster is checked by the ProtectionZone controller and reported in status.clusters[].status.

Usage

Inspecting controller RBAC errors

Controller permission errors appear in the manager pod logs on the TSR-Manager cluster. The manager pod runs a single container named manager:

# Find the manager pod in the zone namespace
oc get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane

# Stream logs
oc logs -n dr-<zone-name> <pod-name> -c manager -f

Look for lines containing forbidden, cannot, or is not permitted. These indicate the manager's ServiceAccount lacks a verb on a resource.

Inspecting node-agent RBAC errors on workload clusters

The DRBD node-agent runs as a DaemonSet on each worker node of both workload clusters. The DaemonSet object name follows the pattern <release>-site-recovery-workload-control-plane-agent. To read logs from the agent on a specific node:

# Step 1: find the pod on the target node
oc --kubeconfig <workload-kubeconfig> \
  get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}'

# Step 2: read logs from that pod
oc --kubeconfig <workload-kubeconfig> \
  logs -n trilio-site-recovery-system <pod-name> -f

Checking admission webhook rejections

When a kubectl apply or oc apply is rejected at admission, the error message comes from the webhook validator. Common patterns:

ProtectionZone rejected with "Linstor backend is not supported yet; use DRBD"

The storageBackendMode field was set to Linstor. Only DRBD is supported. Remove the field (the mutating webhook defaults it to DRBD) or set it explicitly:

spec:
  storageBackendMode: DRBD

ProtectionRequest rejected with "ReplicationProtocolA is not supported yet"

Protocol A is not supported. The only valid value is C. Either omit replicationProtocol (it defaults to C) or set it explicitly:

spec:
  replicationConfig:
    replicationProtocol: C
    rpo: 0

ProtectionRequest rejected with "rpo must be 0 when spec.replicationConfig.replicationProtocol is C"

Protocol C is synchronous with RPO=0. A non-zero rpo value contradicts the protocol. Set rpo: 0 or omit it (the schema default is 0).

RPOEvent or ReplicationGroupStatus rejected with "only the replication-monitor service account may create"

The monitorServiceAccountName field is set on the validator and the request was made by a different user. These objects are written exclusively by the replication-monitor controller. Do not create, update, or delete them manually. If you need to force-remove a stale record, contact Trilio support.

TestFailover rejected with "protectionGroupRef.namespace: must be set explicitly"

The protectionGroupRef.namespace field was omitted. Although the mutating webhook defaults it from the TestFailover's own namespace, if the webhook is unreachable (for example, due to a missing webhook-server-cert), the backstop validator rejects the request. Verify the webhook serving certificate exists and set the field explicitly:

spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-<zone-name>

ProtectionRequest rejected with a duplicate-VM error

Another ProtectionRequest in the same zone namespace already references the same VM. The webhook enforces one ProtectionRequest per VM per namespace, including requests that are Terminating. Wait for the previous request to finish its teardown before creating a new one. Re-protecting always requires a full unprotect and a full initial resync.

Checking ProtectionZone cluster reachability

If the TSR-Manager cluster cannot reach a workload cluster using the kubeconfig Secret, the zone condition reflects this:

oc get pz <zone-name> -n dr-<zone-name> -o jsonpath='{.status.clusters}'

Each cluster entry has a status field (Reachable, Unreachable, or Degraded) and a message. An Unreachable status means the kubeconfig Secret content is invalid, the cluster API server is not reachable from the TSR-Manager cluster on port 6443, or the credentials have expired. Update the Secret with a valid kubeconfig and the ProtectionZone controller will retry on its next reconcile.

Checking the zone-exclusivity lease

Each workload cluster allows only one ProtectionZone to claim it at a time. The claim is recorded as an annotation on the trilio-site-recovery-system Namespace:

oc --kubeconfig <workload-kubeconfig> \
  get namespace trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations}'

Look for the annotations siterecovery.trilio.io/owner-zone and siterecovery.trilio.io/owner-zone-namespace-uid. If these point to a different zone than the one you are deploying, the zone controller will refuse to install the workload stack on that cluster. Remove the old zone first before creating a new one for the same workload cluster.

Examples

Example 1 — Valid ProtectionZone with both kubeconfig Secrets

This manifest is admitted at creation. Both clusters use different clusterID slots and have non-empty kubeconfigSecretRef.name and .key:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod-dr
  namespace: dr-prod-dr
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: cluster1
      clusterID: cluster1
      kubeconfigSecretRef:
        name: primary-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      kubeconfigSecretRef:
        name: dr-kubeconfig
        key: kubeconfig

Example 2 — ProtectionZone rejected: duplicate clusterID slot

Both clusters claim cluster1. The ProtectionZone admission webhook rejects this with a clusterID duplicate error:

# This manifest is REJECTED at admission
spec:
  clusters:
    - name: primary
      clusterID: cluster1          # First cluster claims cluster1
      kubeconfigSecretRef:
        name: primary-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster1          # Duplicate — rejected
      kubeconfigSecretRef:
        name: dr-kubeconfig
        key: kubeconfig

Example 3 — ProtectionZone rejected: Linstor backend

# This manifest is REJECTED at admission
spec:
  storageBackendMode: Linstor     # Rejected: "Linstor backend is not supported yet; use DRBD"
  clusters: [...]

Example 4 — ProtectionRequest rejected: Protocol A

# This manifest is REJECTED at admission
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: vm-protect
  namespace: dr-prod-dr
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: app-ns
  replicationConfig:
    replicationProtocol: A        # Rejected: "ReplicationProtocolA is not supported yet"

Example 5 — ProtectionRequest admitted: Protocol C with RPO 0

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: vm-protect
  namespace: dr-prod-dr
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: app-ns
  replicationConfig:
    replicationProtocol: C
    rpo: 0

Example 6 — TestFailover rejected: missing protectionGroupRef.namespace

# This manifest is REJECTED at admission
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-01
  namespace: dr-prod-dr
spec:
  protectionGroupRef:
    name: pg-web-tier
    # namespace omitted — rejected by the validator backstop
  cleanupPolicy: Automatic

Example 7 — TestFailover admitted: fully specified

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-01
  namespace: dr-prod-dr
spec:
  protectionGroupRef:
    name: pg-web-tier
    namespace: dr-prod-dr
  cleanupPolicy: Automatic
  schedulingCheck: Fail

Example 8 — Collecting a diagnostic bundle including RBAC state

Use oc adm must-gather with the Site Recovery gather image against the TSR-Manager cluster. The gather tool collects RBAC objects, CRD state, and logs from all clusters in the DR estate:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  --dest-dir=./tsr-bundle

Replace <release-tag> with the pinned tag for your release. The resulting bundle under ./tsr-bundle contains ClusterRole, ClusterRoleBinding, Role, RoleBinding, and ServiceAccount objects for inspection.

Troubleshooting

Issue 1 — Controller pod is stuck Pending or fails to start on a workload cluster

Symptom: The workload control plane pod in trilio-site-recovery-system stays in Pending or enters CrashLoopBackOff.

Likely cause: The ServiceAccount does not have the privileged SCC on OpenShift, or the webhook-server-cert Secret is missing, causing the webhook server to fail TLS setup.

Fix:

  1. Check pod events: oc --kubeconfig <workload-kubeconfig> describe pod -n trilio-site-recovery-system <pod-name>
  2. Look for SCC denial or TLS errors.
  3. Confirm the SCC binding exists: oc --kubeconfig <workload-kubeconfig> get clusterrolebinding -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane
  4. Confirm webhook-server-cert exists: oc --kubeconfig <workload-kubeconfig> get secret webhook-server-cert -n trilio-site-recovery-system
  5. If the Secret is missing, create it with a valid TLS certificate for the webhook service DNS name and restart the pod.

Issue 2 — ProtectionZone stays in Provisioning with cluster status Unreachable

Symptom: oc get pz -n dr-<zone-name> shows STATUS: Provisioning. The zone's status.clusters[].status for one or both clusters is Unreachable.

Likely cause: The kubeconfig Secret content is invalid, the cluster API server is not reachable from the TSR-Manager cluster on TCP port 6443, or the credentials have expired.

Fix:

  1. Retrieve the status message: oc get pz <zone-name> -n dr-<zone-name> -o jsonpath='{.status.clusters}'
  2. Verify the Secret content: oc get secret <secret-name> -n dr-<zone-name> -o jsonpath='{.data.kubeconfig}' | base64 -d | oc --kubeconfig /dev/stdin cluster-info
  3. If the credentials are expired, update the Secret with fresh credentials. The ProtectionZone controller retries on the next reconcile.
  4. Verify TCP port 6443 is open from the TSR-Manager cluster nodes to the workload cluster API server.

Issue 3 — ProtectionRequest rejected with a forbidden error from the webhook

Symptom: oc apply of a ProtectionRequest returns a webhook forbidden error.

Likely cause:

  • replicationProtocol: A was specified (not supported; the webhook rejects it).
  • rpo is non-zero with replicationProtocol: C.
  • virtualMachine.namespace is empty (cannot be inferred from the TSR-Manager-tenant namespace).

Fix: Check the error message substring:

  • ReplicationProtocolA: change to replicationProtocol: C and rpo: 0.
  • rpo must be 0: set rpo: 0.
  • namespace: set spec.virtualMachine.namespace explicitly to the VM's namespace on the workload cluster.

Issue 4 — RPOEvent or ReplicationGroupStatus writes rejected with "only the replication-monitor service account may create"

Symptom: Manual oc apply of an RPOEvent or ReplicationGroupStatus is rejected at admission.

Likely cause: monitorServiceAccountName is configured and the request is not coming from that service account. These objects are written exclusively by the internal replication-monitor controller and must not be created or deleted manually.

Fix: Do not create, update, or delete RPOEvent or ReplicationGroupStatus objects directly. They are managed entirely by the replication-monitor. If a stale record is preventing teardown, contact Trilio support—the webhook exempts Kubernetes garbage-collection and namespace-controller identities, so namespace deletion will eventually clean up these objects.

Issue 5 — oc adm drain does not complete on a node hosting a protected VM

Symptom: oc adm drain <node> hangs or reports that it cannot evict pods on the node.

Likely cause: A protected VM is pinned to that node. Protected VMs are immovable for the lifetime of the protection: live migration and cold migration are denied by the webhook, and a drain that would evict the VM's anchor pod cannot complete while the protection is active.

Fix: Unprotect the VM (delete the ProtectionRequest for that VM and wait for teardown to complete) before draining the node. Re-protecting the VM after the drain requires a full resync.

Issue 6 — Node-agent DaemonSet pods fail with SCC errors on a workload cluster

Symptom: The DRBD node-agent pods on a workload cluster fail to start, and pod events mention unable to validate against any security context constraint.

Likely cause: The chart's SCC binding for the node-agent ServiceAccount is missing or references the wrong account name.

Fix:

  1. Find the node-agent DaemonSet: oc --kubeconfig <workload-kubeconfig> get daemonset -n trilio-site-recovery-system -l app.kubernetes.io/component=drbd-node-agent
  2. Note the serviceAccountName in the DaemonSet pod spec.
  3. Confirm a ClusterRoleBinding grants that service account the privileged SCC: oc --kubeconfig <workload-kubeconfig> get clusterrolebinding -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane -o yaml
  4. If the binding references a different ServiceAccount name, the zone was deployed with a non-default serviceAccount.agentName. Because customers do not run Helm directly, contact Trilio support to reconcile the mismatch by recreating the ProtectionZone.

Issue 7 — oc get drbdresource against the TSR-Manager cluster returns "server doesn't have a resource type"

Symptom: Running oc get dres or oc get drbdresource against the TSR-Manager cluster fails with server doesn't have a resource type.

Likely cause: The DRBDResource CRD is registered only on workload clusters, not on the TSR-Manager cluster. Running the command against the wrong cluster's context produces this error.

Fix: Switch to the workload cluster context:

oc --kubeconfig <workload-kubeconfig> get dres -A

DRBDResource CRs exist on the primary and DR workload clusters in the protected VM's namespace. They do not exist on the TSR-Manager cluster.

Issue 8 — Zone-exclusivity lease blocks a new ProtectionZone from deploying to a workload cluster

Symptom: The ProtectionZone controller reports that the workload cluster is already claimed by another zone, even though no zone appears to be active.

Likely cause: A previous ProtectionZone was deleted without fully releasing the zone-exclusivity lease on the trilio-site-recovery-system Namespace of the workload cluster.

Fix:

  1. Inspect the lease annotations: oc --kubeconfig <workload-kubeconfig> get namespace trilio-site-recovery-system -o jsonpath='{.metadata.annotations}'
  2. Check whether the owning zone referenced by siterecovery.trilio.io/owner-zone still exists on the TSR-Manager cluster.
  3. If the owning zone no longer exists, remove the annotations manually: oc --kubeconfig <workload-kubeconfig> annotate namespace trilio-site-recovery-system siterecovery.trilio.io/owner-zone- siterecovery.trilio.io/owner-zone-namespace-uid-
  4. The new ProtectionZone controller will claim the namespace on its next reconcile.