RBAC Issues

This page helps you diagnose and resolve RBAC-related failures in Trilio Site Recovery (TSR), including Permission denied errors, service account misconfiguration, and access problems that prevent the quorum or workload control planes from operating correctly. Because TSR is a Kubernetes operator whose controllers run across three clusters — quorum, primary, and DR — RBAC problems can surface in multiple places: in admission webhook rejections, in controller logs on the quorum cluster, or in the DRBD node agent on workload clusters. Understanding where each component runs and which service account it uses is essential for diagnosing these failures quickly.

Before troubleshooting RBAC issues you need:

• oc CLI (required — kubectl-only operation is not a supported mode for OpenShift-specific paths)
• Kubeconfig files for all three clusters: quorum, primary, and DR
• Cluster-admin or equivalent access on all three clusters, sufficient to read Roles, ClusterRoles, and their bindings
• Trilio Site Recovery installed via OLM in the trilio-site-recovery-system namespace on the quorum cluster, with a ProtectionZone CR in a dr-<name> namespace
• OpenShift 4.17 or later on all three clusters (4.17 is the supported floor for the quorum cluster and both workload clusters)
• Familiarity with the two-namespace split: quorum components and all quorum CRs (ProtectionZone, ProtectionRequest, ProtectionGroup, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent, DRBDReplicationPolicy) live in the dr-<name> namespace; workload components (DRBDResource, the DRBD node agent DaemonSet) live in trilio-site-recovery-system on each workload cluster

No additional RBAC installation steps are required for a correctly deployed TSR environment. The OLM bundle and the Helm charts embedded in the operator binary create all necessary ServiceAccounts, Roles, ClusterRoles, and bindings automatically when the ProtectionZone controller reconciles a zone to Available. The steps below describe how to verify that the automatically created RBAC is intact, and how to repair it if it has drifted.

Step 1 — Confirm the operator bundle installed correctly on the quorum cluster

The OLM Subscription deploys the protection-zone-controller Deployment in trilio-site-recovery-system. Verify it is running:

oc --context=<quorum-context> get deployment \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller

Expected output shows 1/1 READY. If the pod is stuck in Pending or CrashLoopBackOff, check events:

oc --context=<quorum-context> describe deployment \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller

Step 2 — Confirm the quorum control plane Deployment is running in the zone namespace

Once the ProtectionZone reaches Available, the embedded site-recovery-quorum-control-plane chart is deployed into the dr-<name> namespace. The manager runs a single container named manager:

oc --context=<quorum-context> get deployment \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

Expected output shows 1/1 READY.

Step 3 — Confirm the workload control plane is running on each workload cluster

The site-recovery-workload-control-plane chart is deployed by the ProtectionZone controller to trilio-site-recovery-system on both primary and DR clusters:

# Primary cluster
oc --context=<primary-context> get deployment \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

# DR cluster
oc --context=<dr-context> get deployment \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

Step 4 — Confirm the DRBD node agent DaemonSet is running on each workload cluster

The node agent DaemonSet name follows the pattern <release>-site-recovery-workload-control-plane-agent. List it by component label:

oc --context=<primary-context> get daemonset \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent

If any pods are Not Ready, see the troubleshooting section for node-agent-specific RBAC and SCC failures.

Step 5 — Verify the webhook serving certificate Secret exists

Webhook serving certificates are not managed by the charts. The Secret named by webhook.certSecretName (default: webhook-server-cert) must be created out-of-band in each release namespace before the webhook server can start. If the Secret is missing, webhook pods fail to start and all admission requests are rejected:

# Quorum zone namespace
oc --context=<quorum-context> get secret webhook-server-cert -n dr-<name>

# Workload clusters
oc --context=<primary-context> get secret webhook-server-cert -n trilio-site-recovery-system
oc --context=<dr-context> get secret webhook-server-cert -n trilio-site-recovery-system

If the Secret is absent, create it with a TLS certificate and key valid for the webhook Service's DNS name before proceeding.

RBAC-relevant configuration is controlled by chart values. All values are set internally by the ProtectionZone controller; you do not run helm install or helm upgrade directly. The table below documents the values that affect RBAC so you can interpret controller logs and understand what is deployed.

rbac section

serviceAccount section

webhook section (relevant to admission RBAC)

monitorServiceAccountName (quorum chart)

OpenShift SCC binding (workload clusters)

On OpenShift, the workload chart binds its ServiceAccount to the privileged SCC. This binding is created automatically by the chart (rbac.create: true). The DRBD node agent requires privileged access to load the DRBD kernel module and manage block devices. If this binding is absent or has been removed, node agent pods will fail to start with SCC admission errors — see the troubleshooting section.

Cross-cluster access: kubeconfig Secrets, not remote RBAC

The quorum controller accesses the primary and DR clusters by mounting kubeconfig Secrets, not through in-cluster RBAC on the remote cluster. The ProtectionZone controller reads the kubeconfig from spec.clusters[].kubeconfigSecretRef. If you see cross-cluster access errors, verify the Secret exists and contains a valid kubeconfig — there is no remote ClusterRole or ClusterRoleBinding to inspect.

# Verify the kubeconfig Secret exists in the zone namespace
oc --context=<quorum-context> get secret <secret-name> -n dr-<name>

Checking which service account a TSR pod is running as

All TSR controller pods use the ServiceAccount created by the chart. To identify it:

# Quorum control plane
oc --context=<quorum-context> get pod \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

# Workload control plane (primary)
oc --context=<primary-context> get pod \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

# DRBD node agent (primary)
oc --context=<primary-context> get pod \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

Checking RBAC permissions for a TSR service account

Once you have the ServiceAccount name, use oc auth can-i to verify it holds the expected permissions. The custom resource group for all TSR CRDs is siterecovery.trilio.io.

# Example: verify the quorum controller can list ProtectionRequests
oc --context=<quorum-context> auth can-i list protectionrequests \
  --as=system:serviceaccount:dr-<name>:<sa-name> \
  -n dr-<name>

# Verify it can update ProtectionGroup status
oc --context=<quorum-context> auth can-i update protectiongroups/status \
  --as=system:serviceaccount:dr-<name>:<sa-name> \
  -n dr-<name>

Reading admission webhook rejection messages

When a TSR admission webhook rejects a CR, the error is returned directly to oc apply or oc create. Common rejections to recognise:

• Linstor backend is not supported yet; use DRBD — storageBackendMode: Linstor was specified. Change it to DRBD or omit it (the defaulter sets DRBD automatically).
• ReplicationProtocolA is not supported yet — replicationProtocol: A was specified on a ProtectionRequest. Only Protocol C is supported in Tech Preview.
• rpo must be 0 when spec.replicationConfig.replicationProtocol is C — Protocol C is synchronous (RPO=0); a non-zero rpo contradicts this. Set rpo: 0 or omit it.
• protectionGroupRef.namespace must be set explicitly — ProtectionRequest, TestFailover, ReplicationGroupStatus, or RPOEvent was submitted without a protectionGroupRef.namespace. This field cannot be inferred and must be set explicitly.
• only the replication-monitor service account may create RPOEvents — monitorServiceAccountName is configured and the request came from a different identity. Only the replication-monitor ServiceAccount may write RPOEvent and ReplicationGroupStatus objects.
• kubeconfigSecretRef name/key must not be empty — A ProtectionZone cluster entry has an empty kubeconfigSecretRef.name or .key. Both are required.
• clusterID duplicate — Both clusters in the ProtectionZone claim the same slot (cluster1 or cluster2). Each cluster must have a distinct clusterID.

Checking whether a ProtectionRequest is blocked by a RoleConflict

If a ProtectionRequest shows Ready=False with reason RoleConflict, the DRBD volume role on the workload cluster does not match what the controller expects. This is a status condition on the ProtectionRequest, not an RBAC error — but it is commonly confused with one.

oc --context=<quorum-context> get pr <pr-name> -n dr-<name> -o yaml | grep -A5 'RoleConflict'

Checking logs for RBAC errors on the quorum controller

The quorum Deployment runs a single container named manager. To stream its logs:

oc --context=<quorum-context> logs -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager --follow

Filter for RBAC or permission errors:

oc --context=<quorum-context> logs -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -c manager | grep -iE 'forbidden|permission|rbac|unauthorized'

Collecting a full diagnostic bundle

For persistent or multi-cluster RBAC issues, collect a diagnostic bundle using tsr-gather:

oc --context=<quorum-context> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --dest-dir=./tsr-gather-output

This collects logs, CRD state, and RBAC configuration from all clusters in the DR estate into a single bundle.

Example 1 — Verifying RBAC for the quorum controller ServiceAccount

This example confirms the quorum controller's ServiceAccount can perform the operations it needs on TSR custom resources.

# Get the ServiceAccount name used by the quorum control-plane pod
SA=$(oc --context=<quorum-context> get pod \
  -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -o jsonpath='{.items[0].spec.serviceAccountName}')

echo "Quorum controller ServiceAccount: $SA"

# Verify key permissions in the zone namespace
for VERB in get list watch create update patch delete; do
  RESULT=$(oc --context=<quorum-context> auth can-i $VERB protectionrequests \
    --as=system:serviceaccount:dr-<name>:$SA \
    -n dr-<name>)
  echo "  $VERB protectionrequests: $RESULT"
done

Expected output (all yes):

Quorum controller ServiceAccount: <release>-site-recovery-quorum-control-plane
  get protectionrequests: yes
  list protectionrequests: yes
  watch protectionrequests: yes
  create protectionrequests: yes
  update protectionrequests: yes
  patch protectionrequests: yes
  delete protectionrequests: yes

If any verb returns no, the ClusterRole or RoleBinding created by the chart has drifted. Because the ProtectionZone controller re-applies chart CRDs and resources on every reconcile using server-side-apply with field manager site-recovery-pz-controller, re-reconciling the ProtectionZone (by touching its annotation, for example) will restore the bindings.

Example 2 — Diagnosing an SCC admission failure for the DRBD node agent on a workload cluster

Symptom: node agent pods are stuck in Pending with an event like unable to validate against any security context constraint.

# Identify the failing pod on the primary cluster
oc --context=<primary-context> get pod \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent

# Read the event that describes the SCC rejection
oc --context=<primary-context> describe pod \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent \
  | grep -A5 'Events:'

Expected event when SCC binding is missing:

Warning  Failed  ...  unable to validate against any security context constraint:
         [provider "privileged": Forbidden: not usable by user or serviceaccount ...]

Verify the SCC binding exists:

SA=$(oc --context=<primary-context> get pod \
  -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].spec.serviceAccountName}')

oc --context=<primary-context> get clusterrolebinding \
  -o json | jq --arg sa "$SA" --arg ns "trilio-site-recovery-system" \
  '.items[] | select(.subjects[]? | .kind=="ServiceAccount" and .name==$sa and .namespace==$ns) | .metadata.name'

The workload chart binds its ServiceAccount to the privileged SCC automatically when rbac.create: true. If the binding is absent, re-reconcile the ProtectionZone or contact Trilio support with the must-gather bundle.

Example 3 — Identifying a ProtectionRequest rejected by the admission webhook for Protocol A

cat <<EOF | oc --context=<quorum-context> apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: pr-example
  namespace: dr-<name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: my-vm
    namespace: my-app-ns
  replicationConfig:
    replicationProtocol: A
    rpo: 60
EOF

Expected rejection:

Error from server: error when creating "STDIN": admission webhook
"vprotectionrequest-v1alpha1.siterecovery.trilio.io" denied the request:
spec.replicationConfig.replicationProtocol: Invalid value: "A":
ReplicationProtocolA is not supported yet

Fix: remove replicationProtocol (it defaults to C) or set it explicitly to C, and set rpo: 0:

spec:
  sourceCluster: cluster1
  virtualMachine:
    name: my-vm
    namespace: my-app-ns
  replicationConfig:
    replicationProtocol: C
    rpo: 0

Example 4 — Verifying the kubeconfig Secret that grants the quorum controller access to workload clusters

# List kubeconfig Secrets referenced by the ProtectionZone
oc --context=<quorum-context> get pz -n dr-<name> -o jsonpath=\
  '{range .items[0].spec.clusters[*]}{.name}{"\t"}{.kubeconfigSecretRef.name}{"\t"}{.kubeconfigSecretRef.key}{"\n"}{end}'

Expected output:

cluster1    cluster1-kubeconfig    kubeconfig
cluster2    cluster2-kubeconfig    kubeconfig

Verify each Secret exists and contains the key:

oc --context=<quorum-context> get secret cluster1-kubeconfig \
  -n dr-<name> -o jsonpath='{.data.kubeconfig}' | base64 -d | head -5

If the Secret is missing or the key is absent, create or update it and then re-reconcile the ProtectionZone (the controller will re-read it on the next reconcile).

Use a consistent format for each issue: Symptom, Likely cause, Fix.

Issue 1 — forbidden: not usable by user or serviceaccount on node agent pods

Symptom: DRBD node agent pods on a workload cluster are stuck in Pending. Events show an SCC admission error: unable to validate against any security context constraint.

Likely cause: The workload chart's ServiceAccount is not bound to the privileged SCC. This binding is created automatically by the chart when rbac.create: true. It can be removed accidentally by a cluster administrator or by a policy enforcement tool.

Fix: Check whether the SCC binding exists (see Example 2 above). If it is missing, re-reconcile the ProtectionZone CR on the quorum cluster by patching a mutable annotation — the ProtectionZone controller re-applies the workload chart including RBAC on every reconcile. If the binding cannot be restored automatically, contact Trilio support with a must-gather bundle.

Issue 2 — kubectl get drbdresource against the quorum cluster returns server doesn't have a resource type

Symptom: Running oc --context=<quorum-context> get dres -n dr-<name> returns error: the server doesn't have a resource type "drbdresources".

Likely cause: DRBDResource CRDs are registered only on workload clusters, not on the quorum cluster. This is by design: DRBDResource CRs live in the protected VM's namespace on each workload cluster. Running the command against the quorum cluster will always return this error — it is not an RBAC or permissions problem.

Fix: Run the command against the correct cluster:

oc --context=<primary-context> get dres -n <vm-namespace>

Issue 3 — Webhook admission rejected with only the replication-monitor service account may create RPOEvents

Symptom: An attempt to create or delete an RPOEvent or ReplicationGroupStatus object fails with a Forbidden error referencing replication-monitor.

Likely cause: monitorServiceAccountName is set in the quorum chart values, and the request is not coming from that service account. The webhook enforces that only the replication-monitor service account may write these audit-trail objects. Manual creation or deletion by an administrator is blocked by design.

Fix: Do not create or delete RPOEvent and ReplicationGroupStatus objects manually. They are managed exclusively by the replication-monitor reconciler. If you need to inspect them:

# List RPOEvents for a Protection Group (filter by label — field selectors on spec.* are not supported)
oc --context=<quorum-context> get rpoe -n dr-<name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

# List ReplicationGroupStatuses
oc --context=<quorum-context> get rgs -n dr-<name>

If the replication-monitor itself cannot write these objects, verify monitorServiceAccountName matches the fully-qualified service account username and that the ServiceAccount exists in the zone namespace.

Issue 4 — protectionGroupRef.namespace must be set explicitly rejection

Symptom: Creating a ProtectionRequest, TestFailover, ReplicationGroupStatus, or RPOEvent fails with a webhook error about protectionGroupRef.namespace.

Likely cause: The protectionGroupRef.namespace field was omitted. Although the mutating webhook defaults this field from the CR's own namespace when possible, ProtectionRequest has no mutating webhook and the field is always required there. For TestFailover, ReplicationGroupStatus, and RPOEvent, the defaulter fills the namespace from the CR's own namespace only when protectionGroupRef.name is non-empty.

Fix: Always set protectionGroupRef.namespace explicitly. The namespace must be the namespace on the quorum cluster where the ProtectionGroup lives (typically the same dr-<name> tenant namespace), not the VM's application namespace on the workload cluster:

spec:
  protectionGroupRef:
    name: my-pg
    namespace: dr-<name>   # quorum-side tenant namespace

Issue 5 — oc get pz or oc get pr returns empty against the wrong cluster or namespace

Symptom: oc get pz -n trilio-site-recovery-system returns empty, or oc get pr against a workload cluster returns server doesn't have a resource type.

Likely cause: The namespace split is not being respected. Quorum-side CRs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent, DRBDReplicationPolicy) live in the dr-<name> namespace on the quorum cluster. Workload-side CRs (DRBDResource) live in the VM's namespace on each workload cluster. Using trilio-site-recovery-system for quorum queries or querying the quorum cluster for DRBDResource objects will always return nothing.

Fix: Target the correct cluster and namespace:

# Quorum-side CRs — quorum cluster, dr-<name> namespace
oc --context=<quorum-context> get pz -n dr-<name>
oc --context=<quorum-context> get pr -n dr-<name>
oc --context=<quorum-context> get pg -n dr-<name>

# Workload-side CRs — workload cluster, VM application namespace
oc --context=<primary-context> get dres -n <vm-namespace>

Issue 6 — ProtectionZone shows Unreachable for a workload cluster

Symptom: oc get pz -n dr-<name> shows one or both workload clusters with status: Unreachable. The ProtectionZone controller cannot reach the cluster's API server.

Likely cause: The kubeconfig Secret referenced by spec.clusters[].kubeconfigSecretRef is missing, contains an invalid kubeconfig, or the credentials in it have expired. The quorum controller accesses workload clusters exclusively through these kubeconfig Secrets — there is no RBAC on the remote cluster that you can fix directly.

Fix:

1. Verify the Secret exists and the key is present:

   Copy

   oc --context=<quorum-context> get secret <secret-name> -n dr-<name> \
     -o jsonpath='{.data.<key>}' | base64 -d | oc --kubeconfig=/dev/stdin cluster-info
   

2. If the credentials have expired, rotate the kubeconfig and update the Secret:

   Copy

   oc --context=<quorum-context> create secret generic <secret-name> \
     --from-file=kubeconfig=/path/to/new-kubeconfig \
     -n dr-<name> \
     --dry-run=client -o yaml | oc --context=<quorum-context> apply -f -
   

3. The ProtectionZone controller re-reads the Secret on the next reconcile. Verify the zone recovers:

   Copy

   oc --context=<quorum-context> get pz -n dr-<name> -w
   

Issue 7 — Quorum manager logs show forbidden errors for a TSR resource

Symptom: The quorum controller logs contain repeated errors like "verb":"update","resource":"protectionrequests","reason":"Forbidden".

Likely cause: The ClusterRole or RoleBinding created by the site-recovery-quorum-control-plane chart has been removed or modified by an out-of-band change. Because the field manager site-recovery-pz-controller applies chart resources with ForceOwnership on every reconcile, any out-of-band edit to a chart-owned resource will be reverted within one reconcile cycle — but a deletion is not automatically recreated until the next ProtectionZone reconcile.

Fix: Trigger a ProtectionZone reconcile by adding or updating a mutable annotation:

oc --context=<quorum-context> annotate pz <zone-name> -n dr-<name> \
  kubectl.kubernetes.io/last-applied-configuration- --overwrite

This causes the controller to re-apply all chart resources including RBAC. Monitor the quorum manager logs to confirm the forbidden errors stop. If they persist after a full reconcile, collect a must-gather bundle:

oc --context=<quorum-context> adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather \
  --dest-dir=./tsr-gather-output