RBAC Issues

This page helps you diagnose and resolve RBAC-related failures in Trilio Site Recovery—permission denied errors when applying custom resources, service account misconfiguration, and cluster-scoped access problems that prevent the failover, protection, and replication controllers from reconciling state. Because Site Recovery operates across multiple OpenShift clusters (primary, DR, and quorum), RBAC issues can appear on any cluster and are often subtle: a controller may start successfully but silently fail to watch or patch resources on a remote cluster. Understanding which service accounts each controller uses and which API groups they require is essential for diagnosing these failures quickly.

Before working through this guide, ensure you have:

• OpenShift 4.14 or later on all clusters (primary, DR, and quorum)
• KubeVirt or OpenShift Virtualization (CNV) ≥ 1.0 installed on workload clusters
• kubectl or oc CLI with kubeconfig files for all clusters
• Cluster-admin or equivalent access on each cluster so you can inspect and modify ClusterRoles and ClusterRoleBindings
• The tsr-gather must-gather tool available, in case you need to collect a diagnostic bundle
• Helm ≥ 3.0 installed locally, to redeploy or upgrade chart-managed RBAC resources if needed
• Basic familiarity with Kubernetes RBAC (Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, and ServiceAccounts)

Site Recovery RBAC resources are installed automatically by the three Helm charts. If you are seeing permission errors, the most common root cause is a chart that was installed without the correct values, a service account that was manually deleted, or an upgrade that did not refresh ClusterRoleBindings. Re-applying the charts is the safest corrective action.

Step 1 — Re-apply the ProtectionZone controller chart on the quorum cluster

This chart must be installed first because it creates the ProtectionZone CRDs and admission webhook RBAC that the quorum control plane chart depends on.

helm upgrade --install site-recovery-protectionzone-controller \
  trilio/site-recovery-protectionzone-controller \
  --namespace dr-<deployment-name> \
  --create-namespace \
  --kubeconfig <path-to-quorum-kubeconfig>

Step 2 — Re-apply the quorum control plane chart on the quorum cluster

This deploys the failover-controller, protection-controller, pg-sync-controller, and test-failover-controller, together with their ServiceAccounts, ClusterRoles, and ClusterRoleBindings.

helm upgrade --install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  --namespace dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig> \
  -f quorum-values.yaml

Step 3 — Re-apply the workload control plane chart on each workload cluster

Run this command separately for the primary cluster and the DR cluster. The chart installs the drbd-node-agent DaemonSet, admission webhooks, and the RBAC required for VM protection and replication.

# Primary cluster
helm upgrade --install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  --namespace dr-<deployment-name> \
  --kubeconfig <path-to-primary-kubeconfig> \
  -f workload-values.yaml

# DR cluster
helm upgrade --install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  --namespace dr-<deployment-name> \
  --kubeconfig <path-to-dr-kubeconfig> \
  -f workload-values.yaml

Step 4 — Verify that ServiceAccounts exist after chart application

# Quorum cluster
oc get serviceaccounts -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

# Primary cluster
oc get serviceaccounts -n dr-<deployment-name> \
  --kubeconfig <path-to-primary-kubeconfig>

If any ServiceAccount is missing, the chart did not apply cleanly. Check for Helm errors and re-run the relevant upgrade command.

Site Recovery RBAC is chart-managed. You do not edit ClusterRoles or ClusterRoleBindings directly in normal operation—changes are applied by re-running the Helm charts with updated values. The key configuration knobs that affect RBAC behaviour are:

rbac.create (all charts) Default: true When true, the chart creates all ServiceAccounts, ClusterRoles, ClusterRoleBindings, and Roles needed by that component. Set to false only if you are supplying pre-existing RBAC resources manually (advanced use case). Leaving this as false after a chart upgrade without pre-creating the required resources is a common cause of permission errors.

serviceAccount.name (all charts) Default: auto-generated from the chart release name. Override this if your organisation requires a specific naming convention. The name you set here must match what appears in the RoleBinding's subjects field. A mismatch between the ServiceAccount name in the chart and the one bound in the ClusterRoleBinding is a frequent source of Forbidden errors.

serviceAccount.annotations (all charts) Default: {} Use this to attach an IAM role ARN or OpenShift service account token annotation if your clusters use workload identity. On OpenShift, the DRBD node agent DaemonSet needs access to host-level kernel interfaces; an incorrect annotation here causes the DaemonSet pods to start but immediately fail privilege checks.

Cross-cluster kubeconfig secrets The quorum control plane controllers (failover-controller, protection-controller, pg-sync-controller, test-failover-controller) reach the primary and DR API servers using kubeconfig credentials stored as Kubernetes Secrets in the DR namespace on the quorum cluster. These secrets are not managed by RBAC but must grant at minimum the permissions listed in the ClusterRoles created by the workload control plane chart. If you rotate cluster credentials, update these secrets and restart the quorum control plane deployment:

oc rollout restart deployment/site-recovery-quorum-control-plane \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

Webhook RBAC The ProtectionZone admission webhook runs on the quorum cluster and must be able to read ProtectionZone CRs cluster-wide. If the webhook's ServiceAccount loses its ClusterRoleBinding, the webhook will return 403 Forbidden for all ProtectionGroup and ProtectionRequest admission requests, blocking all protection operations. Re-applying the site-recovery-protectionzone-controller chart restores this binding.

Use the following patterns when investigating and resolving RBAC issues in your Site Recovery deployment.

Check whether a controller can access a resource it needs

Use oc auth can-i with the service account identity to simulate permission checks without restarting anything:

# Can the quorum control plane service account list FailoverRequests?
oc auth can-i list failoverrequests.siterecovery.trilio.io \
  --as=system:serviceaccount:dr-<deployment-name>:<service-account-name> \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

Repeat this pattern for every CRD kind relevant to the controller you are debugging. The full list of kinds is: FailoverRequest, ProtectionRequest, ProtectionGroup, TestFailover, DRBDReplicationPolicy, DRBDResource, RPOEvent, ReplicationGroupStatus, ProtectionZone.

Inspect controller logs for Forbidden errors

The controllers log 403 Forbidden responses as errors. Because the quorum cluster hosts multiple reconcilers inside a single deployment, filter by controller name:

# All Forbidden errors from any controller on the quorum cluster
oc logs deployment/site-recovery-quorum-control-plane \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig> \
  | grep -i 'forbidden\|permission denied\|RBAC'

# Forbidden errors from the drbd-node-agent on the primary cluster
oc logs daemonset/site-recovery-workload-control-plane \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-primary-kubeconfig> \
  | grep -i 'forbidden\|permission denied'

Describe a ClusterRoleBinding to verify it targets the correct service account

oc get clusterrolebindings \
  --kubeconfig <path-to-quorum-kubeconfig> \
  | grep site-recovery

oc describe clusterrolebinding <binding-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

Confirm that subjects[].namespace matches the DR namespace (for example, dr-production) and subjects[].name matches the ServiceAccount name used by the chart.

Collect a diagnostic bundle when you cannot identify the root cause

tsr-gather --kubeconfig <path-to-quorum-kubeconfig> \
  --output-dir ./tsr-bundle

The bundle includes CRD state, controller logs, and configuration from all clusters. Share it with Trilio support without risk of exposing secrets—tsr-gather does not capture Secret objects.

Example 1 — Verify that the failover-controller service account can create FailoverRequests

This confirms the controller has the write access it needs to update FailoverRequest status after orchestrating a failover.

oc auth can-i update failoverrequests/status \
  --as=system:serviceaccount:dr-production:site-recovery-quorum-control-plane \
  -n dr-production \
  --kubeconfig /etc/kube/quorum-kubeconfig

Expected output:

yes

If the output is no, the ClusterRole is missing the update verb on failoverrequests/status. Re-apply the quorum control plane chart.

Example 2 — Check that the drbd-node-agent can read DRBDResource objects on the primary cluster

oc auth can-i get drbdresources.siterecovery.trilio.io \
  --as=system:serviceaccount:dr-production:site-recovery-workload-control-plane \
  -n dr-production \
  --kubeconfig /etc/kube/primary-kubeconfig

Expected output:

yes

Example 3 — List all ClusterRoleBindings installed by Site Recovery charts

Use this to confirm that bindings were not accidentally deleted after an upgrade.

oc get clusterrolebindings \
  -l app.kubernetes.io/managed-by=Helm \
  --kubeconfig /etc/kube/quorum-kubeconfig \
  | grep site-recovery

Expected output (names are illustrative; exact names are set by your Helm release):

site-recovery-quorum-control-plane-crb   ClusterRole/site-recovery-quorum-control-plane   10d
site-recovery-protectionzone-controller-crb  ClusterRole/site-recovery-protectionzone-controller  10d

If this list is empty, the chart was installed with --set rbac.create=false or the bindings were manually deleted. Re-apply the charts without that override.

Example 4 — Re-create a missing ClusterRoleBinding by re-running the chart

If you find a missing binding, re-running the chart is safer than writing the YAML manually because it ensures the Role, ServiceAccount, and Binding all stay in sync.

helm upgrade --install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  --namespace dr-production \
  --kubeconfig /etc/kube/quorum-kubeconfig \
  -f quorum-values.yaml

Expected outcome: Helm reports Release "site-recovery-quorum-control-plane" has been upgraded. The binding reappears immediately.

Example 5 — Confirm the ProtectionZone webhook service account has cluster-wide read on ProtectionZone CRs

This must be checked on the quorum cluster. A missing permission here causes every ProtectionGroup admission request to be rejected with a webhook 403 error.

oc auth can-i get protectionzones.siterecovery.trilio.io \
  --as=system:serviceaccount:dr-production:site-recovery-protectionzone-controller \
  --all-namespaces \
  --kubeconfig /etc/kube/quorum-kubeconfig

Expected output:

yes

Use the format below for each issue: Symptom → Likely cause → Fix.

Issue: Error from server (Forbidden) when applying a ProtectionGroup, ProtectionRequest, or FailoverRequest CR

Symptom: Running oc apply -f <manifest>.yaml returns a message such as failoverrequests.siterecovery.trilio.io is forbidden: User "<user>" cannot create resource.

Likely cause: The OpenShift user or service account you are authenticated as does not have create access to the relevant CRD in the target namespace. Site Recovery CRDs are cluster-scoped RBAC resources; a plain edit or view role in the namespace is not sufficient.

Fix: Ask your cluster administrator to grant a ClusterRole that includes create, get, list, patch, update, and watch verbs on the siterecovery.trilio.io API group in the DR namespace, or log in as a user with cluster-admin during testing. Verify your access with:

oc auth can-i create failoverrequests.siterecovery.trilio.io \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

Issue: The failover-controller or protection-controller pod is running but FailoverRequest or ProtectionRequest objects stay in Pending indefinitely

Symptom: A CR is created successfully but its .status.phase never advances. Controller logs show repeated 403 Forbidden or 401 Unauthorized errors when trying to reach the primary or DR cluster API server.

Likely cause: The kubeconfig Secret that the quorum controller uses to reach the workload clusters contains expired or revoked credentials. This is not a ClusterRole issue—it is an authentication failure at the remote cluster.

Fix: Rotate the kubeconfig Secret in the DR namespace on the quorum cluster with fresh credentials, then restart the quorum control plane deployment:

# Replace the secret content with refreshed kubeconfig
oc create secret generic <kubeconfig-secret-name> \
  --from-file=kubeconfig=<new-kubeconfig-path> \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig> \
  --dry-run=client -o yaml | oc apply -f -

# Restart the controller to pick up the new secret
oc rollout restart deployment/site-recovery-quorum-control-plane \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

Issue: drbd-node-agent DaemonSet pods fail to start with forbidden: unable to validate against any security context constraint

Symptom: On OpenShift, drbd-node-agent pods are in Pending or Error state. The pod events show an SCC (Security Context Constraint) denial.

Likely cause: The workload control plane chart's ServiceAccount was not granted the required SCC (typically privileged) because the chart was applied without cluster-admin rights, or the SCC binding was removed during a cluster upgrade.

Fix: Grant the privileged SCC to the drbd-node-agent ServiceAccount, then re-apply the chart:

# Grant the privileged SCC to the workload control plane service account
oc adm policy add-scc-to-user privileged \
  -z site-recovery-workload-control-plane \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-primary-kubeconfig>

# Re-apply the chart to ensure all RBAC is in sync
helm upgrade --install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  --namespace dr-<deployment-name> \
  --kubeconfig <path-to-primary-kubeconfig> \
  -f workload-values.yaml

Repeat for the DR cluster.

Issue: ProtectionGroup or ProtectionRequest admission webhook returns 403 Forbidden for all requests, blocking protection operations entirely

Symptom: Every oc apply of a ProtectionGroup or ProtectionRequest fails with a webhook-generated error. The error message references the ProtectionZone admission webhook.

Likely cause: The ProtectionZone controller's ClusterRoleBinding was deleted or the ServiceAccount was removed, so the webhook cannot authenticate its own read requests against the API server.

Fix: Re-apply the ProtectionZone controller chart to restore the ServiceAccount and ClusterRoleBinding:

helm upgrade --install site-recovery-protectionzone-controller \
  trilio/site-recovery-protectionzone-controller \
  --namespace dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

After the chart applies, verify the webhook pod is running and re-try your original CR apply.

Issue: oc auth can-i returns yes for all verbs but controllers still log Forbidden errors

Symptom: Manual oc auth can-i checks pass for the service account in question, but controller logs continue to show 403 Forbidden against specific sub-resources (for example, failoverrequests/status or protectiongroups/finalizers).

Likely cause: OpenShift RBAC sub-resources require explicit grants. A ClusterRole that allows update on failoverrequests does not automatically allow update on failoverrequests/status. If the chart was customised or partially applied, sub-resource entries may be missing.

Fix: Check the exact sub-resource that is failing in the controller log, then verify it explicitly:

oc auth can-i update failoverrequests/status \
  --as=system:serviceaccount:dr-<deployment-name>:<service-account-name> \
  -n dr-<deployment-name> \
  --kubeconfig <path-to-quorum-kubeconfig>

If the result is no, re-apply the quorum control plane chart without any rbac.create=false overrides to restore the full ClusterRole definition.

Issue: After adding a new DR deployment (multi-tenant scenario), controllers in the new namespace cannot access resources

Symptom: A second DR deployment (for example, dr-tenant-b) is created on the same quorum cluster, but its controllers fail with Forbidden errors even though the first deployment (dr-tenant-a) works correctly.

Likely cause: ClusterRoleBindings from the first chart installation scope subjects to the original namespace (dr-tenant-a). The second Helm release in namespace dr-tenant-b creates its own ServiceAccount but may share or conflict with existing ClusterRoleBindings if release names collide.

Fix: Use a distinct Helm release name for each deployment so that each gets its own isolated ServiceAccount and ClusterRoleBindings:

# Each release name is unique per DR deployment
helm upgrade --install site-recovery-quorum-cp-tenant-b \
  trilio/site-recovery-quorum-control-plane \
  --namespace dr-tenant-b \
  --create-namespace \
  --kubeconfig <path-to-quorum-kubeconfig> \
  -f tenant-b-values.yaml

Verify that separate ClusterRoleBindings exist for each namespace after installation:

oc get clusterrolebindings \
  --kubeconfig <path-to-quorum-kubeconfig> \
  | grep site-recovery