--- title: RBAC Issues product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/troubleshooting-rbac --- # RBAC Issues _Permission denied errors and service account misconfiguration affecting recovery operations_ ## Overview This page helps you diagnose and resolve permission-related failures that prevent Site Recovery controllers, the Site Manager UI, and `pgctl` from operating correctly across the quorum, primary, and DR clusters. RBAC misconfigurations are among the most common causes of stalled FailoverRequests, failed ProtectionRequests, and controllers that loop in error states without clear messages. Because Site Recovery controllers run in-cluster and reconcile state continuously, a missing ClusterRole binding or an expired service account token will silently block recovery operations at exactly the moment you need them most. ## Prerequisites Before working through this guide, ensure you have: - `kubectl` configured with administrative access to the quorum, primary, and DR clusters - Kubeconfig files for all three clusters (or two clusters minimum for DRBD Operator deployments) - Sufficient privileges to inspect and create `ClusterRole`, `ClusterRoleBinding`, `Role`, `RoleBinding`, and `ServiceAccount` resources on each cluster - Site Recovery deployed via the standard Ansible playbooks (the `replication-monitor` and other agents are deployed automatically as part of this process) - Kubernetes ≥ 1.28 on all clusters - Familiarity with Kubernetes RBAC concepts: service accounts, role bindings, and token review ## Installation No additional software installation is required for RBAC troubleshooting. All steps use `kubectl` and, optionally, `pgctl` and `quorum-deployments.sh`, which are installed as part of the standard Site Recovery deployment. If `pgctl` is not available on your PATH, locate it in the deployment package and ensure it is executable: ```bash chmod +x /usr/local/bin/pgctl pgctl version ``` If `quorum-deployments.sh` is not on your PATH: ```bash bash /path/to/quorum-deployments.sh ``` The interactive menu includes a **health check** option that performs automated RBAC validation across all registered deployments. ## Configuration Site Recovery uses Kubernetes RBAC to control which service accounts can read and write each CRD, and uses OpenShift OAuth token passthrough for the Site Manager UI. The key RBAC surface areas are: ### Controller service accounts (quorum cluster) Each DR deployment runs in its own namespace (`dr-` on the quorum cluster). The controllers in that namespace — `failover-controller`, `protection-controller`, `pg-sync-controller` — each run under a dedicated service account. These service accounts require: | Resource | Verbs | |---|---| | `FailoverRequest` | `get`, `list`, `watch`, `update`, `patch` | | `ProtectionRequest` | `get`, `list`, `watch`, `update`, `patch` | | `ProtectionGroup` | `get`, `list`, `watch`, `update`, `patch` | | `DRBDVolume` | `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` | | `DRBDReplicationPolicy` | `get`, `list`, `watch` | | `RPOEvent` | `get`, `list`, `watch`, `create`, `update`, `patch` | | `ReplicationGroupStatus` | `get`, `list`, `watch`, `create`, `update`, `patch` | | `events` | `create`, `patch` | | `secrets` | `get`, `list` (for kubeconfig access) | ### Controller service accounts (primary and DR clusters) The `protection-group-controller` and `test-failover-controller` run on the primary and DR clusters and require access to: | Resource | Verbs | |---|---| | `ProtectionGroup` | `get`, `list`, `watch`, `update`, `patch` | | `TestFailover` | `get`, `list`, `watch`, `update`, `patch` | | `VirtualMachine` (KubeVirt) | `get`, `list`, `watch`, `update`, `patch` | | `PersistentVolumeClaim` | `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` | | `VolumeSnapshot` | `get`, `list`, `watch`, `create`, `delete` | ### Site Manager UI token passthrough The Site Manager UI uses a two-tier authentication model: - **Quorum cluster API calls**: forwarded using the authenticated user's OpenShift OAuth bearer token. Kubernetes RBAC is enforced on the quorum cluster for every request the UI makes on behalf of the user. - **Workload cluster calls (primary and DR)**: the API backend uses stored kubeconfigs from the deployment namespace. Access to a given deployment is gated by a `SelfSubjectAccessReview` check — the user's token is tested for `get` on `secrets` in the `dr-` namespace before workload cluster data is returned. This means a user who cannot `get secrets` in the relevant `dr-` namespace will see cluster health and VM data as unavailable in the UI, even if the underlying clusters are healthy. ### `replication-monitor` service account The `replication-monitor` agent, deployed automatically by the Ansible playbooks, writes `RPOEvent` and `ReplicationGroupStatus` resources. Its service account requires `create` and `update` on both CRDs in the deployment namespace. A missing binding here causes replication health data to stop updating without any controller error being surfaced. ## Usage Use the following patterns to systematically identify and fix RBAC issues. ### Check controller pod logs for permission errors On the quorum cluster, inspect the logs for each controller in the affected deployment namespace: ```bash # Replace with your deployment, e.g. dr-production kubectl logs -n dr- -l app=failover-controller --tail=100 | grep -i "forbidden\|rbac\|permission\|unauthorized" kubectl logs -n dr- -l app=protection-controller --tail=100 | grep -i "forbidden\|rbac\|permission\|unauthorized" kubectl logs -n dr- -l app=pg-sync-controller --tail=100 | grep -i "forbidden\|rbac\|permission\|unauthorized" ``` On the primary and DR clusters, inspect the per-cluster controllers: ```bash kubectl --kubeconfig /path/to/primary-kubeconfig logs -n site-recovery -l app=protection-group-controller --tail=100 | grep -i "forbidden\|rbac" kubectl --kubeconfig /path/to/dr-kubeconfig logs -n site-recovery -l app=test-failover-controller --tail=100 | grep -i "forbidden\|rbac" ``` ### Verify service account bindings List all role bindings in a deployment namespace on the quorum cluster: ```bash kubectl get rolebindings,clusterrolebindings -n dr- -o wide ``` Confirm the expected service accounts are present: ```bash kubectl get serviceaccounts -n dr- ``` ### Test specific permissions with `kubectl auth can-i` Impersonate a controller service account to verify its effective permissions: ```bash # Check if the failover-controller service account can update FailoverRequests kubectl auth can-i update failoverrequests \ --as=system:serviceaccount:dr-:failover-controller \ -n dr- # Check if the protection-controller can create DRBDVolumes kubectl auth can-i create drbdvolumes \ --as=system:serviceaccount:dr-:protection-controller \ -n dr- # Check if replication-monitor can create RPOEvents kubectl auth can-i create rpoevents \ --as=system:serviceaccount:dr-:replication-monitor \ -n dr- ``` Repeat these checks on the primary and DR clusters for `protection-group-controller` and `test-failover-controller` using their respective kubeconfigs. ### Validate UI namespace access for a specific user If a user sees missing data in the Site Manager UI for a deployment, check whether their token has the required namespace-level access: ```bash # As the target user (or using their token) kubectl auth can-i get secrets -n dr- ``` If this returns `no`, grant the user access to the deployment namespace (see the Examples section). ### Use `pgctl` to validate deployment configuration ```bash pgctl validate --deployment ``` This command checks that the deployment context is reachable and that the expected CRDs and controller service accounts are present. It surfaces RBAC misconfigurations as validation failures. ### Use `quorum-deployments.sh` for guided health checks ```bash bash quorum-deployments.sh ``` Select the **Health Check** option from the menu. The script tests connectivity and RBAC posture across all registered deployments and reports failures. ## Examples ### Example 1: Identify a forbidden error on a stalled FailoverRequest A FailoverRequest is stuck in `Pending` state. You inspect the failover-controller logs: ```bash kubectl logs -n dr-production -l app=failover-controller --tail=50 ``` Expected output showing the root cause: ``` {"level":"error","msg":"failed to update FailoverRequest status","name":"pg-web-failover","error":"failoverrequests.site-recovery.io is forbidden: User \"system:serviceaccount:dr-production:failover-controller\" cannot update resource \"failoverrequests\" in API group \"site-recovery.io\" in the namespace \"dr-production\""} ``` Fix — create a missing RoleBinding: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: failover-controller-failoverrequests namespace: dr-production roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: site-recovery:failover-controller subjects: - kind: ServiceAccount name: failover-controller namespace: dr-production ``` ```bash kubectl apply -f failover-controller-rolebinding.yaml ``` After applying, the controller reconciles immediately and the FailoverRequest transitions out of `Pending`. --- ### Example 2: Grant a user access to a deployment namespace for the Site Manager UI A platform engineer reports that the Site Manager UI shows all VMs as unavailable for the `staging` deployment, although the clusters are healthy. The two-tier auth check requires `get` on `secrets` in the deployment namespace. Check their current access: ```bash kubectl auth can-i get secrets -n dr-staging \ --as= ``` Output: ``` no ``` Create a Role and RoleBinding in the deployment namespace: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: dr-deployment-viewer namespace: dr-staging rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] - apiGroups: ["site-recovery.io"] resources: ["protectiongroups", "failoverrequests", "replicationgroupstatuses", "rpoevents"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: eng-dr-staging-viewer namespace: dr-staging roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: dr-deployment-viewer subjects: - kind: User name: apiGroup: rbac.authorization.k8s.io ``` ```bash kubectl apply -f dr-staging-viewer.yaml ``` The user refreshes the Site Manager UI and the cluster health and VM protection status become visible. --- ### Example 3: Diagnose a replication-monitor that stopped writing RPOEvents ReplicationGroupStatus resources are no longer updating. Inspect the replication-monitor pod: ```bash kubectl logs -n dr-production -l app=replication-monitor --tail=50 | grep -i "forbidden\|error" ``` Output: ``` {"level":"error","msg":"cannot create RPOEvent","error":"rpoevents.site-recovery.io is forbidden: User \"system:serviceaccount:dr-production:replication-monitor\" cannot create resource \"rpoevents\" in API group \"site-recovery.io\""} ``` Verify the service account exists: ```bash kubectl get serviceaccount replication-monitor -n dr-production ``` If the service account is missing (for example, after a namespace was partially re-created), re-run the Ansible deployment playbook for the affected deployment to restore the full service account and binding set. The `replication-monitor` is deployed and managed automatically by the Ansible playbooks and should not be manually reconstructed. ```bash # Re-run the relevant Ansible play targeting the quorum deployment ansible-playbook site-recovery-deploy.yml -e deployment_name=production ``` --- ### Example 4: Confirm protection-group-controller permissions on the DR cluster Before executing a failover, validate that the `protection-group-controller` on the DR cluster can update ProtectionGroup resources: ```bash kubectl --kubeconfig /etc/site-recovery/dr-kubeconfig \ auth can-i update protectiongroups \ --as=system:serviceaccount:site-recovery:protection-group-controller \ -n site-recovery ``` Expected output: ``` yes ``` If the output is `no`, re-apply the ClusterRole and ClusterRoleBinding from the Ansible-managed manifests for the DR cluster. ## Troubleshooting Use the following issue patterns to narrow down and resolve RBAC failures. --- **Issue: FailoverRequest stays in `Pending` or `Failed` with no progress** _Symptom_: A FailoverRequest resource exists in the `dr-` namespace but the `failover-controller` never transitions it past `Pending`. The controller pod is running. _Likely cause_: The `failover-controller` service account is missing the `update` or `patch` verb on `FailoverRequest` resources, or is missing entirely from the deployment namespace. _Fix_: 1. Check logs: `kubectl logs -n dr- -l app=failover-controller | grep forbidden` 2. Verify the service account: `kubectl get sa failover-controller -n dr-` 3. Check the binding: `kubectl get rolebindings -n dr- | grep failover-controller` 4. If the binding is absent, re-run the deployment Ansible playbook or manually apply the RoleBinding referencing the `site-recovery:failover-controller` ClusterRole. --- **Issue: ProtectionRequest stalls at `Validating` or `CreatingVolumes`** _Symptom_: A ProtectionRequest for a VM never progresses past the `Validating` or `CreatingVolumes` phase. The `protection-controller` logs show API errors. _Likely cause_: The `protection-controller` service account lacks permission to create `DRBDVolume` resources, or to read the target VM's `PersistentVolumeClaim`. _Fix_: 1. `kubectl auth can-i create drbdvolumes --as=system:serviceaccount:dr-:protection-controller -n dr-` 2. If the result is `no`, check that the ClusterRole covering DRBD CRDs is bound to the protection-controller service account. 3. Re-run the Ansible deployment playbook to restore correct bindings. --- **Issue: Site Manager UI shows deployments as unavailable or VMs as unknown** _Symptom_: In the Site Manager UI, a deployment's clusters appear as unreachable or VM protection status shows as unavailable. The clusters are confirmed healthy via `kubectl`. _Likely cause_: The logged-in user's OpenShift OAuth token does not have `get` access to `secrets` in the `dr-` namespace. The Site Manager API performs a `SelfSubjectAccessReview` before returning workload cluster data; if this check fails, the API treats the deployment as inaccessible for that user. _Fix_: 1. Run: `kubectl auth can-i get secrets -n dr-` as the affected user. 2. If the result is `no`, create a Role with `get`/`list` on `secrets` in the `dr-` namespace and bind it to the user (see Example 2 above). 3. Confirm access is restored: the API returns HTTP 200 and the UI populates. --- **Issue: `pgctl` returns "forbidden" when listing or creating Protection Groups** _Symptom_: Running `pgctl` commands such as `pgctl pg list` or `pgctl pg create` returns a forbidden error referencing the user's identity. _Likely cause_: The user's kubeconfig context does not have the required ClusterRole binding on the quorum cluster, or the kubeconfig is pointing to the wrong cluster. _Fix_: 1. Verify the active context: `pgctl config current-context` 2. Confirm the user identity: `kubectl whoami` or `kubectl auth whoami` 3. Check that the user has `get`/`list`/`watch` on `protectiongroups` in the target namespace: `kubectl auth can-i list protectiongroups -n dr-` 4. If not, bind the appropriate role, or work with a cluster administrator to obtain the necessary ClusterRoleBinding. --- **Issue: RPOEvent and ReplicationGroupStatus resources stop updating** _Symptom_: The `ReplicationGroupStatus` for a Protection Group shows a stale `lastSyncTime` and no new `RPOEvent` resources are being created despite known replication lag. _Likely cause_: The `replication-monitor` service account has lost its binding to create or update `RPOEvent` and `ReplicationGroupStatus` resources. This can happen after namespace recreation or a partial deployment rollback. _Fix_: 1. `kubectl logs -n dr- -l app=replication-monitor | grep forbidden` 2. `kubectl auth can-i create rpoevents --as=system:serviceaccount:dr-:replication-monitor -n dr-` 3. Re-run the Ansible deployment playbook for the affected deployment. The `replication-monitor` and its RBAC are managed by the playbook and must not be manually patched in isolation. --- **Issue: `test-failover-controller` cannot create VolumeSnapshots on the primary cluster** _Symptom_: A TestFailover resource (used only in DRBD Operator deployments) transitions to `Failed` with an error indicating snapshots could not be created. _Likely cause_: The `test-failover-controller` service account on the primary cluster lacks `create` access to `volumesnapshots` in the namespace where production VMs run. VolumeSnapshot support requires the CSI snapshotter to be installed and the service account to have the corresponding RBAC. _Fix_: 1. Confirm the CSI snapshotter is installed: `kubectl get crds | grep volumesnapshots` 2. Check the service account permission: `kubectl --kubeconfig /path/to/primary-kubeconfig auth can-i create volumesnapshots --as=system:serviceaccount:site-recovery:test-failover-controller -n ` 3. If the result is `no`, bind a Role with `create`/`get`/`list`/`delete` on `volumesnapshots` and `volumesnapshotcontents` to the `test-failover-controller` service account in the VM namespace. --- **Issue: `pg-sync-controller` fails to sync Protection Group metadata to the DR cluster** _Symptom_: Standby VM specs on the DR cluster are out of date. The `pg-sync-controller` logs show errors when attempting to write to the DR cluster. _Likely cause_: The kubeconfig stored in the `dr-` namespace for the DR cluster has expired credentials, or the service account used in that kubeconfig has been deleted or had its bindings removed on the DR cluster. _Fix_: 1. Extract the DR kubeconfig from the deployment secret: `kubectl get secret -n dr- -o yaml | grep kubeconfig` 2. Test connectivity and permissions using that kubeconfig: `kubectl --kubeconfig /tmp/dr-kubeconfig auth can-i update protectiongroups -n site-recovery` 3. If credentials are expired, re-onboard the DR cluster through `quorum-deployments.sh` (select the credential rotation or cluster re-registration option) or re-run the Ansible playbook to refresh the stored kubeconfig.