RBAC Issues

This page helps you diagnose and resolve permission-related failures that prevent Site Recovery controllers, the Site Manager UI, and pgctl from operating correctly across the quorum, primary, and DR clusters. RBAC misconfigurations are among the most common causes of stalled FailoverRequests, failed ProtectionRequests, and controllers that loop in error states without clear messages. Because Site Recovery controllers run in-cluster and reconcile state continuously, a missing ClusterRole binding or an expired service account token will silently block recovery operations at exactly the moment you need them most.

Before working through this guide, ensure you have:

• kubectl configured with administrative access to the quorum, primary, and DR clusters
• Kubeconfig files for all three clusters (or two clusters minimum for DRBD Operator deployments)
• Sufficient privileges to inspect and create ClusterRole, ClusterRoleBinding, Role, RoleBinding, and ServiceAccount resources on each cluster
• Site Recovery deployed via the standard Ansible playbooks (the replication-monitor and other agents are deployed automatically as part of this process)
• Kubernetes ≥ 1.28 on all clusters
• Familiarity with Kubernetes RBAC concepts: service accounts, role bindings, and token review

No additional software installation is required for RBAC troubleshooting. All steps use kubectl and, optionally, pgctl and quorum-deployments.sh, which are installed as part of the standard Site Recovery deployment.

If pgctl is not available on your PATH, locate it in the deployment package and ensure it is executable:

chmod +x /usr/local/bin/pgctl
pgctl version

If quorum-deployments.sh is not on your PATH:

bash /path/to/quorum-deployments.sh

The interactive menu includes a health check option that performs automated RBAC validation across all registered deployments.

Site Recovery uses Kubernetes RBAC to control which service accounts can read and write each CRD, and uses OpenShift OAuth token passthrough for the Site Manager UI. The key RBAC surface areas are:

Controller service accounts (quorum cluster)

Each DR deployment runs in its own namespace (dr-<name> on the quorum cluster). The controllers in that namespace — failover-controller, protection-controller, pg-sync-controller — each run under a dedicated service account. These service accounts require:

Controller service accounts (primary and DR clusters)

The protection-group-controller and test-failover-controller run on the primary and DR clusters and require access to:

Site Manager UI token passthrough

The Site Manager UI uses a two-tier authentication model:

• Quorum cluster API calls: forwarded using the authenticated user's OpenShift OAuth bearer token. Kubernetes RBAC is enforced on the quorum cluster for every request the UI makes on behalf of the user.
• Workload cluster calls (primary and DR): the API backend uses stored kubeconfigs from the deployment namespace. Access to a given deployment is gated by a SelfSubjectAccessReview check — the user's token is tested for get on secrets in the dr-<name> namespace before workload cluster data is returned.

This means a user who cannot get secrets in the relevant dr-<name> namespace will see cluster health and VM data as unavailable in the UI, even if the underlying clusters are healthy.

replication-monitor service account

The replication-monitor agent, deployed automatically by the Ansible playbooks, writes RPOEvent and ReplicationGroupStatus resources. Its service account requires create and update on both CRDs in the deployment namespace. A missing binding here causes replication health data to stop updating without any controller error being surfaced.

Use the following patterns to systematically identify and fix RBAC issues.

Check controller pod logs for permission errors

On the quorum cluster, inspect the logs for each controller in the affected deployment namespace:

# Replace <deployment-name> with your deployment, e.g. dr-production
kubectl logs -n dr-<deployment-name> -l app=failover-controller --tail=100 | grep -i "forbidden\|rbac\|permission\|unauthorized"
kubectl logs -n dr-<deployment-name> -l app=protection-controller --tail=100 | grep -i "forbidden\|rbac\|permission\|unauthorized"
kubectl logs -n dr-<deployment-name> -l app=pg-sync-controller --tail=100 | grep -i "forbidden\|rbac\|permission\|unauthorized"

On the primary and DR clusters, inspect the per-cluster controllers:

kubectl --kubeconfig /path/to/primary-kubeconfig logs -n site-recovery -l app=protection-group-controller --tail=100 | grep -i "forbidden\|rbac"
kubectl --kubeconfig /path/to/dr-kubeconfig logs -n site-recovery -l app=test-failover-controller --tail=100 | grep -i "forbidden\|rbac"

Verify service account bindings

List all role bindings in a deployment namespace on the quorum cluster:

kubectl get rolebindings,clusterrolebindings -n dr-<deployment-name> -o wide

Confirm the expected service accounts are present:

kubectl get serviceaccounts -n dr-<deployment-name>

Test specific permissions with kubectl auth can-i

Impersonate a controller service account to verify its effective permissions:

# Check if the failover-controller service account can update FailoverRequests
kubectl auth can-i update failoverrequests \
  --as=system:serviceaccount:dr-<deployment-name>:failover-controller \
  -n dr-<deployment-name>

# Check if the protection-controller can create DRBDVolumes
kubectl auth can-i create drbdvolumes \
  --as=system:serviceaccount:dr-<deployment-name>:protection-controller \
  -n dr-<deployment-name>

# Check if replication-monitor can create RPOEvents
kubectl auth can-i create rpoevents \
  --as=system:serviceaccount:dr-<deployment-name>:replication-monitor \
  -n dr-<deployment-name>

Repeat these checks on the primary and DR clusters for protection-group-controller and test-failover-controller using their respective kubeconfigs.

Validate UI namespace access for a specific user

If a user sees missing data in the Site Manager UI for a deployment, check whether their token has the required namespace-level access:

# As the target user (or using their token)
kubectl auth can-i get secrets -n dr-<deployment-name>

If this returns no, grant the user access to the deployment namespace (see the Examples section).

Use pgctl to validate deployment configuration

pgctl validate --deployment <deployment-name>

This command checks that the deployment context is reachable and that the expected CRDs and controller service accounts are present. It surfaces RBAC misconfigurations as validation failures.

Use quorum-deployments.sh for guided health checks

bash quorum-deployments.sh

Select the Health Check option from the menu. The script tests connectivity and RBAC posture across all registered deployments and reports failures.

Example 1: Identify a forbidden error on a stalled FailoverRequest

A FailoverRequest is stuck in Pending state. You inspect the failover-controller logs:

kubectl logs -n dr-production -l app=failover-controller --tail=50

Expected output showing the root cause:

{"level":"error","msg":"failed to update FailoverRequest status","name":"pg-web-failover","error":"failoverrequests.site-recovery.io is forbidden: User \"system:serviceaccount:dr-production:failover-controller\" cannot update resource \"failoverrequests\" in API group \"site-recovery.io\" in the namespace \"dr-production\""}

Fix — create a missing RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: failover-controller-failoverrequests
  namespace: dr-production
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: site-recovery:failover-controller
subjects:
- kind: ServiceAccount
  name: failover-controller
  namespace: dr-production

kubectl apply -f failover-controller-rolebinding.yaml

After applying, the controller reconciles immediately and the FailoverRequest transitions out of Pending.

Example 2: Grant a user access to a deployment namespace for the Site Manager UI

A platform engineer reports that the Site Manager UI shows all VMs as unavailable for the staging deployment, although the clusters are healthy. The two-tier auth check requires get on secrets in the deployment namespace.

Check their current access:

kubectl auth can-i get secrets -n dr-staging \
  --as=<username-or-service-account>

Output:

no

Create a Role and RoleBinding in the deployment namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: dr-deployment-viewer
  namespace: dr-staging
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
- apiGroups: ["site-recovery.io"]
  resources: ["protectiongroups", "failoverrequests", "replicationgroupstatuses", "rpoevents"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eng-dr-staging-viewer
  namespace: dr-staging
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: dr-deployment-viewer
subjects:
- kind: User
  name: <username>
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f dr-staging-viewer.yaml

The user refreshes the Site Manager UI and the cluster health and VM protection status become visible.

Example 3: Diagnose a replication-monitor that stopped writing RPOEvents

ReplicationGroupStatus resources are no longer updating. Inspect the replication-monitor pod:

kubectl logs -n dr-production -l app=replication-monitor --tail=50 | grep -i "forbidden\|error"

Output:

{"level":"error","msg":"cannot create RPOEvent","error":"rpoevents.site-recovery.io is forbidden: User \"system:serviceaccount:dr-production:replication-monitor\" cannot create resource \"rpoevents\" in API group \"site-recovery.io\""}

Verify the service account exists:

kubectl get serviceaccount replication-monitor -n dr-production

If the service account is missing (for example, after a namespace was partially re-created), re-run the Ansible deployment playbook for the affected deployment to restore the full service account and binding set. The replication-monitor is deployed and managed automatically by the Ansible playbooks and should not be manually reconstructed.

# Re-run the relevant Ansible play targeting the quorum deployment
ansible-playbook site-recovery-deploy.yml -e deployment_name=production

Example 4: Confirm protection-group-controller permissions on the DR cluster

Before executing a failover, validate that the protection-group-controller on the DR cluster can update ProtectionGroup resources:

kubectl --kubeconfig /etc/site-recovery/dr-kubeconfig \
  auth can-i update protectiongroups \
  --as=system:serviceaccount:site-recovery:protection-group-controller \
  -n site-recovery

Expected output:

yes

If the output is no, re-apply the ClusterRole and ClusterRoleBinding from the Ansible-managed manifests for the DR cluster.

Use the following issue patterns to narrow down and resolve RBAC failures.

Issue: FailoverRequest stays in Pending or Failed with no progress

Symptom: A FailoverRequest resource exists in the dr-<name> namespace but the failover-controller never transitions it past Pending. The controller pod is running.

Likely cause: The failover-controller service account is missing the update or patch verb on FailoverRequest resources, or is missing entirely from the deployment namespace.

Fix:

1. Check logs: kubectl logs -n dr-<name> -l app=failover-controller | grep forbidden
2. Verify the service account: kubectl get sa failover-controller -n dr-<name>
3. Check the binding: kubectl get rolebindings -n dr-<name> | grep failover-controller
4. If the binding is absent, re-run the deployment Ansible playbook or manually apply the RoleBinding referencing the site-recovery:failover-controller ClusterRole.

Issue: ProtectionRequest stalls at Validating or CreatingVolumes

Symptom: A ProtectionRequest for a VM never progresses past the Validating or CreatingVolumes phase. The protection-controller logs show API errors.

Likely cause: The protection-controller service account lacks permission to create DRBDVolume resources, or to read the target VM's PersistentVolumeClaim.

Fix:

1. kubectl auth can-i create drbdvolumes --as=system:serviceaccount:dr-<name>:protection-controller -n dr-<name>
2. If the result is no, check that the ClusterRole covering DRBD CRDs is bound to the protection-controller service account.
3. Re-run the Ansible deployment playbook to restore correct bindings.

Issue: Site Manager UI shows deployments as unavailable or VMs as unknown

Symptom: In the Site Manager UI, a deployment's clusters appear as unreachable or VM protection status shows as unavailable. The clusters are confirmed healthy via kubectl.

Likely cause: The logged-in user's OpenShift OAuth token does not have get access to secrets in the dr-<name> namespace. The Site Manager API performs a SelfSubjectAccessReview before returning workload cluster data; if this check fails, the API treats the deployment as inaccessible for that user.

Fix:

1. Run: kubectl auth can-i get secrets -n dr-<name> as the affected user.
2. If the result is no, create a Role with get/list on secrets in the dr-<name> namespace and bind it to the user (see Example 2 above).
3. Confirm access is restored: the API returns HTTP 200 and the UI populates.

Issue: pgctl returns "forbidden" when listing or creating Protection Groups

Symptom: Running pgctl commands such as pgctl pg list or pgctl pg create returns a forbidden error referencing the user's identity.

Likely cause: The user's kubeconfig context does not have the required ClusterRole binding on the quorum cluster, or the kubeconfig is pointing to the wrong cluster.

Fix:

1. Verify the active context: pgctl config current-context
2. Confirm the user identity: kubectl whoami or kubectl auth whoami
3. Check that the user has get/list/watch on protectiongroups in the target namespace: kubectl auth can-i list protectiongroups -n dr-<name>
4. If not, bind the appropriate role, or work with a cluster administrator to obtain the necessary ClusterRoleBinding.

Issue: RPOEvent and ReplicationGroupStatus resources stop updating

Symptom: The ReplicationGroupStatus for a Protection Group shows a stale lastSyncTime and no new RPOEvent resources are being created despite known replication lag.

Likely cause: The replication-monitor service account has lost its binding to create or update RPOEvent and ReplicationGroupStatus resources. This can happen after namespace recreation or a partial deployment rollback.

Fix:

1. kubectl logs -n dr-<name> -l app=replication-monitor | grep forbidden
2. kubectl auth can-i create rpoevents --as=system:serviceaccount:dr-<name>:replication-monitor -n dr-<name>
3. Re-run the Ansible deployment playbook for the affected deployment. The replication-monitor and its RBAC are managed by the playbook and must not be manually patched in isolation.

Issue: test-failover-controller cannot create VolumeSnapshots on the primary cluster

Symptom: A TestFailover resource (used only in DRBD Operator deployments) transitions to Failed with an error indicating snapshots could not be created.

Likely cause: The test-failover-controller service account on the primary cluster lacks create access to volumesnapshots in the namespace where production VMs run. VolumeSnapshot support requires the CSI snapshotter to be installed and the service account to have the corresponding RBAC.

Fix:

1. Confirm the CSI snapshotter is installed: kubectl get crds | grep volumesnapshots
2. Check the service account permission: kubectl --kubeconfig /path/to/primary-kubeconfig auth can-i create volumesnapshots --as=system:serviceaccount:site-recovery:test-failover-controller -n <vm-namespace>
3. If the result is no, bind a Role with create/get/list/delete on volumesnapshots and volumesnapshotcontents to the test-failover-controller service account in the VM namespace.

Issue: pg-sync-controller fails to sync Protection Group metadata to the DR cluster

Symptom: Standby VM specs on the DR cluster are out of date. The pg-sync-controller logs show errors when attempting to write to the DR cluster.

Likely cause: The kubeconfig stored in the dr-<name> namespace for the DR cluster has expired credentials, or the service account used in that kubeconfig has been deleted or had its bindings removed on the DR cluster.

Fix:

1. Extract the DR kubeconfig from the deployment secret: kubectl get secret -n dr-<name> -o yaml | grep kubeconfig
2. Test connectivity and permissions using that kubeconfig: kubectl --kubeconfig /tmp/dr-kubeconfig auth can-i update protectiongroups -n site-recovery
3. If credentials are expired, re-onboard the DR cluster through quorum-deployments.sh (select the credential rotation or cluster re-registration option) or re-run the Ansible playbook to refresh the stored kubeconfig.