--- title: Webhook Issues product: trilio-site-recovery-for-kubernetes-openshift-virtualization doc_type: guide version: feature-tsr-24 source: git2docs (code-derived, validation-filtered) canonical: https://git2docs.com/murali-balcha/docs/trilio-site-recovery-for-kubernetes-openshift-virtualization/site-recovery/troubleshooting-webhooks --- # Webhook Issues _Admission webhook failures and validation errors blocking resource creation or updates_ ## Overview Admission webhooks in Site Recovery validate and mutate Kubernetes custom resources before they are persisted to the API server. When a webhook rejects a request or becomes unavailable, resource creation and updates fail immediately — blocking protection workflows, failover operations, and configuration changes. This page explains the most common webhook failure modes you will encounter when working with `ProtectionGroup`, `ProtectionRequest`, `FailoverRequest`, `TestFailover`, `DRBDReplicationPolicy`, and `DRBDVolume` resources, along with the steps to diagnose and resolve each one. ## Prerequisites Before troubleshooting webhook issues, ensure you have: - `kubectl` installed and configured with administrative access to the quorum cluster, primary cluster, and DR cluster - Kubeconfig contexts for all relevant clusters - Kubernetes ≥ 1.28 on all clusters - Sufficient RBAC permissions to read `ValidatingWebhookConfiguration`, `MutatingWebhookConfiguration`, and `Secret` resources in the `kube-system` and Site Recovery controller namespaces - Access to controller pod logs on the quorum cluster (for `failover-controller`, `protection-controller`, `pg-sync-controller`) and on the primary and DR clusters (for `protection-group-controller` and `test-failover-controller`) - Familiarity with Site Recovery CRDs and their validation rules, particularly geo-replication and storage class requirements for `ProtectionGroup` ## Installation Webhooks in Site Recovery are deployed automatically by the Ansible playbooks that install the controllers. You do not install webhooks separately. If you need to confirm that webhooks are present after a controller deployment, run the following steps. 1. **Verify webhook configurations exist on the quorum cluster:** ```bash kubectl get validatingwebhookconfigurations kubectl get mutatingwebhookconfigurations ``` Look for entries associated with `siterecovery.trilio.io`. 2. **Verify webhook configurations exist on the primary or DR cluster (for protection-group-controller and test-failover-controller):** ```bash kubectl --kubeconfig get validatingwebhookconfigurations kubectl --kubeconfig get validatingwebhookconfigurations ``` 3. **Confirm the controller pods backing the webhooks are running:** ```bash # On the quorum cluster kubectl get pods -n dr- -l app=failover-controller kubectl get pods -n dr- -l app=protection-controller kubectl get pods -n dr- -l app=pg-sync-controller # On the primary or DR cluster kubectl get pods -n dr- -l app=protection-group-controller kubectl get pods -n dr- -l app=test-failover-controller ``` If any pod is not in `Running` state, address the pod issue before troubleshooting webhook errors — a controller pod that is not running will cause all webhook calls to that controller to fail with a connection timeout. 4. **Confirm the webhook service endpoints are ready:** ```bash kubectl get endpoints -n dr- ``` Any service backing a webhook must show at least one ready endpoint. An empty endpoint list means the controller pod is not ready to serve admission requests. ## Configuration Site Recovery webhooks enforce validation rules that correspond directly to the constraints built into each controller. Understanding these rules helps you write valid manifests on the first attempt. ### ProtectionGroup validation rules The `protection-group-controller` validates every `ProtectionGroup` at create and update time. The following rules are enforced: | Rule | Requirement | Effect if violated | |---|---|---| | VM existence | Each VM listed in `spec.virtualMachines` must exist in the same namespace | Request rejected with `ValidationFailed` condition | | PVC existence | Every PVC attached to each listed VM must exist | Request rejected | | Storage class replication | Every PVC's storage class must support geo-replication with `placementCount >= 2` | Request rejected | | Provisioner type | Storage class provisioner must be compatible with the deployment model in use | Request rejected | | Cross-site placement (best practice) | Storage class should include `replicasOnDifferent` parameter | Warning added to `status.warnings`; request is not rejected | Validation runs on both `create` and `update` events. Adding a VM to an existing `ProtectionGroup` triggers full re-validation of all VMs in the group, not just the newly added one. ### ProtectionRequest validation rules The `protection-controller` on the quorum cluster validates `ProtectionRequest` resources for DRBD Operator deployments. The VM referenced in `spec.virtualMachine` must exist on the primary cluster, and the associated PVCs must be compatible with the `DRBDReplicationPolicy` in the same namespace. ### FailoverRequest validation rules The `failover-controller` validates `FailoverRequest` resources. The `spec.protectionGroup` field must reference an existing `ProtectionGroup`, and the `ProtectionGroup` must not already have an active failover lock (a Kubernetes Lease named `failover-lock-` in the same namespace). ### TestFailover validation rules The `test-failover-controller` validates `TestFailover` resources. Test failover is only supported in DRBD Operator deployment models. Submitting a `TestFailover` resource in a centralized storage deployment will result in a rejection. ### Webhook failure policy Site Recovery webhooks use `Fail` failure policy by default. This means that if the webhook service is unreachable — for example because the controller pod is restarting — the API server will reject the resource operation rather than allow it through. This is intentional: permitting an unvalidated resource could create an invalid DR configuration that only fails at failover time. ## Usage Most webhook failures surface as an error returned immediately when you run `kubectl apply` or `kubectl patch`. The error message comes from the admitting controller and describes which validation rule was violated. ### Reading webhook rejection messages When a webhook rejects a request, `kubectl` prints output like the following: ``` Error from server: error when creating "my-pg.yaml": admission webhook "protectiongroup.siterecovery.trilio.io" denied the request: PVC prod-vm-1-disk for VM prod-vm-1: Storage class linstor-local has placementCount=1, minimum 2 required for geo-replication ``` The message contains three useful pieces of information: - The webhook name (identifies which controller rejected it) - The resource and field that failed validation - The specific rule that was violated Fix the underlying configuration issue described in the message, then resubmit. ### Checking resource status after a rejection If a resource was partially created before a validation step failed (for example, on an update to an existing `ProtectionGroup`), check the `status` field for a `ValidationFailed` condition: ```bash kubectl get protectiongroup -n -o yaml ``` Look for: ```yaml status: state: Failed conditions: - type: ValidationFailed message: "PVC invalid-vm-disk: placementCount=1, minimum 2 required" ``` An existing resource in `Failed` state is unchanged from its last valid configuration. Correct the spec and reapply. ### Inspecting controller logs for validation details Webhook rejection messages are intentionally concise. For full diagnostic context, inspect the controller logs: ```bash # For ProtectionGroup validation failures (on the cluster where the PG lives) kubectl logs -n dr- deployment/protection-group-controller # For ProtectionRequest validation failures (on the quorum cluster) kubectl logs -n dr- deployment/protection-controller # For FailoverRequest validation failures (on the quorum cluster) kubectl logs -n dr- deployment/failover-controller # For TestFailover validation failures kubectl logs -n dr- deployment/test-failover-controller ``` Filter for the resource name to reduce log volume: ```bash kubectl logs -n dr- deployment/protection-group-controller \ | grep "" ``` ## Examples ### Example 1: ProtectionGroup rejected due to insufficient storage class placement count You apply a `ProtectionGroup` manifest referencing a VM whose PVC uses a storage class with only one replica: ```bash kubectl apply -f production-pg.yaml -n dr-prod ``` **Output:** ``` Error from server: error when creating "production-pg.yaml": admission webhook "protectiongroup.siterecovery.trilio.io" denied the request: PVC prod-vm-1-disk for VM prod-vm-1: Storage class linstor-local has placementCount=1, minimum 2 required for geo-replication ``` Correct the storage class to set `placementCount: 2` or higher, or provision a new PVC on a qualifying storage class and update the VM before resubmitting the `ProtectionGroup`. --- ### Example 2: ProtectionGroup update rejected when adding a VM with an invalid PVC You patch an existing `ProtectionGroup` to add a second VM: ```bash kubectl patch protectiongroup production-pg -n dr-prod --type merge -p '{ "spec": { "virtualMachines": [ {"name": "prod-vm-1"}, {"name": "new-vm"} ] } }' ``` **Output:** ``` Error from server: admission webhook "protectiongroup.siterecovery.trilio.io" denied the request: PVC new-vm-disk for VM new-vm: Storage class nfs-storage does not use the required CSI provisioner ``` The existing `ProtectionGroup` is unchanged. Resolve the storage issue for `new-vm` before retrying the patch. --- ### Example 3: FailoverRequest blocked by an active failover lock You submit a `FailoverRequest` for a `ProtectionGroup` that already has a failover in progress: ```bash kubectl apply -f failover-request.yaml -n dr-prod ``` **Output:** ``` Error from server: admission webhook "failoverrequest.siterecovery.trilio.io" denied the request: Failed to acquire failover lock for production-protection-group Failover lock is held by: failover-controller-12345 ``` Check whether the existing failover is still running: ```bash kubectl get failoverrequests -n dr-prod kubectl get lease failover-lock-production-protection-group -n dr-prod ``` If the original failover completed but the lock was not released (for example, due to a crashed controller pod), delete the stale Lease: ```bash kubectl delete lease failover-lock-production-protection-group -n dr-prod ``` Then resubmit your `FailoverRequest`. --- ### Example 4: Webhook service unreachable — controller pod not running You apply a `ProtectionGroup` and receive a timeout rather than a validation message: ```bash kubectl apply -f production-pg.yaml -n dr-prod ``` **Output:** ``` Error from server (InternalError): error when creating "production-pg.yaml": Internal error occurred: failed calling webhook "protectiongroup.siterecovery.trilio.io": Post "https://protection-group-controller.dr-prod.svc:443/validate": dial tcp: connect: connection refused ``` The `protection-group-controller` pod is not running. Check pod status: ```bash kubectl get pods -n dr-prod -l app=protection-group-controller kubectl describe pod -n dr-prod -l app=protection-group-controller ``` Review pod events and logs, resolve any image pull or resource issue, and wait for the pod to reach `Running` state before retrying. ## Troubleshooting Use the following reference to match the symptom you observe with its most likely cause and resolution. --- **Symptom:** `kubectl apply` returns `admission webhook denied the request: ... placementCount=1, minimum 2 required for geo-replication` **Cause:** The PVC attached to a VM in your `ProtectionGroup` uses a storage class configured with only one replica. Site Recovery requires at least two replicas for geo-replication to function. **Fix:** Identify the storage class in question from the error message. Update its `placementCount` to `2` or higher, or migrate the VM to a PVC that already uses a qualifying storage class. Re-validate by re-applying the `ProtectionGroup` manifest. --- **Symptom:** `kubectl apply` returns `admission webhook denied the request: ... does not use the required CSI provisioner` **Cause:** The PVC's storage class uses a provisioner that is not compatible with the Site Recovery deployment model in use. **Fix:** Provision the VM's disk on a storage class that uses the correct provisioner for your deployment model. Check the `DRBDReplicationPolicy` in the same namespace for the expected storage class mappings, then update the VM and resubmit. --- **Symptom:** `kubectl apply` returns `admission webhook denied the request: PVC : PVC has no storage class` **Cause:** The PVC attached to a listed VM was created without a storage class annotation, or the storage class was deleted after the PVC was created. **Fix:** Identify the PVC from the error message. If no storage class is set, recreate the PVC with an appropriate storage class. Update the VM to use the new PVC before resubmitting the `ProtectionGroup`. --- **Symptom:** `kubectl apply` returns `admission webhook denied the request: Failed to acquire failover lock ... Failover lock is held by: failover-controller-` **Cause:** A `FailoverRequest` was submitted for a `ProtectionGroup` that is already being processed by the `failover-controller`. Concurrent failovers of the same Protection Group are prevented by a Kubernetes Lease. **Fix:** Check whether the in-progress failover is still active: ```bash kubectl get failoverrequests -n dr- kubectl get lease failover-lock- -n dr- -o yaml ``` If the failover completed or the controller crashed, delete the stale Lease and resubmit: ```bash kubectl delete lease failover-lock- -n dr- ``` Leases expire automatically after 300 seconds if the holder does not renew them, so you can also wait for automatic expiry. --- **Symptom:** `kubectl apply` returns `Internal error occurred: failed calling webhook ... connection refused` or `dial tcp: ... i/o timeout` **Cause:** The controller pod that serves the webhook is not running or is not yet ready. Because Site Recovery webhooks use a `Fail` failure policy, the API server rejects the request rather than permitting an unvalidated resource. **Fix:** 1. Identify which controller serves the webhook from the webhook name in the error message. 2. Check the pod status in the appropriate namespace: ```bash # For protection-group-controller (runs on primary or DR cluster) kubectl get pods -n dr- -l app=protection-group-controller # For failover-controller (runs on quorum cluster) kubectl get pods -n dr- -l app=failover-controller # For test-failover-controller (runs on primary or DR cluster) kubectl get pods -n dr- -l app=test-failover-controller ``` 3. If the pod is in `CrashLoopBackOff` or `Error` state, inspect logs and events: ```bash kubectl logs -n dr- --previous kubectl describe pod -n dr- ``` 4. Resolve the underlying pod issue (image pull error, OOMKilled, missing Secret, etc.) and wait for the pod to reach `Running` with all containers ready before retrying the original operation. --- **Symptom:** `kubectl apply` returns `admission webhook denied the request: TestFailover is only supported for DRBD Operator deployment models` **Cause:** You submitted a `TestFailover` resource in a deployment that uses the centralized storage model. Test failover is only available for DRBD Operator deployments. **Fix:** Verify your deployment model by checking whether a `DRBDReplicationPolicy` exists in the namespace: ```bash kubectl get drbdreplicationpolicies -n dr- ``` If no `DRBDReplicationPolicy` exists, the deployment uses the centralized storage model and `TestFailover` is not supported. Non-disruptive DR validation must be performed through another method appropriate for your deployment model. --- **Symptom:** A `ProtectionGroup` was successfully created, but its `status.state` is `Failed` with a `ValidationFailed` condition after an update **Cause:** An update to the `ProtectionGroup` spec (such as adding a new VM) passed the webhook but one of the newly referenced VMs or PVCs failed post-admission validation during controller reconciliation. The `ProtectionGroup` reverts to its last valid configuration. **Fix:** Read the condition message: ```bash kubectl get protectiongroup -n dr- \ -o jsonpath='{.status.conditions[*].message}' ``` Also check the `protection-group-controller` logs for the full validation trace: ```bash kubectl logs -n dr- deployment/protection-group-controller \ | grep ``` Resolve the reported issue with the VM or PVC, then reapply the updated `ProtectionGroup` spec. --- **Symptom:** Webhook warnings appear in `status.warnings` but the resource was accepted **Cause:** The `ProtectionGroup` passed all blocking validation rules but one or more VMs use a storage class that lacks recommended geo-placement configuration (for example, missing `replicasOnDifferent`). The controller admits the resource but records a warning because replicas may be placed on the same site, reducing DR effectiveness. **Fix:** Review the warnings: ```bash kubectl get protectiongroup -n dr- \ -o jsonpath='{.status.warnings}' ``` Update the storage class to add cross-site placement parameters, then reapply the `ProtectionGroup`. The warnings will be cleared on the next reconciliation once the storage class is corrected.