Webhook Issues

Admission webhooks in Site Recovery validate and mutate Kubernetes custom resources before they are persisted to the API server. When a webhook rejects a request or becomes unavailable, resource creation and updates fail immediately — blocking protection workflows, failover operations, and configuration changes. This page explains the most common webhook failure modes you will encounter when working with ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, DRBDReplicationPolicy, and DRBDVolume resources, along with the steps to diagnose and resolve each one.

Before troubleshooting webhook issues, ensure you have:

• kubectl installed and configured with administrative access to the quorum cluster, primary cluster, and DR cluster
• Kubeconfig contexts for all relevant clusters
• Kubernetes ≥ 1.28 on all clusters
• Sufficient RBAC permissions to read ValidatingWebhookConfiguration, MutatingWebhookConfiguration, and Secret resources in the kube-system and Site Recovery controller namespaces
• Access to controller pod logs on the quorum cluster (for failover-controller, protection-controller, pg-sync-controller) and on the primary and DR clusters (for protection-group-controller and test-failover-controller)
• Familiarity with Site Recovery CRDs and their validation rules, particularly geo-replication and storage class requirements for ProtectionGroup

Webhooks in Site Recovery are deployed automatically by the Ansible playbooks that install the controllers. You do not install webhooks separately. If you need to confirm that webhooks are present after a controller deployment, run the following steps.

1. Verify webhook configurations exist on the quorum cluster:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Look for entries associated with siterecovery.trilio.io.

2. Verify webhook configurations exist on the primary or DR cluster (for protection-group-controller and test-failover-controller):

kubectl --kubeconfig <primary-kubeconfig> get validatingwebhookconfigurations
kubectl --kubeconfig <dr-kubeconfig> get validatingwebhookconfigurations

3. Confirm the controller pods backing the webhooks are running:

# On the quorum cluster
kubectl get pods -n dr-<deployment-name> -l app=failover-controller
kubectl get pods -n dr-<deployment-name> -l app=protection-controller
kubectl get pods -n dr-<deployment-name> -l app=pg-sync-controller

# On the primary or DR cluster
kubectl get pods -n dr-<deployment-name> -l app=protection-group-controller
kubectl get pods -n dr-<deployment-name> -l app=test-failover-controller

If any pod is not in Running state, address the pod issue before troubleshooting webhook errors — a controller pod that is not running will cause all webhook calls to that controller to fail with a connection timeout.

4. Confirm the webhook service endpoints are ready:

kubectl get endpoints -n dr-<deployment-name>

Any service backing a webhook must show at least one ready endpoint. An empty endpoint list means the controller pod is not ready to serve admission requests.

Site Recovery webhooks enforce validation rules that correspond directly to the constraints built into each controller. Understanding these rules helps you write valid manifests on the first attempt.

ProtectionGroup validation rules

The protection-group-controller validates every ProtectionGroup at create and update time. The following rules are enforced:

Validation runs on both create and update events. Adding a VM to an existing ProtectionGroup triggers full re-validation of all VMs in the group, not just the newly added one.

ProtectionRequest validation rules

The protection-controller on the quorum cluster validates ProtectionRequest resources for DRBD Operator deployments. The VM referenced in spec.virtualMachine must exist on the primary cluster, and the associated PVCs must be compatible with the DRBDReplicationPolicy in the same namespace.

FailoverRequest validation rules

The failover-controller validates FailoverRequest resources. The spec.protectionGroup field must reference an existing ProtectionGroup, and the ProtectionGroup must not already have an active failover lock (a Kubernetes Lease named failover-lock-<protection-group-name> in the same namespace).

TestFailover validation rules

The test-failover-controller validates TestFailover resources. Test failover is only supported in DRBD Operator deployment models. Submitting a TestFailover resource in a centralized storage deployment will result in a rejection.

Webhook failure policy

Site Recovery webhooks use Fail failure policy by default. This means that if the webhook service is unreachable — for example because the controller pod is restarting — the API server will reject the resource operation rather than allow it through. This is intentional: permitting an unvalidated resource could create an invalid DR configuration that only fails at failover time.

Most webhook failures surface as an error returned immediately when you run kubectl apply or kubectl patch. The error message comes from the admitting controller and describes which validation rule was violated.

Reading webhook rejection messages

When a webhook rejects a request, kubectl prints output like the following:

Error from server: error when creating "my-pg.yaml": admission webhook
"protectiongroup.siterecovery.trilio.io" denied the request:
PVC prod-vm-1-disk for VM prod-vm-1: Storage class linstor-local has
placementCount=1, minimum 2 required for geo-replication

The message contains three useful pieces of information:

• The webhook name (identifies which controller rejected it)
• The resource and field that failed validation
• The specific rule that was violated

Fix the underlying configuration issue described in the message, then resubmit.

Checking resource status after a rejection

If a resource was partially created before a validation step failed (for example, on an update to an existing ProtectionGroup), check the status field for a ValidationFailed condition:

kubectl get protectiongroup <name> -n <namespace> -o yaml

Look for:

status:
  state: Failed
  conditions:
    - type: ValidationFailed
      message: "PVC invalid-vm-disk: placementCount=1, minimum 2 required"

An existing resource in Failed state is unchanged from its last valid configuration. Correct the spec and reapply.

Inspecting controller logs for validation details

Webhook rejection messages are intentionally concise. For full diagnostic context, inspect the controller logs:

# For ProtectionGroup validation failures (on the cluster where the PG lives)
kubectl logs -n dr-<deployment-name> deployment/protection-group-controller

# For ProtectionRequest validation failures (on the quorum cluster)
kubectl logs -n dr-<deployment-name> deployment/protection-controller

# For FailoverRequest validation failures (on the quorum cluster)
kubectl logs -n dr-<deployment-name> deployment/failover-controller

# For TestFailover validation failures
kubectl logs -n dr-<deployment-name> deployment/test-failover-controller

Filter for the resource name to reduce log volume:

kubectl logs -n dr-<deployment-name> deployment/protection-group-controller \
  | grep "<protectiongroup-name>"

Example 1: ProtectionGroup rejected due to insufficient storage class placement count

You apply a ProtectionGroup manifest referencing a VM whose PVC uses a storage class with only one replica:

kubectl apply -f production-pg.yaml -n dr-prod

Output:

Error from server: error when creating "production-pg.yaml": admission webhook
"protectiongroup.siterecovery.trilio.io" denied the request:
PVC prod-vm-1-disk for VM prod-vm-1: Storage class linstor-local has
placementCount=1, minimum 2 required for geo-replication

Correct the storage class to set placementCount: 2 or higher, or provision a new PVC on a qualifying storage class and update the VM before resubmitting the ProtectionGroup.

Example 2: ProtectionGroup update rejected when adding a VM with an invalid PVC

You patch an existing ProtectionGroup to add a second VM:

kubectl patch protectiongroup production-pg -n dr-prod --type merge -p '{
  "spec": {
    "virtualMachines": [
      {"name": "prod-vm-1"},
      {"name": "new-vm"}
    ]
  }
}'

Output:

Error from server: admission webhook "protectiongroup.siterecovery.trilio.io"
denied the request: PVC new-vm-disk for VM new-vm: Storage class nfs-storage
does not use the required CSI provisioner

The existing ProtectionGroup is unchanged. Resolve the storage issue for new-vm before retrying the patch.

Example 3: FailoverRequest blocked by an active failover lock

You submit a FailoverRequest for a ProtectionGroup that already has a failover in progress:

kubectl apply -f failover-request.yaml -n dr-prod

Output:

Error from server: admission webhook "failoverrequest.siterecovery.trilio.io"
denied the request: Failed to acquire failover lock for production-protection-group
Failover lock is held by: failover-controller-12345

Check whether the existing failover is still running:

kubectl get failoverrequests -n dr-prod
kubectl get lease failover-lock-production-protection-group -n dr-prod

If the original failover completed but the lock was not released (for example, due to a crashed controller pod), delete the stale Lease:

kubectl delete lease failover-lock-production-protection-group -n dr-prod

Then resubmit your FailoverRequest.

Example 4: Webhook service unreachable — controller pod not running

You apply a ProtectionGroup and receive a timeout rather than a validation message:

kubectl apply -f production-pg.yaml -n dr-prod

Output:

Error from server (InternalError): error when creating "production-pg.yaml":
Internal error occurred: failed calling webhook
"protectiongroup.siterecovery.trilio.io": Post
"https://protection-group-controller.dr-prod.svc:443/validate": dial tcp:
connect: connection refused

The protection-group-controller pod is not running. Check pod status:

kubectl get pods -n dr-prod -l app=protection-group-controller
kubectl describe pod -n dr-prod -l app=protection-group-controller

Review pod events and logs, resolve any image pull or resource issue, and wait for the pod to reach Running state before retrying.

Use the following reference to match the symptom you observe with its most likely cause and resolution.

Symptom: kubectl apply returns admission webhook denied the request: ... placementCount=1, minimum 2 required for geo-replication

Cause: The PVC attached to a VM in your ProtectionGroup uses a storage class configured with only one replica. Site Recovery requires at least two replicas for geo-replication to function.

Fix: Identify the storage class in question from the error message. Update its placementCount to 2 or higher, or migrate the VM to a PVC that already uses a qualifying storage class. Re-validate by re-applying the ProtectionGroup manifest.

Symptom: kubectl apply returns admission webhook denied the request: ... does not use the required CSI provisioner

Cause: The PVC's storage class uses a provisioner that is not compatible with the Site Recovery deployment model in use.

Fix: Provision the VM's disk on a storage class that uses the correct provisioner for your deployment model. Check the DRBDReplicationPolicy in the same namespace for the expected storage class mappings, then update the VM and resubmit.

Symptom: kubectl apply returns admission webhook denied the request: PVC <name>: PVC <name> has no storage class

Cause: The PVC attached to a listed VM was created without a storage class annotation, or the storage class was deleted after the PVC was created.

Fix: Identify the PVC from the error message. If no storage class is set, recreate the PVC with an appropriate storage class. Update the VM to use the new PVC before resubmitting the ProtectionGroup.

Symptom: kubectl apply returns admission webhook denied the request: Failed to acquire failover lock ... Failover lock is held by: failover-controller-<pid>

Cause: A FailoverRequest was submitted for a ProtectionGroup that is already being processed by the failover-controller. Concurrent failovers of the same Protection Group are prevented by a Kubernetes Lease.

Fix: Check whether the in-progress failover is still active:

kubectl get failoverrequests -n dr-<deployment-name>
kubectl get lease failover-lock-<pg-name> -n dr-<deployment-name> -o yaml

If the failover completed or the controller crashed, delete the stale Lease and resubmit:

kubectl delete lease failover-lock-<pg-name> -n dr-<deployment-name>

Leases expire automatically after 300 seconds if the holder does not renew them, so you can also wait for automatic expiry.

Symptom: kubectl apply returns Internal error occurred: failed calling webhook ... connection refused or dial tcp: ... i/o timeout

Cause: The controller pod that serves the webhook is not running or is not yet ready. Because Site Recovery webhooks use a Fail failure policy, the API server rejects the request rather than permitting an unvalidated resource.

Fix:

1. Identify which controller serves the webhook from the webhook name in the error message.
2. Check the pod status in the appropriate namespace:

# For protection-group-controller (runs on primary or DR cluster)
kubectl get pods -n dr-<deployment-name> -l app=protection-group-controller

# For failover-controller (runs on quorum cluster)
kubectl get pods -n dr-<deployment-name> -l app=failover-controller

# For test-failover-controller (runs on primary or DR cluster)
kubectl get pods -n dr-<deployment-name> -l app=test-failover-controller

3. If the pod is in CrashLoopBackOff or Error state, inspect logs and events:

kubectl logs -n dr-<deployment-name> <pod-name> --previous
kubectl describe pod -n dr-<deployment-name> <pod-name>

4. Resolve the underlying pod issue (image pull error, OOMKilled, missing Secret, etc.) and wait for the pod to reach Running with all containers ready before retrying the original operation.

Symptom: kubectl apply returns admission webhook denied the request: TestFailover is only supported for DRBD Operator deployment models

Cause: You submitted a TestFailover resource in a deployment that uses the centralized storage model. Test failover is only available for DRBD Operator deployments.

Fix: Verify your deployment model by checking whether a DRBDReplicationPolicy exists in the namespace:

kubectl get drbdreplicationpolicies -n dr-<deployment-name>

If no DRBDReplicationPolicy exists, the deployment uses the centralized storage model and TestFailover is not supported. Non-disruptive DR validation must be performed through another method appropriate for your deployment model.

Symptom: A ProtectionGroup was successfully created, but its status.state is Failed with a ValidationFailed condition after an update

Cause: An update to the ProtectionGroup spec (such as adding a new VM) passed the webhook but one of the newly referenced VMs or PVCs failed post-admission validation during controller reconciliation. The ProtectionGroup reverts to its last valid configuration.

Fix: Read the condition message:

kubectl get protectiongroup <name> -n dr-<deployment-name> \
  -o jsonpath='{.status.conditions[*].message}'

Also check the protection-group-controller logs for the full validation trace:

kubectl logs -n dr-<deployment-name> deployment/protection-group-controller \
  | grep <protectiongroup-name>

Resolve the reported issue with the VM or PVC, then reapply the updated ProtectionGroup spec.

Symptom: Webhook warnings appear in status.warnings but the resource was accepted

Cause: The ProtectionGroup passed all blocking validation rules but one or more VMs use a storage class that lacks recommended geo-placement configuration (for example, missing replicasOnDifferent). The controller admits the resource but records a warning because replicas may be placed on the same site, reducing DR effectiveness.

Fix: Review the warnings:

kubectl get protectiongroup <name> -n dr-<deployment-name> \
  -o jsonpath='{.status.warnings}'

Update the storage class to add cross-site placement parameters, then reapply the ProtectionGroup. The warnings will be cleared on the next reconciliation once the storage class is corrected.