Trilio Site Recovery for Kubernetes
Docs/site-recovery/Troubleshooting/Webhook Issues
Guide

Webhook Issues

Admission webhook failures and validation errors on recovery resource creation

Overview

This page explains how to diagnose and resolve admission webhook failures that occur when creating or updating Site Recovery custom resources such as ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, and DRBDReplicationPolicy. Admission webhooks enforce validation rules before any DR resource reaches the cluster API—if a webhook is unavailable or a resource fails validation, the operation is rejected immediately with an error. Understanding these failure modes is critical because a misconfigured or unavailable webhook can block VM protection, failover triggers, and test failover operations entirely.

Prerequisites

Before troubleshooting webhook issues, confirm the following:

  • OpenShift 4.14 or later
  • KubeVirt or OpenShift Virtualization (CNV) 1.0 or later
  • kubectl or oc CLI available and configured with kubeconfig files for the quorum cluster, primary cluster, and DR cluster
  • Helm 3.0 or later (if you need to reinstall or upgrade webhook components)
  • The site-recovery-protectionzone-controller chart was installed on the quorum cluster before the site-recovery-quorum-control-plane chart—the ProtectionZone admission webhook and its CRDs must be present before the quorum control plane initializes
  • Network connectivity: TCP port 6443 open from the quorum cluster to the primary and DR cluster API servers
  • Sufficient RBAC permissions to inspect ValidatingWebhookConfiguration, MutatingWebhookConfiguration, pods, and events in the DR namespace (dr-<name>) on the quorum cluster
Installation

Webhook components are installed as part of the Site Recovery Helm charts. If a webhook is missing or broken, reinstall the relevant chart.

Step 1: Verify the ProtectionZone webhook is installed first

The site-recovery-protectionzone-controller chart must be installed before the quorum control plane chart. Check whether it is present:

helm list -n dr-<name> --kubeconfig ~/.kube/config-quorum

Expected output includes both site-recovery-protectionzone-controller and site-recovery-quorum-control-plane entries. If the ProtectionZone chart is missing, install it first:

helm install site-recovery-protectionzone-controller \
  trilio/site-recovery-protectionzone-controller \
  -n dr-<name> \
  --kubeconfig ~/.kube/config-quorum

Step 2: Install the quorum control plane chart

If the quorum control plane is missing or was installed before the ProtectionZone chart:

helm install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  -n dr-<name> \
  --kubeconfig ~/.kube/config-quorum

Step 3: Install the workload control plane on primary and DR clusters

Admission webhooks on the workload side are installed by the site-recovery-workload-control-plane chart. Run this against both clusters:

# Primary cluster
helm install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  -n dr-<name> \
  --kubeconfig ~/.kube/config-primary

# DR cluster
helm install site-recovery-workload-control-plane \
  trilio/site-recovery-workload-control-plane \
  -n dr-<name> \
  --kubeconfig ~/.kube/config-dr

Step 4: Confirm webhook configurations are registered

# Quorum cluster — ProtectionZone and quorum control plane webhooks
oc get validatingwebhookconfiguration,mutatingwebhookconfiguration \
  --kubeconfig ~/.kube/config-quorum

# Primary cluster — workload-side webhooks
oc get validatingwebhookconfiguration,mutatingwebhookconfiguration \
  --kubeconfig ~/.kube/config-primary

If expected webhook configurations are absent, the relevant Helm chart did not install successfully. Check Helm release status:

helm status site-recovery-protectionzone-controller \
  -n dr-<name> \
  --kubeconfig ~/.kube/config-quorum
Configuration

Webhook behavior is determined by the Helm chart values used at install time and by the Site Recovery CRDs themselves. The following describes the key areas that affect webhook validation.

ProtectionZone admission webhook

The site-recovery-protectionzone-controller chart installs the webhook that validates ProtectionZone CRs before they are admitted. A ProtectionZone requires:

  • spec.displayName (required string)
  • spec.clusters (required array — must list valid cluster references)
  • spec.storageBackendMode (required string — controls whether DRBD Operator mode is used)

If any required field is absent or invalid, the webhook rejects the create or update with a descriptive error.

ProtectionGroup namespace restriction

The validating webhook enforces that all VMs in a ProtectionGroup belong to the same namespace. Any create or update that would result in member VMs spanning more than one namespace is rejected. This restriction applies even to ProtectionGroup objects that existed before the rule was introduced. If you have a multi-namespace ProtectionGroup, you must delete it and recreate it as separate single-namespace groups.

FailoverRequest required fields

The webhook validates FailoverRequest resources before admission. The following fields are required and must be present and well-formed:

  • spec.protectionGroupRef (required object — must reference an existing ProtectionGroup)
  • spec.targetCluster (required string)

Optional fields such as spec.failoverType, spec.drainTimeoutSeconds, and spec.batchBootTimeoutSeconds are validated for type correctness if provided.

TestFailover required fields

The webhook validates TestFailover resources. The only required field is:

  • spec.protectionGroupRef (required object)

Optional fields (spec.cleanupPolicy, spec.retentionTime, spec.timeout, spec.verification, spec.batchBootTimeoutSeconds) are type-checked if present.

DRBDReplicationPolicy required fields

For DRBD Operator deployments, the webhook validates DRBDReplicationPolicy resources. Required:

  • spec.clusters (required array)

Optional fields such as spec.drbdProtocol, spec.replicationMode, spec.rpo, spec.networkConfig, spec.diskConfig, and spec.resyncConfig are type-checked if provided.

Webhook failure policy

By default, the Site Recovery webhooks use a Fail failure policy, meaning that if the webhook service is unreachable, all resource create and update operations for Site Recovery CRDs are blocked. This is intentional to prevent invalid DR state from being admitted to the cluster. If a webhook pod is down, restore it before attempting any DR operations.

Usage

Use the following patterns to investigate and resolve webhook issues in day-to-day operations.

Check whether a webhook rejection is a validation error or a connectivity error

When you run oc apply or kubectl apply and the operation is rejected, the error message will indicate which:

  • A validation error contains a human-readable message from the webhook controller, such as "all VMs in a ProtectionGroup must be in the same namespace" or "spec.protectionGroupRef is required".
  • A connectivity error will reference connection refused, timeout, no endpoints available, or a TLS handshake failure rather than a policy message.

Inspect webhook pods on the quorum cluster

The site-recovery-quorum-control-plane deployment on the quorum cluster hosts the admission webhook reconcilers. Check whether its pods are running:

oc get pods -n dr-<name> \
  -l app.kubernetes.io/name=site-recovery-quorum-control-plane \
  --kubeconfig ~/.kube/config-quorum

If pods are in CrashLoopBackOff, Pending, or Error state, retrieve logs:

oc logs -n dr-<name> \
  deployment/site-recovery-quorum-control-plane \
  --kubeconfig ~/.kube/config-quorum

Inspect the ProtectionZone controller pod

oc get pods -n dr-<name> \
  -l app.kubernetes.io/name=site-recovery-protectionzone-controller \
  --kubeconfig ~/.kube/config-quorum

oc logs -n dr-<name> \
  deployment/site-recovery-protectionzone-controller \
  --kubeconfig ~/.kube/config-quorum

Inspect workload-side webhook pods

For rejections that occur on the primary or DR cluster when applying ProtectionGroup or DRBDReplicationPolicy resources:

oc get pods -n dr-<name> \
  -l app.kubernetes.io/name=site-recovery-workload-control-plane \
  --kubeconfig ~/.kube/config-primary

oc logs -n dr-<name> \
  deployment/site-recovery-workload-control-plane \
  --kubeconfig ~/.kube/config-primary

Collect a full diagnostic bundle

If you cannot identify the root cause from logs alone, use tsr-gather to collect logs, CRD state, and configuration from all clusters:

oc adm must-gather \
  --image=<tsr-gather-image> \
  --kubeconfig ~/.kube/config-quorum

This produces a single bundle containing webhook configurations, controller logs, and CRD states across the DR estate without capturing secrets.

Examples

Example 1: ProtectionGroup rejected due to multi-namespace VMs

Attempting to add a VM from a different namespace to an existing ProtectionGroup:

oc patch protectiongroup production-pg \
  -n workloads \
  --type merge \
  --kubeconfig ~/.kube/config-primary \
  -p '{"spec":{"virtualMachines":[{"name":"vm-a"},{"name":"vm-b"}]}}'

Expected rejection output:

Error from server: admission webhook "protectiongroup.siterecovery.trilio.io" denied the request:
all VMs in a ProtectionGroup must be in the same namespace; vm-b is in namespace "other-ns", expected "workloads"

Resolution: Ensure all VMs listed in spec.virtualMachines exist in the same namespace as the ProtectionGroup. If you need to protect VMs from multiple namespaces, create separate ProtectionGroup objects per namespace.

Example 2: FailoverRequest rejected due to missing required field

Applying a FailoverRequest without a targetCluster:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-production
  namespace: dr-production
spec:
  protectionGroupRef:
    name: production-pg
    namespace: workloads
  failoverType: planned

oc apply -f failover-production.yaml \
  --kubeconfig ~/.kube/config-quorum

Expected rejection output:

Error from server: admission webhook "failoverrequest.siterecovery.trilio.io" denied the request:
spec.targetCluster is required

Resolution: Add the spec.targetCluster field:

spec:
  protectionGroupRef:
    name: production-pg
    namespace: workloads
  targetCluster: dr-cluster
  failoverType: planned

Example 3: Webhook connectivity failure blocking all DR resource operations

The site-recovery-protectionzone-controller pod is down. Any attempt to create or modify a ProtectionZone is rejected:

oc apply -f protection-zone.yaml \
  --kubeconfig ~/.kube/config-quorum

Expected error:

Error from server: Internal error occurred: failed calling webhook
"protectionzone.siterecovery.trilio.io": Post "https://site-recovery-protectionzone-controller.dr-production.svc:443/validate":
connect: connection refused

Check pod status:

oc get pods -n dr-production \
  -l app.kubernetes.io/name=site-recovery-protectionzone-controller \
  --kubeconfig ~/.kube/config-quorum

Expected output showing the problem:

NAME                                                   READY   STATUS             RESTARTS
site-recovery-protectionzone-controller-7d9f6b-xk2jp   0/1     CrashLoopBackOff   5

Retrieve logs to identify the root cause:

oc logs -n dr-production \
  deployment/site-recovery-protectionzone-controller \
  --kubeconfig ~/.kube/config-quorum

Example 4: ProtectionZone rejected due to missing required fields

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: zone-east
  namespace: dr-production
spec:
  clusters:
    - name: primary-cluster
    - name: dr-cluster
  description: "East region DR zone"

oc apply -f zone-east.yaml \
  --kubeconfig ~/.kube/config-quorum

Expected rejection:

Error from server: admission webhook "protectionzone.siterecovery.trilio.io" denied the request:
spec.displayName is required; spec.storageBackendMode is required

Resolution: Add the missing required fields:

spec:
  displayName: "East Region DR Zone"
  storageBackendMode: drbd-operator
  clusters:
    - name: primary-cluster
    - name: dr-cluster
  description: "East region DR zone"
Troubleshooting

Use the consistent format below—Symptom → Likely Cause → Fix—for each issue.

Issue: connection refused or timeout when applying any Site Recovery CR

Symptom: Every oc apply for a ProtectionGroup, FailoverRequest, TestFailover, ProtectionRequest, DRBDReplicationPolicy, or ProtectionZone returns an error containing connection refused, i/o timeout, or no endpoints available for service.

Likely cause: The webhook pod (site-recovery-quorum-control-plane, site-recovery-protectionzone-controller, or site-recovery-workload-control-plane) is not running or its service has no ready endpoints. Because the webhook failure policy is Fail, all admission calls block until the webhook pod is healthy.

Fix:

  1. Identify the affected cluster and namespace from the error message.
  2. Check pod status: oc get pods -n dr-<name> --kubeconfig ~/.kube/config-<cluster>
  3. If pods are in CrashLoopBackOff or Pending, retrieve logs: oc logs -n dr-<name> deployment/<component> --kubeconfig ~/.kube/config-<cluster>
  4. Address the root cause (image pull failure, OOMKilled, missing secret) and wait for the pod to reach Running state.
  5. Retry your oc apply.

Issue: ProtectionGroup create or update rejected with a namespace mismatch error

Symptom: admission webhook denied the request: all VMs in a ProtectionGroup must be in the same namespace

Likely cause: You are attempting to add a VM that lives in a different namespace from the other members of the ProtectionGroup. The validating webhook enforces single-namespace membership. This restriction also applies to ProtectionGroup objects created before the restriction was introduced—any spec change on a pre-existing multi-namespace group is rejected.

Fix:

  1. Verify the namespace of each VM listed in spec.virtualMachines: oc get vm <name> -n <namespace> --kubeconfig ~/.kube/config-primary
  2. Move all VMs to the same namespace, or create separate ProtectionGroup objects—one per namespace.
  3. If an existing ProtectionGroup has members in multiple namespaces and cannot be edited, delete it and recreate it as single-namespace groups.

Issue: FailoverRequest or TestFailover rejected with a missing required field error

Symptom: admission webhook denied the request: spec.targetCluster is required or spec.protectionGroupRef is required

Likely cause: The submitted manifest is missing one or more fields that the webhook requires. FailoverRequest requires both spec.protectionGroupRef and spec.targetCluster. TestFailover requires spec.protectionGroupRef.

Fix:

  1. Review the error message to identify which field is missing.
  2. Add the missing field to your manifest. For FailoverRequest: 
    spec:
  protectionGroupRef:
    name: <protection-group-name>
    namespace: <namespace>
  targetCluster: <dr-cluster-name>
  3. Reapply the manifest.

Issue: ProtectionZone rejected with missing displayName or storageBackendMode

Symptom: admission webhook denied the request: spec.displayName is required or spec.storageBackendMode is required

Likely cause: The ProtectionZone manifest omits one or both required fields. The site-recovery-protectionzone-controller webhook enforces these fields at admission time.

Fix:

  1. Add spec.displayName (a human-readable string) and spec.storageBackendMode to your ProtectionZone manifest.
  2. Reapply: oc apply -f <protectionzone-manifest>.yaml --kubeconfig ~/.kube/config-quorum

Issue: site-recovery-protectionzone-controller was installed after site-recovery-quorum-control-plane, causing CRD or webhook initialization errors

Symptom: The quorum control plane pod logs contain errors referencing missing CRDs or webhook configurations immediately after installation. ProtectionZone resources cannot be created.

Likely cause: The site-recovery-protectionzone-controller chart installs the ProtectionZone CRD and its admission webhook. If this chart is installed after site-recovery-quorum-control-plane, the quorum control plane initializes before the webhook and CRD exist.

Fix:

  1. Uninstall both charts: helm uninstall site-recovery-quorum-control-plane -n dr-<name> --kubeconfig ~/.kube/config-quorum and helm uninstall site-recovery-protectionzone-controller -n dr-<name> --kubeconfig ~/.kube/config-quorum
  2. Reinstall in the correct order—ProtectionZone controller first, then quorum control plane: 
    helm install site-recovery-protectionzone-controller \
  trilio/site-recovery-protectionzone-controller \
  -n dr-<name> --kubeconfig ~/.kube/config-quorum

helm install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  -n dr-<name> --kubeconfig ~/.kube/config-quorum

Issue: Webhook TLS certificate error

Symptom: Error contains x509: certificate signed by unknown authority or TLS handshake error.

Likely cause: The webhook service certificate has expired, or the caBundle in the ValidatingWebhookConfiguration or MutatingWebhookConfiguration does not match the certificate currently served by the webhook pod.

Fix:

  1. Check webhook configuration: oc get validatingwebhookconfiguration -o yaml --kubeconfig ~/.kube/config-<cluster>
  2. Verify the caBundle field is populated and current.
  3. Reinstall the relevant Helm chart with helm upgrade --install to regenerate certificates and update the webhook configuration: 
    helm upgrade --install site-recovery-quorum-control-plane \
  trilio/site-recovery-quorum-control-plane \
  -n dr-<name> --kubeconfig ~/.kube/config-quorum

Issue: Cannot determine root cause from pod logs alone

Symptom: Pod logs are insufficient, truncated, or the problem is intermittent.

Likely cause: Multi-cluster issues, transient API server connectivity problems, or a configuration state that is not visible from a single cluster.

Fix:

  1. Run tsr-gather against the quorum cluster to collect logs, CRD state, and webhook configurations from all clusters in the DR estate: 
    oc adm must-gather \
  --image=<tsr-gather-image> \
  --kubeconfig ~/.kube/config-quorum
  2. Attach the resulting bundle when opening a support case.