Webhook Issues

Admission webhooks are the first line of defense for Trilio Site Recovery's custom resources: every ProtectionZone, ProtectionRequest, ProtectionGroup, FailoverRequest, TestFailover, ReplicationGroupStatus, and RPOEvent passes through a validating (and in some cases mutating) webhook before it is persisted to etcd. This page describes the webhook architecture, the exact validation rules each webhook enforces, the certificate prerequisites that must be satisfied before the webhooks can serve, and how to diagnose and resolve common admission failures. Understanding these rules is essential because a rejected admission request immediately blocks a DR workflow — protection, failover, or test failover — with an error message that names the offending field.

Before troubleshooting webhook issues, confirm the following:

• OpenShift 4.17 or later on all three clusters (quorum, primary, DR). Earlier 4.x releases are not supported.
• oc CLI — required for OpenShift-specific paths including log collection with oc adm must-gather. kubectl-only operation is not a supported mode.
• Kubeconfig access to the quorum cluster — all admission webhooks run there. Primary and DR clusters host only the workload control plane (DRBD node agent, webhooks for DRBDResource).
• webhook-server-cert Secret pre-created in every release namespace before install. Webhook serving certificates are not managed by the charts; they must be created out of band. The chart key is webhook.certSecretName (default: webhook-server-cert).
• webhook.caBundle — a base64-encoded CA bundle injected into each ValidatingWebhookConfiguration and MutatingWebhookConfiguration clientConfig.caBundle. Without it the API server cannot verify the webhook's TLS certificate and will reject or timeout every admission request.
• OLM-installed operator bundle in namespace trilio-site-recovery-system on the quorum cluster. The ProtectionZone controller deploys the quorum and workload control planes automatically once a ProtectionZone CR reaches Available; customers do not run helm install.
• ProtectionZone CR in Available phase before any other CR can be created. The quorum control plane (which hosts the admission webhooks for all CRDs except ProtectionZone itself) is not deployed until the zone is available.

Webhooks are installed automatically as part of the Trilio Site Recovery operator deployment. No manual webhook registration is required, but the TLS certificate Secret must exist before the webhook server can start.

Step 1 — Create the webhook-server-cert Secret

Create the TLS certificate Secret in the zone namespace (format: dr-<name>) on the quorum cluster before creating the ProtectionZone CR. The Secret name must match webhook.certSecretName (default: webhook-server-cert).

# Generate a self-signed cert (replace <zone-namespace> with your dr-<name> namespace)
oc create secret tls webhook-server-cert \
  --cert=tls.crt \
  --key=tls.key \
  -n <zone-namespace>

If you use a CA, also capture the CA bundle — you will need it for the caBundle field in the next step.

Step 2 — Verify the ProtectionZone controller is running

The protection-zone-controller (installed by OLM) handles the ProtectionZone admission webhook. Confirm it is ready on the quorum cluster:

oc get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller

Expected output (one pod, 1/1 Ready):

NAME                                         READY   STATUS    RESTARTS   AGE
<release>-pz-controller-<hash>              1/1     Running   0          5m

Step 3 — Create the ProtectionZone CR

Once the certificate Secret exists and the protection-zone-controller is ready, create your ProtectionZone. The mutating webhook defaults storageBackendMode to DRBD when omitted. The validating webhook enforces all other rules described in the Configuration section below.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod-zone
  namespace: dr-prod-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      kubeconfigSecretRef:
        name: primary-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      kubeconfigSecretRef:
        name: dr-kubeconfig
        key: kubeconfig

oc apply -f protection-zone.yaml -n dr-prod-zone

Step 4 — Wait for the quorum control plane to become ready

Once the ProtectionZone reaches Available, the ProtectionZone controller deploys the site-recovery-quorum-control-plane Helm chart (embedded in the operator binary) into the zone namespace. This deployment hosts the webhooks for all remaining CRDs.

oc get pz prod-zone -n dr-prod-zone -w
# Wait for STATUS = Available

oc get pods -n dr-prod-zone \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

Expected output (single container named manager, 1/1 Ready):

NAME                                                    READY   STATUS    RESTARTS   AGE
<release>-site-recovery-quorum-control-plane-<hash>    1/1     Running   0          2m

Webhook serving configuration

All webhook behaviour is governed by chart values under the webhook key. These values are set internally by the ProtectionZone controller when it deploys the embedded charts — customers do not run helm install or pass --set flags.

Per-resource validation rules

The following rules are enforced at admission time. Schema-level constraints (CRD enums, minLength, minimum/maximum) are enforced by the API server before the webhook runs; the webhook adds only cross-field and cross-resource rules.

ProtectionZone

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-protectionzone Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-protectionzone Verbs: create, update

• storageBackendMode defaults to DRBD when omitted (mutating webhook).
• storageBackendMode: Linstor is rejected with: "Linstor backend is not supported yet; use DRBD". DRBD is the only accepted value.
• Each cluster's kubeconfigSecretRef.name must be non-empty (the schema permits an empty name for backwards compatibility; the webhook closes that gap).
• Each cluster's kubeconfigSecretRef.key must be non-empty.
• The two clusters must have distinct clusterID values (cluster1 and cluster2). Both claiming the same slot is rejected with a Duplicate error on clusterID.
• clusters[].name has a schema maxLength of 63 (DNS-label-sized).
• clusters[].distributionName defaults to RHEL-CoreOS-9; accepted values are RHEL-CoreOS-9 and RHEL-CoreOS-10.
• On update, only displayName, description, and each cluster's kubeconfigSecretRef and distributionName may change. storageBackendMode, cluster name, and cluster clusterID are immutable for the zone's lifetime.

ProtectionRequest

Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-protectionrequest Verbs: create, update, delete Note: There is no mutating webhook for ProtectionRequest.

• spec.virtualMachine.namespace must be set explicitly. The VM lives on a workload cluster in its application namespace, which is unrelated to the ProtectionRequest's quorum-tenant namespace and cannot be inferred.
• replicationConfig.replicationProtocol: A is rejected with: "ReplicationProtocolA is not supported yet". C (synchronous, RPO=0) is the only supported protocol.
• replicationConfig.replicationProtocol: C with a non-zero rpo is rejected: "rpo must be 0 when spec.replicationConfig.replicationProtocol is C (synchronous)".
• replicationConfig defaults to protocol C, rpo: 0.
• replicationConfig.drbdMetadataConfig.size defaults to 128Mi; values below 128Mi are rejected by a CEL rule.
• On create, if another ProtectionRequest in the same namespace already references the same VM (name + namespace), the new request is rejected — including when the predecessor is Terminating. Re-protect requires the full teardown of the previous request to complete first.
• On delete, deletion is blocked when an in-flight FailoverRequest or TestFailover is operating on a ProtectionGroup that includes this VM.

TestFailover

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-testfailover Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-testfailover Verbs (mutating): create, update — Verbs (validating): create, update, delete

• protectionGroupRef.namespace is defaulted from the TestFailover's own namespace when omitted (mutating webhook). If the ProtectionGroup lives in a different namespace, set it explicitly — the defaulter does not overwrite an explicit value.
• protectionGroupRef.namespace must be non-empty after defaulting; the validator keeps a backstop field.Required.
• The referenced ProtectionGroup must exist on the quorum cluster at admission time.
• cleanupPolicy must be Manual or Automatic (schema enum).
• A Pending or InProgress FailoverRequest for the same ProtectionGroup blocks a TestFailover creation.
• Reachability probe cross-field rules:
  • endpoint and expectedStatus must not be set on tcp probes.
  • Probe timeout must not exceed 10m.
  • vmName, when set, must reference a virtualMachine in the target ProtectionGroup.

FailoverRequest

• spec.failoverType must be planned or unplanned (lowercase).
• spec.targetCluster must be cluster1 or cluster2.
• spec.drainTimeoutSeconds is accepted and range-checked but has no effect on failover behaviour in Tech Preview; do not rely on it.

ReplicationGroupStatus

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-replicationgroupstatus Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-replicationgroupstatus Verbs: create, update, delete

• protectionGroupRef.namespace is defaulted from the ReplicationGroupStatus's own namespace when omitted.
• The referenced ProtectionGroup must exist at admission time.
• spec.protectionGroupRef and spec.replicationProtocol are immutable once set.
• spec.pollingIntervalSeconds must be between 5 and 300 (schema). Defaults to 15.
• Only the replication-monitor service account may create, update, or delete ReplicationGroupStatus objects when MonitorServiceAccountName is configured. Kubernetes garbage-collector and namespace-controller identities are exempt.

RPOEvent

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-rpoevent Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-rpoevent Verbs: create, update, delete

• protectionGroupRef.namespace is defaulted from the RPOEvent's own namespace when omitted.
• The referenced ProtectionGroup must exist.
• RPOEvent is fully immutable — all updates are rejected.
• Only the replication-monitor service account may create or delete RPOEvent objects when MonitorServiceAccountName is configured.

Checking if a webhook is healthy

Webhook readiness depends on the quorum control plane pod being 1/1 Ready. Check it first:

oc get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

If the pod is 0/1 or CrashLoopBackOff, the webhook server is not serving — all admission requests will fail with a connection-refused or timeout error.

Reading a rejected admission response

When a resource is rejected, oc apply or oc create prints the webhook's field-level error. The error always names the offending field path:

Error from server: error when creating "protection-request.yaml":
admission webhook "vprotectionrequest-v1alpha1.siterecovery.trilio.io" denied the request:
spec.replicationConfig.replicationProtocol: Invalid value: "A": ReplicationProtocolA is not supported yet

The webhook name in the error (vprotectionrequest-v1alpha1.siterecovery.trilio.io) tells you which resource type and which webhook (mutating m... vs validating v...) rejected the request.

Reviewing webhook logs

Webhook validation runs inside the manager container of the quorum control plane pod. To follow webhook decision logs:

# Get the quorum pod name
QUORUM_POD=$(oc get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane \
  -o jsonpath='{.items[0].metadata.name}')

# Stream logs
oc logs -n dr-<zone-name> "$QUORUM_POD" -c manager --follow

Each admission decision is logged with the resource name and the validation result. Look for lines containing Validation for <Kind> upon creation or denied the request.

Checking webhook configurations

List the registered webhook configurations on the quorum cluster:

oc get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -l app.kubernetes.io/part-of=site-recovery

If these objects are missing, the quorum control plane chart has not been deployed yet (the ProtectionZone is not Available) or the chart manager has not yet completed its first reconcile.

Checking the webhook-server-cert Secret

oc get secret webhook-server-cert -n dr-<zone-name>

If the Secret is missing, the manager container cannot start its HTTPS server and every admission request will fail with a TLS handshake error.

Adjusting runtime log verbosity for webhook decisions

Webhook log verbosity follows the per-namespace site-recovery-logging ConfigMap. To enable debug level on the quorum cluster:

oc patch configmap site-recovery-logging -n dr-<zone-name> \
  --type=merge \
  -p '{"data":{"level":"debug"}}'

Accepted values for level: error, info, debug, trace. There is no warn level.

Example 1 — Rejected ProtectionZone: Linstor backend

Attempting to create a zone with storageBackendMode: Linstor:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: bad-zone
  namespace: dr-bad-zone
spec:
  displayName: "Bad Zone"
  storageBackendMode: Linstor   # rejected
  clusters:
    - name: primary
      clusterID: cluster1
      kubeconfigSecretRef:
        name: primary-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      kubeconfigSecretRef:
        name: dr-kubeconfig
        key: kubeconfig

oc apply -f bad-zone.yaml -n dr-bad-zone

Expected error output:

Error from server: error when creating "bad-zone.yaml":
admission webhook "vprotectionzone-v1alpha1.kb.io" denied the request:
spec.storageBackendMode: Invalid value: "Linstor": Linstor backend is not supported yet; use DRBD

Fix: Set storageBackendMode: DRBD or omit the field entirely (the mutating webhook defaults it to DRBD).

Example 2 — Rejected ProtectionRequest: Protocol A

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: pr-vm1
  namespace: dr-prod-zone
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm1
    namespace: workload-ns
  replicationConfig:
    replicationProtocol: A   # rejected

oc apply -f pr-vm1.yaml -n dr-prod-zone

Expected error output:

Error from server: error when creating "pr-vm1.yaml":
admission webhook "vprotectionrequest-v1alpha1.siterecovery.trilio.io" denied the request:
spec.replicationConfig.replicationProtocol: Invalid value: "A": ReplicationProtocolA is not supported yet

Fix: Omit replicationProtocol (it defaults to C) or set it explicitly to C. Protocol A is not supported in Tech Preview.

Example 3 — Rejected ProtectionRequest: non-zero RPO with Protocol C

spec:
  replicationConfig:
    replicationProtocol: C
    rpo: 5   # rejected — Protocol C is synchronous (RPO=0)

Expected error:

spec.replicationConfig.rpo: Invalid value: 5:
rpo must be 0 when spec.replicationConfig.replicationProtocol is C (synchronous)

Fix: Set rpo: 0 or omit it (default is 0).

Example 4 — Rejected ProtectionZone update: immutable field

# Attempt to rename a cluster after the zone is created
oc patch pz prod-zone -n dr-prod-zone --type=merge \
  -p '{"spec":{"clusters":[{"name":"primary-renamed","clusterID":"cluster1","kubeconfigSecretRef":{"name":"primary-kubeconfig","key":"kubeconfig"}}]}}'

Expected error:

admission webhook "vprotectionzone-v1alpha1.kb.io" denied the request:
spec.clusters[0].name: Forbidden: field is immutable

Fix: Cluster name and clusterID are immutable for the zone's lifetime. Only kubeconfigSecretRef, distributionName, displayName, and description may be changed after creation.

Example 5 — Rejected TestFailover: in-flight FailoverRequest

# A FailoverRequest is Pending for protection-group pg-prod
oc get fr -n dr-prod-zone
# NAME         PROTECTION GROUP   TARGET     STATE     PHASE
# fr-planned   pg-prod            cluster2   Pending   Initializing

# Attempt to create a TestFailover for the same PG
oc apply -f test-failover.yaml -n dr-prod-zone

Expected error:

admission webhook "vtestfailover-v1alpha1.siterecovery.trilio.io" denied the request:
a FailoverRequest is Pending or InProgress for ProtectionGroup pg-prod

Fix: Wait for the FailoverRequest to reach Completed or Failed before creating a TestFailover for the same ProtectionGroup.

Example 6 — Rejected ProtectionRequest: missing VM namespace

spec:
  sourceCluster: cluster1
  virtualMachine:
    name: vm1
    # namespace omitted — cannot be inferred from the PR's quorum-tenant namespace

Expected error:

spec.virtualMachine.namespace: Required value: must be set explicitly:
the VM lives on the workload cluster and its namespace cannot be inferred
from the ProtectionRequest's (quorum-tenant) namespace

Fix: Add namespace: <vm-application-namespace> to spec.virtualMachine.

Example 7 — Rejected duplicate ProtectionRequest

# pr-vm1 already exists for vm1 in namespace workload-ns (phase: Protected)
oc get pr -n dr-prod-zone
# NAME     VM    NAMESPACE     PHASE
# pr-vm1   vm1   workload-ns   Protected

# Attempting a second ProtectionRequest for the same VM
oc apply -f pr-vm1-duplicate.yaml -n dr-prod-zone

Expected error:

admission webhook "vprotectionrequest-v1alpha1.siterecovery.trilio.io" denied the request:
VM "workload-ns/vm1" is already protected by ProtectionRequest "pr-vm1"

Fix: You must unprotect the VM completely (delete the existing ProtectionRequest and wait for its teardown finalizer to complete) before creating a new one.

Use a consistent approach for every webhook failure: read the error message to identify the webhook and the rule that fired, inspect the webhook configuration to confirm it is registered and has a non-empty caBundle, verify the webhook server pod is running and its serving certificate is valid, then correct the manifest and reapply.

Issue 1 — connection refused or no endpoints available for service when applying any resource

Symptom:

Error from server (InternalError): error when creating "...": Internal error occurred:
failed calling webhook "...": Post "https://site-recovery-webhook-service.dr-<name>.svc:443/...": 
dial tcp ...: connect: connection refused

Likely causes:

• The webhook server pod is not running or not yet ready.
• The webhook Service has no healthy endpoints (pod label mismatch).

Fix:

# 1. Check that the manager pod is running and 1/1 ready
oc get pods -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

# 2. Check that the Service has endpoints
oc get endpoints -n dr-<name>

# 3. If the pod is in CrashLoopBackOff, check its logs
oc logs -n dr-<name> \
  -l control-plane=controller-manager \
  -c manager --previous

# 4. If there are no endpoints, check the Service selector
# The Service must select pods with label control-plane=controller-manager
oc describe service site-recovery-webhook-service -n dr-<name>

Issue 2 — x509: certificate signed by unknown authority or TLS handshake failure

Symptom:

Error from server (InternalError): ...: x509: certificate signed by unknown authority

Likely cause: The CA bundle in the webhook configuration does not match the CA that signed the certificate in webhook-server-cert. This happens when the certificate and CA were generated separately, or when the Secret was rotated. The ProtectionZone controller reconciles the webhook configuration from the Secret, so the fix is to correct the Secret and let it re-reconcile — there is no chart for you to upgrade.

Fix:

# 1. Extract the current serving certificate from the secret
oc get secret webhook-server-cert -n dr-<name> \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > current-tls.crt

# 2. Inspect it
openssl x509 -in current-tls.crt -noout -text | grep -A5 Issuer

# 3. Recreate webhook-server-cert from a certificate and CA that match
oc delete secret webhook-server-cert -n dr-<name>
oc create secret tls webhook-server-cert -n dr-<name> \
  --cert=tls.crt --key=tls.key

# 5. Verify the webhook configuration was updated
oc get validatingwebhookconfiguration -o yaml | grep caBundle

Issue 3 — ProtectionZone creation rejected: Linstor backend is not supported yet; use DRBD

Symptom:

admission webhook "protectionzone.webhook.siterecovery.trilio.io" denied the request:
Linstor backend is not supported yet; use DRBD

Likely cause: spec.storageBackendMode is set to Linstor. This value is defined in the API enum but the admission webhook unconditionally rejects it. DRBD is the only supported value.

Fix:

# Edit your ProtectionZone manifest and set:
#   spec.storageBackendMode: DRBD
# then reapply
oc apply -f protectionzone.yaml

Issue 4 — ProtectionZone rejected: cluster count or clusterID validation error

Symptom:

spec.clusters: Too few items, minimum is 2
# or
spec.clusters[0].clusterID: Unsupported value: "primary": supported values: "cluster1", "cluster2"

Likely cause: The clusters array has fewer than two entries, more than two entries, or uses a free-form string for clusterID instead of the enum values cluster1 or cluster2.

Fix:

Ensure your ProtectionZone spec has exactly two cluster entries and that each clusterID is either cluster1 or cluster2:

spec:
  clusters:
    - name: my-primary
      clusterID: cluster1          # must be exactly "cluster1"
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: my-dr
      clusterID: cluster2          # must be exactly "cluster2"
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Issue 5 — ProtectionGroup update rejected: VMs span multiple namespaces

Symptom:

admission webhook "protectiongroup.webhook.siterecovery.trilio.io" denied the request:
all VMs in a ProtectionGroup must be in the same namespace

Likely cause: You are attempting to add a VM from a different namespace to an existing ProtectionGroup, or creating a new ProtectionGroup whose member VMs are spread across namespaces. This is a Tech Preview limitation (TSR-162); there are no exceptions.

Fix:

All VMs in a ProtectionGroup must reside in the same namespace. If you have an existing multi-namespace ProtectionGroup that predates this restriction, you cannot edit it—any spec change will be rejected. Delete it and recreate it as single-namespace groups:

# Delete the existing multi-namespace group
oc delete protectiongroup <name> -n dr-<name>

# Create separate single-namespace groups
oc apply -f pg-namespace-a.yaml
oc apply -f pg-namespace-b.yaml

Issue 6 — Webhook timeout: context deadline exceeded

Symptom:

Error from server (InternalError): ...: context deadline exceeded

Likely cause: The webhook server pod is overloaded, the network path between the API server and the webhook Service is broken, or the pod is running but too slow to respond within the API server's timeout.

Fix:

# 1. Check pod resource usage
oc top pod -n dr-<name> \
  -l control-plane=controller-manager

# 2. Check for recent restarts indicating OOM
oc get pods -n dr-<name> \
  -l control-plane=controller-manager

# 3. Stream logs for errors during the window the timeout occurred
oc logs -n dr-<name> \
  -l control-plane=controller-manager \
  -c manager --since=10m

# 4. Verify network connectivity from the API server to the webhook Service
# The webhook Service must be reachable on port 443 from the control plane nodes
oc describe validatingwebhookconfiguration | grep -A5 clientConfig

If the pod is repeatedly OOM-killed, increase its memory request in the Helm values and upgrade the release.

Issue 7 — webhook-server-cert secret not found; controller pod fails to start

Symptom:

Error: secret "webhook-server-cert" not found

(from oc describe pod events or controller logs)

Likely cause: The TLS serving certificate secret was not created before the Helm chart was installed. The chart does not create this secret.

Fix:

Create the secret before (re)installing the chart, as described in the Installation section:

# Confirm the secret is missing
oc get secret webhook-server-cert -n dr-<name>

# Create it (replace with your actual cert and key files)
oc create secret tls webhook-server-cert \
  --cert=tls.crt \
  --key=tls.key \
  -n dr-<name>

# Restart the controller pod so it can mount the new secret
oc rollout restart deployment/site-recovery-quorum-control-plane -n dr-<name>

Collecting a diagnostic bundle

If you cannot determine the root cause from the steps above, collect a full diagnostic bundle using tsr-gather and attach it to your support case:

oc adm must-gather \
  --image=registry.connect.redhat.com/trilio/tsr-gather:latest \
  --dest-dir=./tsr-support-bundle

The bundle captures webhook configuration state, controller logs, and CRD status from all clusters in the DR estate without collecting secrets or sensitive credentials.