Trilio Site Recovery for Kubernetes
Docs/site-recovery/Troubleshooting/Webhook Issues
Guide

Webhook Issues

Admission webhook failures and validation errors on recovery resource creation

Troubleshooting

Use a consistent approach for every webhook failure: read the error message to identify the webhook and the rule that fired, inspect the webhook configuration to confirm it is registered and has a non-empty caBundle, verify the webhook server pod is running and its serving certificate is valid, then correct the manifest and reapply.

Issue 1 — connection refused or no endpoints available for service when applying any resource

Symptom:

Error from server (InternalError): error when creating "...": Internal error occurred:
failed calling webhook "...": Post "https://site-recovery-webhook-service.dr-<name>.svc:443/...": 
dial tcp ...: connect: connection refused

Likely causes:

  • The webhook server pod is not running or not yet ready.
  • The webhook Service has no healthy endpoints (pod label mismatch).

Fix:

# 1. Check that the manager pod is running and 1/1 ready
oc get pods -n dr-<name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=quorum-control-plane

# 2. Check that the Service has endpoints
oc get endpoints -n dr-<name>

# 3. If the pod is in CrashLoopBackOff, check its logs
oc logs -n dr-<name> \
  -l control-plane=controller-manager \
  -c manager --previous

# 4. If there are no endpoints, check the Service selector
# The Service must select pods with label control-plane=controller-manager
oc describe service site-recovery-webhook-service -n dr-<name>

Issue 2 — x509: certificate signed by unknown authority or TLS handshake failure

Symptom:

Error from server (InternalError): ...: x509: certificate signed by unknown authority

Likely cause: The CA bundle in the webhook configuration does not match the CA that signed the certificate in webhook-server-cert. This happens when the certificate and CA were generated separately, or when the Secret was rotated. The ProtectionZone controller reconciles the webhook configuration from the Secret, so the fix is to correct the Secret and let it re-reconcile — there is no chart for you to upgrade.

Fix:

# 1. Extract the current serving certificate from the secret
oc get secret webhook-server-cert -n dr-<name> \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > current-tls.crt

# 2. Inspect it
openssl x509 -in current-tls.crt -noout -text | grep -A5 Issuer

# 3. Recreate webhook-server-cert from a certificate and CA that match
oc delete secret webhook-server-cert -n dr-<name>
oc create secret tls webhook-server-cert -n dr-<name> \
  --cert=tls.crt --key=tls.key

# 5. Verify the webhook configuration was updated
oc get validatingwebhookconfiguration -o yaml | grep caBundle

Issue 3 — ProtectionZone creation rejected: Linstor backend is not supported yet; use DRBD

Symptom:

admission webhook "protectionzone.webhook.siterecovery.trilio.io" denied the request:
Linstor backend is not supported yet; use DRBD

Likely cause: spec.storageBackendMode is set to Linstor. This value is defined in the API enum but the admission webhook unconditionally rejects it. DRBD is the only supported value.

Fix:

# Edit your ProtectionZone manifest and set:
#   spec.storageBackendMode: DRBD
# then reapply
oc apply -f protectionzone.yaml

Issue 4 — ProtectionZone rejected: cluster count or clusterID validation error

Symptom:

spec.clusters: Too few items, minimum is 2
# or
spec.clusters[0].clusterID: Unsupported value: "primary": supported values: "cluster1", "cluster2"

Likely cause: The clusters array has fewer than two entries, more than two entries, or uses a free-form string for clusterID instead of the enum values cluster1 or cluster2.

Fix:

Ensure your ProtectionZone spec has exactly two cluster entries and that each clusterID is either cluster1 or cluster2:

spec:
  clusters:
    - name: my-primary
      clusterID: cluster1          # must be exactly "cluster1"
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: my-dr
      clusterID: cluster2          # must be exactly "cluster2"
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Issue 5 — ProtectionGroup update rejected: VMs span multiple namespaces

Symptom:

admission webhook "protectiongroup.webhook.siterecovery.trilio.io" denied the request:
all VMs in a ProtectionGroup must be in the same namespace

Likely cause: You are attempting to add a VM from a different namespace to an existing ProtectionGroup, or creating a new ProtectionGroup whose member VMs are spread across namespaces. This is a Tech Preview limitation (TSR-162); there are no exceptions.

Fix:

All VMs in a ProtectionGroup must reside in the same namespace. If you have an existing multi-namespace ProtectionGroup that predates this restriction, you cannot edit it—any spec change will be rejected. Delete it and recreate it as single-namespace groups:

# Delete the existing multi-namespace group
oc delete protectiongroup <name> -n dr-<name>

# Create separate single-namespace groups
oc apply -f pg-namespace-a.yaml
oc apply -f pg-namespace-b.yaml

Issue 6 — Webhook timeout: context deadline exceeded

Symptom:

Error from server (InternalError): ...: context deadline exceeded

Likely cause: The webhook server pod is overloaded, the network path between the API server and the webhook Service is broken, or the pod is running but too slow to respond within the API server's timeout.

Fix:

# 1. Check pod resource usage
oc top pod -n dr-<name> \
  -l control-plane=controller-manager

# 2. Check for recent restarts indicating OOM
oc get pods -n dr-<name> \
  -l control-plane=controller-manager

# 3. Stream logs for errors during the window the timeout occurred
oc logs -n dr-<name> \
  -l control-plane=controller-manager \
  -c manager --since=10m

# 4. Verify network connectivity from the API server to the webhook Service
# The webhook Service must be reachable on port 443 from the control plane nodes
oc describe validatingwebhookconfiguration | grep -A5 clientConfig

If the pod is repeatedly OOM-killed, increase its memory request in the Helm values and upgrade the release.

Issue 7 — webhook-server-cert secret not found; controller pod fails to start

Symptom:

Error: secret "webhook-server-cert" not found

(from oc describe pod events or controller logs)

Likely cause: The TLS serving certificate secret was not created before the Helm chart was installed. The chart does not create this secret.

Fix:

Create the secret before (re)installing the chart, as described in the Installation section:

# Confirm the secret is missing
oc get secret webhook-server-cert -n dr-<name>

# Create it (replace with your actual cert and key files)
oc create secret tls webhook-server-cert \
  --cert=tls.crt \
  --key=tls.key \
  -n dr-<name>

# Restart the controller pod so it can mount the new secret
oc rollout restart deployment/site-recovery-quorum-control-plane -n dr-<name>

Collecting a diagnostic bundle

If you cannot determine the root cause from the steps above, collect a full diagnostic bundle using tsr-gather and attach it to your support case:

oc adm must-gather \
  --image=registry.connect.redhat.com/trilio/tsr-gather:latest \
  --dest-dir=./tsr-support-bundle

The bundle captures webhook configuration state, controller logs, and CRD status from all clusters in the DR estate without collecting secrets or sensitive credentials.