Runbook

Troubleshooting

Common API errors and debugging

Objective

Use this runbook to diagnose and resolve common errors encountered when deploying the CSI Hostpath driver, provisioning volumes, or verifying driver operation in a Kubernetes cluster.

Scope

This runbook covers troubleshooting for:

Failed or incomplete driver deployment
Pods not reaching Running status after deployment
PVCs stuck in Pending state
Volumes not mounting correctly inside application pods
VolumeAttachment objects missing or in an error state
Snapshot provisioning failures

This runbook does not cover:

Rebuilding the driver binary from source
Multi-node cluster networking issues unrelated to CSI
Kubernetes control plane failures
Snapshot feature gate configuration beyond enabling VolumeSnapshotDataSource

Prerequisites

Before following this runbook, ensure you have:

A running Kubernetes cluster at version 1.13 or later
kubectl installed and configured with access to the cluster
The CSI Hostpath driver previously deployed via deploy/kubernetes-<version>/deploy-hostpath.sh
Sufficient RBAC permissions to describe pods, inspect events, and exec into containers
For snapshot-related issues: the VolumeSnapshotDataSource feature gate enabled (required for Kubernetes v1.12–v1.16 where snapshots are alpha)

Steps

Work through these steps in order. Stop at the step that resolves your issue.

Check that all driver pods are running.

Run:

kubectl get pods

Expected output shows all four pods with STATUS: Running and no unexpected restarts:

csi-hostpath-attacher-0      1/1     Running   0
csi-hostpath-provisioner-0   1/1     Running   0
csi-hostpath-snapshotter-0   1/1     Running   0
csi-hostpathplugin-0         2/2     Running   0

If any pod is in Pending, CrashLoopBackOff, or Error state, proceed to step 2.

Inspect the failing pod for errors.

Replace <pod-name> with the name of the unhealthy pod:
```
kubectl describe pod <pod-name>
```
Scroll to the Events section at the bottom of the output. Common issues to look for:
- FailedScheduling — the node may lack resources or a required label
- ImagePullBackOff — the container image (e.g., quay.io/k8scsi/hostpathplugin:v1.0.1) could not be pulled; verify network access to quay.io
- CrashLoopBackOff — the container is starting and immediately exiting; retrieve logs in step 3
Retrieve container logs for the crashing component.

For the main plugin container:
```
kubectl logs csi-hostpathplugin-0 -c hostpath
```
For the node driver registrar sidecar:
```
kubectl logs csi-hostpathplugin-0 -c node-driver-registrar
```
For the liveness probe sidecar:
```
kubectl logs csi-hostpathplugin-0 -c liveness-probe
```
Look for stack traces, permission errors, or socket-related failures. The liveness probe monitors CSI service health; repeated failures here indicate the driver gRPC server is not responding.
Diagnose a PVC stuck in Pending state.

Run:
```
kubectl get pvc
kubectl describe pvc csi-pvc
```
In the Events section, look for provisioner errors. A healthy bound PVC looks like:
```
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
csi-pvc   Bound    pvc-58d5ec38-03e5-11e9-be51-000c29e88ff1   1Gi        RWO            csi-hostpath-sc
```
If the PVC remains Pending:
- Confirm the csi-hostpath-sc StorageClass exists: kubectl get storageclass
- Confirm the provisioner pod csi-hostpath-provisioner-0 is running (step 1)
- Check provisioner logs: kubectl logs csi-hostpath-provisioner-0
Diagnose an application pod that fails to start or mount the volume.

Run:
```
kubectl describe pods/my-csi-app
```
Check the Events section for FailedMount or FailedAttach errors. Then inspect the attacher logs:
```
kubectl logs csi-hostpath-attacher-0
```
A successfully attached volume produces a VolumeAttachment object with Status.Attached: true. Verify this in step 6.
Verify the VolumeAttachment object exists and is healthy.

Run:
```
kubectl describe volumeattachment
```
Look for Status.Attached: true in the output:
```
Status:
  Attached: true
```
If Attached is false or the object is missing entirely, the external attacher is not communicating with the driver. Re-check attacher pod logs (step 5) and confirm the csi-hostpath-attacher service is present:
```
kubectl get svc csi-hostpath-attacher
```
Confirm volume data path is working end-to-end.

Exec into the application pod and write a test file:
```
kubectl exec -it my-csi-app /bin/sh
/ # touch /data/hello-world
/ # exit
```
Then exec into the hostpath plugin container and search for the file:
```
kubectl exec -it $(kubectl get pods --selector app=csi-hostpathplugin -o jsonpath='{.items[*].metadata.name}') -c hostpath /bin/sh
/ # find / -name hello-world
/tmp/057485ab-c714-11e8-bb16-000c2967769a/hello-world
/ # exit
```
If the file is not found under /tmp inside the hostpath container, the volume bind-mount is broken. Recheck pod scheduling — both the app pod and the plugin DaemonSet pod must be on the same node.
Diagnose snapshot provisioning failures.

Confirm the snapshot class exists:
```
kubectl get volumesnapshotclass
```
Expected output:
```
NAME                     AGE
csi-hostpath-snapclass   11s
```
If the class is missing, re-run the deployment script which creates it. If snapshots fail after the class exists, verify the VolumeSnapshotDataSource feature gate is enabled on your cluster's API server and that the snapshotter pod is healthy:
```
kubectl logs csi-hostpath-snapshotter-0
```
Inspect the snapshot object for status details:
```
kubectl describe volumesnapshot
```
Look for Status.Ready: true. A false value accompanied by an error message in Events points to a driver-side failure.

Verification

After completing the relevant steps above, confirm the full workflow is healthy by checking all of the following observable outcomes:

All pods running:
```
kubectl get pods
```
All four driver pods report STATUS: Running with RESTARTS: 0 (or a stable low number).
PVC bound:
```
kubectl get pvc
```
The csi-pvc entry shows STATUS: Bound.
PV exists:
```
kubectl get pv
```
A corresponding PersistentVolume with STATUS: Bound is listed.
Application pod running:
```
kubectl describe pods/my-csi-app
```
Status: Running and all conditions (Initialized, Ready, ContainersReady, PodScheduled) show True.
VolumeAttachment attached:
```
kubectl describe volumeattachment
```
Status.Attached: true.
Data path verified: The hello-world file written in the app pod is discoverable under /tmp/<volume-id>/hello-world inside the hostpath container.

Rollback

If you need to remove the CSI Hostpath driver deployment and return the cluster to a clean state:

Delete the example application resources (pod, PVC, StorageClass):
```
kubectl delete -f ./examples
```
Wait until the PVC is fully deleted before proceeding, as the Delete reclaim policy will trigger volume cleanup on the driver side.

Delete snapshot resources if you created them:

kubectl delete volumesnapshot new-snapshot-demo
kubectl delete volumesnapshotclass csi-hostpath-snapclass

Delete the driver components. Re-run the deployment manifests with delete instead of apply:
```
kubectl delete -f deploy/kubernetes-<version>/hostpath/
```
This removes the StatefulSets and Services for the attacher, provisioner, snapshotter, and plugin.
Remove RBAC rules by deleting the service accounts, cluster roles, and bindings created during deployment for csi-provisioner, csi-attacher, and csi-snapshotter.

Escalation

If you have followed all steps in this runbook and the issue is not resolved, escalate through the following channels:

Kubernetes CSI community (primary channel):

Slack: http://slack.k8s.io/ — search for the #csi or #sig-storage channels
Mailing List: https://groups.google.com/forum/#!forum/kubernetes-dev

When escalating, collect and provide the following information:

Kubernetes version: kubectl version
Full pod listing: kubectl get pods -o wide
Logs from all four driver pods (attacher, provisioner, snapshotter, plugin — all containers)
Output of kubectl describe pvc, kubectl describe volumeattachment, and kubectl describe pods/my-csi-app
The exact error message or unexpected behavior you observed
The deployment script and Kubernetes version directory used (e.g., deploy/kubernetes-1.13/)
Any customizations made to the default manifests or example files