Kubernetes: pod can't mount a PVC even though the latter seems to be properly bound to a PV

Question

​Hello.

I'm experiencing a problem while trying to deploy a pod that has been configured to use a Persistent Volume. My installation is hosted on Hetzner Cloud and Kubernetes has been configured according to the instructions made available by the provider. The basic configuration seems ok since I can deploy e.g. Portainer from one of the provided manifests: it correctly mounts the PVC and runs. Since the default configuration doesn't create a persistent storage, every time I undeploy and redeploy Portainer all data are lost.

So I proceeded with the creation of a PersistentVolume, but the final step doesn't work. I can't tell whether the problem is a generic Kubernetes one or it's with my integration with Hetzner - but I think it makes sense to check first the first hypothesis. So I'm asking here before going to the Hetzner forum.

Steps:

1. I created a Volume using the Hetzner Cloud user interface.

2. Then I deployed a PV:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: portainer
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-portainer-test
      namespace: portainer
      labels:
        env: test
    spec:
      storageClassName: portainer-test
      accessModes:
        - ReadWriteOnce
      capacity:
        storage: 10Gi
      persistentVolumeReclaimPolicy: Retain
      volumeMode: Filesystem
      csi:
        driver: csi.hetzner.cloud
        fsType: ext4
        volumeHandle: "pv_id"

pv_id is the volume id that I can see with the Hetzner user interface.

I can see the PV with kubelet get pv -A:

NAMESPACE   NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS     REASON   AGE
            persistentvolume/pv-portainer-test   10Gi       RWO            Retain           Available           portainer-test            71s

Note that the namespace has not been defined... but I suppose this is not a serious issue. Maybe I'm wrong.

3. I deployed Portainer from here and the following patch:​

    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: portainer
      namespace: portainer
    spec:
      storageClassName: portainer-test
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: "10Gi"

Now with kubectl get pvc,pv -A I see that the PVC has been bound to the PV:

​NAMESPACE   NAME                              STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS     AGE
portainer   persistentvolumeclaim/portainer   Bound    pv-portainer-test   10Gi       RWO            portainer-test   16m

NAMESPACE   NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS     REASON   AGE
            persistentvolume/pv-portainer-test   10Gi       RWO            Retain           Bound    portainer/portainer   portainer-test            17m

4. But the Portainer pod stays in the ContainerCreating status forever.​ With kubectl describe I see the reason:

---

Warning  FailedAttachVolume  2s (x7 over 38s)  attachdetach-controller  AttachVolume.Attach failed for volume "pv-portainer-test" : rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached

---

So it seems it is complaining because the PV is already attached... but indeed it should have been attached precisely to that pod.

Answer 1

Usually you don't have to create PV first, when you create PVC, PV is created automatically. However, let's keep as it is and there is doubt about your storage class portainer-test, how have you created it?

If you start from the beginning, I suppose you've created the secret for your API token to Hetzner and enabled CSI drivers, right? Then requesting PVC and deploying Pod shouldn't be a problem, unless the driver type is supported by Kubernetes. I don't know what type you're having, but please note that Kubernetes supports for csi.hetzner.cloud only v0.3, v1.0 versions.

Let's assume that you did everything correct before and the Kubernetes version (1.13 or newer) and driver versions are good.So then, here is the guide on how to manually create a PersistentVolume object to represent the existing volume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-portainer-tes
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: csi.hetzner.cloud
    volumeHandle: portainer
    readOnly: false
    fsType: ext4
    volumeAttributes:
      foo: bar
    controllerPublishSecretRef:
      name: mysecret1
      namespace: mynamespace
    nodeStageSecretRef:
      name: mysecret2
      namespace: mynamespace
    nodePublishSecretRef
      name: mysecret3
      namespace: mynamespace

Then attach and mount the volume by deploying a Pod. When the pod referencing a CSI volume is scheduled, Kubernetes will trigger the appropriate operations against the external CSI plugin (ControllerPublishVolume, NodeStageVolume, NodePublishVolume, etc.) to ensure the specified volume is attached, mounted, and ready to use by the containers in the pod.

And note: there is a limit for csi drivers up to 16, please check if your limit is not almost 16

Answer 2

The problem was not related to k8s, but specific to Hetzner. In the project configuration I had left the volume attached to a fixed server, which obviously made it unavailable for attaching through the CSI driver. A user at the Hetzner Cloud forum pointed me to that.

I'd like to thank Bazhikov anyway as he gave me some hints that were useful in fixing other problems.