Kubernetes CSI on OpenStack adds bad nodeid annotations to nodes

Question

I'm trying to configure the Cinder CSI plugin (https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-cinder-csi-plugin.md) on my Kubernetes cluster (installed on OpenStack instances).

Everything works fine (the PV, PVC and actual Cinder volume are created just fine) but when the csi-attacher tries to attach my Cinder volume to my node, it fails with:

Error processing "csi-0c1aee9c7e99bc90a586541a4bb7974e38978dc3a2c441a1ad1413d7435d886c": failed to attach: rpc error: code = Unknown desc = failed to attach 445d9931-9e32-48b5-b765-209a4526691f volume to i-0031336c compute: Resource not found

Note that the volume ID is right though:

> cinder list | grep 445d9931-9e32-48b5-b765-209a4526691f
| 445d9931-9e32-48b5-b765-209a4526691f | available | pvc-788eb00a-99bc-11e9-b7dc-fa163eb3415c | 1    | classic     | false    |                                      |

My node is not i-0031336c but an actual UUID:

> nova list | grep worker03-dev
| 9d7cd2bb-a365-4276-86ed-f4884132c361 | worker03-dev     | ACTIVE | -          | Running     | Ext-Net=<redacted>    |

I don't know why, but something (I'm not yet quite expert on the CSI side) is adding an annotation with the wrong ID to my nodes:

> kubectl get node worker03-dev.node.consul -o yaml | grep nodeid
    csi.volume.kubernetes.io/nodeid: '{"cinder.csi.openstack.org":"i-0031336c"}'

When querying the OpenStack API from the host, it returns the right UUID too:

root@worker03-dev ~ # curl -s 169.254.169.254/openstack/latest/meta_data.json | jq '.uuid'
"9d7cd2bb-a365-4276-86ed-f4884132c361"

Do I have something completely wrong?

I run these CSI containers:

image: quay.io/k8scsi/csi-attacher:v1.1.1
image: quay.io/k8scsi/csi-provisioner:v1.0.1
image: quay.io/k8scsi/csi-snapshotter:v1.0.1
image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
image: docker.io/k8scloudprovider/cinder-csi-plugin:latest

(I started with v1.0.1 for all k8scsi containers but tried to upgrade some of them without success)

Answer 1

The issue was not with CSI but with my cloud-init configuration which was fetching EC2 metadata instead of OpenStack metadata (my /etc/cloud/cloud.cfg.d/90_dpkg.cfg was referring to Ec2 instead of OpenStack) therefore didn't retrieve the good instance ID:

root@worker03-dev ~ # cat /var/lib/cloud/data/instance-id 
i-0031336c

After fixing the issue, cloud-init was retrieving the right instance ID:

root@worker03-dev ~ # cat /var/lib/cloud/data/instance-id 
9d7cd2bb-a365-4276-86ed-f4884132c361

Now the right NodeID is attached to my nodes.

I don't know why my cloud provider (OVH) returns wrong information for the EC2 metadata URL though:

root@worker03-dev ~ # curl http://169.254.169.254/latest/meta-data/instance-id
i-0031336c