So I have a 4 node (VMs) Kubernetes cluster spun up with Kubespray. I have a Ceph cluster set up from Proxmox, and a pool is available to k8s. I can make deployments using Ceph just fine.
But when I want to resize the container, I run into a long list of issues.
When I tried to resize, instead of using the kube-system/rbd-provisioner pod, it tries to use kube-system/kube-controller-manager to resize it. This failed initially because the container doesn't have the rbd command installed. I fixed this by switching the image in /etc/kubernetes/manifests/kube-controller-manager.yaml from gcr.io/google-containers/kube-controller-manager:v1.15.1 to image: gcr.io/google_containers/hyperkube:v1.14.6, as this image includes the rbd binary.
Now it can attempt to resize volumes. But it still fails when it tries. The error it shows is:
'VolumeResizeFailed' error expanding volume "default/test-pvc" of plugin "kubernetes.io/rbd": rbd info failed, error: exit status 110
The StorageClass is as follows:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: kubedata-rbd
provisioner: ceph.com/rbd
allowVolumeExpansion: true
parameters:
monitors: 10.10.10.2:6789
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: kube-system
pool: Kubedata
userId: kube
userSecretName: ceph-secret-kube
userSecretNamespace: kube-system
imageFormat: "2"
imageFeatures: layering
How can I be able to resize PVCs created by Ceph? I've even tried manually resizing them on a ceph node then editing the claim size, but that doesn't update to reflect the new size.
