NFS-Client Provisioner stucked at ContainerCreating

Question

I followed this(https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-volumes-example-nfs-persistent-volume.html) link and installed an NFS server in my cluster with blow yaml file:

kind: Service
apiVersion: v1
metadata:
  name: nfs-service
spec:
  selector:
    role: nfs
  ports:
    # Open the ports required by the NFS server
    # Port 2049 for TCP
    - name: tcp-2049
      port: 2049
      protocol: TCP

    # Port 111 for UDP
    - name: udp-111
      port: 111
      protocol: UDP

---
kind: Pod
apiVersion: v1
metadata:
  name: nfs-server-pod
  labels:
    role: nfs
spec:
  containers:
    - name: nfs-server-container
      image: cpuguy83/nfs-server
      securityContext:
        privileged: true
      args:
        # Pass the paths to share to the Docker image
        - /exports

I installed the server via below commnad:

kubectl create namespace nfs
kubectl apply -f nfs-server.yaml -n nfs

The server installed smoothly and is up and running.

I installed nfs-common on all nodes then I installed the nfs-client provisioner helm chart via below command:

helm install stable/nfs-client-provisioner --set nfs.server=nfs-service --set nfs.path=/exported/path --name=nfs-client --namespace=nfs

But the nfs-client stuck at ContainerCreating and this is the output of describe:

Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nfs-client-root:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    nfs-service
    Path:      /exported/path
    ReadOnly:  false
  nfs-client-nfs-client-provisioner-token-k9n8n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nfs-client-nfs-client-provisioner-token-k9n8n
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age   From                      Message
  ----     ------       ----  ----                      -------
  Warning  FailedMount  59m   kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-rdc5765db89a64abf87610ba4d853f48d.scope
mount.nfs: Connection timed out
  Warning  FailedMount  57m  kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r0b472b9b183240bd8a4a3020bc13a129.scope
mount.nfs: Connection timed out
  Warning  FailedMount  55m  kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r8f149b9cfb4d414cb15834537f4fd88a.scope
mount.nfs: Connection timed out
  Warning  FailedMount  52m  kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r3f2e2388086f42f987f0e1475ef95385.scope
mount.nfs: Connection timed out
  Warning  FailedMount  50m  kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r8db7f40d8a7948598deae1fb07162372.scope
mount.nfs: Connection timed out
  Warning  FailedMount  48m  kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-rff3d9f71b0e44148ba70c3f9f706d8e4.scope
mount.nfs: Connection timed out
  Warning  FailedMount  46m  kubelet, serflex-argus-1  MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r05af2a5659ad47bea7a0e7010d280444.scope
mount.nfs: Connection timed out
  Warning  FailedMount  20m (x5 over 56m)     kubelet, serflex-argus-1  Unable to attach or mount volumes: unmounted volumes=[nfs-client-root], unattached volumes=[nfs-client-nfs-client-provisioner-token-k9n8n nfs-client-root]: timed out waiting for the condition
  Warning  FailedMount  9m7s (x15 over 59m)   kubelet, serflex-argus-1  Unable to attach or mount volumes: unmounted volumes=[nfs-client-root], unattached volumes=[nfs-client-root nfs-client-nfs-client-provisioner-token-k9n8n]: timed out waiting for the condition
  Warning  FailedMount  3m20s (x15 over 43m)  kubelet, serflex-argus-1  (combined from similar events): MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs nfs-service:/exported/path /var/lib/kubelet/pods/811e6fd7-0e48-40a3-ad4d-2f677d49178c/volumes/kubernetes.io~nfs/nfs-client-root
Output: Running scope as unit: run-r27616bcceccc4ff194ac009997604d31.scope
mount.nfs: Connection timed out

I have to mention that I am running k8s v1.16.

How I can solve this problem?

Update

I don't know what was the problem but when I removed the nfs-server from k8s and used a server located on a bare-metal machine the client started to work properly.

mario · Answer 1 · 2019-10-18T12:58:37.840

1

Make sure you have installed nfs-utils (rpm-base distros) or nfs-common (deb-based distros) packages on all Kubernetes nodes.

edited Oct 18 '19 at 12:58

answered Oct 18 '19 at 12:20

mario

525
3
8

As I mentioned I already installed `nfs-common`. `nfs-utils` is for fedora and centos so I don't need it since my nodes are using Ubuntu. – AVarf Oct 18 '19 at 12:39
Yes, you're right. I've just checked it as well and I see that `nfs-common` for Ubuntu already contains `nfs-utils`. Could you make sure that there are no firewall rules blocking traffic on 111 and 2049 ( both TCP and UDP ) ? – mario Oct 18 '19 at 12:50
I checked 3 out of 5 nodes and there is no rule blocking those ports. – AVarf Oct 18 '19 at 12:58

score 1 · Answer 2 · answered Oct 20 '19 at 13:44

If you use NFS v3, then you need yet another port and daemon to run - mountd. Usually. mountd picks a random port number. Thus you have to pin it in /etc/sysconfig/nfs

MOUNTD_PORT=892

And, as @mario have mentioned, make sure that all ports are TCP.

For client's that use NFSv4.x, probably you need to specify it on the command line:

mount -vers=4.1 ....

or

mountOptions:
 - tcp
 - nfsvers=4.1

info deployment.yaml

score 0 · Answer 3 · answered Apr 03 '22 at 11:06

0

Not sure whether this issue resolved. But when i got this issue, executing below resolves the error

systemctl enable nfs-server rpcbind

answered Apr 03 '22 at 11:06

Antony Rajesh

1

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/517084) – Davidw Apr 06 '22 at 03:30

NFS-Client Provisioner stucked at ContainerCreating

Update

3 Answers3