1

I'm following guide from Linux Foundation "Kubernetes Administrator" course and stuck on deploying simple app. I think trouble is even earlier than with app deployment.

I've created master and worker, seems that they are ok:

$ kubectl get nodes
NAME                       STATUS   ROLES    AGE   VERSION
ubuntu-training-server-1   Ready    master   63m   v1.19.1
ubuntu-training-server-2   Ready    <none>   57m   v1.19.1

But here is something wrong:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                               READY   STATUS              RESTARTS   AGE
default       nginx-6799fc88d8-556z4                             0/1     ContainerCreating   0          50m
kube-system   calico-kube-controllers-69496d8b75-thcl8           1/1     Running             1          63m
kube-system   calico-node-gl885                                  0/1     CrashLoopBackOff    20         58m
kube-system   calico-node-jvc59                                  1/1     Running             1          63m
kube-system   coredns-f9fd979d6-hjfst                            1/1     Running             1          64m
kube-system   coredns-f9fd979d6-kvx42                            1/1     Running             1          64m
kube-system   etcd-ubuntu-training-server-1                      1/1     Running             1          64m
kube-system   kube-apiserver-ubuntu-training-server-1            1/1     Running             1          64m
kube-system   kube-controller-manager-ubuntu-training-server-1   1/1     Running             1          64m
kube-system   kube-proxy-9899t                                   1/1     Running             1          58m
kube-system   kube-proxy-z6b22                                   1/1     Running             1          64m
kube-system   kube-scheduler-ubuntu-training-server-1            1/1     Running             1          64m

I mean not all are ready.

If I try to get details about trouble node, I see:

$ kubectl logs -n kube-system calico-node-gl885
Error from server (NotFound): the server could not find the requested resource ( pods/log calico-node-gl885)

And when I try to deploy nginx I get:

$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

and

$ kubectl get deployments
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/1     1            0           52m

Here are troubles too:

$ kubectl get events
...
92s         Warning   FailedCreatePodSandBox    pod/nginx-6799fc88d8-556z4      Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4c451bc2c92f555c930f84e4e8b7082a03dd2824cf50948d348893ebea488d93" network for pod "nginx-6799fc88d8-556z4": networkPlugin cni failed to set up pod "nginx-6799fc88d8-556z4_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
...

I see no /var/lib/calico/nodename on worker node, only on master and in guide there was speaking only about kubectl apply -f calico.yaml on master.

Could anybody help me get rid of calico errors? Tried to search, have seen similar cases but looks like that they are about something different.

UPDATE

I've found possible networking conflict (Calico config contained 192.168.0.0/16 and my VirtualBox adapter was 192.168.56.0/24) so I reset cluster, changed Calico config and networking/podSubnet in kubeadm-config.yaml to 192.168.0.0/24 and init cluster again.

New status is following.

Seem OK:

$ kubectl get nodes
NAME                       STATUS   ROLES    AGE   VERSION
ubuntu-training-server-1   Ready    master   39m   v1.19.1
ubuntu-training-server-2   Ready    <none>   38m   v1.19.1

Seem OK too:

$ kubectl get events
LAST SEEN   TYPE     REASON                    OBJECT                          MESSAGE
36m         Normal   Starting                  node/ubuntu-training-server-1   Starting kubelet.
36m         Normal   NodeHasSufficientMemory   node/ubuntu-training-server-1   Node ubuntu-training-server-1 status is now: NodeHasSufficientMemory
36m         Normal   NodeHasNoDiskPressure     node/ubuntu-training-server-1   Node ubuntu-training-server-1 status is now: NodeHasNoDiskPressure
36m         Normal   NodeHasSufficientPID      node/ubuntu-training-server-1   Node ubuntu-training-server-1 status is now: NodeHasSufficientPID
36m         Normal   NodeAllocatableEnforced   node/ubuntu-training-server-1   Updated Node Allocatable limit across pods
36m         Normal   NodeReady                 node/ubuntu-training-server-1   Node ubuntu-training-server-1 status is now: NodeReady
35m         Normal   RegisteredNode            node/ubuntu-training-server-1   Node ubuntu-training-server-1 event: Registered Node ubuntu-training-server-1 in Controller
35m         Normal   Starting                  node/ubuntu-training-server-1   Starting kube-proxy.
35m         Normal   Starting                  node/ubuntu-training-server-2   Starting kubelet.
35m         Normal   NodeHasSufficientMemory   node/ubuntu-training-server-2   Node ubuntu-training-server-2 status is now: NodeHasSufficientMemory
35m         Normal   NodeHasNoDiskPressure     node/ubuntu-training-server-2   Node ubuntu-training-server-2 status is now: NodeHasNoDiskPressure
35m         Normal   NodeHasSufficientPID      node/ubuntu-training-server-2   Node ubuntu-training-server-2 status is now: NodeHasSufficientPID
35m         Normal   NodeAllocatableEnforced   node/ubuntu-training-server-2   Updated Node Allocatable limit across pods
22s         Normal   CIDRNotAvailable          node/ubuntu-training-server-2   Node ubuntu-training-server-2 status is now: CIDRNotAvailable
35m         Normal   Starting                  node/ubuntu-training-server-2   Starting kube-proxy.
35m         Normal   RegisteredNode            node/ubuntu-training-server-2   Node ubuntu-training-server-2 event: Registered Node ubuntu-training-server-2 in Controller
35m         Normal   NodeReady                 node/ubuntu-training-server-2   Node ubuntu-training-server-2 status is now: NodeReady

And here is new trouble, calico-kube-controllers-69496d8b75-gdbd7 is starting for more than half of an hour:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                               READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-69496d8b75-gdbd7           0/1     ContainerCreating   0          37m
kube-system   calico-node-8xjsm                                  0/1     CrashLoopBackOff    13         37m
kube-system   calico-node-zktsh                                  1/1     Running             0          37m
kube-system   coredns-f9fd979d6-7bkwn                            1/1     Running             0          39m
kube-system   coredns-f9fd979d6-rsws5                            1/1     Running             0          39m
kube-system   etcd-ubuntu-training-server-1                      1/1     Running             0          39m
kube-system   kube-apiserver-ubuntu-training-server-1            1/1     Running             0          39m
kube-system   kube-controller-manager-ubuntu-training-server-1   1/1     Running             0          39m
kube-system   kube-proxy-2tvjp                                   1/1     Running             0          39m
kube-system   kube-proxy-jkzbz                                   1/1     Running             0          39m
kube-system   kube-scheduler-ubuntu-training-server-1            1/1     Running             0          39m

UPDATE 2

Details about my setup.

$ cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.19.1
controlPlaneEndpoint: "k8smaster:6443"
networking:
  podSubnet: 192.168.0.0/24

Cluster was initialized with:

kubeadm init --config=kubeadm-config.yaml --upload-cert
Dmitriy Vinokurov
  • 153
  • 1
  • 10
  • How did you install calico, is this happening with kubeadm? What pod network CIDR have you specified? What tutorial have you used ? – Malgorzata Apr 02 '21 at 07:43
  • I've downloaded https://docs.projectcalico.org/manifests/calico.yaml, uncommented `CALICO_IPV4POOL_CIDR` there, it's value is "192.168.0.0/16" and `podSubnet` networking option in `kubeadm-config.yaml` I've set to be same. I've applied Calico using `kubectl apply -f calico.yaml` after `kubeadm init` on master host. I've followed tutorial from Linux Foundation Kubernetes course, it is not public. – Dmitriy Vinokurov Apr 02 '21 at 13:11
  • Found possible networking confict, fixed it and recreated cluster, see UPDATE. Now `kubectl get events` seem OK, but `calico-kube-controllers` is starting too long. – Dmitriy Vinokurov Apr 02 '21 at 15:54
  • 1
    @Malgorzata. Found solution, k8s was using wrong IP of VM. Added answer with few details. Thank you for trying to help me. – Dmitriy Vinokurov Apr 02 '21 at 16:35

2 Answers2

1

Got it. In my setup I've got two VirtualBox VMs both with two network interfaces - one to connect with outer world (10.0.2.15) and one to talk to each other (192.168.56.104, 192.168.56.105). In kubeadm init log I've found that it was using first one, so I've explicitly told kubeadm to use internal IP. Here is command with which I've succeeded to create cluster and deploy simple app into it

kubeadm init --apiserver-advertise-address=192.168.56.104 --apiserver-cert-extra-sans=192.168.56.104 --node-name k8smaster --pod-network-cidr=192.168.0.0/24 --kubernetes-version=1.19.1

One sad thing - unfortunately I could not find how to add to config options which I've used in command line.

Dmitriy Vinokurov
  • 153
  • 1
  • 10
  • Adding --pod-network-cidr in config https://github.com/kubernetes/kubeadm/issues/1899 – gout Jun 16 '21 at 06:29
1

Thank you so much Dmitriy. This Definitely saves my day. I'm also working on setting up the cluster using virtualbox for the LFD course and encountered the same error. The calico pod is not running on the worker node and the error message is the same.

I would highly recommend to use a different network than 192.168 as the ip address of the VMs on virtualbox. It's pretty simeple. All you need to do is to create a new host network in virtualbox and add that network as a network adapter to your VMs. I created a 172.16.16.0/24 network for both of my VMs and therefore I don't need to update any config. All I need to do was to update the kubeadm init cmd and put the corrent ip address there.

Bruce.Fu
  • 11
  • 1