2

Every time I install Kubernetes 1.15.3 it works great, until I reboot the VM and then it no longer listens on port 6443. Swap is off. /etc/tab only contains the root volume on my Ubuntu 18.04 server VM. Note, using flannel

cat /proc/swaps
Filename                                Type            Size    Used    Priority

kubectl only returns refused

kubectl get nodes
The connection to the server <myip>:6443 was refused - did you specify the right host or port?

sudo netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      
770/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
1694/sshd
tcp        0      0 127.0.0.1:36957         0.0.0.0:*               LISTEN      
1269/containerd
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      
2720/kubelet
tcp        0      0 127.0.0.1:33385         0.0.0.0:*               LISTEN      
2720/kubelet
tcp6       0      0 :::22                   :::*                    LISTEN      
1694/sshd
tcp6       0      0 :::10250                :::*                    LISTEN      
2720/kubelet
udp        0      0 127.0.0.53:53           0.0.0.0:*                           
770/systemd-resolve

systemctl status kubelet

Oct 21 12:47:23 kubernetes1 kubelet[2720]: E1021 12:47:23.811807    2720 
reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list 
*v1beta1.RuntimeClass: Get https://<MYIP>:6443/apis/nodOct 21 
12:47:23 kubernetes1 kubelet[2720]: E1021 12:47:23.824760    2720 
kubelet.go:2248] node "kubernetes1" not found

ps -ef | grep kube

root      2720     1  0 12:32 ?        00:00:07 /usr/bin/kubelet -- 
bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf -- 
kubeconfig=/etc/kubernetes/kubelet.conf -- 
config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network- 
plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv- 
conf=/run/systemd/resolve/resolv.conf

Using flannel

sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Any ideas? The "interweb" only tells me to check swap, but I've checked it literally 6 different ways and all indications is that it's off.

My the admin conf file copied to .kube shows the right server and right port. The API-server is just not up. Thoughts anyone?

journalctl -xeu kubelet

Note, kubernetes1 is the name of my master server.

Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.521057    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.610016    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://<myip>:6443/api/v1/podsOct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.621170    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.721293    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.810053    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://<myip>:6443/api/v1/servicesOct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.823444    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:28:59 kubernetes1 kubelet[2720]: E1022 02:28:59.923615    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.010365    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://<myip>:6443/api/v1/nodes?fieldOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.023809    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.123954    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.210384    2720 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://<myip>:6443/apis/nodOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.225717    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.325890    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.410429    2720 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://<myip>:6443/apis/storagOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.425998    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.526126    2720 kubelet.go:2248] node "kubernetes1" not found
Oct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.610341    2720 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://<myip>:6443/api/v1/podsOct 22 02:29:00 kubernetes1 kubelet[2720]: E1022 02:29:00.626218    2720 kubelet.go:2248] node "kubernetes1" not found
Dave Brunkow
  • 222
  • 5
  • 11
  • Have you tried to restart `docker` and `kubelet`? `service docker restart` `systemctl restart kubelet` – PjoterS Oct 21 '19 at 14:20
  • @PjoterS yes, no change. – Dave Brunkow Oct 21 '19 at 15:46
  • Nothing in the Kubernetes 1.15.4 change logs that looks like the issue to me; https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md#changelog-since-v1153 – Dave Brunkow Oct 21 '19 at 17:51
  • show ```journalctl -xeu kubelet``` result – sanjayparmar Oct 21 '19 at 18:43
  • Journalctl added to end of issue above. – Dave Brunkow Oct 22 '19 at 02:34
  • Are you using Kubeadm or Cloud? Did you try with the latest Kubernetes version 1.16? Still get the same error? – PjoterS Oct 22 '19 at 10:54
  • Not an option to go to 1.16. May try 1.15.5. If 1.15.3 has such a bug, I can’t find any doc on it. – Dave Brunkow Oct 22 '19 at 11:41
  • Unfortunately I was not able to reproduce your case. Could you provide whole installation procedure? Version of all packages (Docker, Kubelet, Kubctl, Kubeadm, CRI) would be also much appreciate. Did you deploy/execute/change config before turning off the VM? – PjoterS Oct 22 '19 at 14:24
  • Vm running on local esxi server... kubeadm. – Dave Brunkow Oct 22 '19 at 15:03
  • I scripted the setup. I’ll post it this evening. – Dave Brunkow Oct 22 '19 at 15:05
  • Looking at the docker install. I'm thinking there's an issue there. Docker run ubuntu returns docker: Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:55820->[::1]:53: read: connection refused. S – Dave Brunkow Oct 25 '19 at 03:08
  • It looks like your ubuntu cannot reach Docker repository. Could you try below things and let me know if any of them helped: 1. SSH to the master and `sudo vi /etc/resolv.conf`, add nameserver 8.8.8.8 (not sure if still valid but `resolv.conf` recognized only first 3 nameserver records inside). Eventually restart docker and kubelet. 2. Reinstall Docker 3. If you are using any proxy you could check this: https://blog.codeship.com/using-docker-behind-a-proxy/ Please let me know if any of the followings steps helped or still have issue. – PjoterS Oct 25 '19 at 10:34

3 Answers3

1

After re-installing the OS and Kubernetes 1.15.5 the issue is with DNS. Docker images would not pull because the docker server could not resolve after a restart.

I previously tried to modify the resolv.conf directly as mentioned above, but Ubuntu wants to manage resolv.conf itself with netplan. Making manual changes and rebooting looks like it results in those changes being ignored.

If I modify the netplan yaml, and run netplan apply, it puts 127.0.0.53 in the resolv.conf and runs a local dns proxy, which then forwards properly to the bind9 dns server I have configured. I expect to see my dns servers in the resolv.conf and that's what caused the problem.

Do not modify the resolv.conf directly on Ubuntu. Use netplan.

Dave Brunkow
  • 222
  • 5
  • 11
0

Due to lack of OP's confirmation I am posting it as Community Wiki.

Error like:

Get registry-1.docker.io/v2: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:55820->[::1]:53: read: connection refused.

Means that VM cannot reach Docker repository. I found a few fimilar issues like this.

I will post a few actions which may help to resolve this issue:

1. Restart docker and kubelet using:

$ service docker restart
$ systemctl restart kubelet

2. Add nameserver manually

- SSH to the master
- sudo vi /etc/resolv.conf
- add "nameserver 8.8.8.8" and save this change.
- Eventually restart docker and kubelet

3. Reinstall docker or install newest version

4. Using docker behind proxy

If you are using any proxy, check this tutorial.

EDIT:

Based on the information from OP it seems that issue was caused by Netplan which is utility for configuring networking on a linux system. It enforced other network configuration due to Docker couldn't pull the images.

PjoterS
  • 615
  • 3
  • 11
-2

You just need to kill kubelet service and restart again. pods and container will be running as well as before reboot.

pkill kubelet

and

systemctl restart kubelet

good luck

mehrdad rafiee
  • 1