9

After updating Arch Linux (sudo pacman -Syu) and restarting the system, Minikube fails to start because of kube-proxy. The logs show it was trying to modify /proc/sys/net/netfilter/nf_conntrack_max but it ran into a permission denied issue.

I tried to run sudo chmod 777 /proc/sys/net/netfilter/nf_conntrack_max but the system didn't allow me. I also removed ~/.minikube and started again, but it still refused to work. Same issue when starting kind. Can anyone tell me how to fix this?

$ kubectl get pods -n kube-system

NAME                               READY   STATUS    RESTARTS   AGE
coredns-74ff55c5b-2hkpd             0/1     Running   0          22s
etcd-minikube                      0/1     Running   0          30s
kube-apiserver-minikube            1/1     Running   0          30s
kube-controller-manager-minikube   0/1     Running   0          30s
kube-proxy-qhhx9                   0/1     Error     2          22s
kube-scheduler-minikube            0/1     Running   0          30s
storage-provisioner                1/1     Running   0          35s


kubectl logs kube-proxy-qhhx9 -n kube-system

I0511 04:47:08.189373       1 node.go:172] Successfully retrieved node IP: 192.168.49.2
I0511 04:47:08.189422       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.49.2), assume IPv4 operation
W0511 04:47:08.207109       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0511 04:47:08.207314       1 server_others.go:185] Using iptables Proxier.
I0511 04:47:08.208192       1 server.go:650] Version: v1.20.2
I0511 04:47:08.209006       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
F0511 04:47:08.209060       1 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

Kernel version

$ uname -r
5.12.2-arch1-1

$ iptables --version
iptables v1.8.7 (legacy)
RedGiant
  • 211
  • 3
  • 14
  • Hello @RedGiant. How exactly did you try to start the Minikube again? Could you please provide the steps you followed? – Wytrzymały Wiktor May 11 '21 at 13:56
  • I just use `minikube start`. It happens every time I reboot my desktop. When I posted this thread a few days ago the problem was gone an hour without doing anything on my side. Today I rebooted the desktop and it happened again. – RedGiant May 16 '21 at 06:35
  • According to the issue https://github.com/kubernetes/minikube/issues/11418 this will be fixed with minikube [v1.21.0](https://github.com/kubernetes/minikube/milestone/40) (due date is June 4) – aemaem Jun 04 '21 at 05:58
  • I can confirm upgrading to v1.21.0 fixes the issue. – xuhdev Jun 21 '21 at 08:48

4 Answers4

7

change maxPerCore to 0 in configMap of kube-proxy to leave the limit as-is and ignore conntrack-min

Mehdi Shakeri
  • 171
  • 1
4

Getting the same issue although using kind. FWIW it helps downgrading the kernel to 5.11.

Kim
  • 41
  • 1
4

There's a known bug in kind (see issue #2240), caused by a change in version 5.12.2 of the kernel to make nf_conntrack_max read-only in non-init network namespaces (more info in the relevant commit)

A workaround was applied in kind (PR), so as other answers have mentioned, upgrading kind should fix it.

Joe Malt
  • 141
  • 2
3

Upgrading to kind v0.11.1 solves this issue.

$ kind version
kind v0.11.1 go1.16.4 linux/amd64
$ uname -r
5.4.0-80-generic
Fabrice Jammes
  • 131
  • 3