0

I've created a virtual kubernetes cluster using vagrant boxes. All my boxes have 2 network interfaces

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:1d:76:b3:3c:fe brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global enp0s3
   valid_lft forever preferred_lft forever
inet6 fe80::1d:76ff:feb3:3cfe/64 scope link 
   valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:35:39:9f brd ff:ff:ff:ff:ff:ff
inet 192.168.70.11/24 brd 192.168.70.255 scope global enp0s8
   valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe35:399f/64 scope link 
   valid_lft forever preferred_lft forever

enp0s3 is created by default and has the same ip address on all the boxes. Apparently this is a virtual box thing https://github.com/hashicorp/vagrant/issues/6456 . I added the enp0s8 interface as a private network to allow my boxes to talk to each other, this one has a different IP on each box so I want to use it for kubernetes.

When I set up my cluster I used this command which allowed me to create the cluster on the first node and join the second node

$ kubeadm init --apiserver-advertise-address=192.168.70.11 --pod-network-cidr=10.244.0.0/16

The problem is all the kube-system pods are still using the enp0s3 interface

$ kubectl -n kube-system get pods -owide
NAME                             READY     STATUS    RESTARTS   AGE       IP           NODE
coredns-78fcdf6894-8tpnc         1/1       Running   0          11m       10.244.0.2   node-1
coredns-78fcdf6894-tbqxk         1/1       Running   0          11m       10.244.0.3   node-1
etcd-node-1                      1/1       Running   0          10m       10.0.2.15    node-1
kube-apiserver-node-1            1/1       Running   0          10m       10.0.2.15    node-1
kube-controller-manager-node-1   1/1       Running   0          10m       10.0.2.15    node-1
kube-flannel-ds-5wm74            1/1       Running   0          11m       10.0.2.15    node-2
kube-flannel-ds-wx77l            1/1       Running   0          11m       10.0.2.15    node-1
kube-proxy-gmst8                 1/1       Running   0          11m       10.0.2.15    node-2
kube-proxy-sbqqs                 1/1       Running   0          11m       10.0.2.15    node-1
kube-scheduler-node-1            1/1       Running   0          10m       10.0.2.15    node-1

I'd like to force kubernetes to use enp0s8 exclusively. Is there anyway to do that without removing the the enp0s3 interface?

This is what the nodes look like

$ kubectl get nodes -owide
NAME      STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1    Ready     master    4m        v1.11.2   <none>        Ubuntu 16.04.5 LTS   4.4.0-134-generic   docker://17.3.2
node-2    Ready     <none>    3m        v1.11.2   <none>        Ubuntu 16.04.5 LTS   4.4.0-134-generic   docker://17.3.2

And this is the version info

$ sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:08:19Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
rvabdn
  • 235
  • 2
  • 11

1 Answers1

0

There is an issue on GitHub related to your problem: kubelet reports wrong IP address #44702

At the end of the discussion, yujuhong explained why this happens:

kubelet uses the IP address reported by the cloud provider if it exists, or the first non-loopback ipv4 address (code here) if there is no cloud provider. In addition, it could be overwritten by kubelet flags.

I've updated the links in the quote to v1.11. Here what is mentioned in the code comments for v1.11:

    // 1) Use nodeIP if set
    // 2) If the user has specified an IP to HostnameOverride, use it
    // 3) Lookup the IP from node name by DNS and use the first valid IPv4 address.
    //    If the node does not have a valid IPv4 address, use the first valid IPv6 address.
    // 4) Try to get the IP from the network interface used as default gateway

Options of kubelet mentioned in code comments are copied from kubelet documentation:

  1. --node-ip string - IP address of the node. If set, kubelet will use this IP address for the node
  2. --hostname-override string - If non-empty, will use this string as identification instead of the actual hostname.

If we look at the code, we will see that it expects IPv4 or IPv6 address here:

if addr := net.ParseIP(kl.hostname); addr != nil

Last two options are not very convenient and reliable to use in your case, so I skipped them.

The first option was also suggested by dcbw in the comment and had positive feedback:

If you're not using a cloud provider, try modifying the kubelet command-line options to pass "--node-ip=" or setting the NodeIP configuration in the kubelet config yaml file.

Anyway, it is up to you to choose what suits you best.

VAS
  • 370
  • 1
  • 9