0

I'm currently experimenting with kubernetes and docker on my laptop, and I've run into an issue where I'll try start hyperkube on my computer, and it'll take forever to start (or it'll start, but I'll be unable to start containers within it), and my /var/log/syslog becomes saturated with log lines like:

Sep  8 16:51:36 <hostname> kernel: [ 1848.979278] unregister_netdevice: waiting for vethd8a9cb4 to become free. Usage count = 1

I know veth interfaces are used by docker (which is used by kubernetes) to let containers talk to each other, and my understanding is that I ought to be able to find out what's using this interface with ip -d link show, however, that doesn't show this particular interface:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 
2: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
    link/ether 44:1c:a8:e3:d2:c7 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode none 
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:d4:6b:8c:e3 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q addrgenmode eui64 
5: veth9948f07@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 72:33:f7:5d:19:8a brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
7: veth5e9330f@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e6:ac:6a:2e:4b:b3 brd ff:ff:ff:ff:ff:ff link-netnsid 1 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
9: veth758786d@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether fe:d4:1d:c5:9b:39 brd ff:ff:ff:ff:ff:ff link-netnsid 2 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
11: veth7624bf1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 52:5f:75:fc:c5:8c brd ff:ff:ff:ff:ff:ff link-netnsid 3 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
13: veth7297ec8@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 06:b3:19:c3:29:01 brd ff:ff:ff:ff:ff:ff link-netnsid 4 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
17: vethab2eaf2@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 2e:d7:4e:09:b1:58 brd ff:ff:ff:ff:ff:ff link-netnsid 6 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
324: veth37bddd1@veth7960192: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 02:42:ac:11:00:07 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    veth addrgenmode eui64 
325: veth7960192@veth37bddd1: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master docker0 state LOWERLAYERDOWN mode DEFAULT group default 
    link/ether 3e:4f:24:3f:24:7c brd ff:ff:ff:ff:ff:ff promiscuity 1 
    veth 
    bridge_slave state disabled priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
327: veth1e74463@if326: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 86:6f:7c:f9:64:a6 brd ff:ff:ff:ff:ff:ff link-netnsid 12 promiscuity 1 
    veth 
    bridge_slave state forwarding priority 32 cost 2 hairpin on guard off root_block off fastleave off learning on flood on addrgenmode eui64 
221: veth5fd9673@if220: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 s

Does anyone know what this means? Is it normal that syslog would refer to a veth interface that's not in ip link show? How can I release this veth device?

As far as this goes:

$ uname -a
Linux <hostname> 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Thank you for any help or suggestions!!

Edit: Updated to add:

$ docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
maths
  • 101
  • 3

1 Answers1

1

AFAIK this is a kernel bug (https://github.com/docker/docker/issues/5618), Kube worked around it in the 1.2 time frame by flagging the container bridge as promiscuous. You can't recover without a restart. Hyperkube just adopts whatever settings are present on docker0, so you can make docker0 promisoucs too (ip link set docker0 promisc on).

beeps
  • 951
  • 6
  • 2