I'm trying out vxlan with linux, and this problem has kept me stuck for days.
Simple Vxlan works fine
Simple vxlan with muticast works for cross-host communication, this simply create a vxlan vtep and assign an ip address:
ip link add vxlan100 type vxlan id 100 group 239.1.1.1 dev enp0s8
ip addr add 10.20.1.2/24 dev vxlan100
ip link set vxlan100 up
After running the above commands on both hosts, the topology is like:
And this works fine!
Bridged vxlan does not work
Then I try to setup bridged vxlan, to connect containers with vxlan, it does not work. Here is what I did to setup bridge and vxlan:
ip link add br0 type bridge
ip link add vxlan100 type vxlan id 100 group 239.1.1.1 dev enp0s8
ip link set dev vxlan100 master br0
ip link set vxlan100 up
ip link set br0 up
As for vms/containers, I simply use network namespace and veth peer for testing purpose:
ip link add veth0 tyep veth peer name veth1
ip link set dev veth0 master br0
ip link set veth0 up
ip netns add container1
ip link set dev veth1 netns container1
ip netns exec container1 ip link set lo up
ip netns exec contianer1 ip link set veth1 name eth0
ip netns exec container1 ip addr add 10.20.1.2/24 dev eth0
ip netns exec container1 ip link set eth0 up
And the topology is like the following diagram:
When I try to ping VM2 from VM1, it prints out destination Host Unreachable error:
[root@localhost ~]# ip netns exec container2 ping -c 3 10.20.1.3
PING 10.20.1.3 (10.20.1.3) 56(84) bytes of data.
From 10.20.1.2 icmp_seq=1 Destination Host Unreachable
From 10.20.1.2 icmp_seq=2 Destination Host Unreachable
From 10.20.1.2 icmp_seq=3 Destination Host Unreachable
--- 10.20.1.3 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
Using tcpdump to capture the packet on br0, the result:
[root@localhost vagrant]# tcpdump -e -nn -i br0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:35:02.533609 0e:f3:f2:c1:9a:b5 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.1.3 tell 10.20.1.2, length 28
15:35:02.533609 0e:f3:f2:c1:9a:b5 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.1.3 tell 10.20.1.2, length 28
15:35:02.534184 76:c2:07:e6:c2:7b > 0e:f3:f2:c1:9a:b5, ethertype ARP (0x0806), length 42: Reply 10.20.1.3 is-at 76:c2:07:e6:c2:7b, length 28
15:35:03.534274 0e:f3:f2:c1:9a:b5 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.1.3 tell 10.20.1.2, length 28
15:35:03.534274 0e:f3:f2:c1:9a:b5 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.1.3 tell 10.20.1.2, length 28
15:35:03.535261 76:c2:07:e6:c2:7b > 0e:f3:f2:c1:9a:b5, ethertype ARP (0x0806), length 42: Reply 10.20.1.3 is-at 76:c2:07:e6:c2:7b, length 28
15:35:04.536105 0e:f3:f2:c1:9a:b5 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.1.3 tell 10.20.1.2, length 28
15:35:04.536105 0e:f3:f2:c1:9a:b5 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.20.1.3 tell 10.20.1.2, length 28
15:35:04.536696 76:c2:07:e6:c2:7b > 0e:f3:f2:c1:9a:b5, ethertype ARP (0x0806), length 42: Reply 10.20.1.3 is-at 76:c2:07:e6:c2:7b, length 28
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel
As the output show, ARP request was sent in vxlan, and got a response to br0, but the bridge does not forward it to VM1. There are two issues I fully does not understand:
- Why there are two ARP requests for each ICMP ping?
- Why doesn't
br0forward ARP response toVM1, even when the destination MAC address is exactlyVM1?
For your reference I'm reading this 2017 vxlan linux post by vincent bernat.
Not sure if I did something wrong, or miss some configuration. Really need a solution or debug tips.
