0

I have a virtual machine created with libvirt/qemu/kvm attached with a TAP to a Linux bridge (virbr1). Internally the VM has an IP of 10.99.0.9. And has the following routing setup.

default via 10.99.0.1 dev enp1s0 proto static onlink 
10.99.0.1 via 10.99.0.1 dev enp1s0 proto static onlink

Here 10.99.0.1 is the IP of the bridge

I wanted all the traffic coming from that VM to go to a VPN route (wg0) I had setup, so on the host I did

ip route add default dev wg0 table 42
ip rule add from 10.99.0.9 table 42

This worked fine, but I noticed that inside the VM I could still ping the host machine IP 192.168.2.1. I eventually realised this is because the lookup local ip rule has a preference of 0 which is higher than that of the rule I added. I thought I'd simply swap the order, and all would be well.

ip rule add preference 300 lookup local # 300 here is arbitrarily higher than 0
ip rule del preference 0
ip rule add from 10.99.0.9 table 42 preference 0

However upon doing this, I now have no connectivity within the VM. If I sniff on the linux bridge I see it's continually sending the ARP request looking for who has 10.99.0.1 , and no response is forthcoming. I didn't think these routing decisions should affect ARP at all, since it's acting alongside IP, so this has be confused. I've confirmed it's those exact lines that cause the issue

Why are ARP replies not making it back to the VM interface?

joshuao
  • 1
  • 1

1 Answers1

1

My impression is that your answer is provided in your question.

You've changed the local ip rule preference of 0 to 300 therfore pushed the default route to be the most prefered route:

ip route add default dev wg0 table 42
ip rule add from 10.99.0.9 table 42
#and lastly:
ip rule add from 10.99.0.9 table 42 preference 0

which caused the kernel to prefer this route exclusively.

you said: "I see it's continually sending the ARP request looking for who has 10.99.0.1, and no response is forthcoming."

which is reasonable as ARP broadcasts are going to the wg0 interface where no device has 10.99.0.1 IP address (I assume).

solution ?

My advice is not to mess with route priority and if needed - firewall the access which is not required for the VM by iptables.

Roman Spiak
  • 519
  • 2
  • 9
  • I was under the impression that /IP/ packets coming from 10.99.0.9 would be routed to the wg interface, however an ARP message is not an IP packet. I did not think the ip rules would even do anything until actual IP packets started flying, which only happens once the interface within the VM knows which MAC to send its packets too. Is this not correct? – joshuao Dec 28 '21 at 09:42
  • looking closely at https://en.wikipedia.org/wiki/Address_Resolution_Protocol#Packet_structure you are correct to assume ARP operates on Layer2 of OSI and IP routes are Layer3 so ARP is not affected by the IP routing. However please lookup what this command does closely : `ip route add default dev wg0 table 42` by sniffing the bridge before and after applying this as I suspect it may interfere with layer2 as well... – Roman Spiak Dec 28 '21 at 10:22
  • I think the local routes are what determines whether ARP request should be replied, like if the system is configured to ignore all its local route, then it won't respond to ARP requests. (For the details one may have to check the kernel code; just note that the priority 0 is quite special. It's possible that it won't check local table when there's no priority 0 rule that look it up.) – Tom Yan Dec 28 '21 at 12:34