Hosts on subnet can only access hosts subnet directly

Question

I recently setup Proxmox as a virtualization host. It is just KVM + Debian in a nice linux distribution. My home network is 192.168.12.0/24. I wanted to use 192.168.14.0/24 as the subnet for all of my VMs on the proxmox machine. I use DHCP to configure almost all of my hosts. If I want a 'static-ip' I generally just configure a DHCP reservation for the device manually. So, I wanted my VMs to get DHCP addresses in the 192.168.14.0/24 subnet.

When I installed Proxmox it created a bridge called vmbr0 that is bridged with eth0. The IP address of the bridge is 192.168.12.12.

I added a bridge in /etc/network/interfaces called vmbr14 bridged with eth0.14. I gave it an IP of 192.168.14.12. I also modprobe'd in the 8021q module and added it to /etc/modules.

My DHCP server lives on another machine 192.168.12.95 with an interface of eth0 on it. I added another interface called eth0.14 and gave it the IP of 192.168.14.95. I updated /etc/dhcp/dhcpd.conf with another pool for the subnet 192.168.14.0. The default route specified is 192.168.14.12, the Proxmox machine. DHCP worked immediately with no problems.

To get hosts on the 192.168.12.0/24 network to know how to route 192.168.14.0/24 I went to my internet gateway (192.168.12.2) and added a route as follows

192.168.14.0    192.168.12.12   255.255.255.0   UG    1      0        0 eth2

The proxmox box already has forwarding enabled for IPv4

ericu@basov:~$ cat /proc/sys/net/ipv4/ip_forward 
1

After that, things just worked. I can SSH and ping my VMs from my desktop without issue.

But on the VMs themselves I can't reach 192.168.12.95 only 192.168.14.95. I did a ping on one of the VMs to 192.168.12.95 and packet captured using tcpdump

ericu@katz:~$ sudo tcpdump -n -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:50:16.007898 IP 192.168.14.130 > 192.168.12.95: ICMP echo request, id 11498, seq 18, length 64
18:50:17.010380 IP 192.168.14.130 > 192.168.12.95: ICMP echo request, id 11498, seq 19, length 64
18:50:18.012969 IP 192.168.14.130 > 192.168.12.95: ICMP echo request, id 11498, seq 20, length 64
18:50:19.015433 IP 192.168.14.130 > 192.168.12.95: ICMP echo request, id 11498, seq 21, length 64
18:50:20.018043 IP 192.168.14.130 > 192.168.12.95: ICMP echo request, id 11498, seq 22, length 64

The packets obviously get to the machine, but I have no idea what happens after that. The machine just doesn't reply. TCP also doesn't work.

ericu@katz:~$ sudo tcpdump -n tcp port 3000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:53:21.048128 IP 192.168.14.130.58499 > 192.168.12.95.3000: Flags [S], seq 2306774687, win 14600, options [mss 1460,sackOK,TS val 665263 ecr 0,nop,wscale 7], length 0
18:53:22.051442 IP 192.168.14.130.58499 > 192.168.12.95.3000: Flags [S], seq 2306774687, win 14600, options [mss 1460,sackOK,TS val 666264 ecr 0,nop,wscale 7], length 0
18:53:24.060540 IP 192.168.14.130.58499 > 192.168.12.95.3000: Flags [S], seq 2306774687, win 14600, options [mss 1460,sackOK,TS val 668268 ecr 0,nop,wscale 7], length 0

If I try and access the IP of 192.168.14.95 from the VMs everything works fine.

If I do ifdown eth0.14 on the machine 192.168.12.95 is then reachable from the VMs. Obviously this isn't practical as the DHCP server needs to be reachable.

Why can the machines on the 192.168.14.0/24 subnet only reach the machine using the 192.168.14.95 address and not 192.168.12.95 address as well?

The is the routing table on 192.168.12.95

ericu@katz:~$ ip route show
default via 192.168.12.2 dev eth0  metric 100 
192.168.12.0/24 dev eth0  proto kernel  scope link  src 192.168.12.95 
192.168.14.0/24 dev eth0.14  proto kernel  scope link  src 192.168.14.95 
ericu@katz:~$ 

This the routing table on 192.168.14.130

[ericu@squid3 ~]$ ip route show
default via 192.168.14.12 dev ens18  proto static  metric 1024 
192.168.14.0/24 dev ens18  proto kernel  scope link  src 192.168.14.130 
[ericu@squid3 ~]$ 

In order to troubleshoot this issue more I created a VM on my desktop. I edited /etc/network/interfaces as follows

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp


auto eth0.14
iface eth0.14 inet static
    address 192.168.14.14
    netmask 255.255.255.0

The machine got a DHCP address on eth0 of 192.168.12.172. From 192.168.12.130 I still could not access it using 192.168.12.172. On the test VM I changed /etc/iproute2/rt_tables to look like the following

#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1  inr.ruhep
14 vlan14

I then ran the following commands

root@ubuntuvmdesktop:/home/ericu# ip route show
default via 192.168.12.2 dev eth0 
192.168.12.0/24 dev eth0  proto kernel  scope link  src 192.168.12.172 
192.168.14.0/24 dev eth0.14  proto kernel  scope link  src 192.168.14.14 
root@ubuntuvmdesktop:/home/ericu# ip route del 192.168.14.0/24 dev eth0.14 src 192.168.14.14
root@ubuntuvmdesktop:/home/ericu# ip route show
default via 192.168.12.2 dev eth0 
192.168.12.0/24 dev eth0  proto kernel  scope link  src 192.168.12.172 
root@ubuntuvmdesktop:/home/ericu# ip route add 192.168.14.0/24 dev eth0.14 src 192.168.14.14 table vlan14
root@ubuntuvmdesktop:/home/ericu# ip route add default via 192.168.14.12 dev eth0.14 src 192.168.14.14 table  vlan14
RTNETLINK answers: Network is unreachable
root@ubuntuvmdesktop:/home/ericu# ip rule add from 192.168.14.14 table vlan14
root@ubuntuvmdesktop:/home/ericu# ip route show table vlan14
192.168.14.0/24 dev eth0.14  scope link  src 192.168.14.14 

After doing this the VM with IP can reach the machine using 192.168.14.14 and 192.168.12.172

[ericu@squid3 ~]$ nc -v 192.168.12.172 22
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.12.172:22.
SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
^C
[ericu@squid3 ~]$ nc -v 192.168.14.14 22
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.14.14:22.
SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
^C
[ericu@squid3 ~]$ 

So this is at least one way to fix the problem. However, I don't really understand what it does. I do understand that I added a separate routing table called 'vlan14' to the machine. My guess it that using ip rule add somehow directs traffic destined for 192.168.14.14 into its own routing table. Does this isolate the routing for the two subnets into their own routing table? However, all the route changes I made using ip route do not persist across a restart. It seems like I need to update /etc/network/interfaces to somehow indicate that routes should go into this alternate routing table.

Can someone please explain how the separate routing table works? Could someone please explain how to change my configuration so the changes made using the ip command are persistent across reboots?

Answer 1

All linux routing is done via routing tables.

A default installation without any modifications will have a "main" table, a "default" table and a "local" table. The "main" table is shown if you run ip route show without additional options; the "local" table is for the loopback addresses (127.0.0.0/8, ff00::/8). The "default" table is used as a last resort and is usually empty.

you can add special routing instructions which apply to destinations that are selected by an ip rule command. You add rules with a priority, lowest numbers are matched first. By default these are the default rules:

$ ip rule show
0:  from all lookup local 
32766:  from all lookup main 
32767:  from all lookup default

A typical usage for rules is if you have a multi-homed system, i.e. a system with more than one connection to the internet. In this setup you need to ensure that reply packets are sent out over the same interface (i.e. internet connection) as the original packet came in on. Typically this means ensuring that packets with a certain source address are sent out from the corresponding interface:

# ip rule add from 1.2.3.4 pref 30000 lookup first
# ip rule add from 5.6.7.8 pref 30000 lookup second
# ip route add table first  default via 1.2.3.254 dev if1 src 1.2.3.4
# ip route add table second default via 5.6.7.254 dev if2 src 5.6.7.8

Now add a default route which goes over the preferred connection (e.g. the fastest or cheapest in volume):

# ip route add table default via 5.6.7.254 dev if2 src 5.6.7.8

Append a second default route for when the preferred interface does down:

# ip route add table default via 1.2.3.254 dev if1 src 1.2.3.4

You'll have to ensure that when the preferred interface comes back up, its default route is again first in the table. I generally do that by adding that route, deleting the other and appending it again; there's probably some better way...

You can use different selectors in ip rule add, e.g. make it depend on a firewall mark that's been set on the packet by iptables. Use ip rule add help for what you can put there; this applies in general to all ip commands (e.g. ip help, ip route help).

The Linux Advanced Routing Guide describes everything about this subject.

---

To make your manual route configuration apply after reboots etc., add them to the /etc/network/interfaces interface definitions with post-up added in front, those will be executed after the interface comes up. man interfaces describes all the possible things you can put in the interfaces file.