2
0
I am running a Debian-based Linux container under Proxmox 4.4. This host has five network interfaces (though only two come into play in the problem I'm having).
While I am shelled into this host, I ping the IP address associated with eth1. What is happening and what I believe should happen are two very different things.
What I want to happen is for the ping packet to egress eth3, where it will be routed to eth1.
What is happening is that the IP stack sees I'm pinging a local interface and it then sends the reply right back up the stack. I know the packet is not going out and coming back for two reasons:
- A packet capture shows nothing hitting either eth1 or eth3.
- The ping latency averages 0.013 ms. If the packet were going out and back as intended, the latency would be about 60 ms.
Of course, I desire corresponding behavior when I ping the IP address associated with eth3. In that case, I want the packet to egress eth1 where it will be routed to eth3. Unfortunately, similar behavior as described above happens.
Below, I show the static routes I've set up to try and induce the desired behavior. Such routes work as intended on a Windows machine, but they do not work under the Linux setup I am using.
How may I configure this host to forward as intended?
root@my-host:~# uname -a
Linux my-host 4.4.35-1-pve #1 SMP Fri Dec 9 11:09:55 CET 2016 x86_64 GNU/Linux
root@my-host:~#
root@my-host:~# cat /etc/debian_version
8.9
root@my-host:~#
root@my-host:~# ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:192.0.2.65 Bcast:192.0.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:195028 errors:0 dropped:0 overruns:0 frame:0
TX packets:12891 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:92353608 (88.0 MiB) TX bytes:11164530 (10.6 MiB)
eth1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:128.66.100.10 Bcast:128.66.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:816 errors:0 dropped:0 overruns:0 frame:0
TX packets:486 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:149517 (146.0 KiB) TX bytes:34107 (33.3 KiB)
eth2 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:203.0.113.1 Bcast:203.0.113.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:738 errors:0 dropped:0 overruns:0 frame:0
TX packets:880 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:423603 (413.6 KiB) TX bytes:94555 (92.3 KiB)
eth3 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:128.66.200.10 Bcast:128.66.200.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:611 errors:0 dropped:0 overruns:0 frame:0
TX packets:182 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:43921 (42.8 KiB) TX bytes:13614 (13.2 KiB)
eth4 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:198.51.100.206 Bcast:198.51.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:183427 errors:0 dropped:0 overruns:0 frame:0
TX packets:83 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:85706791 (81.7 MiB) TX bytes:3906 (3.8 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:252 errors:0 dropped:0 overruns:0 frame:0
TX packets:252 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:22869 (22.3 KiB) TX bytes:22869 (22.3 KiB)
root@my-host:~#
root@my-host:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
128.66.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
203.0.113.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
128.66.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
198.51.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
root@my-host:~#
root@my-host:~# route -v add 128.66.200.10/32 gw 128.66.100.1
root@my-host:~# route -v add 128.66.100.10/32 gw 128.66.200.1
root@my-host:~#
root@my-host:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
203.0.113.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
198.51.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
128.66.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
128.66.100.10 128.66.200.1 255.255.255.255 UGH 0 0 0 eth3
128.66.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
128.66.200.10 128.66.100.1 255.255.255.255 UGH 0 0 0 eth1
root@my-host:~#
root@my-host:~# ping -c 3 128.66.100.10
PING 128.66.100.10 (128.66.100.10) 56(84) bytes of data.
64 bytes from 128.66.100.10: icmp_seq=1 ttl=64 time=0.008 ms
64 bytes from 128.66.100.10: icmp_seq=2 ttl=64 time=0.014 ms
64 bytes from 128.66.100.10: icmp_seq=3 ttl=64 time=0.017 ms
--- 128.66.100.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.008/0.013/0.017/0.003 ms
root@my-host:~#
THURSDAY, 8/17/2017 8:12 AM PDT UPDATE
Per the request of dirkt, I am elaborating on our architecture and the reason for my question.
The virtual host that is the subject of this post (i.e. the host with network interfaces eth1, eth3, and three other network interfaces unrelated to my question), is being used to test a physical, wired TCP/IP networking infrastructure we have set up. Specifically, it is the routing functionality of this TCP/IP networking infrastructure that we are testing.
We used to have two virtual hosts, not one as I've described in my original post. A ping between these two hosts would be our smoke test to ensure that the TCP/IP networking infrastructure under test was still working.
For reasons too detailed to get into, having two hosts made it difficult to collect the logs we need to. So, we switched to one host, gave it two NICs, set up static routes so that anything destined for NIC 2 would egress NIC 1 and vice versa. The problem is, as I've stated, they're not egressing.
This one host / two NIC setup has worked under Windows for us for years. I don't know if that is because Windows is broken and we were inadvertently taking advantage of a bug, or if Windows is fine (i.e. RFC-compliant) and we just need to get the configuration right on our Linux VMs to get the same behavior.
To summarize and distill down the long block of shell text above:
Two Interfaces:
eth1: 128.66.100.10/24; the router on this interface's network has IP address 128.66.100.1
eth3: 128.66.200.10/24; the router on this interface's network has IP address 128.66.200.1
Relevant Routes:
Destination Gateway Genmask Flags Metric Ref Use Iface
128.66.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
128.66.100.10 128.66.200.1 255.255.255.255 UGH 0 0 0 eth3
128.66.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
128.66.200.10 128.66.100.1 255.255.255.255 UGH 0 0 0 eth1
Command I'm Executing:
ping -c 3 128.66.100.10
The destination of 128.66.100.10 matches two of the above routes:
Destination Gateway Genmask Flags Metric Ref Use Iface
128.66.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
128.66.100.10 128.66.200.1 255.255.255.255 UGH 0 0 0 eth3
The route with the longest prefix match is:
Destination Gateway Genmask Flags Metric Ref Use Iface
128.66.100.10 128.66.200.1 255.255.255.255 UGH 0 0 0 eth3
What I am trying to understand is why, given the existence of this route, the packet won't egress eth3, travel through our TCP/IP networking infrastructure, come back and hit eth1 from the outside.
The TCP/IP stack is apparently not consulting the forwarding table. It's as if when it sees that I'm pinging a locally-connected interface, the TCP/IP stack just says, "Oh, this is local interface. So, I'm not going to consult the forwarding table. Instead, I'll just send an echo reply right back up the stack".
Is the behavior I desire RFC-compliant? If it is not, I must abandon the attempt. But if it is RFC-compliant, I would like to learn how to configure the Linux TCP/IP stack to allow this behavior.
MONDAY, 8/21/2017 UPDATE
I've discovered the sysctl rp_filter and accept_local kernel parameters. I have set them as follows:
root@my-host:~# cat /proc/sys/net/ipv4/conf/eth1/accept_local
1
root@my-host:~# cat /proc/sys/net/ipv4/conf/eth3/accept_local
1
root@my-host:~# cat /proc/sys/net/ipv4/conf/all/accept_local
1
root@my-host:~# cat /proc/sys/net/ipv4/conf/default/accept_local
1
root@my-host:~# cat /proc/sys/net/ipv4/conf/eth1/rp_filter
0
root@my-host:~# cat /proc/sys/net/ipv4/conf/eth3/rp_filter
0
root@my-host:~# cat /proc/sys/net/ipv4/conf/all/rp_filter
0
root@my-host:~# cat /proc/sys/net/ipv4/conf/default/rp_filter
0
Setting this kernel parameters, rebooting, verifying they survived the reboot, and testing again showed no difference in behavior.
Please note that my-host is an lxc Linux container running under Proxmox 4.4. I have also set rp_filter and accept_local as shown above on the hypervisor interfaces that corresponds to the eth1 and eth3 interfaces on my-host.
To re-summarize my objective, I have a Linux host with two NICs, eth1 and eth3. I am trying to ping out eth1, have the ping packet get routed through a TCP/IP network infrastructure under test, and make its way back to eth3.
Nothing I've tried above has allowed me to do so. How may I do so?
8/27/2017 UPDATE
Per a note by dirkt that I had failed to mention if eth1 and eth3 are purely virtual or if they correspond to a physical interface... eth1 and eth3 both correspond to the same physical interface on the hypervisor. The intent is that a packet that egresses eth1 actually physically leave the hypervisor box, go out onto a real TCP/IP network, and get routed back.
8/27/2017 UPDATE #2
Per dirkt, I have investigated network namespaces as it seemed quite promising. However, it doesn't "just work".
I am using LXC containers, and it seems that some of the isolation mechanisms present in containers are preventing me from creating a network namespace. Were I not running in a container, I think I'd have no problem adding the network namespace.
I am finding some references to making this work in LXC containers, but they are quite obscure and arcane. Not there yet, and have to throw in the towel for today... Should anyone have any suggestions in this regard, please advise...
Wrong site. You're looking for [unix.se] or [su] instead. This site is for programming related questions. – Ken White – 2017-08-17T01:24:13.823
Yes, agreed and apologies. Posted here by accident. Intended SuperUser. – Dave – 2017-08-17T01:34:10.673
Move has been requested via flag. – Dave – 2017-08-17T01:35:49.887
1Your description of what you want is a bit confusing and doesn't match the way routing works in the kernel. When a packet egresses on
eth3
, and will leave the physical interface, so it can't be routed toeth1
afterwards. Routing is always by destination. Can you explain the reason why you need your ping to behave this way? If you want to test forwarding frometh3
toeth1
, you need to pingeth3
from the outside, i.e. from another computer connected to that LAN segment. Or there may be a way to simulate ingressing packets oneth3
, but not withping
. – dirkt – 2017-08-17T09:29:40.700dirkt, I have expanded my original post to answer your questions. – Dave – 2017-08-17T16:42:20.060
I believe this might be an RFC 1122 "strong host model" vs. "weak host model" issue. I am looking into how to configure Linux to use the weak host model (if it is indeed using the strong host model). – Dave – 2017-08-17T19:14:32.253
Moderators: Is it possible to place a link in the "Unix & Linux" Stack Exchange group so that this thread appears there as well? There may well be some folks in that forum that could answer this question. – Dave – 2017-08-21T16:12:56.933
1
You might find your answer here.
– harrymc – 2017-08-23T18:41:07.830