Strange traceroute behavior when using iptables/nat to redirect an IP

Question

At a client of mine, who has a LAN of 10.0.1.0/24, they have a Windows machine with ip 10.0.1.8 that hosts a rather old legacy application (that of course no one has the source code for) that wants to connect to a hard-coded IP address of 192.168.173.93.

That IP used to belong to a machine on a remote network 192.168.173.0/24, which is accessed over a VPN via 10.0.1.4 (which is a Linux server, separate from the main router, 10.0.1.254) and to which all computers on the LAN have a static route to via 10.0.1.4.

The problem is that the machine that used to be 192.168.173.93 at the remote network no longer exists and it's software has been moved to a public VPS, accessible via a public ip, lets say 1.2.3.4.

So a solution that I came up with was to add a couple of iptables rules on the 10.0.1.4 (the Linux server) to redirect packets from 10.0.1.8 bound for 192.168.173.93 to 1.2.3.4:

iptables -t nat -I PREROUTING  -i br0 -s 10.0.1.8 -d 192.168.173.93 -j DNAT --to-destination 1.2.3.4
iptables -t nat -I POSTROUTING -o br0 -s 10.0.1.8 -d 1.2.3.4        -j SNAT --to-source      10.0.1.4

And after I can ping and make general connections to 192.168.173.93 from 10.0.1.8 just fine:

C:\Work>ping 192.168.173.93

Pinging 192.168.173.93 with 32 bytes of data:

Reply from 192.168.173.93: bytes=32 time=37ms TTL=115
Reply from 192.168.173.93: bytes=32 time=36ms TTL=115
Reply from 192.168.173.93: bytes=32 time=36ms TTL=115
Reply from 192.168.173.93: bytes=32 time=36ms TTL=115

Ping statistics for 192.168.173.93:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 36ms, Maximum = 37ms, Average = 36ms

The ping times above are consistent with what I see from directly pinging 1.2.3.4, so that looks good. But when I test with tracert, I see something really strange:

C:\Work>tracert 192.168.173.93

Tracing route to 192.168.173.93 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.173.93
  2    <1 ms    <1 ms    <1 ms  192.168.173.93
  3     8 ms     8 ms     8 ms  192.168.173.93
  4    12 ms    13 ms    11 ms  192.168.173.93
  5    15 ms    31 ms    15 ms  192.168.173.93
  6    13 ms    13 ms    13 ms  192.168.173.93
  7    21 ms    21 ms    21 ms  192.168.173.93
  8    20 ms    19 ms    23 ms  192.168.173.93
  9    21 ms   108 ms    74 ms  192.168.173.93
 10    31 ms    31 ms    38 ms  192.168.173.93
 11    37 ms    37 ms    37 ms  192.168.173.93
 12    36 ms    39 ms    36 ms  192.168.173.93
 13    37 ms    40 ms    37 ms  192.168.173.93

Trace complete.

Which is similar in length to a direct run of tracert:

C:\Work>tracert -d 1.2.3.4

Tracing route to 1.2.3.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  10.0.1.254
  2     7 ms     8 ms     8 ms  HOP-A
  3    19 ms    11 ms    11 ms  HOP-B
  4    12 ms    11 ms    11 ms  HOP-C
  5    22 ms    14 ms    16 ms  HOP-D
  6    21 ms    21 ms    22 ms  HOP-E
  7    95 ms    59 ms    26 ms  HOP-F
  8    20 ms    21 ms    32 ms  HOP-G
  9    31 ms    30 ms    30 ms  HOP-H
 10    36 ms    36 ms    36 ms  HOP-I
 11    37 ms    36 ms    36 ms  HOP-J
 12    36 ms    36 ms    37 ms  1.2.3.4

Trace complete.

(10.0.1.254 is the main router on the local network.)

So I ran the trace route for 192.168.173.93 again while running tcpdump on 10.0.1.4 (To make it easier to read, I added line breaks to show what parts correspond to each hop and to each of the three pings in the first tracert above.):

$ tcpdump -nni br0 -e icmp --no-promiscuous-mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:09:17.288490 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 34304, length 72
09:09:17.288554 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 134: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 100
----- ping -----
09:09:17.289002 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 34560, length 72
09:09:17.289070 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 134: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 100
----- ping -----
09:09:17.289520 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 34816, length 72
09:09:17.289589 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 134: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 100

09:09:18.297270 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 35072, length 72
09:09:18.297339 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 35072, length 72
09:09:18.297544 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 134: 10.0.1.254 > 10.0.1.4: ICMP time exceeded in-transit, length 100
09:09:18.297584 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 134: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 100
----- ping -----
09:09:18.298043 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 35328, length 72
09:09:18.298094 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 35328, length 72
09:09:18.298213 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 134: 10.0.1.254 > 10.0.1.4: ICMP time exceeded in-transit, length 100
09:09:18.298242 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 134: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 100
----- ping -----
09:09:18.298667 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 35584, length 72
09:09:18.298713 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 35584, length 72
09:09:18.298818 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 134: 10.0.1.254 > 10.0.1.4: ICMP time exceeded in-transit, length 100
09:09:18.298846 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 134: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 100

09:09:19.305326 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 35840, length 72
09:09:19.305389 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 35840, length 72
09:09:19.327923 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-A > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:19.327967 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36
----- ping -----
09:09:19.328407 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 36096, length 72
09:09:19.328457 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 36096, length 72
09:09:19.337999 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-A > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:19.338041 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36
----- ping -----
09:09:19.338521 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 36352, length 72
09:09:19.338567 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 36352, length 72
09:09:19.351300 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-A > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:19.351343 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36

09:09:20.344214 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 36608, length 72
09:09:20.344278 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 36608, length 72
09:09:20.355855 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-B > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:20.355898 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:20.356847 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 36864, length 72
09:09:20.356871 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 36864, length 72
09:09:20.367650 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-B > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:20.367698 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:20.368114 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 37120, length 72
09:09:20.368161 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 37120, length 72
09:09:20.379480 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-B > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:20.379526 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76

09:09:21.375622 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 37376, length 72
09:09:21.375688 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 37376, length 72
09:09:21.388676 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-C > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:21.388720 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:21.389293 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 37632, length 72
09:09:21.389330 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 37632, length 72
09:09:21.404654 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-C > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:21.404695 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:21.405334 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 37888, length 72
09:09:21.405374 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 37888, length 72
09:09:21.416980 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-C > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:21.417018 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76

09:09:22.412744 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 38144, length 72
09:09:22.412798 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 38144, length 72
09:09:22.426660 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-D > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:22.426704 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:22.427382 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 38400, length 72
09:09:22.427438 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 38400, length 72
09:09:22.441656 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-D > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:22.441698 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:22.442238 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 38656, length 72
09:09:22.442274 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 38656, length 72
09:09:22.455759 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-D > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:22.455802 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76

09:09:23.456528 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 38912, length 72
09:09:23.456601 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 38912, length 72
09:09:23.478954 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-E > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:23.478997 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:23.479470 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 39168, length 72
09:09:23.479519 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 39168, length 72
09:09:23.500600 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-E > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:23.500644 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:23.501168 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 39424, length 72
09:09:23.501201 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 39424, length 72
09:09:23.522284 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-E > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:23.522335 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76

09:09:24.509376 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 39680, length 72
09:09:24.509419 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 39680, length 72
09:09:24.529265 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-F > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:24.529315 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36
----- ping -----
09:09:24.529805 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 39936, length 72
09:09:24.529854 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 39936, length 72
09:09:24.549402 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-F > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:24.549429 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36
----- ping -----
09:09:24.549822 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 40192, length 72
09:09:24.549868 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 40192, length 72
09:09:24.570871 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-F > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:24.570915 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36

09:09:25.556159 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 40448, length 72
09:09:25.556226 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 40448, length 72
09:09:25.577631 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 182: HOP-G > 10.0.1.4: ICMP time exceeded in-transit, length 148
09:09:25.577675 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 182: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 148
----- ping -----
09:09:25.578113 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 40704, length 72
09:09:25.578162 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 40704, length 72
09:09:25.599536 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 182: HOP-G > 10.0.1.4: ICMP time exceeded in-transit, length 148
09:09:25.599582 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 182: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 148
----- ping -----
09:09:25.600110 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 40960, length 72
09:09:25.600144 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 40960, length 72
09:09:25.620280 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 182: HOP-G > 10.0.1.4: ICMP time exceeded in-transit, length 148
09:09:25.620322 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 182: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 148

09:09:26.608931 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 41216, length 72
09:09:26.608977 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 41216, length 72
09:09:26.642008 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 182: HOP-H > 10.0.1.4: ICMP time exceeded in-transit, length 148
09:09:26.642048 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 182: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 148
----- ping -----
09:09:26.642506 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 41472, length 72
09:09:26.642555 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 41472, length 72
09:09:26.675939 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 182: HOP-H > 10.0.1.4: ICMP time exceeded in-transit, length 148
09:09:26.675976 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 182: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 148
----- ping -----
09:09:26.676626 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 41728, length 72
09:09:26.676666 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 41728, length 72
09:09:26.707232 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 182: HOP-H > 10.0.1.4: ICMP time exceeded in-transit, length 148
09:09:26.707274 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 182: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 148

09:09:27.687132 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 41984, length 72
09:09:27.687170 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 41984, length 72
09:09:27.724543 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-I > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:27.724578 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:27.725211 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 42240, length 72
09:09:27.725252 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 42240, length 72
09:09:27.762221 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-I > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:27.762265 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76
----- ping -----
09:09:27.762704 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 42496, length 72
09:09:27.762753 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 42496, length 72
09:09:27.799409 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 110: HOP-I > 10.0.1.4: ICMP time exceeded in-transit, length 76
09:09:27.799434 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 110: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 76

09:09:28.770910 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 42752, length 72
09:09:28.770966 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 42752, length 72
09:09:28.808115 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-J > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:28.808154 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36
----- ping -----
09:09:28.808666 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 43008, length 72
09:09:28.808701 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 43008, length 72
09:09:28.845294 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-J > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:28.845337 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36
----- ping -----
09:09:28.845927 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 43264, length 72
09:09:28.845976 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 43264, length 72
09:09:28.882803 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 70: HOP-J > 10.0.1.4: ICMP time exceeded in-transit, length 36
09:09:28.882859 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 70: 192.168.173.93 > 10.0.1.8: ICMP time exceeded in-transit, length 36

09:09:29.854005 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 43520, length 72
09:09:29.854043 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 43520, length 72
09:09:29.890198 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 1.2.3.4 > 10.0.1.4: ICMP echo reply, id 512, seq 43520, length 72
09:09:29.890253 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 106: 192.168.173.93 > 10.0.1.8: ICMP echo reply, id 512, seq 43520, length 72
----- ping -----
09:09:29.890813 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 43776, length 72
09:09:29.890862 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 43776, length 72
09:09:29.927667 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 1.2.3.4 > 10.0.1.4: ICMP echo reply, id 512, seq 43776, length 72
09:09:29.927700 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 106: 192.168.173.93 > 10.0.1.8: ICMP echo reply, id 512, seq 43776, length 72
----- ping -----
09:09:29.928462 MAC_10.0.1.8_____ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 10.0.1.8 > 192.168.173.93: ICMP echo request, id 512, seq 44032, length 72
09:09:29.928503 MAC_10.0.1.4_eth0 > MAC_10.0.1.254___, ethertype IPv4 (0x0800), length 106: 10.0.1.4 > 1.2.3.4: ICMP echo request, id 512, seq 44032, length 72
09:09:29.964340 MAC_10.0.1.254___ > MAC_10.0.1.4_eth0, ethertype IPv4 (0x0800), length 106: 1.2.3.4 > 10.0.1.4: ICMP echo reply, id 512, seq 44032, length 72
09:09:29.964370 MAC_10.0.1.4_eth0 > MAC_10.0.1.8_____, ethertype IPv4 (0x0800), length 106: 192.168.173.93 > 10.0.1.8: ICMP echo reply, id 512, seq 44032, length 72
^C
150 packets captured
150 packets received by filter
0 packets dropped by kernel

So it seems to me that the responses coming from each hop are not making their way back to 10.0.1.8 properly.

Is this a limitation of iptables? Or am I missing something related to conntrack/etc, that would allow hop responses to be seen as related? Or am I missing a better approach?

More info:

10.0.1.4:

iptables v1.4.12 tcpdump version 4.6.2 libpcap version 1.6.2

Note about ``br0`` on ``10.0.1.4``, that is used in the iptables rules; this is a left over bridge that presently only has ``eth0`` and (unused) ``eth1`` as members. ``br0`` and ``eth0`` show the same MAC address in ``ip link`` and ``ifconfig``. — gordonfish, Nov 05 '19 at 22:17
Because Windows uses ICMP for tracert, the ICMP query ID is used for the NAT table lookup, and the source of the replies will be translated to the destination of the request that was sent. This is destination NAT, so the destination address of requests is what is translated, and the source of the replies will be translated back to the destination address of the requests. — Ron Maupin, Nov 10 '19 at 19:41

Peter Green · Answer 1 · 2019-11-10T21:04:23.880

I am not 100% sure but I believe this is a feature and if the network is otherwise working fine you should stop worrying about it.

Traceroute works by sending out probe packets with increasing TTLs and looking for ICMP errors. As well as being used for traceroute ICMP errors are important to the correct functioning of the network stack, for example path MTU discovery relies on ICMP errors.

In order to allow ICMP errors to be matched to the communication session that caused them they contain a portion of the packet that caused the error. In order for ICMP errors relating to a natted connection to be correctly matched to the session that caused them, the addresses in the embedded partial packet copy must be translated.

Furthermore NAT is often used at the boundary between a private network and the public Internet. If packets with private source addresses are routed onto the public Internet then it is very likely they will be dropped by ingress filtering.

Thus I believe that what is happening here is that iptables is changing the source addresses of the ICMP error packets to reduce the risk that they will fall victim to ingress filtering.

It sounds like you two are talking more about masquerading when it comes to translating the address within the ICMP packets?

An ICMP error packet normally contains FOUR ip addresses, two of which are normally equal.

The source address of the error packet (normally the system on which the error happened).
The destination address of the ICMP error packet (equal to the source addres of the packet that caused the error).
The source address in the partial copy of the packet that caused the error.
The destination address in the partial copy of the packet that caused the error.

When you send your trace packets through your NAT, the source and destination addresses are changed and the NAT box creates an entry for the connection* in it's NAT mapping tables.

When an ICMP error arrives at the NAT box, the NAT engine will try to match it against it's table of connections. If it matches then the NAT engine will perform transation on it.

The NAT needs to map the source and destination in the embedded packet copy to match what the client expects. It also needs to change the destination of the ICMP packet so the packet is delivered to the client. It does not strictly-speaking need to change the source of the ICMP packet itself, but it seems that it does anyway. As I said above this is most likely to reduce the risk of the error getting caught in an ingress filter.

In your network we have two NATs, a one-armed NAT performing the redirection and a regular NAT between your network and the internet. So the flow looks something like.

client->one armed NAT: 10.0.1.8 > 192.168.173.93: ICMP echo request id 512, seq 36096
one-armed nat->regular NAT: 10.0.1.4 > 1.2.3.4: ICMP echo request id 512, seq 36096
regular nat->internet: ???.???.???.??? > 1.2.3.4: ICMP echo request id 512, seq 36096
internet->regular NAT: HOP-A > ???.???.???.???: ICMP time exceeded in-transit with embedded partial packet copy ???.???.???.??? > 1.2.3.4: ICMP echo request id 512, seq 36096
regular NAT->one armed NAT: HOP-A > 10.0.1.4: ICMP time exceeded in-transit with embedded partial packet copy 10.0.1.4 > 1.2.3.4: ICMP echo request id 512, seq 36096
one armed NAT->client: 192.168.173.93 -> 10.0.1.8 ICMP time exceeded in-transit with embedded partial packet copy 10.0.1.8 > 192.168.173.93: ICMP echo request id 512, seq 36096

* Yes I know ICMP queries don't strictly speaking have connections, nevertheless they do create entries in NAT tracking tables. The query ID is effectively treated like a source port for the connection. Iptables NAT uses a port-preservative strategy by default, so the query ID will only be changed if nessacery to disambiguate.

Yes, it appears NAT is set up to translate the source address of incoming packets because it is destination NAT, so the source of replies will need to match the destination of what was sent. The ICMP query ID is what will determine the lookup in the NAT table to change the source address on the replies (Windows uses ICMP for tracert).. — Ron Maupin, Nov 10 '19 at 19:37
Thanks for the responses. It sounds like you two are talking more about masquerading when it comes to translating the address within the ICMP packets? — gordonfish, Nov 10 '19 at 20:01
I've read your update, Peter. So it looks like the trouble part is ``HOP > 10.0.1.4`` being rewritten to ``192.168.173.93 > 10.0.1.8`` (for the icmp time exceeded) instead of to ``HOP > 10.0.1.8``, if I'm understanding correctly? — gordonfish, Nov 10 '19 at 22:32
Yes, that is why your traceroute looks the way it does, as I said I belive this was likely a deliberate design desicion on the part of the iptables authors. — Peter Green, Nov 10 '19 at 23:58
Do you know of any way to alter how iptables handles those icmp replies/timeouts? I have been searching and experimenting but haven't been able to get that behavior to change. — gordonfish, Nov 11 '19 at 21:30

Strange traceroute behavior when using iptables/nat to redirect an IP

1 Answers1