Why does traceroute ended with additional hop after the destination?

Question

If nothing block the traffic, traceroute normally ends with the destination IP as the last hop. (10.1.1.10 in this case)

Normal traceroute would be like this.

user@linux:~$ traceroute 10.1.1.10
traceroute to 10.1.1.10 (10.1.1.10), 30 hops max, 60 byte packets
 1  10.2.8.2 (10.2.8.2)  0.572 ms  0.692 ms  0.837 ms
 2  10.1.9.50 (10.1.9.50)  202.638 ms 10.1.9.78 (10.1.9.78)  202.547 ms 10.1.9.50 (10.1.9.50)  202.139 ms
 3  10.1.4.9 (10.1.4.9)  202.508 ms  202.483 ms 10.1.4.13 (10.1.4.13)  204.149 ms
 4  10.1.1.10 (10.1.1.10)  202.133 ms  202.100 ms  202.692 ms
user@linux:~$ 

Recently, I encountered an issue whereby there was an additional hop (10.1.1.9) in the traceroute output (look at hop 5).

Source IP Address: 10.2.8.8

user@linux:~$ ifconfig | head -2
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.2.8.8  netmask 255.255.255.0  broadcast 10.2.8.255
user@linux:~$ 

Destination IP Address: 10.1.1.10

Additional hop: 10.1.1.9 ???

user@linux:~$ traceroute 10.1.1.10
traceroute to 10.1.1.10 (10.1.1.10), 30 hops max, 60 byte packets
 1  10.2.8.2 (10.2.8.2)  0.572 ms  0.692 ms  0.837 ms
 2  10.1.9.50 (10.1.9.50)  202.638 ms 10.1.9.78 (10.1.9.78)  202.547 ms 10.1.9.50 (10.1.9.50)  202.139 ms
 3  10.1.4.9 (10.1.4.9)  202.508 ms  202.483 ms 10.1.4.13 (10.1.4.13)  204.149 ms
 4  10.1.1.10 (10.1.1.10)  202.133 ms  202.100 ms  202.692 ms
 5  10.1.1.9 (10.1.1.9)  6201.720 ms !H * *
user@linux:~$ 

Also, if you look at hop 2 and 3, there is additional IP Addresses (10.1.9.78 & 10.1.9.50)

Why did this happen? I've never seen anything like this before.

Was this because of the server configuration?

Answer 1

traceroute works by sending UDP or ICMP Echo packets with sequentially increasing Time-to-Live (TTL) fields. TTL is decremented by 1 by each router that processes the packet.

It expects two types of responses to its probes.

• If TTL reaches 0 when a router decrements it, it responds with an ICMP Time-Exceeded message and doesn't forward the packet.
• The final destination responds with an ICMP Port-Unreachable message (in the UDP case) or ICMP Echo-Response message (in the ICMP case).

When it receives the second type, it knows it has reached the destination, and stops sending probes with higher TTLs. Note that it doesn't base this decision on the address that the response comes from -- if it receives a Time-Exceeded message from the target address, it will keep incrementing.

So your trace indicates that a router responded with Time-Exceeded and the target address as its source. This probably means that it's a router performing NAT, and the target address is the public address corresponding to a private address behind it.

What's strange, though, is that the final response came from a different address. Normally you'd expect the response to go back through the router, so its private IP would be translated back to the public IP. In this case you'd see two rows with 10.1.1.10 as the address.

Apparently in this case the path from 10.1.1.9 to the original machine does not need to go through the NAT router, so its address doesn't get translated. Asymmetric routing can often produce anomalous traceroute results. In this case, all your machines are within the 10.0.0.0/8 private address space, so it's not completely surprising that there are direct paths available.

Answer 2

Without knowing that router's config there's no way to be sure - however, a likely reason is destination NAT aka reverse NAT aka port forwarding or - in this case rather - an exposed host.

The 10.1.1.10 router may be configured to forward/DNAT everything to the 10.1.1.9 host (=exposed host), including requests from private IPs. From a public IP you'd see a double last hop because the actual destination 10.1.1.9 is hidden by the NAT router. In this case it's possible that just the request is DNATed and the reply is just forwarded as is.

It's also possible that both 10.1.1.10 and 10.1.1.9 are DNATed elsewhere and the latter is the default reply address. That would explain the large RTT increase.

Answer 3

Take your attention on times. Traceroute sends packets which ask switches or routers to answer to source with response " I am processing". But this packets may take its destination, your machine in this example, with various order, depending on TTL and response time configurations, dependent both on config of devices between, routing tables and topology of the network.