It needs a couple of explanations:
First of all, how traceroute works:
- Traceroute sends a package to the destination IP (94.2.55.112) with TLL=1
- The receving router decrements TTL and because TTL=0 discards the package and returns a
TTL exceeded
- Traceroute sends a package to the destination IP (94.2.55.112) with TLL=2
- The receving router decrements TTL and pass it to the next hop.
- The next hop decrements TTL and because TTL=0 discards the package and returns a
TTL exceeded
- Traceroute sends a package to the destination IP (94.2.55.112) with TLL=3
.
.
.
And so on, until the package reach its destination (94.2.55.112).
Now, let's check how routers work:
When a router receives a packet, its only mission is to send it to the next hop as soon as possible.
To achieve this mission, the router makes a decision based on a routing table; the routing table for an internet table could be huge. Checking the table and making a decision is called process switching and is the slowest way to do it. So there are some tricks to speed it up.
There is fast switching. With it, after a packet has been sent to the next hop, the routing information about how to get to the destination is stored in a fast cache. When the router receives another packet that is directed to the same destination it uses the cache. It's faster than the traditional way.
Cisco has developed a technique called Cisco Express Forwarding or CEF, that is even faster. CEF is a propietary system that stores information in a way that can be queried very quickly.
Now let's get all together:
When the destination of a traceroute is a host (94.2.55.112, for example), on the first packet the routers will use the traditional way (process switching) to know where is the destination, but after that in the next packets the routers will use fast switching or CEF. From origin to destination there will be a fixed path.
When the destination of a traceroute is a router (99.34.xx.xx), the intermediate routers will do the same as before (first packet with process switching and from then cache), but the destination router won't do it because there are no next hop. So, the destination router will do process switching to respond to each received packet.
The big problem arises when the destination router has two or more connections to different ISPs and the connections are doing load balancing.
Load balancing means that to avoid that a single connection gets overloaded until the router uses the next one, it uses a mechanism as round-robin to evenly distribute the traffic among all its connections.
Now you have the explosive mix for your situation: A traceroute to a router, it will use process switching for every package, that is influenced by load balancing and you have every time a different path.
Sorry for the long explanation, but as the result is a sum of multiple mechanisms, I wanted to make them as clear as possible.
Who says the routing table changed? ISP 1 may know a good route to IPS2 and dump packets for unknown destinations to ISP3. Maybe for no good reason. Maybe to balance a load. More importantly, why do you care? As long as it is outside your network and you get a good connection this should not influence you at all. – Hennes – 2015-08-11T10:34:13.207
It seems you have only given us what you think the problem might be, rather than what you are trying to resolve overall. Is this actually causing a problem? Or are you just frustrated because it is not taking the route you would like? – CharlieRB – 2015-08-11T11:21:36.123
@CharlieRB Yes. And as a result, the packets goes back and forth between overseas which increase the ping/latency from 30-50ms(good route) to 240-250ms(bad route) and neither you, me or anyone would agree to this confusing behavior right? That's why it DOES NOT MAKE A SENSE. – Semphie94 – 2015-08-11T13:19:38.903