How does mtr work / interpreting pain points using mtr

Question

I have been recently trying out mtr to get network congestion pain points. Following are sample mtr requests

Example 1

$ mtr --report -c 10 my.example.com 

HOST: ansh0l-Lenovo               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.0.1                0.0%    10    1.3   5.2   1.3  22.4   8.0
  2.|-- 10.10.20.1                 0.0%    10    3.9   2.5   1.6   4.6   1.2
  3.|-- NSG-Static-*.*.*.*        10.0%    10    7.7   6.7   5.1  10.1   1.5
  4.|-- AES-Static-*.*.*.*        10.0%    10   46.3  48.5  46.2  53.8   2.6
  5.|-- s38895.sgw.equinix.com     0.0%    10   50.3  47.9  46.1  50.3   1.5
  6.|-- 203.83.223.2               0.0%    10   49.0  48.7  47.0  51.1   1.2
  7.|-- 203.83.223.23              0.0%    10   47.8  48.1  46.9  50.0   1.0
  8.|-- ec2-175-*-*-*.ap-sou       0.0%    10   47.7  49.0  47.6  55.8   2.5
  9.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

Example 2

$ mtr --report -c 100 my.example.com 
HOST: ansh0l-Lenovo               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.0.1                2.0%   100    5.5   3.2   1.2  94.6   9.8
  2.|-- 10.10.20.1                 3.0%   100    4.3   3.9   1.5 160.5  16.3
  3.|-- NSG-Static-*.*.*.*         3.0%   100    9.9   8.1   4.3  99.0   9.8
  4.|-- AES-Static-*.*.*.*         3.0%   100   48.6  48.9  45.9 137.0   9.4
  5.|-- s38895.sgw.equinix.com     5.0%   100   46.7  49.6  45.5 155.6  11.5
  6.|-- 203.83.223.2               2.0%   100   52.4  53.0  46.5 213.3  20.8
  7.|-- 203.83.223.23              4.0%   100   49.1  50.0  46.2 145.6  11.5
  8.|-- ec2-175-*-*-*.ap-sou       5.0%   100   49.3  50.8  46.4 169.6  12.8
  9.|-- ???                       100.0   100    0.0   0.0   0.0   0.0   0.0

Questions:

1. Is Packet drops at HOST n = Sum of Packet drops for packets sent exclusively for HOST n? How safe is it to assume, that packets sent to say HOST 7, would have had the same previous hops?

2. In Example 1, At HOST 3 and 4, packet loss is same (10%). Is it safe to assume that all packet loss has thus happened at node 3?

3. In Example 1. When there is a packet loss at HOST 4 for 10%, shouldn't the next hops also be getting affected in terms of performance? If I have a 10% packet loss in one of the intermediate node, the nodes after it should also experience some packet loss, right?

4. In Example 2, some nodes have higher StDev. Should these be interpreted as points of unreliability?

Answer 1

1) Is Packet drops at HOST n = Sum of Packet drops for packets sent exclusively for HOST n?

Yes, they are specifically for that host. MTR relies on sending a packet of a fixed TTL, and expects to receive back a "time exceeded" ICMP response for the ICMP echo it originally sent which will come from the router which the TTL exceeded.

How safe is it to assume, that packets sent to say HOST 7, would have had the same previous hops?

Its pretty safe, I cant speak for all networks but its incredibly unusual on a inter-hop route to expect traffic to be routed to multiple paths -- it could happen but its more the exception than the norm.

2) In Example1, At HOST 3 and 4, packet loss is same (10%). Is it safe to assume that all packet loss has thus happened at node 3?

No, probably not. One would expect to see a derivative loss for all other hops thereafter (so around 10% loss on hops 5, 6, 7, 8 and 9) if it was the case node 3 really was dropping forwarded packets.

3) In Example1. When there is a packet loss at HOST 4 for 10%, shouldn't the next hops also be getting affected in terms of performance? If I have a 10% packet loss in one of the intermediate node, the nodes after it should also experience some packet loss, right?

Yes, if you are receiving genuine packet loss. Things are much more complicated than that unfortunately.

4) In Example2, some nodes have higher StDev. Should these be interpreted as points of unreliablity?

mtr can really only give you a ballpark figure. Many routers will drop ICMP packets as part of a quality of service regime (icmp being less important to it than tcp/udp traffic). Others may delay the traffic, or do both.

All you can really say is that sending ICMP traffic which that router should respond to may result in unreliable performance, but that you cannot say the same holds true for other types of traffic like TCP.

To summarize, if you have genuine loss of packets to a particular destination caused by a router mid-hop, you will see <= loss % all the way down the future hops.

If your destination hop responds with 0% loss, you are not dropping packets.

Some routers deliberately drop ICMP traffic they are responsible for responding to, thus you may get 'additional loss' confined to just that hop. If that hop is BOTH performing some form of traffic shaping AND really losing traffic things get horribly muddled because you cant tell how much loss you really have. Instead the best you can do is take the lowest loss % from a future hop and state that its probably around that %age of loss you are seeing.

Answer 2

In short, routers put a higher priority on processing traffic than they do on responding to packets with 0 ttl. Tools like mtr and traceroute are useful for determining the path you're using. They are not useful for determining the performance of that path. I went into this in more detail in my answer to How much network latency is "typical" for east - west coast USA?