understanding traceroute output to diagnose an issue with Verizon FIOS

Question

Tech support has tried, and failed, to repeatedly fix anything, which makes me think the issues are deep within the Verizon network. But I'd like a quick sanity check, since I'm not very experienced reading traceroutes. Basically, the issue presents as one of websites intermittently not loading---it's not a speed issue (speedtest is fast, streaming video is fast), but some other kind of issue related to making the connection in the first place.

(I believe I'm experiencing issues very similar to the ones described here, and I'm in New Jersey so that makes sense: http://ireport.cnn.com/docs/DOC-1164097 also see here.)

Here are traceroute results:

bash-3.2$ traceroute -S -q 5 www.latimes.com
traceroute: Warning: www.latimes.com has multiple addresses; using 63.88.100.192
traceroute to a1574.w3.akamai.net (63.88.100.192), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  0.711 ms  0.436 ms  0.428 ms  0.450 ms  0.402 ms (0% loss)
 2  l100.nwrknj-vfttp-119.verizon-gni.net (72.88.205.1)  6.323 ms  5.500 ms  5.011 ms  9.535 ms  5.408 ms (0% loss)
 3  g0-14-4-7.nwrknj-lcr-22.verizon-gni.net (130.81.59.168)  9.588 ms  9.821 ms  9.469 ms  10.493 ms  8.836 ms (0% loss)
 4  * ae2-0.nwrk-bb-rtr2.verizon-gni.net (130.81.209.170)  91.449 ms *  82.672 ms
    so-6-1-0-0.nwrk-bb-rtr2.verizon-gni.net (130.81.199.16)  9.064 ms (40% loss)
 5  0.xe-3-0-2.xl4.ewr6.alter.net (152.63.5.193)  41.877 ms  25.767 ms
    0.ae2.xl4.ewr6.alter.net (140.222.228.45)  7.568 ms  11.517 ms
    0.xe-3-0-2.xl4.ewr6.alter.net (152.63.5.193)  17.856 ms (0% loss)
 6  tengige0-7-0-0.gw8.ewr6.alter.net (152.63.17.254)  8.606 ms
    tengige0-5-0-3.gw8.ewr6.alter.net (152.63.21.50)  7.215 ms  8.632 ms
    tengige0-7-0-7.gw8.ewr6.alter.net (152.63.25.30)  14.960 ms
    tengige0-7-0-5.gw8.ewr6.alter.net (152.63.25.22)  14.219 ms (0% loss)
 7  63.88.100.192 (63.88.100.192)  8.519 ms  9.845 ms  8.241 ms  9.538 ms  9.036 ms (0% loss)

bash-3.2$ traceroute -S -q 5 bing.com
traceroute to bing.com (204.79.197.200), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  0.582 ms  0.434 ms  0.412 ms  0.406 ms  0.396 ms (0% loss)
 2  l100.nwrknj-vfttp-119.verizon-gni.net (72.88.205.1)  6.715 ms  5.527 ms  5.293 ms  4.987 ms  5.250 ms (0% loss)
 3  g0-9-1-7.nwrknj-lcr-22.verizon-gni.net (100.41.201.78)  9.006 ms  7.372 ms  8.035 ms  8.325 ms  7.870 ms (0% loss)
 4  ae0-0.nwrk-bb-rtr2.verizon-gni.net (130.81.209.162)  10.290 ms *
    ae4-0.nwrk-bb-rtr2.verizon-gni.net (130.81.199.194)  53.058 ms  7.630 ms
    so-6-1-0-0.nwrk-bb-rtr2.verizon-gni.net (130.81.199.16)  7.587 ms (20% loss)
 5  * * 0.ae4.xl4.nyc1.alter.net (140.222.226.37)  9.048 ms * * (80% loss)
 6  0.ae4.xl4.nyc1.alter.net (140.222.226.37)  8.220 ms  7.500 ms  6.423 ms  7.539 ms  6.716 ms (0% loss)
 7  0.xe-9-0-0.gw13.nyc1.alter.net (152.63.19.61)  8.367 ms
    0.xe-11-1-1.gw13.nyc1.alter.net (152.63.19.57)  8.623 ms
    0.xe-9-2-0.gw13.nyc1.alter.net (152.63.4.145)  17.875 ms
    0.xe-11-0-1.gw13.nyc1.alter.net (152.63.20.173)  7.964 ms
    microsoft-gw.customer.alter.net (152.179.29.234)  7.954 ms (0% loss)
 8  microsoft-gw.customer.alter.net (152.179.29.234)  7.382 ms  8.597 ms
    191.234.230.207 (191.234.230.207)  7.836 ms  7.797 ms
    torl.nycr2.msedge.net (131.253.32.127)  6.827 ms (0% loss)
 9  * torl.nycr2.msedge.net (131.253.32.127)  7.942 ms  7.743 ms  8.254 ms  7.446 ms (20% loss)
10  * * * * * (100% loss)

bash-3.2$ traceroute -S -q 5 www.slashdot.org
traceroute to www.slashdot.org (216.34.181.48), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  0.573 ms  0.445 ms  0.398 ms  0.404 ms  0.405 ms (0% loss)
 2  l100.nwrknj-vfttp-119.verizon-gni.net (72.88.205.1)  6.544 ms  5.406 ms  5.037 ms  5.907 ms  9.706 ms (0% loss)
 3  g0-9-3-2.nwrknj-lcr-22.verizon-gni.net (130.81.133.46)  9.577 ms  6.881 ms  7.517 ms  8.104 ms  8.325 ms (0% loss)
 4  ae2-0.nwrk-bb-rtr2.verizon-gni.net (130.81.209.170)  53.079 ms
    ae4-0.nwrk-bb-rtr2.verizon-gni.net (130.81.199.194)  54.713 ms *
    ae2-0.nwrk-bb-rtr2.verizon-gni.net (130.81.209.170)  26.503 ms  7.601 ms (20% loss)
 5  0.ae1.br1.iad8.alter.net (140.222.229.163)  15.713 ms
    xe-15-0-6-0.res-bb-rtr2.verizon-gni.net (130.81.23.161)  17.231 ms  16.871 ms  16.416 ms
    0.ae1.br1.iad8.alter.net (140.222.229.163)  14.735 ms (0% loss)
 6  * * ber1-ge-7-45.virginiaequinix.savvis.net (208.173.52.81)  15.096 ms  14.452 ms * (60% loss)
 7  * * * * * (100% loss)
 8  0.ae2.br1.iad8.alter.net (140.222.229.165)  17.779 ms  18.334 ms  18.508 ms
    206.28.96.218 (206.28.96.218)  37.857 ms
    0.ae2.br1.iad8.alter.net (140.222.229.165)  17.535 ms (0% loss)
 9  ber1-ge-7-45.virginiaequinix.savvis.net (208.173.52.81)  17.107 ms  17.664 ms  17.768 ms  17.026 ms  16.014 ms (0% loss)
10  cr2-tengig0-8-5-0.washington.savvis.net (204.70.206.241)  20.793 ms  19.787 ms
    das6-v3034.ch3.savvis.net (64.37.207.166)  39.962 ms
    cr2-tengig0-8-5-0.washington.savvis.net (204.70.206.241)  19.116 ms
    das6-v3034.ch3.savvis.net (64.37.207.166)  39.895 ms (0% loss)
11  64.27.160.198 (64.27.160.198)  37.568 ms
    206.28.96.218 (206.28.96.218)  39.595 ms
    64.27.160.198 (64.27.160.198)  39.330 ms
    206.28.96.218 (206.28.96.218)  40.668 ms  39.726 ms (0% loss)
12  hr1-te-12-0-1.elkgrovech3.savvis.net (204.70.198.73)  42.953 ms  46.232 ms  44.017 ms  43.542 ms  42.858 ms (0% loss)
13  star.slashdot.org (216.34.181.48)  41.410 ms
    das6-v3034.ch3.savvis.net (64.37.207.166)  41.214 ms  41.491 ms  42.815 ms  42.055 ms (0% loss)

I know traceroutes can be tricky to interpret which is why I'm posting. I believe these show significant packet loss within the Verizon network, and alter.net seems to be a particular problem. Am I interpreting these correctly? I've sent them to Verizon technicians repeatedly and they haven't indicated one way or another what they think of them...

Are there other diagnostics I should try? I registered on hostmycalls.com to get results from a server running a traceroute to my computer (i.e. in the reverse direction). Here's what those show (sorry, can't post images):

www.dropbox.com/s/w8qye03qqi16b1m/isproute.png?dl=0

UPDATE

Here are MTR reports---I would say this is consistent with the explanation or rate limiting, yes?, and the takeaway is that there isn't any indication of an issue. The one I'm curious about is the last one-- 40% Loss to 192.168.1.1 (and 10% loss at the very end) --- how could that be?:

bash-3.2$ sudo ./mtr --report www.google.com
HOST: iMac-2.local                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.1                   0.0%    10    0.5   0.5   0.4   0.5   0.0
  2. l100.nwrknj-vfttp-119.verizo  0.0%    10  147.5  23.6   4.4 147.5  45.1
  3. g0-9-1-1.nwrknj-lcr-22.veriz  0.0%    10    6.9   9.2   6.3  11.5   1.7
  4. ae2-0.nwrk-bb-rtr2.verizon-g  0.0%    10    8.7  19.1   7.2  47.7  13.9
  5. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
  6. 2.ae0.xt2.nyc4.alter.net      0.0%    10    8.2  16.1   7.9  47.9  15.7
  7. tengige0-7-0-8.gw8.nyc4.alte  0.0%    10   20.5  14.4   9.2  20.8   4.8
  8. google-gw.customer.alter.net 10.0%    10    8.3   9.1   8.0  10.7   1.0
  9. 209.85.255.68                 0.0%    10   24.7  13.6   8.0  40.1  10.6
 10. 72.14.239.245                 0.0%    10   10.6  10.6   9.4  15.2   1.7
 11. lga15s46-in-f20.1e100.net     0.0%    10    9.8   9.5   8.5  10.2   0.6

bash-3.2$ sudo ./mtr --report www.yahoo.com
HOST: iMac-2.local                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.1                   0.0%    10    0.4   0.5   0.4   0.5   0.0
  2. l100.nwrknj-vfttp-119.verizo  0.0%    10    4.8  14.6   4.5  77.9  23.2
  3. g0-9-1-1.nwrknj-lcr-22.veriz  0.0%    10    7.1   8.3   7.0  10.0   1.1
  4. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
  5. 0.ae2.br1.nyc1.alter.net      0.0%    10    8.4   9.6   8.3  11.0   0.8
  6. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
  7. ae-1-60.edge4.newyork1.level  0.0%    10    9.8  17.0   9.5  74.2  20.2
  8. ae-1-60.edge4.newyork1.level  0.0%    10   10.1  17.4   8.6  81.9  22.7
  9. yahoo-inc.edge4.newyork1.lev  0.0%    10    8.4  11.7   8.4  33.2   7.6
 10. ae-2.pat1.bfz.yahoo.com       0.0%    10   20.9  31.9  17.6 100.8  26.1
 11. ae-4.msr1.bf1.yahoo.com       0.0%    10   20.1  23.5  18.6  55.9  11.4
 12. unknown-98-139-130-x.yahoo.c  0.0%    10   20.3  20.7  17.8  22.2   1.3
 13. et-17-1.fab3-1-gdc.bf1.yahoo  0.0%    10   22.3  21.0  18.9  22.6   1.3
 14. po-10.bas1-7-prd.bf1.yahoo.c  0.0%    10   22.3  21.5  18.1  23.8   1.8
 15. ir2.fp.vip.bf1.yahoo.com      0.0%    10   23.0  20.9  19.0  23.0   1.3

bash-3.2$ sudo ./mtr --report bbcnews.com
HOST: iMac-2.local                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 192.168.1.1                  40.0%    10    0.4   0.5   0.4   0.7   0.1
  2. l100.nwrknj-vfttp-119.verizo  0.0%    10    7.5  10.1   5.0  45.2  12.4
  3. g0-14-4-7.nwrknj-lcr-21.veri  0.0%    10    6.6   7.7   6.0  10.7   1.5
  4. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
  5. 2.ae1.xt1.nyc4.alter.net      0.0%    10    8.0  12.4   7.6  37.3  10.0
  6. gigabitethernet4-0-0.gw1.nyc  0.0%    10    7.0   7.2   6.3   7.8   0.4
  7. teliasonera-test.customer.al  0.0%    10    6.6   6.7   6.1   7.0   0.3
  8. nyk-bb2-link.telia.net        0.0%    10   32.3  26.2   9.2  78.9  24.4
  9. ldn-bb2-link.telia.net        0.0%    10   95.6  88.2  83.0 115.9  10.4
 10. ldn-b3-link.telia.net         0.0%    10   81.7  85.5  81.7  88.1   1.9
 11. atos-ic-124708-ldn-b2.c.teli  0.0%    10   82.0  78.3  76.7  82.0   1.5
 12. ???                          100.0    10    0.0   0.0   0.0   0.0   0.0
 13. ae1.er02.thdow.bbc.co.uk      0.0%    10   79.8  79.4  78.0  81.6   1.0
 14. ae5-vrf-mitigate.thdow.bbc.c  0.0%    10   80.0  78.5  76.9  80.0   0.8
 15. ae0.er01.thdow.bbc.co.uk      0.0%    10   79.0  79.7  78.0  84.4   1.8
 16. 132.185.255.92                0.0%    10   79.1  78.5  77.4  80.6   0.9
 17. virtual-vip.thdo.bbc.co.uk   10.0%    10   77.8  80.0  76.6  99.5   7.3

Answer 1

To affectively use traceroute you need to have an understanding of what it actually does. Tracert is a path determination tool. It is not a packet loss analysis tool.

Traceroute sends a packet/datagram (actually several) to each router in the path from the source to the destination. Any or all of those routers may be configured to discard or give low priority to those packets/datagrams directed to themselves because a routers job is to forward real traffic, not reply to ping and traceroute. What you're seeing in your traces is those routers more than likely doing just that. It is possible that those routers are dropping real traffic but the subsequent routers don't bear that out because they don't show the same, or greater, packet loss. If there were real packet loss at those routers then you would see some degree of packet loss at all the subsequent routers as well. Otherwise how would it be possible for one router to drop 40% of traffic flowing through it without subsequent routers showing some degree of packet loss as well? It's not.

The only real problem I see in your traceroute is your traceroute to bing.com and that's not an issue with your ISP, it's an issue with whomever manages the msedge.net network (presumably Microsoft). Now it could be that they're simply dropping/blocking all incoming traceroute, ping, etc. traffic to their network but I was able to successfully traceroute to bing.com so I don't think that's the case. At any rate, I don't see anything in any of your traces that points to a problem with your ISP.

To wrap this up: traceroute is a path determination tool, not a packet loss analysis tool. If you want to test the packet loss of the path you need to use a tool that can test that, such as pathping, MTR or iperf. See this note from the MTR web page regarding packet loss:

Answer 2

@joeqwerty is absolutely correct - to expand on parts of his answer though -

From reading the page at ireport.cnn.com/docs/DOC-1164097 it would not appear that this issue is related to what you are seeing at all.

There is nothing obvious that jumps out (bearing in mind I'm on the other side of the world, although I have run a couple of ISP's). I do note the "HostMyCalls" AvgDelay/ms thing is not neccessarily a major concern - your packet loss is negligeable and the latency - although higher then the ideal of < 30ms is not indicative of any issue of significance. In any event this would have nothing to do with a website not loading.

If the website is not loading but you can ping the site (even if you get packet loss), this normally means either there is a firewall problem or a corruption with your cache. First thing I'd do is see if it works with another browser - if so try clearing the cache of your normal browser. A browser cache corruption causing the page to not load is a surprisingly common problem.