0

I work for SD-WAN company providing an all in one solution edge router(Like FORTINET but not FORTINET).

We have one customer in particular that has such a hard time with our services i.e. gift card servers unreachable, credit card processor unreachable and sometimes the security cameras don't send data back to the DC servers... Also the average ping to 4.2.2.1 is about 36ms but the avg ping to GC or CC processors is usually greater than 100ms.

We use Linux based platform with alot of utilities based on Open standards, e.g OpenVPN.

WAN circuits are my area of troubleshooting. I run pings and traceroutes and sometimes I see some crazy things. I have some traceroutes below and would like to see if the collective experience and brain power of this community can offer some possibilities.

I've taken the GW IP out, the rest I don't mind including.

Heres a good example of a clean traceroute to 4.2.2.1

  • Trace In Progress

traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets

1 x.x.x.x (x.x.x.x) 0.579 ms 0.679 ms 0.798 ms

2 76-202-64-1.lightspeed.brbnca.sbcglobal.net (76.202.64.1) 24.986 ms 25.546 ms 25.666 ms

3 71.147.176.37 (71.147.176.37) 20.833 ms 21.911 ms 22.035 ms

4 75.20.0.202 (75.20.0.202) 24.520 ms 24.838 ms 26.063 ms

5 12.83.38.157 (12.83.38.157) 30.335 ms 12.83.38.153 (12.83.38.153) 31.029 ms 12.83.38.157 (12.83.38.157) 30.416 ms

6 ggr2.la2ca.ip.att.net (12.122.128.101) 31.437 ms 24.566 ms 21.605 ms

7 192.205.37.146 (192.205.37.146) 23.534 ms 24.605 ms 25.294 ms

8 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 27.329 ms ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 28.090 ms ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 28.782 ms

9 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 28.938 ms ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 29.683 ms 30.287 ms

10 a.resolvers.level3.net (4.2.2.1) 29.979 ms 30.654 ms 31.089 ms

Pretty good no problems. Lets look at the next example

This time an AT&T router's latency jumps WAY UP

  • Trace In Progress

    traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets

1 x.x.x.x (x.x.x.x) 0.523 ms 0.637 ms 0.771 ms

2 76-202-64-1.lightspeed.brbnca.sbcglobal.net (76.202.64.1) 19.439 ms 29.179 ms 38.125 ms

3 71.147.176.37 (71.147.176.37) 20.750 ms 21.705 ms 22.158 ms

4 75.20.0.202 (75.20.0.202) 24.994 ms 25.144 ms 27.133 ms

5 12.83.38.153 (12.83.38.153) 29.790 ms 29.935 ms 12.83.38.157 (12.83.38.157) 31.198 ms

6 ggr2.la2ca.ip.att.net (12.122.128.101) 32.876 ms 25.446 ms 280.808 ms

7 192.205.37.146 (192.205.37.146) 279.921 ms 280.059 ms 281.683 ms

8 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 282.984 ms ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 283.906 ms ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 284.473 ms

9 ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 285.152 ms ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 285.917 ms 286.054 ms

10 a.resolvers.level3.net (4.2.2.1) 286.187 ms 286.935 ms 287.514 ms

IP Address: 192.205.37.146 [ IP Lookup ] Hostname: 192.205.37.146 IP Location: - United States (US) ISP: AT&T Services Organization: AT&T Internet Services

Well, this shows all sorts of latency in the First , Middle and Last mile.

  • Trace In Progress

    traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets

1 x.x.x.x (x.x.x.x) 0.539 ms 0.696 ms 0.837 ms

2 76-202-64-1.lightspeed.brbnca.sbcglobal.net (76.202.64.1) 201.040 ms 201.634 ms 202.037 ms

3 71.147.176.37 (71.147.176.37) 191.586 ms 192.707 ms 192.847 ms

4 75.20.0.202 (75.20.0.202) 195.827

ms 196.305 ms 196.800 ms

5 12.83.38.157 (12.83.38.157) 201.911 ms 12.83.38.153 (12.83.38.153) 201.063 ms 12.83.38.157 (12.83.38.157) 202.037 ms

6 ggr2.la2ca.ip.att.net (12.122.128.101) 201.354 ms 222.888 ms 226.604 ms

7 192.205.37.146 (192.205.37.146) 224.932 ms 222.087 ms 221.104 ms

8 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 223.069 ms 220.636 ms ae-3-80.edge3.LosAngeles1.Level3.net (4.69.144.137) 221.221 ms

9 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 221.632 ms 233.260 ms 144.725 ms

10 a.resolvers.level3.net (4.2.2.1) 143.808 ms 144.389 ms 144.739 ms

Finally this last one shows a First and Middle but not a Last mile issue:

  • Trace In Progress

    traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets

1 x.x.x.x (x.x.x.x) 1.184 ms 1.269 ms 1.408 ms

2 76-202-64-1.lightspeed.brbnca.sbcglobal.net (76.202.64.1) 288.124 ms 288.613 ms 290.000 ms

3 71.147.176.37 (71.147.176.37) 289.777 ms 302.201 ms 302.334 ms

4 75.20.0.202 (75.20.0.202) 305.836 ms 309.880 ms 310.032 ms

5 12.83.38.157 (12.83.38.157) 313.163 ms 312.685 ms 316.640 ms

6 ggr2.la2ca.ip.att.net (12.122.128.101) 328.398 ms 421.902 ms 420.155 ms

7 192.205.37.146 (192.205.37.146) 421.869 ms 21.274 ms 21.278 ms

8 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 23.584 ms 25.078 ms ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 25.425 ms

9 ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 26.380 ms 27.567 ms 28.278 ms

10 a.resolvers.level3.net (4.2.2.1) 27.654 ms 28.365 ms 28.907 ms

If anyone can offer any insight on this circuit or path issue please give me some ideas... Thank you

Ixanthylus
  • 9
  • 1
  • 3
  • You should collect metrics to get a better idea of when this is happening and well as related stats such as packet loss and route instability. 200ms latency is not unreachability. Your edge router solution should be monitoring all these things already. – Mark Wagner Apr 01 '19 at 18:27
  • Right Mark, thanks for responding. What I'm trying to show here is the amount of variability in the circuit. These trace routes were done while the business was closed as well. – Ixanthylus Apr 01 '19 at 18:51

0 Answers0