Sporadic high latency on my home network

Question

tl;dr My home network recently has been experiencing jumps from 27ms latency to 600ms. It doesn't happen always, and seems to occur frequently at night. What equipment should I buy and tests should I run to deduce the cause?

Setup

My home has 12Mb/800kb DSL. I live in the mountains, far away from other Wi-Fi sources. Historically (for years) I could ping google.com and get ~27ms times. If something was flooding the network or connection (an iPhone syncing all photos with iCloud) pings would jump into the 2000-6000ms range. But normally everything was good.

Recently, however, the network stays pegged around 600ms for tens of minutes at a time. I cannot find any device that is flooding the network. (It may exist, but I haven't found it.) The connection is generally completely fine in the morning, and generally persistently bad at night (just when we want to stream shows in bed!)

During high latency times pings to other devices on the network (some that I've tried) are unchanged (always <2ms).

Failed and Confusing Troubleshooting

I have purchased all new hardware (DSL modem, Wi-Fi routers, network switches) to rule that out. The problem persists. Here is the setup:

I have tried using the DSL Modem as the router (PPPoE + DHCP + NAT) with the Wi-Fi base stations in bridge mode. I have tried putting the DSL Modem in transparent Bridging mode and having the first Airport Extreme handle PPPoE, DHCP, and NAT. The problem persists.

I have disconnected all wired connections (leaving only the DSL modem and the Wi-Fi base station). The problem persists.

I have used only the DSL Modem (with PPPoE) and used its own Wi-Fi. The problem persists. I have attempted to hunt down every old tablet, phone, laptop on the Wi-Fi and turn them off. The problem persists. I have renamed the Wi-Fi SSID and put a password on it, connecting a single MacBook Pro laptop over Wi-Fi. The problem persists. I have used a different laptop over Wi-Fi. The problem persists.

I have connected a laptop directly to the modem over Ethernet, with Wi-Fi disabled on the modem and nothing else connected. The problem goes away! (I think...it *could* be that the problem just was not exhibiting itself on the three occasions that I tested this.)

At one point, with just a laptop connected over Ethernet, I turned Wi-Fi on for the modem and the problem exhibited itself. Ping latency immediately jumped as soon as I turned on Wi-Fi, though I do not believe that any devices were connected over Wi-Fi.

I have used iStumbler and there does not appear to be any correlation between the bad latency and increases in noise. Indeed, the SNR looks good consistently over Wi-Fi.

Remember that when things are bad they are not ALWAYS bad. Even with every device in the house turned on and connected, there are times when the latency will drop to 30ms or so for a few seconds (or minutes, or hours) before getting bad again.

Next Steps?

I think that iStumbler has shown me that the problem is not related to RF problems. (Maybe I'm wrong?) So I'm thinking it must be real traffic on the network.

The Airport Extreme base station does not support any sort of SNMP logging. Neither does the Actiontec C1000A. I don't have a switch with a monitor port, or a hub. I've never used Wireshark before.

BUT I AM WILLING TO THROW MONEY AND TIME AT THIS PROBLEM TO SOLVE IT

What should I buy? Where should I inject it into my network? What should I look for? How can I watch every packet on the network and build histograms and graphs to determine if one bad device is ruining the situation for everyone?

---

Edit 1: DSL Statistics when everything is fine

+-----------------+-------------+
|   Connection    |   Status    |
+-----------------+-------------+
| DSL Downstream: | 15.869 Mbps |
| DSL Upstream:   | 0.896 Mbps  |
+-----------------+-------------+

DSL Link Statistics

+------------------------------+---------------------+
|        Link Statistic        |       Status        |
+------------------------------+---------------------+
| Broadband Mode Setting:      | Auto Select         |
| Broadband Mode Detected:     | VDSL2 - 8A          |
| DSL Link Uptime:             | 0 Days, 10H:39M:57S |
| Retrains:                    | 1                   |
| Retrains in Last 24 Hours:   | 1                   |
| Loss of Power Link Failures: | 0                   |
| Loss of Signal Link Failure: | 0                   |
| Loss of Margin Link Failure: | 0                   |
| Link Train Errors:           | 0                   |
| Unavailable Seconds:         | 23                  |
| Estimated Loop Length:       | 2250                |
| Uncanceled Echo:             | N/A                 |
| Transport Mode:              | PTM                 |
| Path Parameter:              | 201                 |
| Priority:                    | 0                   |
| Service Type:                | PTM-Tagged          |
+------------------------------+---------------------+

DSL Power

+--------------+-------------------------+------------------------+
|    Levels    |       Downstream        |        Upstream        |
+--------------+-------------------------+------------------------+
| SNR:         | 16 dB                   | 10 dB                  |
| Attenuation: | (DS1)21.7, (DS2)58.8 dB | (US1)4.3, (US2)47.8 dB |
| Power:       | 16.4 dBm                | 7.8 dBm                |
+--------------+-------------------------+------------------------+

DSL Transport

+----------------------+------------------+---------------+
|      Transport       |    Downstream    |   Upstream    |
+----------------------+------------------+---------------+
| Packets:             | 1482864          | 1088249       |
| Error Packets:       | 0                | 0             |
| 24 Hour Usage:       | 1225940.68 Mbits | 2420.93 Mbits |
| Total Usage:         | 1225940.68 Mbits | 2420.93 Mbits |
| 30 Minute Discarded: | 0                | 3930          |
+----------------------+------------------+---------------+

DSL Channel

+----------------+-------------+-------------+
|    Channel     |  Near End   |   Far End   |
+----------------+-------------+-------------+
| Channel Type:  | Interleaved | Interleaved |
| CRC Errors:    | 0           | 0           |
| 30 Minute CRC: | 0           | 0           |
| RS FEC:        | 5873        | 29          |
| 30 Minute FEC: | 372         | 0           |
+----------------+-------------+-------------+

---

Edit 2: DSLReports Bufferbloat report

Running the speedtest during otherwise-normal latency indicates that the problem occurs during uploading

---

Ping times at night and overnight

The spike around 10:35pm was one computer starting to upload to Dropbox.

---

Edit 3: ISP tech support said:

Modem is getting more signals that it is suppose to. If the cables are not enough to carry the load we are sending we can lower it down to 100%. To test this is for me to lower down the signal for 7 days and you can observe if the browsing \ internet is better. After 7 days our server would run test and would boost your signals up again. And by that time we would have enough figures what to do next.

Our server is provisioning you more than your purchase. Technically this should make the internet faster but if pings and delay that are caused by traffic are observed by the customer. We can bring it to the purchased speed\ signal and observe if the DSL line on the customers premise are cable to carry the load.

Actual/Provisioned/Purchased speeds
Down: 15868/15872/12128Mbps
Up: 896/896/896kbps

Answer 1

The symptoms you've reported sound like a bufferbloat problem, where your router, DSL modem, or your ISP's DSLAM buffers too many packets when the link is congested, resulting in high latency. Typically, TCP looks for dropped frames as evidence of congestion, and backs off. But if your router or modem or DSLAM buffers forever and never lets a frame drop, you end up with huge latency increases without TCP getting a chance to back off to relieve the congestion. You should never have a huge latency increase just because your upstream or downstream bandwidth is saturated. If you do, you almost certainly have bufferbloat.

Run the dslreports.com speed test tool. Unlike other speed test tools, this tool also measures and reports bufferbloat problems, which can cause high latency whenever something is using all of your downstream or upstream bandwidth (like when you decide to stream video at night).

The fact that you've already proven that your latency jumps when something is using all of your upload bandwidth (your iCloud Photo sync example) is a good indication that you're suffering from bufferbloat problems.

Your DSL modem is probably the source of any upstream bufferbloat problems. One solution might be to buy a DSL modem known to not have bufferbloat problems. I haven't researched this market though, so I can't help you with any suggestions. Your Google-fu is probably as good as mine.

Alternatively, consider buying a home gateway that can run CeroWrt, OpenWrt, or DD-WRT, all of which now have the anti-bufferbloat technologies such as FQ_CoDel that were first pioneered/developed in CeroWrt. By using a box like that to artificially limit your upstream and downstream bandwidth to something slightly slower than what your DSL link is actually capable of, and having that box actually drop frames and send Explict Congestion Notifications (ECN) when that limit is hit, instead of buffering forever, it allows TCP to detect the congestion and back off like TCP is supposed to do.

You don't necessarily have to ditch your DSL modem or your AirPort Extreme to install this *Wrt box; you can install it as a wired box between your DSL modem and your first AirPort Extreme. Just make sure that all the traffic to/from your home network goes through this box. That is, make sure you don't have any devices directly attached to the DSL modem other than this *Wrt box.

If you know you have bufferbloat, you should probably eliminate it before looking for other potential sources of latency spikes, otherwise it will hinder your attempts to find other sources of latency.

Answer 2

Somethings wrong. 24hr stats say:

312,600 MBytes Down 247,500 Mbytes Up

You didnt include link rates but 8A at 2KM gives you maybe a 15/5 link. At 5Mb US you could only upload around 55GB/24hrs. Even at 10Mb you wouldnt reach 250GB, so dont trust those stats.

Still, this sounds really like peer to peer/sync/malware on your network is self DOSing.

UPDATE:

You're connection is balanced like an older style ADSL connection(8D 0.5U, 12D 0.7U, 15D 1U) vs what you'd normally do with VDSL(2) (15D, 3U). This leaves you in a situation where its very easy to congest your own link.

Anything running on your network can cause an upstream queue where the modem holds a series of frames that are trying to send but are coming faster than it can forward them. So for example instead of 1ms from your laptop to modem, 20ms from modem to exchange, 5 ms from exchange to website you have: 1ms from you to modem, 100ms waiting in the frame buffer, 20ms to exchange and 5ms to site. The more thats sent, the bigger that wait time.

Things to look for: Peer to Peer (bit torrent, game launchers) Syncing apps: Windows 7/8/10 One Drive, Dropbox(esp Camera Sync), iCloud Offsite backup like Crashplan/Backblaze etc VOIP/Video call apps: Skype, TS/Mumble

Anything that sends data out to the web.