Periods of high packet loss, two ethernet cables plugged into TWC modem

Question

A small business for which I am the system administrator has been experiencing intermittent internet dropouts. There seems to be no pattern to the dropouts — sometimes there is nothing for a week, and other times it will drop out dozens of times in a single day.

I have been running mtr (my traceroute) to try to diagnose the issue. I am including the mtr results below, pinging Google's Public DNS. I have also been running a test within the LAN, which shows exactly zero packets dropped, even over the course of a week or more; this leads me to believe the problem lies beyond our router. Everything is connected by gigabit ethernet. 10.0.0.1 is the LAN address our router, and the next hop (Time Warner's server) is where the packet loss begins.

                        Packets               Pings
 Host                 Loss% Drop   Snt   Last   Avg  Best  Wrst StDev
 1. 10.0.0.1           0.0%    0 10722    0.4   0.3   0.2  10.6   0.8
 2. ##.###.##.##       7.9%  845 10722   22.1  21.7   7.7 292.0  14.2
 3. ##.###.##.##       7.9%  845 10722   13.8  20.2   8.4 459.2  13.2
 4. ##.###.##.##       7.9%  847 10722   22.3  22.0   8.9 374.5  13.2
 5. ##.###.##.##       7.9%  852 10722   29.1  24.7  10.4 290.1  13.4
 6. ##.###.##.##       7.9%  848 10722   23.3  25.2   8.3 643.9  15.3
 7. 66.109.6.163       7.9%  849 10722   15.1  23.3   9.0 554.9  17.2
 8. 66.110.96.53       7.9%  848 10722   26.3  21.3   7.9 467.2  14.4
 9. 72.14.195.232      7.9%  846 10721   28.0  21.9   8.9 402.2  14.7
10. ???
11. 108.170.238.201    7.9%  842 10721   19.1  21.8   8.8 498.9  14.5
12. 8.8.8.8            7.9%  844 10721   24.0  21.4   8.7 414.5  13.5

It very recently came to my attention that there is a second device connected to the modem via ethernet. I want to note that the modem is actually a wireless gateway, but it was put into bridge mode and we have confirmed that our router (an Apple AirPort Extreme AC) gets a proper WAN address and DNS servers. So, the gateway is effectively just a modem.

Here is our network topography:

Time Warner Business Modem
├── Mystery Device
└── Apple AirPort Extreme AC
    ├── Netgear Unmanaged Gigabit Switch 1
    │   ├── Apple iMac 1
    │   │   └── Canon Printer
    │   ├── Apple iMac 2
    │   ├── Apple iMac 3
    │   └── HP Ubuntu Laptop
    ├── Netgear Unmanaged Gigabit Switch 2
    │   ├── HP Ubuntu Desktop
    │   ├── Brother Printer 1
    │   └── Brother Printer 2
    └── Apple iMac 4

I work remotely so have not been able to visit the office to inspect the situation myself. We have been in contact with Time Warner and they assert that their modem has had no signal issues for the past few months. Our issues started a few weeks ago, after several years of consistent connection.

The cable between the modem and router was just replaced to rule out a faulty connection between the two, but the issue persists.

The core of my question is this — if this is a typical TW Business Class modem, does it only have one IP address to assign? I don't believe we are paying for additional IPs and I assume we only get one by default. If so, are the two ethernet devices connected (AirPort and mystery device) in competition for that IP address? Or has it been assigned to the router permanently by its MAC address? Would such a competition yield the observed mtr results above, perhaps when that mystery device attempts to connect to the internet?

Any advice would be much appreciated. If you have any idea of how I can further isolate the variables, I am comfortable with UNIX tools. In the meantime I have instructed someone in the office to disconnect the mystery device, but it is difficult to determine whether or not the issue will be fixed by that alone because it is not always observable, sometimes for days at a time.

Update: The second ethernet cable coming out of the modem was not connected to anything, and has been unplugged. I assume that rules out a conflict in IP address assignment. Any other diagnostic ideas would be very much welcomed.

Answer 1

modem has had no signal issues

This doesn't mean that it wasn't dropping packets. The signal is just how strong the connection is on the wire between the modem and the nearest TWC hub, but if there is packet loss, this wouldn't necessarily correlate to a signal drop.

does it only have one IP address to assign? I don't believe we are paying for additional IPs and I assume we only get one by default.

Yes. You only get one IP unless you pay for more. At that point TWC will convert the modem to bridge mode and allow you to configure your router(s) with the appropriate static IP information.

If so, are the two ethernet devices connected (AirPort and mystery device) in competition for that IP address?

Yes.

Or has it been assigned to the router permanently by its MAC address?

No. Only the modem is typically MAC address locked to the ISP network.

Would such a competition yield the observed mtr results above, perhaps when that mystery device attempts to connect to the internet?

Very much so.

The second ethernet cable coming out of the modem was not connected to anything, and has been unplugged

How long ago was this? Have you noticed the issues after this was unplugged?

Additionally, ping is a protocol that's very common to block or QOS since it's not a core technology and meant for diagnostic usage. This is true for most ICMP traffic. Having drops for ICMP traffic is not something I would jump on to claim dropped traffic. Also, Google is not a good permanent health check with pings; I'd suggest using a VPS or a dedicated ping service for that.

As far as how to go about diagnosing this? I would start with replacing the Airport Extreme, even if just temporarily. You can find a better router and still use the Extreme for it's WiFi broadcast if you need to, but this is probably step 1.

I would also encourage you to monitor for bandwidth usage. If something on the network is pulling all of your available bandwidth, this could cause symptoms that seems very much like a drop in network performance, when in fact it's just 1 device using it all. This would also account for the intermittency of the issue.

If you continue having issues, you would probably have to work more closely with TWC to have them monitor the local upstream switch and see if they can find any drops. With a better router installed, you could also start capturing pcaps to find if there are drops, or even monitor the interface itself to see what it's reporting for activity.