Router restarts after big git push or big file upload

20

4

My problem is that modem (Modem / router #1 presented below) restarts itself, when I make git push with a lot of files (I'm not sure how big the push must be to break the connection, but smaller pushes with only a couple of files are working correctly). The same thing happens, when I'm connected with server via OpenVPN and I'm trying to upload a file through Samba.

Here's my home network setup (I have a home server exposed through ddns.net website):

Network setup

  1. PC is my home computer from which I'm sending requests. Its IP is random.
  2. PC is connected to Modem / router #1 via WiFi or LAN.
  3. Modem / router #1 also handles incoming requests - its public IP is used by ddns.net website. Its local IP is 192.168.0.1.
  4. Router #2 is connected to Modem / router #1. Its IP on Modem / router #1 is 192.168.0.103.
  5. Router #2 has its own local network, in which its IP is 192.168.1.1.
  6. Server is connected to Router #2 via LAN. Its IP is 192.168.1.100.
  7. ddns.net website requests go to Modem / router #1 and specific ports are forwarded to Router #2, which further forwards the request to Server, which finally handles the requests and sends a response the same way.
  8. Modem / router #1 model isn't easy to find, as it's just called "UPC Connect Box" (I'm from Poland and UPC is one of our ISPs), but I was able to find that it's software version is CH7465LG-NCIP-6.12.18.25-2p4-NOSH, which is used by Compal Broadband Networks CH7465LG-LC.
  9. Router #2 is TP-Link TL-WR841N.
  10. Server uses Lubuntu. lsb_release -a command returns:

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 18.04.2 LTS

Release: 18.04

Codename: bionic

  1. It's worth noting that such problems didn't happen in the past. I was able to upload files of any size on the exact same setup. I have no idea what could've changed.

Some things I've already tried / checked:

  1. PC connected to a mobile hotspot, instead of Modem / router #1 - in this case, everything works fine.
  2. PC connected to a Modem / router #1 via LAN instead of WiFi - the problem occurs in both cases.
  3. This solution from SO - doesn't change anything.
  4. I am able to see a log of Modem / router #1, which is unfortunately somewhat vague. When the restart happens, it simply states that "Cable Modem Reboot - due to power reset".
  5. Nothing wrong was found in server logs (at least in syslog and Samba logs).

I'm not expecting a ready solution for this problem, but I would really appreciate any advices on where can I start additional troubleshooting? What can I check to find out the cause of this behaviour?

-- Edit #1 --

I checked another possibility. When I'm connected directly to Router #2 via WiFi, it doesn't work as well. I must be completely outside my home network to make an upload. When I'm connected to Router #2, then it's still Modem / router #1 that resets itself.

-- Edit #2 --

Actually, after further analysis, check from Edit #1 doesn't really make sense, because it only adds another "node" to the request journey. I prepared a diagram of all three cases (PC connected to Modem / router #1 (UPC modem), Router #2 (TP-Link) and mobile hotspot). It may not be completely correct, it's really simplified, just to see how much work does the modem have.

Request diagram

-- Edit #3 --

Thanks to the diagram from Edit #2 I came to a conclusion, that I can try to reduce the work required by modem when I'm in the same network simply by accessing server by its local IP, instead of going through ddns domain. So when I pushed files to 192.168.0.103 (which is an IP of TP-Link router on UPC - see the first diagram), it was forwarded directly to the server, thus decreasing a load on UPC modem and everything worked flawlessly. Why more operations on the modem are causing it to reboot is still a mistery to me, especially knowing that it worked a couple of months ago, but at least I have a workaround for now. The main problem still remains though and I'm afraid it may actually be a firmware issue. Any new ideas are still welcome.

-- Edit #4 --

After switching a power adapter for a completely new one, router still resets when doing a big git push. This happens with and without a fan cooling it. I'll contact ISP.

Bartłomiej Zieliński

Posted 2019-07-21T19:17:25.783

Reputation: 303

2Why in all that is holy are you using Router#2 as a router, if you aren't using professional gear? Just disable the second router's DHCP and plug the primary router into the switch side, PLEASE. Put all your machines on the same segment. – Aron – 2019-07-22T03:44:06.900

@Aron First router has to be in a specific room, in which I don't want to have the server. I have a long LAN cable moved from first room to the second room. In the second room, server is not the only machine which requires a cable connection. I've already had an unused TP-Link router, so I've decided to use it as a switch. I'm not entirely sure, what do you mean by: "plug the primary router into the switch side"? – Bartłomiej Zieliński – 2019-07-22T09:04:25.643

2You said so yourself, you want to use it as a switch. A modern cheap consumer router is actually 4 devices, NATing router, DHCP server, WiFi AP and Ethernet Switch. The NATing router connects the WAN network to the LAN network-switch via a Routing Table & NAT. The DHCP server assigns IP addresses. Chances are you only need the WiFi AP and Ethernet Switch. Connect the Router #1 to Router #2 via the LAN ports. – Aron – 2019-07-22T09:36:44.093

@Aron I think I get it now, it sounds great! I didn't know it was possible to use router exactly like a switch. I learned everything by myself, so it's possible I did some basic mistakes in the configuration. It would be awesome if I was able to keep everything in a single IP range - I'll try to do this when I find a moment. Thank you for the suggestion. – Bartłomiej Zieliński – 2019-07-22T13:18:27.137

1Remember to turn off the DHCP before you do this, some router even have a bridge mode – Aron – 2019-07-22T13:21:31.317

3Re Why more operations on the modem are causing it to reboot - most likely router power adapter is slowly failing (caps going bad or smtg similar). If possible, then replace it (same voltage and same connector required, bigger current output is recommended). – Arvo – 2019-07-22T13:28:33.813

@Aron I was able to make router #2 work as a switch. Now everything is in the same IP range. Thanks again. – Bartłomiej Zieliński – 2019-07-22T17:40:06.473

@Arvo Ok, it seems worth a try. I'll get a new power supply and test it again. – Bartłomiej Zieliński – 2019-07-22T17:40:10.130

A similar issue occurred on WRT AC Series routers running OpenWrt, and the issue there was RAM related. A large local file transfer would result with available RAM reaching near 0%, resulting with an immediate reboot of the router. If you have a way to monitor the RAM of Router #1 (I'd also monitor the CPU temp), initiate the git push once monitoring begins and see if the RAM is being exhausted or if the CPU is overheating, prompting the immediate reboot. If that is not the issue, there are only two possible causes, either it's some other hardware issue or a bug in the firmware. – JW0914 – 2019-08-02T12:31:59.473

Answers

2

It's faulty firmware, this did not happen with CH7465LG-NCIP-6.12.18.24-5p4-NOSH. This issue is causing router restart when hairpin or loopback is being performed. With this error, you cannot use public IP or domain name inside your local network, because that is what causes reboot. It is couple of month now and still not fixed. You can ask your ISP to remotely downgrade to CH7465LG-NCIP-6.12.18.24-5p4-NOSH but the router updates itself anyway and ISP cannot stop it from doing so, so we are all screwed.

dwbmb

Posted 2019-07-21T19:17:25.783

Reputation: 36

Thanks for the info! I'll definitely check it with ISP. – Bartłomiej Zieliński – 2019-08-21T08:31:47.210

1I also have UPC and they have already addressed this issue and are working on a fix. There is probably nothing more you can do right now. As I sad before, downgrading FW is temporary workaround, as router auto updates itself again after a day or two. – dwbmb – 2019-08-23T06:57:31.620

Well, we can only hope that when they find an issue in one country, they fix it everywhere :) I've asked our Polish service about this problem, now I'm waiting for a response. – Bartłomiej Zieliński – 2019-08-23T13:59:38.287

I've just got my modem downgraded to the previous firmware version by ISP - it solved the problem completely. Thank you very much for your suggestions. Now we can only hope they'll fix it soon. I'm choosing this answer to be accepted, because it described an exact cause of my problem, but https://superuser.com/a/1462431/424167 is worth checking as well. By the way, I've got another info from UPC - modem should not be automatically updated after a downgrade. We'll see if it's true.

– Bartłomiej Zieliński – 2019-08-26T13:25:30.807

Minor update CH7465LG-NCIP-6.12.18.25-2p6-NOSH released, issue still not fixed. – dwbmb – 2019-09-24T17:05:38.007

Thanks for confirmation that I wasn't going crazy. I have the exact same issue with UPC. The router is performing extremely poorly when I access any services running on my local server using a public domain/IP. When I've used mobile network to connect the performance was as expected. If I keep using public domain it suddenly restarts. – Sylwester Kardziejonek – 2019-11-30T10:56:43.323

The only crazy thing about this issue is how long it takes for UPC to fix this bug. It is not about crying here and hoping, I am reaching UPC very often about this all I am getting back is "we know about it and we are working on it" for like a year. – dwbmb – 2019-12-01T13:47:16.220

14

You’ve made a well written, descriptive post. But, it’s written from the perspective that there must be something external to the modem causing the problem.

You wrote

I am able to see a log of Modem / router #1, which is unfortunately somewhat vague. When the restart happens, it simply states that "Cable Modem Reboot - due to power reset".

Nothing should ever cause a device to physically reset.

There are only three possible problems here:

  1. It’s hot and your modem is overheating. Increase air flow around the unit and test again. Put a fan directly on the unit.
  2. The modem or power adapter is bad. Get it replaced. It could be a firmware bug and you could ask the ISP to confirm that it is running the latest firmware, but I feel this is unlikely.
  3. Something is interrupting the power to the unit. Try a completely different outlet.

In my opinion, #1 is the most likely culprit and it may have already done irreversible damage requiring the modem to be replaced.

This is ISP equipment. You should be talking to them about why the modem is rebooting and then get it replaced if necessary.

Appleoddity

Posted 2019-07-21T19:17:25.783

Reputation: 9 360

1But wouldn't it restart more often if overheating was the cause? It restarts itself only when I'm doing a very specific operation, which is uploading something to the server and the file has to be big enough to cause any problems. Opening websites hosted on the server works ok, as well as downloading huge files on full speed from other hosts.

I'll try to get some info from ISP only when nothing else helps, I don't expect much help from them. Also, I'd say that bug in the firmware seems more likely - I didn't make any updates, but I can't be sure that modem doesn't do it in the background. – Bartłomiej Zieliński – 2019-07-21T20:35:02.773

1@BartłomiejZieliński I think it's the router too, if it's not heating, it can be DPI, if the %CPU of the router choke while dooing the DPI inspection, or bad memory inside it. – yagmoth555 – 2019-07-22T00:09:19.767

6There is another possibility. All of the scenarios the OP posts about are large long lived TCP connections. There could be an overflow occurring in the router's firmware. I would be interesting to see if something similar happens with a) UDP connections and b) large numbers of small TCP connections. You load test both these scenarios with qBittorrent for instance. – Aron – 2019-07-22T03:38:36.293

2@BartłomiejZieliński It's possible that it gets hotter when it's doing more work because more data is going through it. – user253751 – 2019-07-22T03:58:39.463

@Aron That is possible, but I wouldn't expect it to say "due to power reset" then. – user253751 – 2019-07-22T03:59:11.177

5@user20574 Why not? If the kernel dies due to an unrecoverable fault, which it was unable to log (because routers have lots of reliable persistent storage), AND some kind of firmware/watchdog power cycles the main SOC to recover, from the perspective of the micro distro, it did get "reset due to power reset". – Aron – 2019-07-22T06:39:24.040

"Nothing should ever cause a device to physically reset." Embedded devices have watchdog timers for the express purpose of triggering a physical reset on some conditions. – pmf – 2019-07-22T14:22:19.523

1I'd like to add that it also might be a old, now faulty capacitance in the power supply of the router, leading to undervoltage on the router on load, which might trigger the reset. (Had that with a Raspberry Pi after it's been running on the same power supply for two years. Swapping the power supply fixed it.) – orithena – 2019-07-22T15:44:08.217

2If it's a firmware bug, the ISP will probably replace the modem with another one that has the same problem. – Nonny Moose – 2019-07-22T19:55:12.217

@Aron These devices have a hardware register that records the reason for reboot. The SOC has a built-in watchdog which sets the reason to "watchdog timeout" and not "power failure". – user253751 – 2019-07-22T22:40:37.247

@BartłomiejZieliński You’re drawing more power and generating more heat during certain activities. After all of this, have you taken any of the suggestions in this answer in to consideration to further troubleshoot? Put a fan on the unit? Tried a different power adapter? Asked to have the unit replaced, preferably with a different model to eliminate hardware and firmware issues? The answer is succinct and accurate, just wondering what else you’re looking for. We can’t fix hardware and firmware issues in your ISP’s equipment, but it most assuredly either hardware or firmware. Probably hardware. – Appleoddity – 2019-07-23T12:00:17.450

@Appleoddity I'm waiting for a delivery of the new power adapter, didn't have one at hand. I'll try with the fan today or tomorrow, I don't really think it'll help though, but I'll check it anyway. If both suggestions fail, then I'll get to ISP and ask for a replacement. I'll edit my post after each trial - unfortunately, I can't test everything at once, gotta be patient :) – Bartłomiej Zieliński – 2019-07-23T12:09:54.103

@Appleoddity I've added a new edit to the question - switching a power adapter and using a cooling fan didn't help. I'll give another update after I get a response from ISP. – Bartłomiej Zieliński – 2019-08-02T12:06:47.800

1There are also two other potential causes, available RAM shrinking to near 0% (due to a bug or misconfiguration of hardware buffers) resulting in a kernel panic, or a bug in the firmware only presenting itself during large file transfers. – JW0914 – 2019-08-02T12:38:34.853

@JW0914 According to answer from dwbmb, it may be caused by a faulty firmware. I'll ask ISP for a downgrade to see if it helps. – Bartłomiej Zieliński – 2019-08-21T08:51:27.430

Asked: 2019-07-21T19:17:25.783

Viewed: 2 854 times

Active: 2019-08-16T21:30:36.900