1

I made a script to scan a file which contains a portion of ipv4 addresses (about 50 million), it attempts to connect to the website using OpenSSL and extract a small piece of it and writes it into a file. To save some of the details, it uses multiprocessing to run about 150 workers to connect to different sites at the same time, and each connection takes under a second.

My CPU can handle much more workers, so the script is not CPU bounded by any means. The script caps at the 1/5 of my maximum bandwidth at home and not even a 1/30 of the maximum speed at the university labs (I have special permission to run this), even if I bump up the workers to about 300, 400, 500, etc. It is also definitely not I/O bounded, because it keeps it in a list in memory and writes it out at the end. I believe it is due to the router not being handle so many requests at the same time

  1. Can the router be overloaded with request, even if bandwidth is not an issue. I'm more surprised that the university network cannot handle these requests.

  2. What can I do to solve it? Or any other insights?

user153882
  • 113
  • 5
  • You're not the only user of the university router, you're sharing it with thousands of other users. They might have traffic shaping implemented, to prevent one user from monopolizing the connection. – Barmar Apr 04 '17 at 21:04
  • One thing that will limit the number of requests is the maximum number of available ports in the NAT device for the outbound TCP connections. – Tero Kilkanen Apr 05 '17 at 20:02

1 Answers1

3

The answer to your question is that it depends heavily on the router. Business class routers are typically rated not only by bandwidth, but but PPS (Packets per second) this is usually in the seven digit range.

As an example, a Ubiquiti Security Gateway Pro 4 is rated at 2.4 million pps.

https://dl.ubnt.com/datasheets/unifi/UniFi_Security_Gateway_DS.pdf on page 4

Cory Knutson
  • 1,866
  • 12
  • 20