Problem: All other devices on the network experience serious lag and timed out web pages, email, chat dropping, etc while this is running.
This is a cross post from stackoverflow, I may have used the wrong site!
Details:
Using libcurl for this
Debian 7 - Intel i5 - 8 GB of ram
Comcast Business 75/15 connection (Cisco DPC3939B)
Running 75 (tested up to 200) threads with Parallel ForkManager
20 Threads is almost acceptable - but too slow for my needs
Timeout: CURLOPT_CONNECTTIMEOUT_MS,5000 + CURLOPT_TIMEOUT_MS,10000
Round robin dns using options rotate in resolv.conf
All connections are to unique independent websites - no reusing connections here
Is my math wrong in assuming the connections per second is threads * timeout?
Notes: The server resources are virtually untouched - .10 load average and that is mostly MySQL keeping track of things. The spider and the server are performing flawlessly, I can achieve 8 million visits a day if I wanted to. Just can`t actually use the network to do work during this process.
My biggest epiphany in years of tweaking this was changing DNS servers that the other devices use - so as to avoid any sort of timeout or throttling imposed by the dns servers the linux box is using. This didn`t have the effect I was hoping.
The other devices like my Windows 7 laptop cannot visit multiple tabs, websites may timeout - cannot run rsync with a lot of small files transferring - but can run speedtest.net - once it loads the available bandwidth is still nearly 90% of what i`m buying. So downloading a single large file is not a problem, but moving around and opening more connections is.
I once connected the server through a dd-wrt router just to get more info about what is going on - and active connections will eventually reach the max of 4096 if running 200 threads. I really feel like the gateway/modem cannot process all the connections - but I also feel that 100 connections a second should be very acceptable - and something isn't being closed properly.
