0
I need to tune my windows 8.1 for a web scraper
I am using thousands of threads for concurrent crawling
For crawling process, i am using my own written c# HttpWebRequest using crawler
However i am having problems when i use proxies
I dont know what causes error however established connection count keep stacking while nothing on the system can connect anything until i terminate the application, disable network card and enable back again
I am properly disposing every connection even if the connection get terminate unexpectedly
These are the tunings i have found so far but i wonder are there any other tunings i can make?
My computer is very strong so i have no hardware limitation
regedit tweaks TCPIP
- set TcpTimedWaitDelay to 30 seconds
- TcpFinWait2Delay : 30 seconds
- seconds set MaxUserPort to 32k
- EnableDynamicBacklog : 1
- MinimumDynamicBacklog : 20
- MaximumDynamicBacklog : 10k
- DynamicBacklogGrowthDelta : 100
- EnableTCPA : 1
- TcpAckFrequency : 1
- TCPNoDelay : 1
- TcpDelAckTicks : 1
- LocalPriority=4
- HostsPriority=5
- DnsPriority=6
- NetbtPriority=7
cmd commands:
- netsh int tcp set global autotuninglevel=disabled
- netsh int tcp set global ecncapability=enabled
- netsh int tcp set global rss=enabled
- netsh int tcp set global chimney=enabled
- netsh int tcp set heuristics disabled
- netsh int tcp set global dca=enabled
my current settings
my computer is core i7 2600k @ 4.5 ghz = 8 threads 32 gb ram raid 0 ssd disks as C drive
After a while established connection counts never get decreased and keep increasing. When this happens, internet connectivity is lost
MonsterMMORPG
Posted 2017-03-20T23:05:02.773
Reputation: 475
1Using thousands of connections to scrape a site is bordering on abusive, any self respecting site owner would be tempted to find a way to block such behaviour. That said, the fact that you have to disable and reenable the network card is telling. Have you tried a different card? Have you looked for updated drivers? – Mokubai – 2017-03-21T06:38:56.403
@Mokubai the issue is not site related. That is another issue as you have mentioned (i mean having thousands of connections). I did not try different network card because i dont have. Yes drivers are all updated. Some weird bug happens in system and established connection count keep starting to stack. any ideas? – MonsterMMORPG – 2017-03-21T14:51:51.333
1Despite what you've said in your question I'm tempted to say that you are not correctly disposing of the TCP and they are lingering around long after they should have been closed. Are you simply destroying pointers to the connection in your program or are you fully closing them down? If this is truly a problem in your code then it would be better to post an example of what you are doing to [so]. – Mokubai – 2017-03-21T15:03:26.107
If ESTABLISHED sockets are continually increasing, then you are not closing them out. TCP sockets, when no longer needed, should be closed using a specific process (see here: http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm). There are TCP setting that might help if they linger in states like TIME_WAIT, but initially moving from ESTABLISHED to FIN_WAIT is all on the application, not TCP.
– MaQleod – 2017-03-21T15:39:30.667@MaQleod that is the issue. i am following every close handling rules. check here : http://codereview.stackexchange.com/questions/158359/composing-best-web-page-fetcher-function-by-httpclienthandler-for-c . it is not like starting to stack after application starts. application runs perfectly fine as expected for a while. lets say sometimes 30 40 minutes with hundreds of thousands of requests. all handled properly. then somehow some error bug happens. internet connectivity get lost and established count starts to stack
– MonsterMMORPG – 2017-03-21T18:44:46.333