4

I've been reading the following article:

And wondering if there's anything else i need to know about tuning linux to handle 1million tcp connections? So far i've narrowed it down to the following:

  • Configuring the kernel to support 1mil connections, system wide (sysctl.conf)
  • Configuring to have 1mil connections for the specific user (/etc/security/limits.conf)
  • Configuring tcp stack memory settings (sysctl.conf?)

Is there anything else i need to configure? (this is for an EC2 large 64-bit server)

-edit-

It's not apache, its a libev-based custom coded C server FYI. It'll scale to 1mil just fine, its the kernel that's my worry :)

Community
  • 1
Chris
  • 1,201
  • 5
  • 15
  • 17
  • Are you using any sort of firewall on the box itself or in front of it? We are use BSD to handle this sort of load in NGiNX and needed to tune some variables in pf to support that number of connections. – polynomial Aug 26 '11 at 14:00
  • Hi poly, no there is no firewall. Would BSD be a better option rather than linux for this kind of load? – Chris Aug 27 '11 at 02:31
  • 1
    Some lite reading. In particular follow the links provided in the answers. http://serverfault.com/questions/10852/what-limits-the-maximum-number-of-connections-on-a-linux-server http://serverfault.com/questions/107174/what-is-the-maximum-number-of-socket-connections-on-linux http://serverfault.com/questions/160996/more-than-65536-tcp-connections-on-linux http://serverfault.com/questions/69524/im-designing-a-system-to-handle-10000-tcp-connections-per-second-what-problems – user9517 Aug 27 '11 at 08:23

3 Answers3

2

You have most of the tunables configured that I would have set (and had to set). One thing I found when we scaling like this was that you will always have something special to your environment that no one else mentioned. To catch this you need to make sure you are watching and alerting on:

  • errors via syslog
  • errors your program sees like socket() failures, etc
  • network buffer availability (via SNMP or netstat cron)
  • kernel table limits (again via SNMP or /proc file parsing crons)
  • frequent monitoring (very lightweight polls done every 1-10ms, we use OpenNMS which does this really easily, because OpenNMS is awesome).

One other thing you might run into is issues with the HZ value. On our FreeBSD systems we increased this. I was investigating another question on linux and ran into a case where the socket queues are cleaned in relation to the HZ value:

TIME_WAIT connections not being cleaned up after timeout period expires

Regarding the comment I don't think FreeBSD specifically will be any better at this, they both need massive amounts of tuning to work. We are using FreeBSD because the boxes directly connect to the internet and OpenBGPD is currently the best open source BGP implementation available.

Community
  • 1
polynomial
  • 3,968
  • 13
  • 24
  • Hi, I'm not sure what you mean by the 'HZ value' ? – Chris Aug 30 '11 at 01:30
  • More info on the HZ value: http://kerneltrap.org/node/5411 I also found this question interesting because I ran into many of these while scaling: http://unix.stackexchange.com/questions/12985/how-to-check-rx-ring-max-backlog-and-max-syn-backlog-size/ – polynomial Aug 30 '11 at 02:01
1

I was having problems configuring the per-user settings, i kept changing the number of files in the limits.conf but it would then fail to allow me to log in ever again, i had to terminate the ec2 instance. I since discovered that 1024*1024 is the maximum you can put in the limits.conf, any more and you'll bork it:

echo "* soft nofile 1048576" >> /etc/security/limits.conf 
echo "* hard nofile 1048576" >> /etc/security/limits.conf

Other than that, and the additions to sysctl that are mentioned in the mochiweb blog links, is there anything else i need to do?

Thanks a lot guys. I'm really keen to see if i can push this server to handle 1mil connections...

Chris
  • 1,201
  • 5
  • 15
  • 17
-2

It depends on the specs of the server and what type of content it serves. It may not be possible at all, (simultaneous right?) if it's serving active content and media. Database and small websites should do fine. In apache config, try making it start many worker/child threads. In the box itself, just test it to see if Apache will handle it. If apache can but not the box, maybe someone else can help with that. Sorry. Good luck.

Yes, sorry for not reading it thoroughly.

U4iK_HaZe
  • 631
  • 5
  • 13
  • What on earth are you talking about? The user doesn't mention apache anywhere in their question. – Mark Henderson Aug 26 '11 at 04:53
  • It's not apache, its a libev-based custom coded C server FYI. It'll scale to 1mil just fine, its the kernel that's my worry :) – Chris Aug 26 '11 at 04:56