9

This may seem programming related, but this is an OS question.

I'm writing a small high performance daemon that takes thousands of connections per second. It's working fine on Linux (specifically Ubuntu 9.10 on EC2). On Mac OS X if I throw a few thousand connections at it (roughly about 16350) in a benchmark that simply opens a connection, does it's thing and closes the connection, then the benchmark program hangs for several seconds waiting for a socket to become available before continuing (or timing out in the process).

I used both Apache Bench as well as Siege (to make sure it wasn't the benchmark application).

So why/how is Mac OS X limiting the RATE at which sockets can be used, and can I stop it from doing this?

Or is there something else going on?

I know there is a file descriptor limit, but I'm not hitting that. There is no error on accepting a socket, it's simply hangs for a while after the first (roughly) 16000, waiting -- I assume -- for the OS to release a socket. This shouldn't happen since all prior the sockets are closed at that point. They're supposed to come available at the rate they're closed, and do on Ubuntu, but there seems to be some kind of multi (5-10?) second delay on Mac OS X.

I tried tweaking with ulimit every-which-way. Nada.

pbhogan
  • 321
  • 2
  • 7
  • 2
    this was cross-posted to SU: http://superuser.com/questions/145989/does-mac-os-x-throttle-the-rate-of-socket-creation .. @pbhogan: please don't crosspost in the future. – quack quixote May 27 '10 at 18:03

2 Answers2

13

So it turns out the Mac OS X ephemeral port range is fairly low.

Wikipedia informs me that IANA suggests 49152 to 65535 as "dynamic and/or private ports" while many Linux kernels use 32768 to 61000. OS X uses the IANA range. This means Linux has almost twice the available ephemeral ports. Since each closed socket goes through a TIME_WAIT state (that I didn't know about) the rate is just overwhelming my system.

How to fix?

sudo sysctl -w net.inet.ip.portrange.first=32768
sudo sysctl -w net.inet.ip.portrange.hifirst=32768

This will give about double the range.

(Thanks to Spiff who answered in more detail here: https://superuser.com/questions/145989/does-mac-os-x-throttle-the-rate-of-socket-creation)

Community
  • 1
pbhogan
  • 321
  • 2
  • 7
0

Could it be a buffer limit? http://www.psc.edu/networking/projects/tcptune/#MacOS

This site and a few others hint that BSD Unix systems have sockets tied to file descriptors, so may be the file limitation may be tied to your open socket limitations.

Bart Silverstrim
  • 31,092
  • 9
  • 65
  • 87