I have an Apache server that's proxying to Tomcat via mod_jk. I've set up a custom log in Apache to log the access times:
LogFormat "%h %l %u %t \"%r\" %>s %b %D" transfertimes
CustomLog /path/to/transfer-times.log transfertimes
I'm running ApacheBench on the web server itself. I've noticed that as I increase the concurrency, my 99th percentile request times appear to exponentially increase, although the 50th percentile remains relatively stable. As an example, ApacheBench will say that roughly 10 requests of 1000 take > 1 second to respond with a concurrency level of 100.
However, when I look at the transfer-times.log, no requests show up as greater than 1 second, based on the %D in the LogFormat. I'm trying to figure out what would cause the disparity between Apache's log and ApacheBench's reported access times. Normally I could attribute it to network latency, but I'm running this all on a single host. I'm thinking there must be something quirky with Linux TCP parameters or file descriptors I need to tune, but I'm not sure where to start.