Good day,
I want to be able to serve at least 7000 (preferably 10k) concurrent requests from my Apache Httpd. I have configured my httpd.conf with MPM worker with the following setup
ServerLimit 330
StartServers 25
ThreadsPerChild 25
MaxClients 7500
My Apache Httpd server does nothing except do a reverse proxy against a cluster of java app servers and some (disk) caching on some static items like HTML/CSS/JS.
While trying to pound my system using JMeter (with about 1k concurrent requests), in a couple of minutes, my server starts to crash but it shows no additional information.
My /etc/security/limits.conf is configured to
* soft nofile 10000
* hard nofile 30000
apache soft nproc 8192
apache hard nproc 12288
* soft stack 512
* hard stack 1024
Any ideas how I can reach 7k (or preferably 10k) concurrent requests
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] Child 15139 returned a Fatal error... Apache is exiting!
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:46 2012] [warn] child process 14004 still did not exit, sending a SIGTERM
EDIT (Additional Info):
This is me trying to see how many active httpd processes that I have while the test is running (each command is about 1second apart - i.e. me pressing the up key on my keyboard and then pressing enter)
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
5
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
5
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
8
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
8
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
8
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
11
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
11
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
13
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
13
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
17
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
17
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
25
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
25
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
41
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
41
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
37
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
37
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
1
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
Update: If I add ThreadStackTrace, it does not crash anymore: ServerLimit 330 StartServers 25 ThreadsPerChild 25 MaxClients 7500 ThreadStackSize 512
Note: I had stack entries in my limits.conf before adding ThreadStackSize, but I guess that wasn't enough. I had to add both the stack entries in limits.conf and ThreadStackSize in apache httpd.cofn.
However, it still cannot process 7500 concurrent requests. When I do a ps -ef | grep httpd | wc -l, the highest that I see is 42 only (and since one of that process is the grep command, that means it's about 41 apache httpd processes). But I've configured my apache to reach up to 330.
So to see if my apache configuration can really go beyond 40 processes, I tried modifying my StartServers into 50.
ServerLimit 330
StartServers 50
ThreadsPerChild 25
MaxClients 7500
ThreadStackSize 512
The result is the following:
[Wed Apr 11 03:33:40 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Apr 11 03:33:40 2012] [notice] Digest: generating secret for digest authentication ...
[Wed Apr 11 03:33:40 2012] [notice] Digest: done
[Wed Apr 11 03:33:40 2012] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
[Wed Apr 11 03:33:40 2012] [alert] Child 55663 returned a Fatal error... Apache is exiting!
So I guess the main problem is that I cannot spawn more than 40 processes.
Update2: Since I cannot seem to be able to increase the number of processes, I tried increasing the number of threads. I can go only as far as 30 ThreadsPerChild (from the original ThreadsPerChild). After that, Apache is not able to start because it cannot seem to create worker thread.
Anybody out there has some ideas how to reach 10k concurrent requests with Apache HTTPd?
Why this is not a duplicate: Pardon, I am not sure why this was marked as duplicate. If the answer to my question is in Can you help me with my capacity planning?, can you kindly highlight how & where? That question is generic in nature while mine is very specific and a more "show me how to specifically reach c10k with apache httpd or point out what's wrong with my setup".
Thanks,
Franz
