1

I run a site with about 100,000 visits a day and lately the load averages sometimes hit the roof (250+). They are normally about 0.5-1.5, so I assumed a (d)dos attack. The number of apache processes reaches up to 1,000. However, the number of connections to the server does not change (around 600) and the max connections a client has is ~20. It only happens occasionally and it sorts its self out when I restart apache. Why does apache decide to spawn loads of threads and send my load averages to 250?

This is the result of top:

Normal top:

top - 15:07:16 up 1 day,  1:28,  3 users,  load average: 1.44, 2.55, 21.83
Tasks: 307 total,   3 running, 304 sleeping,   0 stopped,   0 zombie
Cpu(s): 30.2%us,  3.2%sy,  0.0%ni, 64.9%id,  1.5%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   2074128k total,  1836852k used,   237276k free,    21732k buffers
Swap:  2096472k total,    56440k used,  2040032k free,   278048k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
16776 apache    16   0 38752  23m 3124 S 20.4  1.2   0:01.32 httpd              
16252 apache    15   0 37760  20m 3240 S  5.6  1.0   0:00.46 httpd              
16333 apache    15   0 36152  21m 3136 S  5.0  1.1   0:01.16 httpd              
17426 apache    15   0 35732  20m 3056 S  3.2  1.0   0:00.16 httpd              
17418 apache    16   0 38536  22m 3020 S  2.8  1.1   0:00.14 httpd              
16161 apache    16   0 36800  21m 3148 S  2.0  1.1   0:00.63 httpd              
15988 apache    15   0 29284  12m 3112 S  1.8  0.6   0:00.20 httpd              
17441 mysql     16   0 66656  19m 1988 R  1.4  1.0   0:00.07 mysqld             
17442 mysql     16   0 66656  19m 1988 R  1.2  1.0   0:00.06 mysqld             
12265 apache    15   0 35916  21m 3148 S  0.8  1.0   0:01.49 httpd              
15987 apache    15   0 37616  20m 3224 S  0.6  1.0   0:00.46 httpd              
13564 root      15   0  2552 1196  804 R  0.4  0.1   0:00.83 top                
17011 apache    15   0 26668  11m 3016 S  0.4  0.6   0:00.05 httpd              
  409 root      10  -5     0    0    0 S  0.2  0.0   0:21.83 scsi_eh_0          
 3920 mysql     15   0 66656  19m 1988 S  0.2  1.0   0:26.90 mysqld             
 9357 apache    18   0 40896  23m 3228 S  0.2  1.2   0:03.36 httpd              
11971 apache    18   0 39888  23m 3868 S  0.2  1.1   0:01.55 httpd              

Abnormal top:

top - 09:12:12 up 3 days, 19:33,  1 user,  load average: 58.75, 69.17, 58.05
Tasks: 568 total,   2 running, 563 sleeping,   0 stopped,   3 zombie
Cpu(s):  5.6%us,  2.3%sy,  0.0%ni, 11.7%id, 79.9%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:   2074128k total,  2004008k used,    70120k free,     9008k buffers
Swap:  2096472k total,  1009264k used,  1087208k free,   128268k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
14534 mysql     16   0 79764  24m 1824 R  6.0  1.2   0:00.20 mysqld             
13903 mysql     16   0 79764  24m 1824 S  2.4  1.2   0:00.13 mysqld             
13906 mysql     16   0 79764  24m 1824 S  2.4  1.2   0:00.08 mysqld             
14068 mysql     16   0 79764  24m 1824 S  1.8  1.2   0:00.10 mysqld             
13862 apache    15   0 35532  15m 1892 D  0.9  0.8   0:00.44 httpd              
14316 apache    16   0 26012 6672 1852 S  0.9  0.3   0:00.14 httpd              
14787 apache    16   0 27992 8332 1820 D  0.9  0.4   0:00.06 httpd              
10588 apache    18   0 39480  13m 1912 D  0.6  0.7   0:01.42 httpd              
12457 apache    16   0 39332  12m 2328 D  0.6  0.6   0:00.42 httpd              
13564 root      16   0  2684 1180  648 S  0.6  0.1   5:18.00 top                
13819 apache    15   0 26688 7352 2276 S  0.6  0.4   0:00.56 httpd              
13853 mysql     16   0 79764  24m 1824 S  0.6  1.2   0:00.09 mysqld             
14335 apache    16   0 28544 8964 1872 S  0.6  0.4   0:00.30 httpd              
  442 root      10  -5     0    0    0 D  0.3  0.0   0:53.61 kjournald          
 6337 apache    15   0 39420  17m 2648 S  0.3  0.9   0:03.46 httpd              
 9976 apache    18   0 38924 7708 2316 S  0.3  0.4   0:01.44 httpd              
 9990 apache    15   0 36432  16m 2556 S  0.3  0.8   0:02.23 httpd
Cameron Martin
  • 153
  • 2
  • 8

1 Answers1

2

It's the value of your ServerLimit setting and others inside httpd.conf:

StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

Dial down your ServerLimit to 100 or so and decrease MaxRequestPerChild to around 1000 so the children reap faster.

  • Why would load averages suddenly rise from ~1 to 250 though? – Cameron Martin Jul 18 '10 at 05:04
  • load is load - do you think it's going to spawn 200 children and just sit there? no, they were spawned to actually do *work* on your system and use the CPU, etc. –  Jul 18 '10 at 14:42
  • Yeah but why would those children suddenly be created and not be created again when I restart apache? – Cameron Martin Jul 20 '10 at 13:18
  • Look at the swap utilization in the second top. That system is probably thrashing (doing more work swapping processes in and out of RAM/on and off disk than real process work) when the load hits the roof. Setting the ServerLimit to an amount you can hold in RAM will also prevent that. – freiheit Jul 24 '10 at 05:28