1

In the last week suddenly I get errors for my server. I try to access my website I get 503 errors.

When I check top and other things I see these:

  • 60-65 httpd (apache) processes. (Normally I have 30-35 httpd processes)
    You can see graph here: httpd process
  • MySQL makes ~350 MB swap. (Normally it makes 40-50 MB swap)
    Tuning primer says that:
    • Current max_connections = 250
    • Current threads_connected = 96
    • Historic max_used_connections = 97
  • Server all graphs: general data
  • Inside httpd.conf
    • ServerLimit 60
    • MaxClients 60
  • In var\www\vhosts\example.com\logs\access_log there is nothing that would seem like (D)DOS attack. I see normal server request.
  • Nothing suspicious in var\log\httpd\access_log
  • Inside my var\www\vhosts\example.com\logs\error_log I see too many errors like this:
    • [Fri Jan 17 10:56:26 2014] [warn] [client 78.180.71.157] mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper
  • Inside \var\log\mysqld_log no error
  • Inside \var\log\error_log no error in that time
  • Inside \var\log\suexec_log no error

How can I diagnose what causes my website to make DISK IO 100%, load average 10, very high swapping, and server doesn't respond requests. And how can I stop this to happen in future ?

When I restart mysql and httpd problem solves. But how can I diagnose the cause

trante
  • 131
  • 5
  • Sounds like resource exhaustion to me...1 GB of RAM is definitely not enough for a decent-sized website. The I/O is likely caused by your server swapping to keep up. – Nathan C Jan 17 '14 at 12:42
  • I'm planning to increase RAM in future. But now I need to solve this issue. Whether I would have 3 GB RAM, this would happen due to high number of requests. Server would swap and swap and can exhaust 3 GB RAM at the end. – trante Jan 17 '14 at 12:45
  • 3GB is not all that much for a system that's both a web and a database server with that kind of web service load. Try and increase to 8GB immediately, monitor, and warn management they might need more. – MadHatter Jan 17 '14 at 13:05
  • This website has 100-150 visitors in 30 minutes time. And in normal operations server needs 30-35 httpd processes. We can't afford 8 GB for this website we don't earn that much now. Besides, increasing RAM is easy I need to find the cause that occurs suddenly. – trante Jan 17 '14 at 13:12

1 Answers1

2

The warning mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper means that Apache has exhausted all resources available to it and can't serve the client. The client will get a 503 error.

From the graphs, it's either a traffic spike or a software misconfiguration that's causing your problems. When RAM requests spike, your server runs out of RAM so it peaks I/O to swap everything it can to disk (hence the 100% I/O) which causes your iowait to also increase. Once all resources are exhausted or your server is sluggish enough to time-out to Apache, it issues this warning.

Restarting likely just frees up this memory. Your answer would be one of two things: 1) increase the RAM available to your server by an actual increase or by offloading MySQL (which is a memory hog) to another server, or 2) reviewing your code to see if there's a leak.

Nathan C
  • 14,901
  • 4
  • 42
  • 62