1

I desperately need help in figuring out how to troubleshoot this problem I'm having. I run a fairly mission critical web server (Debian 7.5, 512MB RAM, 512MB swap, Apache, MySQL). It runs a couple WordPress sites on it. Today I found the websites responding quite slowly, and ssh'd in to find the load average was just above 10.0, and RAM use was at 100% and swap was close to the 512MB limit.

I have no idea how to figure out what's going on. Is Apache or MySQL not tuned properly? Maybe someone is attacking the server with repeated hits (how would I know?). I installed htop but even if I saw that Apache or MySQL was eating up a ton of resources, how would I figure out why?

Any points in the right direction would be massively appreciated. I'm at a loss here and I have to keep this server stable.

Side note: The server was up for 30 days, so maybe this was some sort of slow leak. Now that I've rebooted, load average is at 0.45 1.10 0.88, RAM is 165/512MB and swap is 68/512MB.

UPDATE: Still having issues. I captured a screenshot of htop below.

enter image description here

CaptSaltyJack
  • 628
  • 2
  • 13
  • 34

3 Answers3

4

Congratulations, you've managed to use nearly all of your swap space.

The first obvious problem here is that you went very deep into swap. This is probably what's causing the system to thrash so hard (lots of time spent in system, I/O wait and software interrupts).

First thing to do is to reduce the number of Apache processes that are running. You don't need that many for a small site, and it's just going to throw you deep into swap and kill your performance...which is what already happened. I would recommend you start very small and increase when it becomes necessary. An example:

StartServers            1
MinSpareServers         1
MaxSpareServers         2
MaxClients              5

This limits you to only serving 5 simultaneous requests (everyone else has to wait in line). If at this point you get warnings from Apache about running out of servers, and you still have RAM to spare, then you can increase them, but you are eventually going to run into a point where your VPS simply hasn't got enough RAM to handle all your traffic. At that point you should upgrade the VPS.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • This sounds good, but there's a problem. All of the calls to StartServers, MaxClients, etc, are within blocks such as ``, and the thing is, I don't have that module installed. I removed it and replaced it with `apache2-mpm-itk` so I could use the `AssignUserID` and have each virtual host's files owned by different users (rather than all owned by `www-data`). – CaptSaltyJack Oct 26 '14 at 18:51
  • “This limits you to only serving 5 simultaneous requests (everyone else has to wait in line).” That’s barely enough for a development or production server. A `MaxClients` of 5 pretty much assures that of the page has like 10 elements the server is immediately slow. Apache clients are processes, not per browser numbers. – Giacomo1968 Oct 26 '14 at 18:51
  • @JakeGould I'm aware of that. The idea is to start small and tune upward until you reach your hardware limits. – Michael Hampton Oct 26 '14 at 18:53
  • PS: If I can use StartServers, MaxClients, etc outside those `` blocks, let me know. – CaptSaltyJack Oct 26 '14 at 18:53
  • @CaptSaltyJack Yes, they do not have to be within `` blocks. In fact, you should already have some in your configuration somewhere. – Michael Hampton Oct 26 '14 at 18:54
  • @MichaelHampton I do, but the problem is that these server/client directives are inside ``, which I've replaced with `apache2-mpm-itk`. – CaptSaltyJack Oct 26 '14 at 18:56
3

Before anything, based on the screenshot you have posted with htop output, it seems you have 512MB of RAM on a site running WordPress? I have never seen WordPress happy on servers less than 1GB of RAM. Maybe if you are running a test or development site, 512MB is adequate, but for a production site you need 1GB of RAM. That is the root of your problem. That said, here are some ideas to help you squeeze out better performance from the setup you have:

I have no idea how to figure out what's going on. Is Apache or MySQL not tuned properly? Maybe someone is attacking the server with repeated hits (how would I know?). I installed htop but even if I saw that Apache or MySQL was eating up a ton of resources, how would I figure out why?

First, I would not panic about an attack happening. The reality is that your server is probably just taking on a high load of legitimate traffic, but the server itself is not configured/tuned for your usage. Of course bad configurations can bring your sit down during DDoS (Distributed Denial of Service) attack, but it can all make your life miserable when nice/normal traffic suddenly shows up in high numbers.

I posted a nice list of items you can review to improve your L.A.M.O. stack performance on another similar question, and will repost here for your reference:

  1. Properly configure Apache: Apache is a good piece of software, but right out of the box it’s a memory hog. For example, I believe the default is to allow 255 connections per second? I can assure you most simple sites barely get 40 connections per second on a good day. So adjusting Apache to be realistic to your traffic will help. Also, there is a KeepAlive setting in Apache that works great! But out of the box, I believe it’s set to a MaxKeepAliveRequests of 100 which is fairly nuts. I usually set this to about 30 connections with a small KeepAliveTimeout of 2 to 3 seconds. The key is to have the KeepAliveTimeout to match speed it takes for an average page to download with a little bit of room for overhead/slowness. So if a page loads in 1 second, do a KeepAliveTimeout of 2 seconds.
  2. Review the code for your WordPress sites for potential bottlenecks: Concentrate on the PHP core of it & clear up what you can. Look out for excessive MySQL calls & file system calls. This is where you will be able to make the app fly! Also, check the memory_limit in your php.ini and make sure it’s not higher than necessary. The default is 64M, but in many cases that can be lowered to 32M.
  3. MySQL tuning or moving it onto it’s own server: After writing about MySQL above I realized you might be hosting your MySQL instance on the same box. Look into optimizing MySQL performance by running a script like MySQL tuning primer. Without tuning, MySQL will eat up all resources & big the system down. With tuning, MySQL will run better/faster & resources can be freed for other purposes. Also, consider moving your MySQL DB to a standalone server. You might have to learn how to properly network & firewall the server to allow your servers access but protect against hackers, but the performance benefit will be great.

Regarding the MySQL tuning, that is something that can take a few weeks to nail down at the beginning. The reason being tuning scripts are based on real traffic MySQL sees. So you basically make your site live to the world, wait 2 days (at least), run the tuning scripts & then wait a few more days to tune some more. After a week or so you should be able to tune MySQL to work as well as it can with your setup.

Giacomo1968
  • 3,522
  • 25
  • 38
  • Thanks, this was extremely helpful! I've tuned Apache accordingly, and adjusted the PHP `memory_limit` value. This has already helped. About MySQL: I ran the tuning script. The only issues that came up were my max memory limit "exceeds 90% of physical memory" but I'm not sure how to change that. It also says I should increase my `table_cache` and that 100% of my table cache is in use (400 open tables). I'm not sure what value to set it to though. After a bit of searching, I decided to set it to 1024. – CaptSaltyJack Oct 22 '14 at 06:32
  • Still having issues. :( I updated my Q, see the screenshot above. Why is Apache freaking out? – CaptSaltyJack Oct 26 '14 at 17:38
  • @CaptSaltyJack You only have 512MB on a site running WordPress? I have never seen WordPress happy on servers less than 1GB of RAM. Maybe if you are running a test or development site, 512MB is adequate, but for a production site you need 1GB of RAM. That is the root of your problem. – Giacomo1968 Oct 26 '14 at 18:49
  • @CaptSaltyJack Also, if this answer has helped you please remember to up vote it. And if this is the answer that solves your problem, please remember to check it off as such. – Giacomo1968 Oct 26 '14 at 18:50
0

First install these plugins: super cache (htaccess mode), wpbase, widget cache. Wordpress is known for having these issues. IF that doesn't help, it might be a memory leak on one of your themes (especially if mobile view is enabled).

Post your access and error logs.

I strongly recommend working with nginx instead of apache.

WindArtas
  • 21
  • 4
  • I'd love to use nginx, though doesn't it take a bit of hacking to get WordPress to work with it? And I'm not sure how to translate the .htaccess files into something nginx can work with. (I have no experience with nginx) – CaptSaltyJack Oct 19 '14 at 21:22
  • 1
    Set up a virtual machine and work your way up. It's worth learning, the advantages are huge ... specially for wordpress ;-) – WindArtas Oct 21 '14 at 17:09