5

I've got a server running apache, and have been seeing occasional apache processes go to 100% and stay there. Today, with two processes at 100%, I turned off external access to the server (to prevent further requests to apache). Five minutes later, no requests are coming in to the server but both processes are still at 100%.

I've run lsof on each process, and they've giving me about 9000 lines of output (that might as well be greek to me). No other processes seem to be behaving strangely or waiting etc.

My database is on a second server. Using mytop shows two MySQL connections active from the apache server, both with a state of "sleep". I killed one of those MySQL threads, and there was no change to either process on the Apache server.

This apache server is one of two behind a simple load balancer. I don't know if that could be related.

How can I confirm that the apache issue is related to what I'm seeing on the database server? And is this likely to be the result of a dodgy SQL call, or something else?

Edit: Found the issue. It was a code problem with Magento. The image resizing function was failing to open an image, because the extension was incorrect (it was a BMP with a jpg extension). The error handler for this was invoking the resize again, et voila - a loop. Found this by doing strace on the misbehaving apache process.

Dave Child
  • 297
  • 5
  • 15
  • 2
    Good question/solution. I asked myself what I would have done given the situation and I had no idea where I would start with something like that. It's good to see someone else's train of thought and process. – Safado Oct 28 '11 at 13:48
  • I suggest that you move the solution text out of the question into your own answer, accept your answer, and mark this comment as obsolete. – Rok Strniša Dec 28 '11 at 23:20

2 Answers2

1

Found the issue. It was a code problem with Magento. The image resizing function was failing to open an image, because the extension was incorrect (it was a BMP with a jpg extension). The error handler for this was invoking the resize again, et voila - a loop. Found this by doing strace on the misbehaving apache process.

Dave Child
  • 297
  • 5
  • 15
0

Perhaps your Apache is out of date and vulnerable to the recent range request vulnerability, and someone or something is now exploiting it?

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
  • Thanks for the quick reply! I updated and upgraded, and added one of the recommended prevention measures for the range vulnerability, restarted everything and re-enabled traffic, and the same thing happened again almost immediately - Apache process went to 100% (and is still there, with traffic off again). – Dave Child Oct 28 '11 at 09:25