I have a problem.
I am on a Xen VPN, running CentOS 6 and Apache. I have a medium traffic Zen Cart store and a few low traffic wordpresses. I have one or two very simple PHP form submission scripts, they are properly escaped.
Sometimes, my CPU usage will jump to 100% (actually 200%). This totally crashes the server - I can't login, can't SSH, can't access control panel (interworx), can't do anything. My only recourse is to reboot the server.
When I reboot, it is fine again...for a couple weeks.
If I don't reboot, the CPU will stay plateaued at 200% until I check and reboot. This went on for 24h once. When it does this, the network utilization drops to zero - this rogue process totally murders the whole server.
Because I can't ssh in, I can't use top or any direct tools like that.
Suspecting it is malware, I have combed through all the access logs each time this has happened. Yet, (I'm pretty sure) I can't find anything too suspicious (I get attacked a lot, but they all seem to be rejected). I have done my best to search for any backdoors or scripts that don't seem like they should be there.
My Zen Cart and Wordpresses are up to date. I use very secure passwords and only connect from linux terminals.
How can I investigate this? Are there any tools I can set that will somehow make a log of what the hell is happening when this happens? Is it more likely, from your gut intuition, to be an innocent software bug, or have my servers been compromised somehow and are being used for...I don't know what? Rogue researchers at Folding@Home?!? I can't think what hackers would be doing with maxed out CPU but zero network.
Any insight would be greatly appreciated. This is getting to be a big problem. Thank you!
