1

So, low and behold, a legacy site we've been hosting for a client had a version of FCKEditor that allowed someone to upload the dreaded c99madshell exploit onto our web host.

I'm not a big security buff -- frankly I'm just a dev currently responsible for S/A duties due to a loss of personnel. Accordingly, I'd love any help you server-faulters could provide in assessing the damage from the exploit.

To give you a bit of information:

The file was uploaded into a directory within the webroot, "/_img/fck_uploads/File/". The Apache user and group are restricted such that they can't log in and don't have permissions outside of the directory from which we serve sites.

All the files had 770 permissions (user rwx, group rwx, other none) -- something I wanted to fix but was told to hold off on as it wasn't "high priority" (hopefully this changes that). So it seems the hackers could've easily executed the script.

Now I wasn't able to actually locate c99madshell.php itself -- only a bunch of other HTML files containing russian text and a few .xl and .html files with inline PHP that were renditions of the madshell hack. Upon doing a bit of research, though, it looks like the hack destroys itself after execution -- great.

Anyway, my initial assessment would be this:

  • Not necessary to rebuild the entire host, as given the isolation of the apache user / group, they shouldn't have been able to get system level passwords.

  • It is necessary to fix this vulnerability by restricting uploads to not have the execute permission, updating the FCKEditor version to correct the original exploit target, and add server-level configuration that denies execution of PHP script within the uploads directory.

  • I should change the DB passwords for the app -- given the configuration file for the database connection lies within the web-root, the hacker most likely could've grabbed it and with it the DB password.

Anyway, please provide any input you guys have regarding what I should tell the bossman. Obviously it'd be ideal to avoid rebuilding the whole host -- but if thats what we have to take to ensure we're not running a hacked machine, then thats what its going to take.

I really appreciate your guys help. Also don't hesitate to ask for more information (I'd be happy to run commands / work with you guys to assess the damage). God damn hackers :(.

peterh
  • 4,914
  • 13
  • 29
  • 44
Skone
  • 479
  • 1
  • 3
  • 13
  • Be sure to see these related questions. http://serverfault.com/questions/6159/aftermath-of-hack http://serverfault.com/questions/6190/reinstall-after-a-root-compromise – Zoredache Jan 20 '10 at 18:51
  • THANKS to everyone. I wish I could give you all the "answer" but I can't -- I'll certainly add votes to everyone who is helping me out. Running Ubuntu 8.0.4(Jaunty). No CC transactions or particularly sensitive data -- mainly just marketing materials / corporate identity clients. Sounds like a rebuild is the best option. – Skone Jan 20 '10 at 19:17

5 Answers5

4

Obviously as others have said, the official "secure" response would be to rebuild the machine.

The reality of the situation might prevent that. You can run a few things to check for compromises:

  • Chkrootkit - tests for common signs of the server being compromised
  • If rpm system, 'rpm -Va|grep 5' will check all rpm installed binaries and report a "5" if the MD5 sum has changed. Rebuild necessary if you found any inconsistencies not explained b a customized binary.
  • Look in /tmp for anything suspicious.
  • Check 'ps fax' for any abnormal processes. If 'ps' is compromised or via other techniques you could still have hidden processes running.
  • If you found ANY files in your search that had ownership other than apache, your system accounts were definitely compromised and you need a rebuild.

Corrections to make on your system configuration:

  • Disable uploads by FCKeditor if possible
  • Make sure your temp directories are made NOEXEC to prevent programs executing off of them
  • Any php scripts should be up to date
  • If you want to get fancy install mod_security to look out for exploits during runtime of the php scripts

There is a ton of stuff I am missing out but those would be the first steps I would take. Depending on what you are running on the server and the importance of the security of it (do you handle CC transactions?) a rebuild might be necessary but if it is a 'low security' server you might be ok with checking the above.

Dave Drager
  • 8,315
  • 28
  • 45
3

You gotta rebuild the host. You don't know what kind of privilege escalation attacks they have used against the host, nor can you be absolutly certain that there are not trojans, keyloggers, or rootkits installed.

Once you get compromised, there is no other option besides a rebuild from bare metal.

Zypher
  • 36,995
  • 5
  • 52
  • 95
3

In short - I would rebuild the server.

If they have access to a local user (now they had access as apache user) they can run local exploits. So you should consider that the entire server is compromised. You should check for other servers too.

What distribution are you running? If it is rpm based you can check the signatures of the files. Boot the installation CD and run rpm -V on to check the packages.

Mircea Vutcovici
  • 16,706
  • 4
  • 52
  • 80
  • don't forget to assume all user passwords (including root) are compromised... change all your user/admin passwords. If you have DB accounts in your source code (not best, but common practice) you should change the passwords to those too. – paxos1977 Jan 21 '10 at 22:46
2

First of all. NEVER TRUST A COMPROMISED SERVER.

Even tho you think you have secured the machine, script kiddies aren't stupid, chances are somewhere there's a backdoor that was installed. User/Group makes little difference after you've gained shell.

Backup your files, implement a change of ALL passwords.

  • scritp kiddies are stupid, the people that write the scripts FOR them ... now some of them can be crazy smart (and probably have a check box for the script kiddies "i wanna get back in later") – Zypher Jan 20 '10 at 18:57
1

I know it isn't what you'd prefer to hear, but I really would recommend backing up the data, rebuilding the server, and reimporting all the data.

Also be sure to check through other sites to ensure that this one isn't the only one affected by the hack. If all your server's scripts are running as one Apache user (nodoby/www_data/what-ever-your-distro-uses) anything that can write to one site can almost certainly write to the rest too.

As well as changing all passwords, make sure that any SSH keys on the server are invalidated on all other servers (i.e. revoke the public keys where-ever they are stored rather than just deleting the private key) and that any other systems you might have entered a password for (or stored a password for in a file) on/via that server have the passwords changed.

David Spillett
  • 22,534
  • 42
  • 66