16

I maintain a server to host my files and a number of websites for hobbies and friends. I just noticed that the site summary being returned in google for one of my domains were all porn related (search "fringe.org", 1st result).

Checking the source, I found obfuscated php code (see here) I didn't recognize included in /index.php of that domain after the <\/html>. I deleted it and then also found and deleted it from the index.php of 5 other domains in my web root (although there were 3 or 4 without it).

Server details:

  • LAMP with Ubuntu 10.10 fully updated very regularly
  • PHPinfo: link
  • My password is very secure, but I have given accounts with web-write access to 3 or 4 friends. I'm not sure how secure their passwords are.
  • I believe access is only by ssh or sftp

My question is what steps I need to take now. Data and content are backed up off the server regularly. It makes me feel dirty so I would be inclined to wipe and rebuild the whole thing, but I am 10 days from the big annual event of my largest site.

Thanks so much in advance for any tips.

Well ... shit. Thanks for the feedback and info. It's tricky how easy it is to get yourself into trouble these days. I would have marked all of your answers as accepted if I could, they were all helpful. I'll be doing a full wipe and carefully reconstructing things asap.

AviD
  • 72,138
  • 22
  • 136
  • 218
Sam Swift
  • 263
  • 1
  • 6
  • What do you mean you "*believe* access is only by ssh or sftp"? Are other remote access services running as well? I'd recommend disabling any (especially cleartext ones like telnet) that aren't absolutely needed. – Iszi Apr 06 '11 at 17:50
  • thanks lszi, I thought someone might call me out on that line. I meant that I haven't intentionally installed or set-up anything else. If anything else is open by default on ubuntu server or might have been opened by the intruder, I might not know about it. I just ran a netstat -a and don't see anything unexpected. – Sam Swift Apr 06 '11 at 18:02
  • Was it running Wordpress? Check the uploads folders and the wordpress databases if so. http://sucuri.net has loads of helpful resources for this sort of attack – atdre Apr 06 '11 at 19:08
  • @ Sam Swift: netstat would not necessarily reveal anything odd if the host has been compromised. Really, it's too late to do much forensics analysis because you don't have Suricata (or ModSecurity) running, you don't have OSSEC installed, and you already messed up what was resident in the DRAM by responding to this incident without copying important files out of /proc. Best to notify the FBI and build a new server with new code/content/DBs/passwords – atdre Apr 06 '11 at 19:10

4 Answers4

10

PHP:

  1. allow_url_fopen needs to be Off. I think this is how you got owned. A big mistake!
  2. display_errors needs to be Off. I use these all of the time to find path disclosures.
  3. file_uploads needs to be Off. This could have been another way you got owned.
  4. expose_php needs to be Off
  5. safe_mode needs to be On
  6. magic_quotes_gpc should probably be set to On

These are severe problems. You were very likely hit by an LFI/RFI botnet. You will need to install a new server, but DO NOT COPY any of the code, data from the database, or any flat files (and do not reuse any passwords -- they are all suspect at this point! You have a responsibility to change all of your passwords and tell your users to do the same).

Let me remind you of the seriousness of this attack. Any of the PHP code or other content on your server can (and probably does) contain other backdoors. If you did not have file integrity monitoring in place before this incident, then you will have to build a new server without this PHP code and content. You can't just copy it over -- you will have to manually re-create all of it.

Secondly, you cannot trust the data in the database if you use one, such as MySQL. You can't copy the MySQL data files or even utilize the same data records, columns, or tables to a new machine. You should consider all of this data as having the capability to take over any new machine its put on -- and to send malware to anyone visiting any website connected to this database or that uses any of the data in this database.

Lastly, all flat files also cannot be trusted -- including important ones like your /etc/passwd and /etc/shadow files. You can't re-use any of this data and all of the passwords are suspect. The passwords in your databases are also suspect.

If anyone has ever SSH'd from this machine, consider those remote accounts compromised, as well. If there is local connectivity on a LAN to any machines -- consider those machines as compromised, as well. Any passwords in any PHP files that connect to other services (such as external databases) should also be considered compromised. Everything will need to be reconstituted. You cannot restore from backups!

You are in very serious trouble and will likely need to hire outside help!

atdre
  • 18,885
  • 6
  • 58
  • 107
  • 1
    I should give you a -1 for the magic_quotes_gpc. I am happy that its being removed, its a security nightmare. Also it can't be RFI if `allow_url_include=Off`. – rook Apr 06 '11 at 18:40
  • @Rook: How do you know that the attackers didn't set allow_url_include=Off in order to close out their entry point? – atdre Apr 06 '11 at 18:45
  • @atdre because its been disabled by default for more than 5 years. Also, that requires root.... – rook Apr 06 '11 at 19:01
  • @Rook: I'm not convinced, but ok -- let's pretend it wasn't an RFI. It could have been an LFI that allowed remote code execution. Or whatever. I changed my stance and called it an LFI/RFI botnet because they tend to do both. Have you seen these or not? I have seen ones exactly like these before. They try both RFI and LFI. – atdre Apr 06 '11 at 19:06
  • @atdre yeah I have seen these, vulnerability scanners that look for remote code execution. I don't think they are as successful as someone who turns a exploit form exploit-db.com into an auto-hacker, and then uses google hacks to find vulnerable versions. The PHPMyAdmin worms are a good example of this. – rook Apr 06 '11 at 19:08
  • @Rook: Yes, PHP-based LFI/RFI/RCE vulnerabilities are not quite as common as they used to be in web applications -- their numbers are dwindling (probably because everyone who had them has already been owned a few times). However, it's a very large Internet out there -- and even Ubuntu 10.10 can get easily owned by a poorly constructed PHP config and codebase. MySQL, WordPress, Joomla, web-calendars, PhpMyAdmin, etc only aggravates this problem further. – atdre Apr 06 '11 at 19:16
4

After experiencing a similar (but more serious) compromise myself, I can tell you that atdre is exactly right. There are four basic steps to handling any security compromise:

  1. Shut down: Shut down all compromised systems. Do not wipe them at this point.
  2. Analyze: Use data from the compromised systems in a sterile environment to try to determine proximate cause and extent. I would highly recommend hiring a security consultant at this point.
  3. Disclose: Tell any affected users the nature and consequences of the exploit, and what they should do about it. This is going to hurt but you must do it.
  4. Recover: Nuke the compromised systems from space. This means a complete OS reinstall or, when possible, launch an entirely new system. If you need to recover data from the system, use backups that predate the time of attack. Do not use any data that may be compromised. If you think you can clean it up, you definitely can't. Before reconnecting the new system(s), fix the security vulnerabilities you discovered in step (2) and do a careful audit for other possible vulnerabilities.

A far more thorough answer is in the following question. It helped me greatly in my own time of need.

https://serverfault.com/questions/218005/my-servers-been-hacked-emergency

Community
  • 1
Rein Henrichs
  • 141
  • 2
2

1)Make sure all libraries, modules and applications are fully updated.

2)Stop using FTP. There are worms that spread by sniffing for FTP Logins.

3) Run PHPSecInfo

4)Consider using a Web Application Firewall (WAF) like mod_security.

5) If by chance you can get an IP address of the attacker or someone accessing the backdoor. REPORT THEM TO THE FBI. (A WAF helps with this)

(Bonus points: replace the backdoor with a simple .php script that logs the ip address. If anyone hits it you'll have an ip for the fbi)

rook
  • 46,916
  • 10
  • 92
  • 181
2

You should wipe the whole machine. The attacker had enough access to your server to alter some files; he probably could execute arbitrary code. Unfortunately, local privilege escalation exploits tend to exist in large numbers; a properly up-to-date machine should have no remote exploit, but believing in security against local users is a bit naive. It is probable that one of your users got hacked (password guessed, or his local machine hacked and the attacker just followed the SSH session).

Consequently, it is plausible that the attacker installed a rootkit which you would have a very hard time to detect from the machine itself (a rootkit is designed to make itself invisible). Hence the recommended wipe.

What would be worse ? That you could not optimally reconfigure your big site for its annual event ? Or that the site suddenly turns into a porn-fest right in the middle of said event ?

Thomas Pornin
  • 320,799
  • 57
  • 780
  • 949