47

I recently helped a client who had their server hacked. The hackers added some PHP code into the header of the homepage redirecting the user to a porn website — but only if they came from Google. This made it slightly harder for the client to spot. The client would see the website fine. Only new website visitors from Google would be directed to the porn site.

Last night a similar thing appeared to happen to a different client. I assumed it was a similar hack, but when I checked the codebase I could not find any malicious code. His chrome browser is redirecting from the clients website to www(dot)pc-site(dot)com. I cannot replicate this behaviour. I guess it is possible that malicious code is being added and removed. So I need a more comprehensive way to tell if the server has been hacked.

Only 2 developers have access to this dedicated server (and the hosting company Rackspace). The server is Red Hat Linux.

What are the steps I go through to find out if the server has been hacked?

Boz
  • 595
  • 1
  • 4
  • 8
  • Please change you URL to http://www(dot)pc-site(dot)com of something simular. You don't want viewers to accidently click through to this site. – Bernie White Sep 23 '11 at 11:37

4 Answers4

49

UPDATED

I would check the following:

  1. Logs. If you have root access you should check things like history which will give you command history and log files in /var/logs.

  2. Baseline. If you have a baseline like file hashes to work with for application and system files this will help a lot. You can also use backups to compare a previous state. If using a backup to compare files, use a slightly older one if you can. The site may have been compromised a while before and it is only now that the redirect has been activated.

  3. Check any includes. The files may not be on your server. They may be script includes such as <script src=”http://baddomain.com/s.js” /> or iframe type tags. Also do not exclude images, PDFs of Flash (SWF), video files. It is a fairly common trick to embed links in to files of a different content type. I would suggest you inspect them by hand particularly at the start and end of a file. The file may be completely a link/html/javascript or may be a legitimate image file with a link trailing at the end of the file.

  4. Check for unusual file dates, sizes and permissions e.g. 777.

  5. Check cron jobs for unusual jobs. Someone compromising a system will often leave a back door to get back in again and again. Cron is a very popular way to do this if they managed to get that far.

  6. Check for the absence of files, you may not be able to have access to logs but the absence of them is equally a tell tail sign that someone has cleaned up after themself.

  7. Use search engines. Not surprising search engines are great at finding everything. Use directives like site: e.g. site:yoursitehere.com baddomain.com see if you get any hits.

  8. Often a link or redirect will be obfuscated so long javascript code with single letter variables should be analyzed carefully.

  9. Do a packet capture with a tool like Wireshark or tcpdump from a secure workstation to the site. Save it to file and search the file for a parts of the url.

  10. Check database records that may be queried or updated. The link could be injected in the database not the PHP.

  11. Don't exclude the client's workstation. Use a free online virus scanner if need be. Also check nslookup and see what that resolves to. Check browser extensions, clear cache and check hosts files.

To clean it up (if you are compromised) you really do need to go back to bare metal and reinstall. It is painful but is really the only way to be sure that you have got the whole lot.

To prevent it in the future you should be doing the following (although you may already be doing some of these):

  1. Harden servers, including using vendor recommendations on secure configurations, using up-to-date software. Apply tight security control such as permissions, password policies. Also see folder and file permission shared host advice.

  2. Implement quality control proceedures such as testing on low security environments, code review and testing.

  3. Have your web application / web site vulnerability tested by a professional certified tester at least once. Look for EC-Council, ISO 27001 and PCI certified testers. http://www.eccouncil.org/certification/licensed_penetration_tester.aspx

  4. Check out OWASP www.owasp.org and http://phpsec.org/projects/guide/2.html for web application security resources.

  5. Use Intrusion Prevention System (IPS) tools. However depending on your hosting provider you may have limitations on what you can use. Host based IPS tools should be ok if you have a dedicated virtual machine.

Hope that helps. Otherwise maybe you could provide more information about the systems you are running?

Community
  • 1
Bernie White
  • 2,866
  • 17
  • 18
7

As @Dgarcia said, a quick method is to use something like Tripwire or other tool which monitors files or the hashes of files to check for changes. This works to identify servers compromised by many types of attack.

  1. It may not work for ones where a rootkit has been installed that counteracts this process.
  2. It will not work for servers which have fallen prey to a memory-only compromise or one that doesn't touch the files you are monitoring.

For 1, your only option is a rebuild from scratch

For 2, your best option is a rebuild from scratch, as any compromise could implement backdoors that will break anything you try to fix, but other steps could be useful:

  • check your webserver and php versions and use these to search on an Advisories list for known exploits - this will help you identify areas which may have been compromised. Then
  • check your web application code
  • check your webserver configs
  • check the client's machine (for hosts file, DNS etc) as it may actually be the problem
Rory Alsop
  • 61,367
  • 12
  • 115
  • 320
  • How do I know that a rebuild will not contain the same vulnerabilities? Red hat enterprise keeps software safe(ish) I assume? The web app code could be at fault for sure (this could be the most likely route in). Is there a standard set of web server configs to check i.e. changing passwords and ports on a set number of services. We are also continuing to check the clients machine - he is no longer getting the issue, but we still do not know why it happened. – Boz Sep 23 '11 at 11:57
  • A rebuild may contain the same vulnerabilities, sure, but it gets rid of the compromised code - which could remain if you do not do a full wipe! There are questions on here on hardening servers, eg: http://security.stackexchange.com/q/77/485 – Rory Alsop Sep 23 '11 at 16:07
6

This is a tough question to answer because it is so broad. There are two categories of "hacks" in my book - minor and serious. I would class a rootkit in the serious category and your average script injection attack as minor. While with minor attacks you can clean them up, you can't be 100% certain you've removed them or closed all access to repeat the attack but you can be 99%-ish certain by analyzing the attack for key factors such as "Was this person a good programmer?" and "What was the person's intent?" Rootkits are nasty business. Removing a rootkit requires a complete wipe and restore. Detecting one remotely is nearly impossible - you have to have physical access to the machine and a boot disk in hand to be certain.

More importantly is prevention. The adage "an ounce of prevention is worth a pound of cure" is completely true in this context. Install software that lets you monitor various aspects of the system and sends daily or even hourly reports. Tripwire was mentioned, but there are other tools out there too. I recommend using a couple of different tools - homegrown ones are harder to locate and aren't difficult to author. You want to build a solid defense and limit access to the system. Don't just let anyone in the world have access to the SSH port (at least restrict it by IP address/small range of IPs). Stick a dedicated firewall in front of each server so there is an extra layer of protection. You don't want to let the box itself be the only line of defense. Only manage critical data with the server over SSH/SSL so everything is encrypted and free from prying eyes. Don't ever manage your servers from open WiFi networks.

A lot of sites use MySQL or a similar database. Detecting things like XSS attacks or other rogue data in a database isn't easy because there are schema-dependent issues. I haven't seen any solutions for this problem but I wouldn't doubt they exist.

Linders
  • 61
  • 1
2

A fast method is to have the md5 of the all the files that you know are healthy. If you suspect your site is behaving bad or as a regular inspection you can do the check over the files. If any of the md5 doesn't match you can diff the files and peruse the changes.

Obviously this does not work with dinamic files: logs, database dumps, etc. If you can't track the changes.

There are, of course, multiple methods (inspect logs...) and preventions, but this is an easy and fast way.

dgarcia
  • 476
  • 3
  • 6
  • We are moving to Github - so another method could be to run git on the live server and see what changes have been made through finding files to be committed. The issue however, is that I think the codebase is fine. I'm hoping to find out how the server has been compromised without using the codebase... – Boz Sep 23 '11 at 09:34
  • This also doesn't work in case of rootkits - as a rootkit can intercept system calls, it is trivial for it to subvert the check and return seemingly valid data when probed. It's a chicken-and-egg problem - you're asking the server to check if it is itself all right; thus the server can be programmed to lie to you. It may require offline analysis to discover a rootkit. – Piskvor left the building Sep 23 '11 at 10:10
  • @Piskvor in any case you can check the responses too. This is: You know what a right response looks like. Check the response with yours and examine diff. Otherwise, if you access the code and download the pages you can check the hash locally on your machine. – dgarcia Sep 23 '11 at 10:20
  • 6
    @dgarcia: Check what a [rootkit](http://en.wikipedia.org/wiki/Rootkit) is and why it's undetectable when written correctly. Let's say that there's one file in the system: `/is_hacked.txt`, containing the string "YES" or "NO". If the string is "NO", the computer is not hacked, else it is. Let's assume the system is hacked, and the content changed to "YES". However, part of the hack is a rootkit, serving also to disguise the fact. It has hijacked the `fopen()` system calls, so that any attempt to open (and read) a file needs to pass through the malicious code. – Piskvor left the building Sep 23 '11 at 10:27
  • 1
    ...So, if you just go ahead and try to check `/is_hacked.txt`, the malicious code will detect this, and instead of returning the actual content ("YES"), it will return whatever fake content it wants ("NO"), for every time you open that file. You can verify this all you want, to an outside observer, it will appear as if `/is_hacked.txt` contains "NO" - so there will be no difference seen by the diff. – Piskvor left the building Sep 23 '11 at 10:29
  • 1
    ...Useless? Not at all - this can be done for anything that happens on the system: Replace `/is_hacked.txt` with `/sbin/su` or `C:\WINDOWS\SYSTEM32\kernel32.dll`, and voila! No way to detect that someone messed with them, unless you boot the server from an uninfected source (e.g. a known-good CD). (Of course, such low-level hijinks are hard to pull off, so a badly-written rootkit may be detected by unusually high number of system crashes; see the infamous Sony DRM for an example) – Piskvor left the building Sep 23 '11 at 10:32
  • @Piskvor Thanks, I already knew what a rootkit is. You are assuming all hacked server are rooted and all rooted server has a undetectable rookit installed. Of course it can happens and when it happens you're screw. But seriously, do you think it happens all the time in all the hacked servers? There is not the ultimate method to know when a server is hacked but a least take some measures. If we thought your way then unplug the server, burn it and throw it to the sea. Poster was asking 'How to' not if is ALWAYS possible to know and be sure about then. – dgarcia Sep 23 '11 at 10:43
  • 3
    @dgarcia: Am not ;) All I'm saying is that once your server's been hacked, it is no longer trustworthy; you could scan it all you want, but you can never be *sure* that it's clean. Nuke from orbit and reinstall from known-good media. It's the only way to be sure. But of course, if you want to run a server that may or may not fully under your control, that's your privilege. (Btw no, this doesn't happen all the time, I'm not even remotely suggesting that. By comparison - do you use the airbags in your car *often*? Then why don't you just rip them out?) – Piskvor left the building Sep 23 '11 at 10:53
  • @Piskvor you're right. EOF – dgarcia Sep 23 '11 at 11:07
  • I'm assuming that root access is safe. Only really because I imagine that they would do something much nastier than adding a redirect in the code (on a pretty insignificant website). I've not heard of rootkit so this is a useful conversation - thanks, will check it out. – Boz Sep 23 '11 at 11:50
  • @BozKay Check logs, most of the time are bots exploiting known vulnerabilities massively and they simply append malicious content on your index or similar front file. In fact if you inspect 404 codes you should see interesting things – dgarcia Sep 23 '11 at 12:40