29

I have a Windows Server 2003 SP2 machine with IIS6, SQL Server 2005, MySQL 5 and PHP 4.3 installed on it. This is not a production machine, but it is exposed to the world via a domain name. Remote desktop is enabled on the machine and two administive accounts are active on it.

This morning I found that the machine had been logged off with an uknown user's name still in the login text box. Upon further investigation I've found that two windows users have been created, anti-virus has been uninstalled and a smattering of .exe files have been dropped into the C: drive.

What I would like to know is, what steps should I take to make sure that this does not happen again, and areas should I focus on to determine the avenue of entry. I've already checked netstat -a to see which ports are open, and nothing appears odd there. I did find unknown files in the data folder for MySQL which I'm thinking might have been the entry point but I'm not sure.

I'd really appreciate the steps to conduct a good post-mortem of a server hack so that I can avoid this in the future.

Post Investigation Review

After some investigation I think I found out what happened. First the machine has not been online during the time frame of August '08 to October '09. During that time frame a security vulnerability was discovered, the MS08-067 Vulnerability. "This is a remote code execution vulnerability. An attacker who successfully exploited this vulnerability could take complete control of an affected system remotely. On Microsoft Windows 2000-based, Windows XP-based, and Windows Server 2003-based systems, an attacker could exploit this vulnerability over RPC without authentication and could run arbitrary code." This vulerability was fixed with the KB958644 Security Update which came out in October 2008.

Because the machine was offline at the time and missed out on this update, I believe that this vulnerability was exploited soon after the machine came back online in October of '09. I found references to a bycnboy.exe program which has been described as a backdoor program which then creates much havoc on an infected system. Soon after the machine was online, automatic updates installed the patch which closed off the ability to have remote control of the system. Because the backdoor was now closed, I believe that the attacker then created physical accounts on the machine and was able to utilize the machine for a further week until I noticed what was happening.

After agressively pursing the malicious code, .exes and .dlls, removing self hosting websites and user accounts, the machine is now again in a working state. For the near future I will be monitoring the system and reviewing the server logs to determin if a repeat of the incident is happening.

Thank you for the information and steps that were provided.

Chris
  • 810
  • 1
  • 7
  • 10

1 Answers1

29

Doing a post-mortem is a black art in an of itself. It's a little different each time because really no two break-ins are the same. With that in mind, a basic overview of my recommended process is below, with a few specific notes toward your situation:

  1. Physically Disconnect the machine from the network. (Really. Do it now.)
  2. Optional Step: Make a binary image copy of the hard drive for future use.
  3. Make a copy of all log files, valuable data, etc. onto a removable hard drive
    • Optionally copy any "hacker tools" you find as well
  4. Begin the actual post-mortem. In your case:
    • Note any new or missing user accounts. See if their home folders have any "interesting" contents.
    • Note any new or missing programs/binaries/data files.
    • Check the MySQL logs first - Look for anything "unusual"
    • Check the rest of the server logs. See if you can find the new users being created, addresses they logged in from, etc.
    • Look for evidence of data damage or theft
  5. When you find the cause of the problem, note how to keep it from happening again.
  6. Wipe the server clean: Format and reinstall everything, restore your data & plug the original hole with your notes from #5.

You typically perform step 2 if you're going to be involving law enforcement. You perform Step 3 so you can review information after the server gets rebuilt without having to read the image copy you made in step 2.

How detailed step 4 gets depends on your goals: Just plugging the hole is a different kind of investigation than tracking down who stole some valuable bit of data :)

Step 6 is IMHO critical. You don't "fix" a compromised host: You wipe it and start over from a known good state. This ensures that you won't miss some nugget of nasty left on the box as a time bomb.

This is by no means a complete post-mortem outline. I'm marking this as community wiki as I'm always looking for improvements to the process - I don't use it often :-)

voretaq7
  • 79,345
  • 17
  • 128
  • 213
  • 3
    I have no experience doing anything like this, but Security Monkey's advice if you are going to image a machine for investigation is to pull the power cord, image the hard drive, then start investigating. (Security Monkey: http://it.toolbox.com/blogs/securitymonkey/) – MattB Jan 28 '10 at 20:53
  • 1
    Security Monkey is dead on - You want to freeze the machine cold (yank the power cord) when you go to image it. shutdown and/or startup can trip self destruct or cleanup code & yanking the power prevents that from happening before you can make your image. – voretaq7 Jan 28 '10 at 21:17
  • 2
    Also - I would say that you should not trust the results of "built in" commands on the hacked system like netstat (or dir, etc.) Again, I have no direct experience with this at the enterprise level, but I remember getting hacked on personal machines where part of the hack was to replace built in tools to mask what was really going on. – MattB Jan 28 '10 at 21:22
  • Yup - Anything on the compromised host is suspect & bringing in your forensic tools on a CD or other read-only device is a must. The chances of dir or similar shell built-ins on Windows being broken aren't great, but netstat, task manager, etc. are good targets to mess with. On *NIX systems you'll often find ps/top/netstat/etc replaced. Another old favorite is to backdoor the login command or PAM modules so the attacker always has a way to get back in. – voretaq7 Jan 28 '10 at 21:42
  • 4
    +1 step 6 is vital, you don't know if netstat is showing you the truth without analyzing the actual network traffic - and that in itself can be quite complicated and a test of patience... so, wipe it. It's not your box anymore. Analyze the image all you want, but wipe the damn machine ;) – Oskar Duveborn Jan 29 '10 at 17:09
  • 1
    I'd say that you are probably better off doing step #2 every time, since you're not entirely sure what you'll find during your investigation. Having binary images also means that you can have different people looking at different things, using a copy each. – Vatine Sep 26 '10 at 16:09
  • @Vatine Agreed - I've never skipped step 2 and can't think of any case where doing so would be worth the tradeoff (faster recovery versus having the images). If you have a separate machine to rebuild onto there's absolutely no excuse for skipping step 2 :) – voretaq7 Sep 27 '10 at 04:55
  • Is it possible that your data is also compromised and can be used as backdoor to get back in ? – Lamnk Nov 11 '10 at 16:20
  • @Lamnk data can't be compromised, but let's be clear on what `data` means: A plain text file (not a script of any kind, say the complete text of Pride and Prejudice) is "data". A PHP or ASP script is ***NOT*** data: It's a program that will be executed by your web server, and could be back-doored like any other program. A Word document is also ***NOT*** data, because any macros in it could be compromised (so unless you strip the macros out treat it as suspect). – voretaq7 Apr 27 '12 at 06:04