Finding how a hacked server was hacked

Question

I was just browsing through the site and found this question: My server's been hacked EMERGENCY. Basically the question says: My server has been hacked. What should I do?

The best answer is excellent but it raised some questions in my mind. One of the steps suggested is to:

Examine the 'attacked' systems to understand how the attacks succeeded in compromising your security. Make every effort to find out where the attacks "came from", so that you understand what problems you have and need to address to make your system safe in the future.

I have done no system admin work so I have no idea how I would start doing this. What would be the first step? I know that you could look in the server log files but as an attacker the first thing that I would do would be errasing the log files. How would you "understand" how the attacks succeeded?

Answer 1

I'll start by saying this, if you have NO LOG FILES, then there is a reasonably good chance that you will NEVER understand where or how the attack succeeded. Even with full and proper log files, it can be extremely difficult to understand fully, the who, what, where, when, why and how.

So, knowing how important log files are, you begin to understand just how safe you have to keep them. Which is why companies do and should be investing in Security Information & Event Management or SIEM for short.

In a nutshell, correlating all of your log files into specific events (time-based or otherwise) can be an extremely daunting task. Just take a look at your firewall syslogs in debug mode if you don't believe me. And that's just from one appliance! A SIEM process puts these log files into a series of logical events which makes figuring out what happened, much easier to understand.

To begin to have a better understanding of the how, it's helpful to study penetration methodologies.

It's also helpful to know how a virus is written. Or how to write a rootkit.

It can also be extremely beneficial to setup and study a honeypot.

It also helps to have a log parser and become proficient with it.

It's helpful to gather a baseline for your network and systems. What's "normal" traffic in your situation vs. "abnormal" traffic?

CERT has an excellent guide on what to do after your computer has been hacked, most notably (which directly pertains to your specific question) the section on "Analyze the intrusion":

• Look for modifications made to system software and configuration files
• Look for modifications to data
• Look for tools and data left behind by the intruder
• Review log files
• Look for signs of a network sniffer
• Check other systems on your network
• Check for systems involved or affected at remote sites

There are many questions similar to yours that have been asked on SF:

1. How to do a post-mortem of a server hack
2. Strange Items in Hosts File and Netstat
3. is this a hack attempt?
4. How can I learn Linux from hacking or security point of view

This can be an extremely convoluted and involved process. Most people, me included, would just hire a consultant if it got any more involved than what my SIEM appliances could put together.

And, apparently, if you ever want to FULLY understand how your systems were hacked, you have to spend years studying them and give up women.

Answer 2

The answer to that one little bit can be a million miles wide and high, and unpicking what happened to a hacked server can be almost an artform as much as anything else, so again I'll give starting points and examples rather than a definitive set of steps to follow.

One thing to keep in mind is that once you have faced an intrusion you can audit your code, your systems administration/configuration and procedures with the knowledge that there definitely is a weakness there. That helps drive motivation more than searching for a theoretical weakness that may or may not be there. Quite often people put stuff online while knowing the code could have been audited a bit harder, if only we had the time; or the system locked down a bit more firmly, if only it wasn't so inconvenient; or procedures made a bit tighter, if only it wasn't such a bother for the boss to remember long passwords. We all know where our most likely weak spots are, so start with those.

In an ideal world you will have logs stored on a different (hopefully not compromised) syslog server, not just from servers but from any firewalls, routers, etc that also log traffic. There are also tools like Nessus available that can analyse a system and look for weaknesses.

For software/framworks from third parties, there's often best practice guides you can use to audit your deployment, or you might pay more attention to the security news and patching schedules and uncover some holes that might have been used.

Lastly, most intrusions do leave a spoor... if you have the time and patience to find it. "Drive by" script kiddie intrusions or break ins using hacking toolkits tend to focus on common weaknesses and can leave a pattern that points you in the right direction. The most difficult thing to analyse can be a manual directed intrusion (e.g. someone didn't want to hack "a" website, but instead wanted to hack "your" website specifically), and these of course are the most important things to understand.

For someone who really doesn't know where to start (or even for experienced people who have other duties) the first step is to probably hire someone with good experience of the above steps. Another advantage to that approach is that they will be looking at your setup without any preconceived notions or personal stake in the answers.

Answer 3

"I know that you could look in the server log files but as an attacker the first thing that I would do would be errasing the log files."

Depending on the type of compromise, the attacker may not have high enough privileges on the compromised server to be able to erase logs. It is also best practice to have server logs kept offbox on another server, to prevent tampering (exported automagically at certain intervals).

Beyond the compromised Server logs, there are still networking logs (Firewall, Router, etc) as well as authentication logs from the directory service, if there is one (Active Directory, RADIUS, ect)

So looking at logs is still one of the best things that can be done.

When dealing with a compromised box, sifting through logs is always one of my major ways of piecing together what happened.

-Josh