What are main steps doing forensic analysis of linux box after it was hacked?
Lets say it is a generic linux server mail/web/database/ftp/ssh/samba. And it started sending spam, scanning other systems.. How to start searching for ways hack was done and who is responsible?
Here are some things to try before rebooting:
First of all, if you think you might be compromised unplug your network cable so the machine can't do further damage.
Then, if possible refrain from rebooting, as many traces of an intruder can be removed by re-booting.
If you thought ahead, and had remote logging in place, use your remote logs, not the ones on the machine, as it's all too easy for someone to tamper with the logs on the machine. But if you don't have remote logs, examine the local ones thoroughly.
Check dmesg, as this will be replaced upon reboot as well.
In linux it is possible to have running programs - even after the running file has been deleted. Check for these with the command file /proc/[0-9]*/exe|grep "(deleted)". (these disappear on reboot, of course). If you want to save a copy of the running program to disk, use /bin/dd if=/proc/filename/exe of=filename
If you have known good copies of who/ps/ls/netstat, use these tools to examine what is going on on the box. Note that if a rootkit has been installed, these utilities are usually replaced with copies that won't give accurate information.
That totally depends on what was hacked, but in general,
Check the timestamps of files that were modified inappropriately, and cross-reference those times with succesful ssh (in /var/log/auth*) and ftp (in /var/log/vsftp* if you're using vsftp as server) to find out which account was compromised and from which IP the attack came.
You can probably find out if the account was brute-forced if there were a lot of unsuccessful login attempts on the same account. If there were no or only a few failed login attempts for that account, then probably the password was discovered in some other ways and the owner of that account needs a lecture on password safety.
If the IP is from somewhere nearby it could be an "inside job"
If the root account was compromised, of course you're in big trouble, and I would, if possible, reformat and rebuild the box from the ground up. Of course you should change all passwords anyway.
You have to check all the logs of running applications. For example, Apache logs may tell you how a hacker could execute arbitrary commands on your system.
Also check if you have running processes that scan servers or send spam. If it is the case, the Unix user they are running from can tell you how your box was hacked. If it's www-data then you know it's Apache, etc.
Be aware that sometimes some programs like ps are replaced...
Take a memory dump and analyze it with a memory forensics tool, such as Second Look.
Naaah!
You should shut down, connect the hard disk to a read only interface (it's a special IDE or SATA, or USB, etc... interface that doesn't allow any writes, something like this: http://www.forensic-computers.com/handBridges.php ) and do an exact dupe with DD.
You may do it to another hard drive, or you may do it to a disk image.
Then, store in a profer and totally safe place that hard disk, is the original proof without any tampering!
Later, you can plug that cloned disk, or image in your forensic computer. If it's a disk, you should plug it through a read only interface, and if you are going to work with an image, mount it 'read only'.
Then you can work on it, again and again without changing any data...
FYI, there are "hacked" systems images on the internet for practice, so you can do forensics "at home"...
PS: What about the hacked system bringed down? if I think that system is compromised, I wouldnt leave it connected, I would put a new hard disk there, and restore a backup or put a new server in production until the forensics finish...
I havent read the other responses, but I would make a ghost image of it to preserve the evidence and only examine the image....maybe...
I highly recommend reading "Dead Linux Machines Do Tell Tales", an article by The SANS Institute. It's from 2003, but the information is still valuable today.
You should ask yourself first: "Why?"
Here's are some reasons that make sense to me:
Going beyond that often doesn't make sense. The police often don't care, and if they did, they would impound your hardware and do their own forensic analysis.
Depending on what you find out, you may be able to make your life alot easier. If an SMTP relay gets compromised, and you determine that it was due to a missing patch exploited by an outside party, you're done. Reinstall the box, patch whatever needs patching and move on.
Often when the word "forensics" is brought up, people have visions of CSI and think about figuring out all sorts of excruciating details about what happened. It can be that, but don't make it a huge lift if you don't have to.
