Does anyone have a general step by step list on how to try discover rootkits on a Linux or Solaris server?
I'm looking for to to manually find the rootkit, not by automated software.
For example:
The number of options for places to hide, places to hook etc is so vast that any step-by-step list for manual checking is going to be incomplete.
And of course, there is a whole other story of kernel rootkits, which leave a very small amount of traces in the system, that can usually be discovered with forensic analysis if implemented correctly.
What you ask for is simply wrong: Even if I start listing things to check, it is going to be very misleading for anyone reading it. The average user's best bet is to use tools like chkrootkit and rkhunter, that while incomplete and easily bypassed, contain the 'list of things to check' you ask for and go through it in an automatic way.
Disclosure: My previous area of research was rootkit-related and I have developed various rootkits and rootkit detection software for linux platforms in the past.
Does anyone have a general step by step list on how to try discover rootkits on a Linux or Solaris server?
Step one: Disconnect the machine in question from all communications channels: Ethernet, WiFi, Bluetooth, serial, Firewire, USB, audio cable, Infrared, parallel. Only the power cable, keyboard cable, and monitor cable should remain connected. Check for RF emissions with a calibrated RF spectrum analyzer. Only proceed when you have verified communication disconnection.
Sep two: If the system has a CD-ROM or DVD-ROM drive and is capable of booting from the CD-ROM, on a know clean system prepare a cold boot RAM attack CD-ROM. Otherwise proceed to Step three. Freely available tools for cold boot attack: msramdump and Princeton.
Step three: If you prepared a cold boot attack disk in Step two do not power off the machine. Disconnect the hard drive(s). As each drive is disconnect connected label it with a brightly colored label. On the label write 'Infected', the time and date, the name of the system it was remove from. Place each drive into an appropriate storage container.
Step four: If you did not prepared a cold boot attack disk in Step two precede to Step six. Otherwise insert the cold boot attack disk and reboot the machine. Capture the contents of the RAM and save it a CD-R or DVD-R. Make sure to finalize the disk for every recording. After saving contents of RAM power down the machine.
Step five: On a separate know good machine analyze the RAM dump. Check for executable file format signatures. Check for virus signatures. Compare to a dump of a known good machine running the same operating system.
Step six: On your analysis system disable any automount, autoplay, plug and play, or any service that takes acts on detection of a new drive. Verify that the system takes no action on new drive attachment by attaching a known good drive of the same physical connector type as the drives from the infected system. If the infected system had drives of more than one physical connect type, then test the analysis system with a known good drive for each physical connector type. If there are multiple drives, only attach one drive at a time to the analysis machine. Create an image of each drive and then store the drives in appropriate containers.
Step seven: Begin analysis on the image of the primary boot drive. Using static analysis, check the boot sequence. Perform differential binary analysis on the primary operating system initialization components, including file meta-data structure and creation order.
Step eight: Begin analysis of security critical libraries.
Step nine: Begin analysis of system administration tools, logs, and configuration files.
Step ten: Begin analysis of services and daemons.
Step eleven: Begin analysis of applications.
Step twelve: Begin analysis of user files and remaining files.
Places to look for suspicious files?
Every file should be considered suspicious until you have verified that it is not.
Search commands to pick up weird file behavior?
No commands should be run on a potentially compromised machine. As soon as you suspect a system may be compromised it should be disconnected for all communication channels. If the system is not a candidate for a cold boot attack on the RAM, then it should be powered off immediately and the power cord removed.
Weird looking traffic picked up from sniffer?
No traffic should be sent or received on a potentially compromised machine.
Checking file integrity on important commands in order to know if you can trust it or not?
File integrity may be check on an analysis machine using read only drive images if you have a known good file integrity database. Every file in the database should be checked.
This question presupposes all of the wrong information. First of all, you need to know what to trust before you come into a rootkit investigation. If you didn't have strong read-only or write-once media operating correctly for the right reasons, then you simply can't trust file integrity monitoring (these should be statically compiled on CDR or lock-protected SD material). Logs should be written to WORM drives.
Memory analyzers via LKMs such as 99lb have been available for years. In the past 4-5 years, these have moved towards the hypervisor. Too much research has been done in this area to describe it all in one post. However, it appears that the VMware server virtualization direction is by use of their proprietary VShield technology. VMware is going to abandon the VMsafe API in short order. Open-source implementations/research can be found in XenAccess and InvisibleThings.
If you want a checklist, you might need to check out the book Unix and Linux Forensic Analysis DVD Toolkit.
