I know that a rootkit can hide processes from the OS by fooling around in the userspace. But can a rootkit also modify a processes metadata in a way that it won't even be recognized by a RAM forensic tool like volatility?
Im asking this because even volatility has to search by a specific pattern or method for the processes executed by the system the RAM dump came from.
But can a rootkit also modify a processes metadata in a way that it won't even be recognized by a RAM forensic tool like volatility?
The short answer is, not really.
Rootkits routinely change process metadata. Cosmetic changes are made to bits that aren't vital to the process running, and significant changes are made to the bits that make the process run. But if you make changes "in a way that it won't even be recognized" to a forensic tool, most likely you'll have made changes in a way that it won't be recognized as a process by the operating system.
In other words, the rootkit is constrained to make the process follow the rules enough to allow it to run - and by the same token, it will always follow the rules enough to allow Volatility to explore it.
In fact, a lot of what Volatility does is look for inconsistencies that are unusual in a normal process but are exactly the sort of tampering a rootkit would do. Oh, look, the pointer for a given system call points off to userspace, not kernelspace like it should! That's a giveaway.
Now, I said "not really..." Volatility has a number of predefined plugins that follow rules to find things that are out of place. Is it possible to write a new rootkit that isn't immediately obvious to these plugins? Sure. Clever people come up with new tricks all the time. But then the clever, clever developers and users of Volatility find these things - you can walk through memory manually with Volatility, you aren't constrained by what existing plugins think you should look for - and they write new plugins that recognize the new evasion.
SANS forensic training uses the tagline "Malware can hide, but it must run." For it to run, it's got to follow some of the OS's rules. And when it does that, it puts itself on the map for Volatility to identify and explore.
(minor exception: I have heard horror stories about hardware that visited China and came back with tampering that left it beaconing home with no evidence in system memory. If you let the enemy make changes to your hardware, then they can move beyond memory analysis. But that's a pretty rare and exclusive club.)
