It's a good method of detection, to repeat:
- Memory check for hidden processes
- Network traffic check
- Filesystem check
The above solution is flexible, scalable and secure, however it's not your average scripting if large scale is involved, for which this is best suitable solution. But this doesn't mean it would not work on smaller scales, however effort might be too high. Possible implementation could be done in the following way:
- Memory checks built-in Virtualization Host
- Network traffic monitoring on the network layer (e.g. netflow)
- Filesystem checks on the network storage layer
Now the problem is, since it's quite custom job, you won't be able to really rely on the results before sufficient time and money are spent on the development of such solution, so it works, but on large scale, e.g. a lot of servers, let's say, 10.000 of them or more, that would make sense. Also if these servers run various things and are not the same kind of server. However, there can be done some trade-offs, you might want to evaluate some software to do some of it, and then it may be easier to do, see below.
If you want to try by yourself, try setting up Netflow (sflow) monitoring and with just this, you can successfully detect malicious traffic when it happens. Using ready-made software would be more efficient. So for example, you can select patters for packets to raise alarm. A lot of unusual traffic can be successfully detected not only related to hacking and that helps a lot. It's cheap, easy, multi-purpose and effective. The only thing is you need to invent proper patters (e.g. which packets you want to capture), these could be basically anything unusual but permitted, e.g. outgoing tcp connections, which upon initial detection can be whitelisted if not malicious.
Another easy way would be to run tcpdump, then netstat and run comparison. But this is less useful, it would only detect hidden network connections, and if you search internet, you might find some helpful scripts already.
Regarding filesystems, you can also use normal script to check for the consistency of the binaries (for example, using standard rpm or dpkg commands), and also check consistency of config files. Such script can be run from external host which connects over ssh, uploads the generated script (e.g. with latest checksums calculated from git repo were are configs), run the check and return result. Rundeck may be software to help you with this. Running command from external host assures you it is being run at all in first place, and it's simple and efficient method in the end.
Regarding RAM, this is the most difficult part, and you may skip this part for the moment because it's more advanced. Dumping and analyzing RAM is far from anything easy.
Also look for chkrootkit and rkhunter scripts and similar. There's free antivirus Clamav and paid solutions like Kaspersky. These do check for known backdoors, however, in average corporate / hosting environment there's 50 / 50 chance if it's custom patched sshd with secret key or a known backdoor.
And don't forget at assuring whatever kernel is not patched or there's no malicious module, but that's a different subject. I would try researching it and using same script to check both kernel and hidden processes. There might be some scripts doing that on the internet already.
In summary, Netflow (or Sflow, it's the very same thing) will help you a lot not only with security but with the network as well. It has nothing to do with servers, it works the way, that switch sends every n-th packet to the netflow receiver whre it is checked against predefined patterns and accounted for stats. It should provide you reports as well. Filesystem checks are easy to do with rpm and dpkg, and the rest can be checked from git or with build server. Note that package managers are using crystallographic signatures so you do not need to use LiveCD. Finally, check out the kernel and hidden processes and you are sorted. And before deploying software, scan it with "Kaspersky".
Take time, develop a plan, and slowly bit by bit it's not hard to make it all.
