We have set up zabbix to monitor our infrastructure, and our security team denied us to run custom scripts through zabbix. This makes our lives a bit harder to find solutions which don't require customized scripts.
I've been researching for a while how can we monitor spawn rate of child processes but with no luck so far.
As far as I know, zabbix checks don't include spawn rate checks, can you suggest how can we achieve this?
You need a number of forks - you can get it by using vmstat -f (number of forks since boot):
vmstat -f
But you can't run any command, because security (OT I know, their default answer is always: no, it's not possible :-D). However you can create Zabbix module, which will execute your command => that's not clever idea. The better option is to read number of forks directly from the kernel counter. You can test it in cmd (it's a equivalent of vmstat -f):
cat /proc/stat | grep ^processes
Your Zabbix module should to read /proc/stat, parse number from the line, which starts with processes and then Zabbix agent uses it as a metric value. Piece of cake for C developer. See dummy Zabbix module and just extend it.
You need a rate so, then configure item in Zabbix UI and store it as Delta (speed per second), so finally it'll be forks/second. Probably Zabbix will handle also overflow. Performance will be amazing, because /proc/stat is not real file, so no IOps. No forks only CPU and mem will be required. Also you can change update interval of item and the result rate will still be correct.
