Amongst many possibilities for local process monitoring (choose your poison) is monit, I do something like this in /etc/monit.d/system.conf
on centos machines;
check system localhost
if loadavg (1min) > 6 then alert
if loadavg (5min) > 6 then alert
if memory usage > 90% then alert
if cpu usage (user) > 90% then alert
if cpu usage (system) > 75% then alert
if cpu usage (wait) > 75% then alert
I imagine that you might want to be more aggressive with the checks, hence you might want to set the daemon to run checks more often, maybe every 30 seconds until you have determined the problem, and hence would use a /etc/monit.conf
something like this;
set daemon 30
set mailserver localhost
#set alert user@gmail.com but not on { instance }
set alert user@gmail.com
include /etc/monit.d/*
set httpd port 2812
allow 127.0.0.1
If monit does not provide enough information in the default mail alert, then you can have monit execute custom scripts on alert conditions like so;
check system localhost
if loadavg (1min) > 6 then exec "/bin/bash -c '/usr/bin/top -n1 -b | /bin/mail -s top-output userXXX@gmail.com'"
if loadavg (5min) > 6 then exec "/bin/bash -c '/usr/bin/top -n1 -b | /bin/mail -s top-output userXXX@gmail.com'"
if cpu usage (user) > 90% then exec "/bin/bash -c '/usr/bin/top -n1 -b | /bin/mail -s top-output userXXX@gmail.com'"
(obviously relies on mail command being setup, but you can use local root instead and just check it manually)