I'm trying to diagnose occasional spikes in memory usage on a Linux server. If the spikes are high enough the OOM Killer kicks in, which results in /var/log/messages containing a useful dump of the process list at the time.
However, I'm also experiencing spikes that aren't severe enough to cause the OOM Killer to kick in, but are still enough to cause heavy swap usage. Any way to automatically log the process table during such periods?
You could use the smem utility to gather information about our systems memory usage and collect it for later analysis. I have used it to find out what processes are using swap by taking a snapshot of the system swap usage at one time and then taking other snapshots later and comparing them.
Montoring tools like nagios or ganglia can collect system metrics and build graphs. More simple solution is to use one-liner and put it in infinite loop, cron etc...
free |awk '/Swap:/ {if(int($3/$2*100) > 50)system("echo Swap usage is over 50% && top -abn 1 > process_dump_`date +%Y-%m-%d_%H:%M`.txt")}'
