Ubuntu how to detect if OOM killer have run

Question

We use Nagios to monitor our servers. Now.. linux uses all the RAM we give them so there is no way to detect if we are about to run out of memory. We have a couple of times had some server crashes due to excessive memory usage.

Is there any way to detect if ubuntu(linux) launches the OOM killer? I know its written in the log files but is there anyway to detect when it is launched?

/RJ

score 2 · Answer 1 · edited Jul 31 '19 at 14:15

Perhaps by logtail.

oom-killer leaves a trace of run in syslog and dmesg. You can put a cron job or a script to invoke via some monitoring tool that "logtails" syslog and warn you if there are lines like the one below between runs

Feb 24 13:35:29 hostname kernel: [22472693.216224] foobar:23 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

dmesg isn't "logtailable" but dmesg -c clears the dmesg buffer after printing so the script can simply call dmesg -c | grep <string> | wc -l A value of 1 or greater means that oom-killer was invoked since the las run/boot. But this will destroy your dmesg stored info.

score 1 · Accepted Answer · answered Aug 08 '12 at 12:51

1

You can not see when OOM passes by, the only way is to check the logs and hope it's there (sometimes it doesn't get recorded).

There are tools to check memory usage with Nagios, I personally use check_mem.pl

answered Aug 08 '12 at 12:51

Lucas Kauffman

16,818
9
57
92

But how can it monitor RAM usage when linux uses all RAM for cache? http://www.linuxatemyram.com/ When I look at a server with 4GB RAM it has almost none free RAM when looking in top. – Ronnie Jespersen Aug 09 '12 at 21:07
There is a switch for that, refer to its manual, it clearly states how you can program it to see the real memory usage and considering how much cache is used. – Lucas Kauffman Aug 09 '12 at 22:41
Do you have an example under which conditions oom-killer will not be recorded? I was convinced that oom-killer will *always* log to dmesg and - provided `init` and `syslog` are not badly misconfigured - will *always* end up in logs. – anx Mar 29 '18 at 15:39

score 1 · Answer 3 · answered Mar 29 '18 at 14:51

1

Create /etc/rsyslog.d/30-oom-killer.conf as below:

if ($msg contains [
    "oom",
    "kill"
]) then {
    /var/log/oom-killer.log
    stop
}

Add /var/log/oom-killer.log to /etc/logrotate.d/rsyslog.

answered Mar 29 '18 at 14:51

Antonio Petricca

225
1
4

score 0 · Answer 4 · answered Aug 08 '12 at 12:49

Obviously it is launched when you don't have enough free virtual memory and an appication requests a memory region from the kernel. So what you really should monitor is your free memory size. And load average is a subject to monitor too because it always gets high when there is not enough free memory.

Ubuntu how to detect if OOM killer have run

4 Answers4