4

I'm using cgroup to partition my processes and I'm getting Out Of Memory messages in my kernel logs.

However, I can't find which partition causes them. I've checked the memory controller cgroup but there are no obvious ways to use it.

The problem is, by the time I see 'task killed' message in system logs the task is dead, its /proc entry is gone and cgroup's tasks file doesn't have the pid of the killed task.

Cyberax
  • 249
  • 1
  • 5
  • I thought you wanted to know what caused the OOM. Your problem seems to be about figuring out how the system responded to the OOM. These are two different things. – David Schwartz Apr 21 '12 at 03:08
  • I need to be able to recover from OOM (which might be caused by a variety of reasons). – Cyberax Apr 23 '12 at 19:08

1 Answers1

8

Answering my own question. I've used SystemTap to hook into the OOM killer:


#!/usr/bin/env stap
%{
#include <linux/cgroup.h>
%}

function find_mem_cgroup:string(task:long) %{
    struct cgroup *cgrp;
    struct task_struct *tsk = (struct task_struct *)((long)THIS->task);

    /* Initialize with an empty value */ 
    strcpy(THIS->__retvalue, "NULL");

    cgroup_lock();
    cgrp = task_cgroup(tsk, mem_cgroup_subsys_id);
    if (cgrp)
        cgroup_path(cgrp, THIS->__retvalue, MAXSTRINGLEN);
    cgroup_unlock();
%}

probe kernel.function("oom_kill_task") {
    cgroup = find_mem_cgroup($p)
    exename = kernel_string($p->comm)
    printf("pid\t%d\tmem-cgroup\t%s\texe-name\t%s\n", $p->pid, cgroup, exename)
}

Works like this:

cyberax@cybnb:~/work/shell$ sudo stap -g oom.stap
pid    3966    mem-cgroup    /task1/1/    exe-name    oom_generator.p
Cyberax
  • 249
  • 1
  • 5