2

When Linux starts swapping you're basically doomed. Very soon the system won't react to any input any more, but happily swap on until the end of days...

Can you think of a command that holds all processes whatsoever, thus (and while) allowing you to open a clean shell where you can examine the source of the problem and kill the process which ate up all the memory? (I guess this won't be easy, because as the memory is probably completely filled up you'd need to swap out some more memory to gather space for opening a shell, on the other hand all other swapping processes must be stopped.)

If you tied such a command to a hot key then maybe you can use this as an emergency button saving you a lot a time. Any ideas if this is possible at all? Has somebody tried something like this before? If one could realize this it would be a cool feature :)

fuenfundachtzig
  • 185
  • 7

5 Answers5

6

Magic SysReq as mentioned in the other answers, is really all you have. If you only have SSH access you can trigger SysReq like so:

# Enable Magic SysReq since most systems default this off
echo 1 > /proc/sys/kernel/sysrq
#call OOM Killer to try to free up memory
echo f > /proc/sysrq-trigger

Making an executable script out of this and executing this as soon a you notice the swapping might give you a chance. Disabling swap is an option too if you have sufficient memory to handle temporary memory spikes, or if you are OK with killing random programs when you run out of memory.

On a related note, if you have a remote system with rocketing load and you need to kill it dead you can use this to force a reboot without cleanly shutting down:

echo 1 > /proc/sys/kernel/sysrq
#Trigger BIOS reset
echo b > /proc/sysrq-trigger
Mark Porter
  • 991
  • 1
  • 5
  • 12
4

One (suboptimal!) solution is to provide less swap.

The idea is that about the time you throw up your hands in despair the OOM Killer kicks in.

Discussion

My experiences using linux on limited memory machine suggests that

For interactive use

  • Up to 100-and-a-few% RAM in use the GUI will do
  • Using from 100 to 150% of RAM works pretty well, though you may experience some command line lag
  • using between 150 and 175--200% of RAM is slow but responsive enough for you to interact on the command line and intelligently kill unnecessary stuff.
  • Above that even with patience it's near hopeless to fix it by hand.

Some special case will allow you to extend these limits. Big, but rarely active background processes will "sit" on memory but will only occasionally become active, so they don't contribute a lot to usability problems.

Anyway, the plan is to tune the available swap so that as soon as the machine is so busy you can't reasonably kill stuff by hand the OOM killer kicks in.

This is not optimal (despite the multiple heuristics available for the OOM killer) it is hard to be sure it will pick "right".

Dennis Williamson
  • 60,515
  • 14
  • 113
  • 148
dmckee --- ex-moderator kitten
  • 1,403
  • 1
  • 12
  • 13
  • +1 Good thinking, you can set `/proc/sys/vm/oom_dump_tasks` to non-zero as well so the information gets dumped so you can go find the offending process after the fact. – Kyle Brandt Apr 15 '10 at 15:07
2

Magic SysRq is the closest thing to your button that I know...

minaev
  • 1,549
  • 12
  • 13
1

Never tried this!, but maybe:

killall -SIGSTOP <main commands>

See man killall to see how you might tailor it your needs. In general you might want to run process with ulimit to keep them under control, and monitor memory usage with something like Nagios (See the gazillion 'What monitoring for ...' questions for alternatives).

Kyle Brandt
  • 82,107
  • 71
  • 302
  • 444
  • Wouldn't that also stop shell, sshd, xwindows, etc? – Zypher Apr 15 '10 at 15:05
  • Zypher: Ya, you can limit it to certain users etc... I like dmckee's option more anyways. – Kyle Brandt Apr 15 '10 at 15:08
  • I know this question is asking about Linux, but please keep in mind that the `killall` command on some commercial Unices (HP-UX et al.) will kill _all processes on the system_. –  Aug 07 '12 at 17:09
1

You can also try with nice -20 bash and you should get a responsive shell...

Dan Andreatta
  • 5,384
  • 2
  • 23
  • 14