monit: check process without pidfile

Question

I'm looking for a way to kill all processes with a given name that have been running for more than X amount of time. I spawn many instances of this particular executable, and sometimes it goes into a bad state and runs forever, taking up a lot of cpu.

I'm already using monit, but I don't know how to check for a process without a pid file. The rule would be something like this:

kill all processes named xxxx that have a running time greater than 2 minutes

How would you express this in monit?

(*you should mark an answer here*) – ewwhite Mar 13 '15 at 11:40 — ewwhite, Mar 13 '15 at 11:40

ewwhite · Answer 1 · 2013-03-03T15:56:43.903

86

In monit, you can use a matching string for processes that do not have a PID. Using the example of a process named "myprocessname",

check process myprocessname
        matching "myprocessname"
        start program = "/etc/init.d/myproccessname start"
        stop program = "/usr/bin/killall myprocessname"
        if cpu usage > 95% for 10 cycles then restart

Maybe if you check to see if CPU load is at a certain level for 10 monitoring cycles (of 30-seconds each), then restart or kill, that could be an option. Or you could use monit's timestamp testing on a file related to the process.

edited Mar 03 '13 at 15:56

answered Jun 30 '11 at 03:12

ewwhite

194,921
91
434
799

1

Be careful: it won't work it there are more than one process – ruX Feb 29 '16 at 16:01
1

you can use a regexp: matchin "otherstuff.*myprocessname" – user174962 Jun 08 '17 at 12:32
@ruX: what happens if multiple related processes match? How can they handled? – kontextify Oct 31 '19 at 10:25
It takes the first match. – ewwhite Oct 31 '19 at 14:02

score 5 · Answer 2 · answered May 17 '11 at 03:25

5

There no ready-to-use tool with that functionality. Let assume you want to kill php-cgi scripts, that runs longer than minute. Do this:

pgrep php-cgi | xargs ps -o pid,time | perl -ne 'print "$1 " if /^\s*([0-9]+) ([0-9]+:[0-9]+:[0-9]+)/ && $2 gt "00:01:00"' | xargs kill

pgrep will select processes by name, ps -o pid,time prints runtime for every pid, and then analyse line, extract time from it, and print pid if time compares with defined one. result passed to kill.

answered May 17 '11 at 03:25

datacompboy

673
2
7
16

the process runnig for **very** long time gets strange runtime (62-13:53:05), so the regexp parsing running time should be ([-0-9]+:[0-9]+:[0-9]+) - look at the minus at the beginning of the expression. – andrej Jun 26 '14 at 11:43

score 3 · Answer 3 · answered May 17 '11 at 05:17

I solved this exact issue with ps-watcher and wrote about it on linux.com a few years back. ps-watcher does allow you to monitor processes and kill them based on accumulated run time. Here's the relevant ps-watcher configuration, assuming your process is named 'foo':

[foo]
  occurs = every
  trigger = elapsed2secs('$time') > 1*HOURS && $ppid != 1
  action = <<EOT
  echo "$command accumulated too much CPU time" | /bin/mail user\@host
  kill -TERM $pid
EOT

[foo?]
   occurs = none
   action = /usr/local/etc/foo restart

The key is the line

trigger = elapsed2secs('$time') > 1*HOURS && $ppid != 1`

which says 'if accumulated process time is > 1 hour AND I'm not the parent process, restart me.

So, I realize that answer doesn't use monit, but it does work. ps-watcher is lightweight and simple to set up, so there's no harm running it in addition to your monit setup.

score 3 · Answer 4 · edited Mar 03 '17 at 08:30

3

Monit can do this as of version 5.4:

if uptime > 3 days then restart

See: the project CHANGES file

edited Mar 03 '17 at 08:30

gertas

1,007
10
11

answered Feb 26 '13 at 20:22

David Radcliffe

146
3

score 0 · Answer 5 · answered May 29 '12 at 03:06

0

You could work this into monit as an exec statement.

    if [[ "$(uname)" = "Linux" ]];then killall --older-than 2m someprocessname;fi

answered May 29 '12 at 03:06

Jodie C

733
6
9

monit: check process without pidfile

5 Answers5

Linked