Monit: how do you stop monit from running the exec statement every time the test fails?

Question

How do you stop monit from running the exec statement every time the test fails? The statement in my monitrc is:

check filesystem tmpfs with path /var                                           
    if space > 90% then exec "/usr/bin/logger -p daemon.crit 'MAJOR: space test'"

This seems weird because someone else asked a question in which he was doing an alert and it had the behaviour I wanted. I'm ready to start choke slamming linux boxes.

Edit: here is the opposite case Repeat monit alerts

Is it because he is using alert not exec?

You control this with the `cycles` directive. How often would you like for this to be logged? — ewwhite, Mar 11 '14 at 20:35

score 5 · Answer 1 · answered May 12 '16 at 08:37

5

The behaivor of 'exec' action (with repeating each time) changed from Monit version 5.16

https://mmonit.com/monit/changes/

The exec action is now executed only once on state change, the same way as the alert action.

answered May 12 '16 at 08:37

Dmitry Dubovitsky

111
1
7

krisFR · Accepted Answer · 2014-03-12T08:06:00.343

I had to deal with a similar issue a few times ago.

The fact is that monit is not able to do this, as far as i know.

With monit you can deal with X times and/or Y cycles directives, but more or less quickly the exec action will be triggered more than once, depending on time you spend to fix the issue.

So, finally, i've decided to write my own check script to handle all the logic, based on flags.

I'm going to share this with you, then you take or not, it's up to you.

First : Write the script to monitor FS usage, let's say /root/check_fsspace.sh :

#!/bin/sh

myFS=/var
myTreshold=90
flagFile=/tmp/flag

spaceused=$(df -h | grep "$myFS" | tr -s " " | cut -d" " -f5 | cut -d"%" -f1)

if [ $spaceused -gt $myTreshold ]; then
  if [ ! -f $flagFile ]; then
     touch $flagFile
     exit 1
  else
     exit 0
  fi
fi

if [ $spaceused -le $myTreshold ]; then
   rm -f $flagFile
   exit 0
fi

Here i assume you can understand the script. If not, tell me, i will explain it.

Second : Setup your monit service definition :

check program check_fs with path "/root/check_fsspace.sh"
  if status != 0 then exec "/usr/bin/logger -p daemon.crit 'MAJOR: space test'"

I ended up going with this. The X cycles directive would be useful for reducing the repeating messages to an acceptable amount, but it would also delay the initial trigging? In my case the disk fills very quickly once something goes wrong. — Terminal, Mar 12 '14 at 16:03

score 0 · Answer 3 · answered Mar 12 '14 at 07:13

You control this with the cycles directive. How often would you like for this to be logged?

Assuming a Monit daemon check interval of 60 seconds, you can say something like, "if this fails for X number of cycles, then execute script".

But Monit is simple... It will alert upon failure each time you hit this disk threshold. That's by design. If you are hitting the threshold often enough for this logging action to be annoying, try changing the threshold.

Monit will log its status each cycle to your messages log.

Mar 12 00:07:06 yo-mama monit[8577]: 'ppro' space usage 92.4% matches resource limit [space usage>85.0%] 
Mar 12 00:08:06 yo-mama monit[8577]: 'ppro' space usage 92.4% matches resource limit [space usage>85.0%] 
Mar 12 00:09:06 yo-mama monit[8577]: 'ppro' space usage 92.4% matches resource limit [space usage>85.0%] 
Mar 12 00:09:07 yo-mama monit[8577]: 'ppro' space usage 92.4% matches resource limit [space usage>85.0%]

It will alert each time the threshold is crossed:

Resource limit matched Service ppro

    Date:        Wed, 12 Mar 2014 00:09:07
    Action:      alert
    Host:        yo-mama
    Description: space usage 92.4% matches resource limit [space usage>85.0%]

Your faithful employee,
Monit

Monit: how do you stop monit from running the exec statement every time the test fails?

3 Answers3