Monit: use content MATCH on “check file …” if match “xxx”

Question

I've asked this question around a week a go at StackOverflow but no answer just yet, probably is no possible but don't know where to look for that answer, I hope someone can help over here.

I'm using monit to scan logs for errors and then push those alert into a monitoring system called DataDog...

All seem to work as expected but now I've need to grab what is causing the alarm.

Using a very simple rule I'm able to catch line on log that provoke this error and run the specific script to alert; until here all ok:

montirc file:

check file testmonit with path /var/log/testmonit.log
      if MATCH "(ERROR.*)" then
         exec "/usr/bin/python /opt/scripts/bin/dd_notify.py test-error"

This config make what I want, it actually raise the alarm I wanted

But now I need to know "What caused this alarm"; so for example if this line appears on the log:

ERROR failure to complete process due lock file....

On monit logs I can see:

[UTC Mar  6 11:59:08] error    : 'testmonit' content match [ERROR failure to complete process due lock file....]
[UTC Mar  6 11:59:08] info     : 'testmonit' exec: /usr/bin/python

Which is perfect... what I want is capture this:

[ERROR failure to complete process due lock file....]

To send this string into my monitoring system (DataDog) I can't find any documentation that actually allows me to use the MATCH content, or groups (which I can see is being supported by the MATCH regex)

So in short:

There's any monit variable(like $DESCRIPTION for mail) that refer to MATCH line that trigger the rule?

(I've tried $DESCRIPTION, $HOST...etc but this seems to work only for email)

I'd a look into google several times (and also here) but I can't find the answer...

Please if you think this has been addressed before feel free to point me to right direction.

An update:

Sorry I forgot to say that I'm running this on:

 Ubuntu 16.04 LTS and 
 Ubuntu 12

Monit version is:

This is Monit version 5.25.1 Built with ssl, with ipv6, with compression, with pam and with large files Copyright (C) 2001-2017 Tildeslash Ltd. All Rights Reserved.

Following DevOps reccommendation, I've upgraded monit version.

And tried to use MONIT_DESCRIPTION or $MONIT_DESCRIPTION without success rule file is like this:

check file pd-error with path /var/log/testmonit.log if CONTENT = "ERROR" then exec "/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error " $MONIT_DESCRIPTION

What I want is pass the content that has been matched as an additional argument for dd_notify.py program;

but what i get is (which is the result of executing dd_notify.py):

{ "ALARM": { "pd_error": 67 }, "MESSAGES": { "pd_error": "$MONIT_DESCRIPTION" } }

what I want is $MONIT_DESCRIPTION content which is actually:

[UTC Apr 3 21:53:22] debug : 'pd-error' Pattern 'ERROR' match on content line [Apr 3 21:52:30 ams01 MainProcess[1376]: cel ery.worker.job ERROR Task tasks.telemetry.gather_and_send_telemetry_info[f090d579-9ec2-40e5-9fb2-91436eb4fc8a] fail]

But I'm not having any luck at the moment... what I'm missing here?

Thank you.

Answer 1

Thanks for DevOps for put me on the rigth track to finish with this issue, I've finally succeed on what I wanted to do, and also can explain (from my understanding) why it wasn't working for me before

Thing is as DevOps says, there's a variable MONIT_DESCRIPTION that in fact has the error string, but this variable is only "reachable" at bash environment.

As I was doing:

check file pd-error with path /var/log/testmonit.log
if CONTENT = "ERROR" then exec "/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error "

and from dd_notify.py I was trying to do :

error = os.environ['MONIT_DESCRIPTION']

but I always got 'key_error' because 'MONIT_DESCRIPTION' wasn't accesible from python

then I tried to call my program using a bash wrapper like:

check file pd-error with path /var/log/testmonit.log
  if CONTENT = "ERROR" then exec "/bin/bash /opt/scripts/bin/wrapper.sh"

and on 'wrapper.sh' code I have:

/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error

and then I got what I was looking for:

{
  "ALARM": {
    "pd_error": 294
  }, 
  "MESSAGES": {
    "pd_error": "content match:\nMay 16 18:07:08 ams01 MainProcess[1358]: celery.worker.job ERROR Task fds.realtime.tasks.telemetry.gather_and_send_telemetry_info[abe35540-55ef-40db-984a-
12287f5648ab] raised unexpected: ConnectionError()#012Traceback (most recent call last):#012  File \"/usr/lib/python2.7/dist-packages/celery/app/trace.py\", line 240, in trace_task#012   
 R = retval = fun(*args, **kwargs)#012  File \"/usr/lib/python2.7/dist-packages/celery/app/trace.py\", line 438, in __protected_call__#012    return self.run(*args, **kwargs)\n...\n"
  }
}

Which is great!

So basically I was not able to access monit enviroment variable from Python... so I wrapped into a bash script and then got it...!

Thank you!

Answer 2

Jsut tried with Monit 5.23.0 and there is an environment variable for it.

MONIT_DESCRIPTION=content match:
[ERROR failure to complete process due lock file....]
[ERROR failure to complete process due lock file....]

Monit will output every occurrence of the matching content

Also the syntax as changed in Monit 5.16.0 but old is still functional. Changelog is available here : https://mmonit.com/monit/changes/

Also be aware that you can change the various limits of Monit https://mmonit.com/monit/documentation/monit.html#LIMITS

If you are willing to upgrade, Monit provides precompiled generic binaries that I'm using to have a newer versions than the ones in the Ubuntu repositories.