I've asked this question around a week a go at StackOverflow but no answer just yet, probably is no possible but don't know where to look for that answer, I hope someone can help over here.
I'm using monit to scan logs for errors and then push those alert into a monitoring system called DataDog...
All seem to work as expected but now I've need to grab what is causing the alarm.
Using a very simple rule I'm able to catch line on log that provoke this error and run the specific script to alert; until here all ok:
montirc file
:
check file testmonit with path /var/log/testmonit.log
if MATCH "(ERROR.*)" then
exec "/usr/bin/python /opt/scripts/bin/dd_notify.py test-error"
This config make what I want, it actually raise the alarm I wanted
But now I need to know "What caused this alarm"; so for example if this line appears on the log:
ERROR failure to complete process due lock file....
On monit logs I can see:
[UTC Mar 6 11:59:08] error : 'testmonit' content match [ERROR failure to complete process due lock file....]
[UTC Mar 6 11:59:08] info : 'testmonit' exec: /usr/bin/python
Which is perfect... what I want is capture this:
[ERROR failure to complete process due lock file....]
To send this string into my monitoring system (DataDog) I can't find any documentation that actually allows me to use the MATCH content
, or groups (which I can see is being supported by the MATCH regex)
So in short:
There's any monit variable(like $DESCRIPTION for mail) that refer to MATCH
line that trigger the rule?
(I've tried $DESCRIPTION, $HOST...etc but this seems to work only for email)
I'd a look into google several times (and also here) but I can't find the answer...
Please if you think this has been addressed before feel free to point me to right direction.
An update:
Sorry I forgot to say that I'm running this on:
Ubuntu 16.04 LTS and
Ubuntu 12
Monit version is:
This is Monit version 5.25.1 Built with ssl, with ipv6, with compression, with pam and with large files Copyright (C) 2001-2017 Tildeslash Ltd. All Rights Reserved.
Following DevOps reccommendation, I've upgraded monit version.
And tried to use MONIT_DESCRIPTION or $MONIT_DESCRIPTION without success rule file is like this:
check file pd-error with path /var/log/testmonit.log
if CONTENT = "ERROR" then exec "/usr/bin/python /opt/scripts/bin/dd_notify.py pd_error " $MONIT_DESCRIPTION
What I want is pass the content that has been matched as an additional argument for dd_notify.py program;
but what i get is (which is the result of executing dd_notify.py
):
{
"ALARM": {
"pd_error": 67
},
"MESSAGES": {
"pd_error": "$MONIT_DESCRIPTION"
}
}
what I want is $MONIT_DESCRIPTION content which is actually:
[UTC Apr 3 21:53:22] debug : 'pd-error' Pattern 'ERROR' match on content line [Apr 3 21:52:30 ams01 MainProcess[1376]: cel
ery.worker.job ERROR Task tasks.telemetry.gather_and_send_telemetry_info[f090d579-9ec2-40e5-9fb2-91436eb4fc8a] fail]
But I'm not having any luck at the moment... what I'm missing here?
Thank you.