With logs getting captured in syslog-ng, I'd like to be able to automatically monitor the logs and receive an alert if NO log events appear that match a certain criteria. For instance, for a subscription-based website, if 6 hours elapse with no orders, then email or text this person or group of people.
What is a good way to do that?
As the information that a certain event should occur at least every six hours is specific to the application writing to the log, it would be best if it could monitor itself and write a log entry if the event does not happen as it should. If the application writing log information is developed locally I would recommend this solution.
If that is not possible, I would keep the mechanism which makes sure the event has occurred as close to the application as possible. Maybe a watchdog could be started in the background from the same start script that starts the application performing logs. When the application is stopped, the script also stops the watchdog.
Another option, in case you would like to keep the start script in its original condition for some reason, would be to create a cron-job which performs the log watching.
In either case. Make sure the level of the log entry is severe enough that you are alerted to take a look at it using whatever tool you are using to monitor logs. It's better, more future proof and inclusive for things you didn't think of but should monitor, then to listen for a log level than a specific log entry. This means at least WARNING level.
Zabbix is another monitoring solution similar to Nagios. Zabbix has the ability to monitor files for various strings and initiate a triggered alert based off of the criteria you specify (found or not found). Zabbix also supports "looking" back or from a various point in time in the log which helps prevent older events from being false positives. Alerts can be configured to send emails or SMS.
Zabbix Website: http://www.zabbix.com/
Some of Zabbix's Log Monitoring Docs: http://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/log_items
http://labs.consol.de/nagios/check_logfiles is a Nagios plugin which is used to monitor logfiles. Usually you check, if there is a certain pattern (error message). But it's alos possible to reverse this. For example, if you run check_logfiles --logfile /var/log/mybackup.log --criticalpattern '!backup succeeded' every morning, you will get an alert if there was no 'backup succeeded' message entry since the last run of check_logfiles.
Gerhard
Nagios can do this for you with ease. Personally, i like any thing that i can easily run on the commandline and/or set up easily via Nagios or Crontab.
LOGROBOT can do exactly what the OP is asking for...in one simple command line entry:
./logrobot autonda /prod/apps/mylogfile 60m 'orders submitted' '.' 1 2 subscription_orders -ndnotfoundn
Just change the bolded strings and of course the log file path to match your needs.
Basically, this command will alert as warning if at least 1 'orders submitted' entry is NOT found in the log file /prod/apps/mylogfile. It will alert as critical if at least 2 'orders submitted' entries are not found in the log.
You could use Shell/Perl Scripts to do this task. You can run the Shell / Perl scripts through Cron.
Nagios would be a better solution. It's an open source tool, so no need to pay anything, but need to fight with it for configuring it. ;)
If you want a commercial product, Splunk is known for the logfile analysis.
The way I've approached this in the past is by forwarding syslog traffic to a database, and then just querying the database for the last received log message from a particular hostname, with it's timestamp.
Which is just one of the positive benefits from logging to a database.
I believe the easiest way to take care of this is by writing very simple script in language of your choice. You just need to grep through the syslog for a specific messages and send email if none found. Including some sanity checks I can't imagine more than 20 lines. Stick this is cron and you are all set. The drawback is that by using this method, the cron job has to run as root in order to read the logs.
What you could also do is to take a look at LogAnalyzer. Unfortunately, I don't remember if it is capable of sending emails. Nevertheless, it uses MySQL and stores the log info there. With LogAnalyzer setup, you would have two copies of your logs, as they go in parallel - one in syslog, one in the DB. If you have an user for this DB, capable of reading its contents, you could avoid running the above mentioned cron job. It may not be that much better but could add another protection layer.
We have done something similar where we get alerted if we don't get a new order on the website within the last N hours. We created a PushMon URL with a schedule of "every 3 hours". We then called the PushMon URL in our thank you page (JavaScript). If we don't get an order, PushMon will notice the missing call, and will send an alert.
PushMon works well for alerting you about events that do not occur.
