8

Under Linux, what's a simple way to automatically watch a logfile, and email me if a certain string appears? I have an application that will log certain failures to a logfile, but has no built-in way of sending alerts or executing scripts on failure. I suppose I could rig something up with tail -f and some shell scripts, but I'd rather use an existing maintained tool if it exists.

davr
  • 1,729
  • 3
  • 14
  • 24

6 Answers6

10

I checked out several of the options mentioned on this page, and ended up using something far simpler: swatch.

Those other systems are great for dealing with existing system logs, or with software where you don't have control over the output. I just didn't want to write a bunch of code to do email notifications just yet. So I just created a swatch file like this:

watchfor /./
    mail addresses=me\@example.com:other\@example.com,subject=log_alert

And then started it up with

swatch -c send-me-everything.swatch -t /my/app/urgentevents

It's crude, but since I control the logfile output, I don't need anything more complicated yet.

William Pietri
  • 233
  • 2
  • 11
  • I have added a tutorial of the installation and updated the link (edit should go through). – Question: It seems that an email is sent for **each line** of the log file entry. Is there a config parameter to send 1 email with the complete log file text? `threshold` or `continue`? – Avatar Aug 14 '18 at 11:09
  • 1
    For beginners, you find the tutorial of how to get swatch running here: https://serverfault.com/posts/180054/revisions (unfortunately the approved edit was removed by the post owner). – Avatar Aug 20 '18 at 12:16
  • Command is `swatchdog` on Ubuntu/Debian (not `swatch`). See https://www.tecmint.com/swatch-linux-log-file-watcher/ – Avatar Apr 03 '21 at 14:52
3

Before we went to a heavyweight solution (Zenoss) we used to use logcheck which is a part of Debian but can easily be ported to other distros as well. I was using it on Gentoo. Distros like RHEL come with logwatch, which does something similar.

Kamil Kisiel
  • 11,946
  • 7
  • 46
  • 68
  • Logcheck is nice, logwatch is balls. – womble Jul 01 '09 at 00:23
  • These are not exactly what I'm looking for. These tools seem to process log files every so often (once a day, every couple of hours), I want something that, upon seeing a string for example "2009-06-29 12:34:56 FooDaemon: Process FAILURE" it should email me right away, within 5 minutes at most. – davr Jul 01 '09 at 00:39
  • 1
    logcheck is scheduled via cron, so you could have it check the logfile as often as 5 minutes. – Kamil Kisiel Jul 01 '09 at 04:05
  • Also, Zenoss will do instantaneous monitoring and you can do all sorts of fancy matching rules as well. Of course, it comes with a lot of other baggage so is probably overkill. – Kamil Kisiel Jul 01 '09 at 04:06
  • I've run logcheck every minute ("* * * * *" in the crontab) for years, and it's been great. – Brandon Rhodes Apr 18 '10 at 03:37
2

I know an answer has already been accepted but rsyslog is much more robust and has built-in filtering, SMTP alerts, and non-syslog based file-watching abilities for applications that do not use syslog. It's now the standard syslog implementation on Ubuntu 10.x.

gravyface
  • 13,947
  • 16
  • 65
  • 100
2

The best way is to use a log analysis program.

OSSEC, for example, is free/open source and allows you to watch as many log files as you want and to generate email alerts (or even active responses) for certain events.

Link: http://www.ossec.net

I know, hacking a shell script is fun, but way less stable than a mature program being developed for years. Plus, if in the future you need to extend your script or add more triggers, it becomes way more complicated. OSSEC (and other tools) have this framework done for you.

sucuri
  • 2,817
  • 1
  • 22
  • 22
  • The poster did say: "I suppose I could rig something up ... but I'd rather use an existing maintained too." Sounds like the poster *wants* to use a "mature program". *smile* – Evan Anderson Jun 30 '09 at 22:36
2

LoFiMo (Log File Monitor) on Sourceforge should get you started or NuHe might work, but I am less familiar with it.

NoahD
  • 407
  • 5
  • 10
2

I found a tool called tenshi that appears to do exactly what I need it to do. It's included in the default Debian repos which is nice, unfortunately not in the RHEL/CentOS ones (I have a mix of both OS's as servers).

davr
  • 1,729
  • 3
  • 14
  • 24