4

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a conditional rule so that let's say "X logfile has the pattern Error 3x in 1 minute email me". This way I don't get overwhelmed with emails.

Here is my current config:

input {
  file {
#    sincedb_path => /path/to/whatever/
    path => "/opt/test.log"
    type => "test_log"
  }
}

filter {
   dns {
      add_field => [ "IPs", "Logs, from %{host}" ]
      type => [ "MESSAGES" ]
      resolve => [ "host" ]
      action => [ "append" ]
     }
}

filter {
  if [message] == "Error" or [message] == "error" {
    throttle {
      before_count => 1
      after_count => 3
      period => 10
      key => "%{message}"
      add_tag => "throttled"
  }
} }

output {
#  stdout { codec => rubydebug }
   redis { host => "redis_IP" data_type => "list" key => "logstash" }
   if "throttled" not in [tags] {
      email {
        from => "logstash@shipper.com"
        to => "sysadmin@something.com"
        subject => "Alert from  %{path}, from %{host}"
        body => "Message is: ]\n'%{message}'. \nLog file:\n %{path}:\n\n%{message}.\n More information can be viewed in Kibana"
        }
    }
}
Gabriel
  • 141
  • 1
  • 1
  • 2
  • 'period' is in seconds. You're reseting the count every 10 seconds. – Alain Collins Sep 08 '14 at 16:54
  • I'm doing that to test. I want to only trigger an email if "error" is parsed into /this/test.log 3x in 15 seconds (or whatever). I have another config that works but emails me EVERY time "error" is parsed which is leading to way too many emails. – Gabriel Sep 08 '14 at 16:58
  • So what is this config doing that you don't want it to? – Alain Collins Sep 08 '14 at 17:06
  • on pastebin: http://pastebin.com/LGsB3qQS – Gabriel Sep 08 '14 at 17:08
  • That's a different config than the one in this question, and contains no throttle. – Alain Collins Sep 08 '14 at 17:12
  • Yes, I just found the throttle. The pastebin one emails me EVERY time. I want to limit this. – Gabriel Sep 08 '14 at 17:16
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/17008/discussion-between-gabriel-and-alain-collins). – Gabriel Sep 08 '14 at 20:41
  • @AlainCollins I know this is a long shot since this was 2 years ago but, did you guys ever find a solution to this problem? – Alpha Sep 20 '16 at 16:25
  • @Alpha, I never got enough information from to OP to help him. If you're having the same problem, perhaps open your own issue or jump in IRC. – Alain Collins Sep 20 '16 at 22:52
  • @AlainCollins Welcome from the past! Thanks for answering! Not yet, I haven't faced this same issue but I might and I thought it would be useful for the community anyways. Thanks a lot! – Alpha Sep 21 '16 at 15:59

1 Answers1

7

We just setup Riemann to handle alerting based on log messages.

Riemann can read a stream of log messages from logstash and send out alerts based on the contents.

One of the advantages with riemann is you can rollup all messages from a certain time into one email. This way you will not get to many e-mails but you will still get all your messages.

Much more examples can be found at http://riemann.io/howto.html

Brian van Rooijen
  • 197
  • 4