0

Im trying to format logs straight from our F5 using HSL and Logstash.

I've copied the example configuration exactly from the site as it contains the majority of the information I wanted logstash example

My F5 is now outputting logs to my elasticsearch cluster marking the sample log data below with the tag "message:" however when I try to apply the example grok filter, which appears to be correct; into my logstash conf file i get a _grokparseerror.

Does anyone see where i'm going wrong with my filter?

SAMPLE LOG

12.123.123.254 [13/Feb/2016:16:04:13 +0000] 123.16.1.23 /SITE/SITE_SSLV2 /SITE/SITE_SSLV2_pool 123.16.1.156 443 '/' 'GET / HTTP/1.1' 200 21003 217 'https://w2.website.co.uk/Planner/Planner' 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36'

SAMPLE GROK

%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}

MY CONFIG

filter {
  if [type] == "f5-access" {
    grok {
        match => { "message" => "%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}"}
}
    geoip {
      source => "clientip"
    }
  }
}
Dan
  • 138
  • 2
  • 11
  • specifically it looks like this bit \"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent} doesnt fit this bit properly '/' 'GET / HTTP/1.1' 200 21003 217 'https://w2.website.co.uk/Planner/Planner' 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36' – Dan Feb 15 '16 at 10:12
  • 1
    Your sample log line has single quotes `'`, while your grok patter is using double quotes `"`. Change your grok to use the former and you should be golden. – GregL Feb 15 '16 at 13:30
  • I looked at my sample just then and thought, my word i'm an idiot he's fixed it - then went back into kibana to check, i'm not sure how thats changed on here but the actual logs do have " and not ' in them - good spot though :) – Dan Feb 15 '16 at 16:37
  • I just tried your log line and grok patter on the [Grok Constructor](http://grokconstructor.appspot.com/do/match#result) and it works, assuming the log line uses all double quotes. I'd suggest you check the format of the lines as they come in to LS to make sure they match, because something seems off. The fact that you're saying it fails at `\"%{DATA:path}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}` tells me that the quotes are part of the problem. – GregL Feb 15 '16 at 16:45
  • yeah i'm thinking its the way i'm calling the pattern, maybe if I save the pattern separately and call the whole thing at once... – Dan Feb 15 '16 at 17:03

0 Answers0