3

I'm using Graylog's sidecar functionality with Filebeat to pickup a number of different log files off my server, including Syslog, Nginx and Java App. All of these flow into the same Graylog input for Beats (I tried to supply multiple inputs, unfortunately Filebeat sends to one and only one location). Everything works great except for Extractors.

How do I specify a specific extractor for a specific log message? Ex: If field 'type' = 'API' (my Java app), I want to apply a JSON extractor. If the type = 'nginx', then apply regex extractors.

Jon
  • 632
  • 5
  • 12

2 Answers2

2

I'm ingesting several log sources on one Input and have 4 Extractors chained to it. From the behavior I've observed, if the extractor fails to match, it simply passes on to the next Extractor. It's only an attempt, not a force.

For example, my extractors:

  1. Decode JSON (input comes in as JSON, this flattens into fields)
  2. Standard App Logging Format (we use an internal standard)
  3. Error Code For App (if ERROR, our apps use a custom Example_Error=Something field)
  4. Mac OS X Hostname

When an app log without an error comes in, it:

  1. Gets decoded from JSON
  2. Matches the format via grok with RegEx
  3. Pass (no "Example_Error=" field)
  4. Pass (no match against Mac OS X logs)

And when a Mac OS X system log comes in:

  1. Gets decoded from JSON
  2. Pass (Doesn't match the app logging format)
  3. Pass (Doesn't match the error code field )
  4. Gets hostname extracted

With some planning and good sets of groks, you can get this to work with lots of Extractors based on the expected formatting of your logs. Of particular use for you might also be the "Only attempt extraction if field contains (string / regular expression)" option within the Extractors.

armani
  • 420
  • 9
  • 26
-1

You can chain extractors in a way that if message contains API, copy message to api_message, do an extractor on that new field.

do the same for nginx, etc.

Jacob Evans
  • 7,636
  • 3
  • 25
  • 55