1

I am using nginx with PHP-FPM and ELK as log file analysis.

When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file.

Problem is: Sometimes those error logs contains line breaks which logstash cannot handle, because line breaks are considered as a new log line. 

2019/04/17 19:23:00 [error] 8356#8356: *4403 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function wp_using_themes() in /htdocs/wp-includes/template-loader.php:7
Stack trace:
#0 /htdocs/wp-blog-header.php(19): require_once()
#1 /htdocs/index.php(17): require('/htdocs/wp-blog...')
#2 {main}
  thrown in /htdocs/wp-includes/template-loader.php on line 7" while reading response header from upstream, client: 123.123.123.123, server: foobar.de, request: "GET /2014/11/foobar/ HTTP/1.1", upstream: "fastcgi://unix:/run/php/php-fpm-foobar.sock:", host: "foobar.de"

How to I either handle those linebreaks with logstash or format those error messages to remove the line breaks?

n.r.
  • 249
  • 1
  • 2
  • 10

2 Answers2

2

Kudos to @USD-Matt thanks to his hint I now know where to look at and that's the solution:

There is a multiline feature in the ELK stack. But as I am using the filebeat module to postprocess the logs, I cannot just activate the multiline feature in logstash as mentioned above.

I have to enable it in filebeat itself (/etc/filebeat/filebeat.yml) as described here: https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html

But heads up: I am also using the filebeat module "nginx" which claims to handle the multiline issue itself. It doesn't. You can force it to do so by adding the above multiline settings to the module config file: /etc/filebeat/modules.d/nginx.yml

So I did, and that's how the particular part looks like now:

  error:
    enabled: true
    input:
      multiline.pattern: '^\d{4}\/\d{2}\/\d{2}'
      multiline.negate: true
      multiline.match: after

    var.paths:
      - /var/nginx/foobar_de/logs/error.log*
n.r.
  • 249
  • 1
  • 2
  • 10
  • https://github.com/elastic/beats/issues/14349 Nginx error module can't handle multiline entries – Slavik Oct 31 '19 at 08:09
1

I've not used ELK for a while, but you'll need to modify the config to support multiline log entries. The following is an example of using the multiline codec to merge together any lines that don't start with a date, taken from the official documentation.

input {
  file {
    path => "/var/log/someapp.log"
    codec => multiline {
      # Grok pattern names are valid! :)
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => "previous"
    }
  }
}

Note that I'm not sure if your log file is using ISO8601 dates as in the example without actually researching that format, so you may not be able to use this example as-is, but it shows the basics of handling multiline logs entries where the date signifies a new entry.

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

USD Matt
  • 5,321
  • 14
  • 23
  • Thank you! It's not the solution, but it points into the right direction. I am using filebeats. At least I know now that I have to enable multiline in filebeat.yml. Thing is: I am also using nginx-Module, which has multiline support "under the hood". The setting in filebeat.yml somehow is being ignored. And: The nginx-under-the-hood-feature seems to be buggy: https://github.com/elastic/beats/issues/6798 – n.r. Apr 18 '19 at 10:51