3

I'm using ELK Stack, and I've got it working pretty well for most of my servers. The exception is that I have a gitlab server that has a ping to/from a gitlab-ci server that happens in the gitlab-access log. This happens every second, and I'd like to ignore it. My regex matches these lines in the regex testers I'm using, but it appears to have stopped all logs coming from that file, instead of the expected single lines.

filebeat:
  prospectors:
    paths:
      - /var/log/gitlab/nginx/gitlab_access.log
    input_type: log
    exclude_lines: ['(.*\bPUT\b)(.*\bgitlab-ci-multi-runner).*']
    document_type: gitlab_access

Below is an example of the log file, and I want to block every line that is a PUT from the gitlab-ci-multi-runner.

**192.168.1.105 - - [07/Feb/2018:07:53:36] "PUT /ci/api/v1/builds/1738.json HTTP/1.1" 404 3082 "" "gitlab-ci-multi-runner 1.3.0 (1-6-stable; go1.3.3; linux/amd64)"**
192.168.1.110 - - [07/Feb/2018:07:53:37] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (7-2-stable; go1.7.5; windows/amd64)"
**192.168.1.105 - - [07/Feb/2018:07:53:39] "PUT /ci/api/v1/builds/1738.json HTTP/1.1" 404 3082 "" "gitlab-ci-multi-runner 1.3.0 (1-6-stable; go1.3.3; linux/amd64)"**
192.168.1.110 - - [07/Feb/2018:07:53:40] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (7-2-stable; go1.7.5; windows/amd64)"
192.168.1.55 - - [07/Feb/2018:07:53:40] "GET / HTTP/2.0" 200 45895 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36"
192.168.1.55 - - [07/Feb/2018:07:53:41] "GET /assets/favicon-075eba763121a0c1f89a89ee81678bcde72e2a47cd3a42.ico HTTP/2.0" 200 1384 "https://gitlab.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.78 Safari/537"
192.168.1.55 - - [07/Feb/2018:07:53:41] "GET /uploads/-/system/user/avatar/21/yodaProfile.jpg HTTP/2.0" 304 0 "https://gitlab.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36"
**192.168.1.105 - - [07/Feb/2018:07:53:42] "PUT /ci/api/v1/builds/1738.json HTTP/1.1" 404 3082 "" "gitlab-ci-multi-runner 1.3.0 (1-6-stable; go1.3.3; linux/amd64)"**
192.168.1.110 - - [07/Feb/2018:07:53:43] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.2.0 (7-2-stable; go1.7.5; windows/amd64)"

I expected the lines that start with ** to be removed, and the rest to go through, but this isn't happening (none of these lines make it through now). If I remove the regex, everything comes through again.

trueCamelType
  • 1,016
  • 5
  • 19
  • 41
  • 1
    What if you simplify the regex to this: `\"PUT.*gitlab-ci-multi-runner` ? You don't need to match the entire line, just enough so you narrow the search down. – pkhamre Feb 07 '18 at 14:21
  • That worked, but I'm not entirely sure why. If you answer with that, and take a guess as to why it worked, I'll mark it as solution. – trueCamelType Feb 07 '18 at 14:29
  • 1
    Okay thanks! I am sure why this works, I am just not sure (yet) why your initial regex does not, because the regex matches the three lines when I enter it into regexr.com – pkhamre Feb 08 '18 at 08:48

1 Answers1

3

If you simplify your exclude_lines-configuration to the following, it will be matched by filebeat.

exclude_lines: ['\"PUT.*gitlab-ci-multi-runner']

I have read through the exclude_lines and the regexp-support documentation, but I didn't figure out the reason why your initial regexp does not match the three lines, since they match when I add it to regexr.com and choose PCRE as the regex engine.

If you want to find out what caused it to not be matched I would suggest that you remove one and one element from the regexp until it matches.

First remove the grouping

exclude_lines: ['.*\bPUT\b.*\bgitlab-ci-multi-runner.*']

Then try to remove the \b entries

exclude_lines: ['.*PUT.*gitlab-ci-multi-runner.*']

Then you should get to something similar to my answer.

exclude_lines: ['PUT.*gitlab-ci-multi-runner']

You can also remove one and one entry at a time, and not all \b elements. When you figure out which entry caused exclude_lines to not match, it will be much easier to find out why.

I hope this answer will help you along the way!

pkhamre
  • 5,900
  • 3
  • 15
  • 27
  • i'd say the usage of \b seams a bit strange but, overhere i'm having similair problems. regex syntax seams broken or so, but if thats the case then why isnt the nr1 issue on filebeat git, is no one using it ?. – Peter Jan 27 '21 at 13:51