How to block Nginx access log statements from specific user agents

Question

I am looking to turn off logging in the Nginx access log file from specific requests from http user agents.

Basically from the Amazon ELB Health Check and our external (Pingdom) monitoring. Since these come every few seconds, makes testing hard to sort through the logs.

"GET / HTTP/1.1" 200 727 "-" "ELB-HealthChecker/1.0"
"GET /login HTTP/1.1" 200 7492 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"

I was able to block logging for image files, but have not seen anything for incoming requests:

location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico|xml|svg)$ {
        access_log        off;
        expires           30d;
}

Thanks in advance!

So I tried the recommendation from @Gnarfoz, but had some interesting side effects. While those two "Health Checks" were not logged, Pingdom started to recognize the server as DOWN even while it was up and running. That is interesting as the Load Balancer did not, which it would have dropped the node we were testing if it had.

I put the MAP section in the HTML block below my logs:

access_log /www/access.log;
error_log /www/error.log;

map $http_user_agent $ignore_ua {
            default                 0;
            "~Pingdom.*"            1;
            "ELB-HealthChecker/1.0" 1;
    }

And I put the IF statement in my server block, with the default location:

location / {
                try_files $uri $uri/ /index.php?$args;

                if ($ignore_ua) {
                       access_log off;
                }
        }

When I did this, Pingdom started to generate 404 errors in the Error Log file:

2012/08/03 17:10:33 [error] 6250#0: *164 open() "/www/public_html/login" failed (2: No such file or directory), client: 10.xx.xx.xx, server: xxx.com, request: "GET /login HTTP/1.1", host: "xxx.com"
2012/08/03 17:11:32 [error] 6250#0: *240 open() "/www/public_html/login" failed (2: No such file or directory), client: 10.xx.xx.xx, server: xxx.com, request: "GET /login HTTP/1.1", host: "xxx.com"

Thanks, wasn't sure I was using the proper terminology with that — briannyc, Aug 03 '12 at 16:52
Ah, my bad. While I haven't had any coffee yet, I think you need to actually return something in that location block, too. So maybe a simple `return 200;` after the `access_log off;` might do the trick. (Non-) Inheritance of nginx configuration directives still is not entirely clear to me at all times... — Gnarfoz, Aug 06 '12 at 08:06
You are correct, adding `return 200;` worked perfectly! Thank you for your help. — briannyc, Aug 06 '12 at 19:56
Good to hear that works. I've updated my answer accordingly. — Gnarfoz, Aug 07 '12 at 08:21

score 7 · Accepted Answer · edited Jul 10 '17 at 23:59

7

Try this:

# map goes *outside* of the "server" block
map $http_user_agent $ignore_ua {
    default                 0;
    "~Pingdom.*"            1;
    "ELB-HealthChecker/1.0" 1;
}

server {
    # Things omitted for brevity

    location / {
        if ($ignore_ua) {
            access_log off;
            return 200;
        }
    }
}

The if part would probably need to be integrated into your appropriate location block.

Relevant nginx documentation: map, if, access_log

edited Jul 10 '17 at 23:59

fideloper

353
3
11

answered Aug 03 '12 at 15:29

Gnarfoz

698
4
10

Tried your recommendation with some interesting side effects. I might have put the IF statement in the wrong location. Will keep playing with it. – briannyc Aug 03 '12 at 17:29
1

This did not work for me. – Mauvis Ledford Sep 09 '14 at 23:30
Didn't work for me either, but I think this is on the right track for how to handle this. – xxdesmus Sep 12 '14 at 16:14
1

Tiny update -- map goes outside of the server block by the way, just in case that helps folks. This does work for me though. – xxdesmus Sep 12 '14 at 16:43

a55 · Answer 2 · 2022-04-20T12:36:29.787

4

Thanks to @gnarfoz

Respect to if is Evil, This is better way to use condition before log

map $http_user_agent $ignore_ua {
    default                 1;
    "~Pingdom.*"            0;
}

server {
    location / {
        access_log /var/log/nginx/access.log if=$ignore_ua;
    }
}

NGINX Documentation Enabling Conditional Logging

edited Apr 20 '22 at 12:36

answered Nov 25 '20 at 14:01

a55

141
3

1

Based on the question, default should be 1, and "~Pingdom.*" should be 0. Because nginx doc: "A request will not be logged if the condition evaluates to “0” or an empty string." http://nginx.org/en/docs/http/ngx_http_log_module.html – Juan Ignacio Barisich Apr 20 '22 at 12:29

How to block Nginx access log statements from specific user agents

2 Answers2

Linked