Parseable NGINX accesslog files with delimiters

Question

The default NGINX format is this:

log_format combined '$remote_addr - $remote_user [$time_local]  '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent"';

Which is a bit hard to parse. I am afraid that people inject " in either requests, referrers or user-agents.

I have thought about using delimiters instead, and use my own format, that uses |P-,| as a delimiter:

log_format parsable '$status |P-,| $time_iso8601 |P-,| $http_host 
|P-,| $bytes_sent |P-,| $http_user_agent |P-,| $http_referer 
|P-,| $request_time |P-,| $request';

However, nothing prevents users from injecting |P-,| into their requests, referrers or user-agents.

I read this article about ASCII delimited text: https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/

I think that could be used to solve this problems, but users would be able to inject ASCII delimiters into their data as well.

Is there a best-practice way to solve this problem?

I prefer to log JSON. It's easy to parse and can be forwarded to a logserver like Graylog or Kibana. — mschuett, Mar 27 '14 at 14:23
I have been doing that as well, but because the JSON module isn't built-in as default in the Ubuntu repos, I haven't used it. I have done what they suggest [here](http://blog.pkhamre.com/2012/08/23/logging-to-logstash-json-format-in-nginx/). They mention a user setting the user-agent to `"` to generate non-valid JSON, but maybe the changes @alexeyton refers to fixes this issue — Kasper Grubbe, Mar 27 '14 at 14:45

Alexey Ten · Accepted Answer · 2014-03-27T13:45:23.363

14

There is no problem.

I am afraid that people inject " in either requests, referrers or user-agents.

" is represented as \x22

Request:

$ curl 'localhost/"?"="' --header 'User-Agent: "'

line in log:

[27/Mar/2014:16:14:42 +0400] localhost 127.0.0.1 "GET /\x22?\x22=\x22 HTTP/1.1" 200 "-" "\x22" "-" "/index.html"

UPDATE

From nginx changelog

Changes with nginx 1.1.6 17 Oct 2011

*) Change: now the 0x7F-0x1F characters are escaped as \xXX in an
   access_log.

Changes with nginx 0.7.0 19 May 2008

*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
   in an access_log.
   Thanks to Maxim Dounin.

edited Mar 27 '14 at 13:45

answered Mar 27 '14 at 12:16

Alexey Ten

7,922
31
35

I didn't see the documentation mention that they escape characters like that, that is good to know, thank you for your answer. – Kasper Grubbe Mar 27 '14 at 13:25
I'm not sure about other characters. That's good reason to stick with double quote as string delimiter in logs. – Alexey Ten Mar 27 '14 at 13:26
Well, in that case I could use " as a delimiter. I will try to contact the Nginx project through their mailinglists and update here. – Kasper Grubbe Mar 27 '14 at 13:30
@KasperGrubbe see update – Alexey Ten Mar 27 '14 at 13:45
How can I switch off the \x22 escaping? – kev May 09 '16 at 09:02
@kev you can't do that – Alexey Ten May 09 '16 at 09:37
Why? I don't understand. – kev May 09 '16 at 09:40
Because nginx's log module does this unconditionally – Alexey Ten May 09 '16 at 09:46

score 1 · Answer 2 · answered Mar 27 '14 at 12:14

Remember a number of the fields are generated by the system, so are safe. If you ensure those fields are to the left and the hackable ones are to the right (http_user_agent should at the end, and the http_referer before that, request should be before that), you can ensure most of the data is sound, and by adding more delimiters to the parser (an optional one on the far right) than can possibly exist without insertion, then your parser will detect records that have been subject to insertion.

Also I recommenced using a tab character as a delimiter, as I believe were someone to attempt to insert it into a url, it'd end up being escaped to a %09

you could also not log referer at the server level, seems it's been missed as an option — MrMesees, Dec 29 '16 at 02:06

Parseable NGINX accesslog files with delimiters

2 Answers2