nginx: Log complete request / response with all headers?

Question

We have an application server that sometimes hangs. We suspect it is due to a bad request from a client.

Can nginx log the complete request/response (like fiddler captures) to files, so we can see the requests that were sent before the hang?

(We probably need to avoid pcap and that approach and do it all in nginx)

If nginx is not the right tool for this, what (other than a network analyzer) might be?

[mitmproxy](http://mitmproxy.org/doc/howmitmproxy.html) in reverse proxy mode should do what you are looking for. — Vivek Thomas, Oct 14 '14 at 05:55
@VivekThomas this is an nginx question.... we are already using nginx and are not going to change. — Jonesome Reinstate Monica, Dec 30 '14 at 15:59
@samsmith Old question, but maybe this helps someone else: you _don't_ have to give up nginx. Depending on the circumstances, you can just re-route nginx to another port temporarily, to allow mitmproxy to intercept the traffic and support the debugging. Then, once complete, you can just re-route nginx to the original port and shutdown mitmproxy. — Per Lundberg, Nov 27 '17 at 13:03
You can use modsecurity module, which can log full requests/responses, see https://www.nginx.com/blog/modsecurity-logging-and-debugging/ — Willem, Oct 16 '18 at 12:54

Xavier Lucas · Accepted Answer · 2014-10-14T08:23:09.140

58

To get the request body sent by visitors, use client_body_in_file_only on; and log the "temporary" file it's written to in the logs by appending var $request_body_file to the log format. "Temporary" files will be located in client_temp directory by default.

You can log request headers $http_<header> too and sent headers with $sent_http_<header>.

If you have request body and headers you should be able to replay it and get the response your visitor had.

Also something like gor should highly be considered so you could replay the traffic on an other environment where you could let nginx write these temporary files without causing IO issues in production (nginx won't purge them with on value that's why It's not that "temporary" in this case).

edited Oct 14 '14 at 08:23

answered Oct 14 '14 at 08:10

Xavier Lucas

12,815
2
44
50

Xavier, Thanks! Works! Now that I have the files, how do I replay them? – Jonesome Reinstate Monica Oct 15 '14 at 02:37
NVM RE Replay. curl works great! – Jonesome Reinstate Monica Oct 15 '14 at 05:33
This doesn't work for me for headers (nginx 1.8.0). I get the _body_ of a request but not the headers. The files are only recorded for POST requests (i.e. one where something is sent in the body). – jwadsack Jul 28 '15 at 03:22
1

@jwadsack Read the answer carefully. – Xavier Lucas Jul 28 '15 at 09:58
4

@XavierLucas I thought you were providing two different approaches. I didn't realize you were saying _both_ `client_body_in_file_only` and `$http_
` would be needed. I get that now.
– jwadsack Jul 28 '15 at 17:31
18

Could you please share a more precise code? – Velkan May 09 '17 at 13:40
12

Surely $http
is only useful if you know all the header names in advance
– Ed Randall Feb 02 '18 at 22:37
12

Can someone share an actual fragment of the nginx config, please? – Nowaker Jun 13 '18 at 15:24
Can we use `$http_
` without installing any module?
– simo Jul 22 '20 at 09:48
Client temp folder, where is that? – jjxtra Mar 02 '21 at 15:50

score 19 · Answer 2 · answered Oct 14 '14 at 06:02

mitmproxy seems to be the right tool to do what you are asking.

mitmproxy is an interactive, SSL-capable man-in-the-middle proxy for HTTP with a console interface.

mitmdump is the command-line version of mitmproxy. Think tcpdump for HTTP.

Features

Intercept HTTP requests and responses and modify them on the fly.

Save complete HTTP conversations for later replay and analysis.

Replay the client-side of an HTTP conversations. Replay HTTP responses of a previously recorded server.

Reverse proxy mode to forward traffic to a specified server.

Transparent proxy mode on OSX and Linux.

Make scripted changes to HTTP traffic using Python.

SSL certificates for interception are generated on the fly.

The reverse proxy mode would let you capture the request and response just like Fiddler does.

In linux context, [`socat`](https://linux.die.net/man/1/socat) is also worth exploring for MITM. — Ross Rogers, May 24 '20 at 17:56

score 1 · Answer 3 · answered Sep 09 '21 at 09:11

I was looking for an answer to this question, and found this: https://tarunlalwani.com/post/request-capturing-nginx-lua/

Basically, uses a lua script to gather and log all headers:

ngx.log(ngx.ERR, "REQUEST capturing started")
json = require("json")

function getval(v, def)
  if v == nil then
     return def
  end
  return v
end

local data = {request={}, response={}}

local req = data["request"]
local resp = data["response"]
req["host"] = ngx.var.host
req["uri"] = ngx.var.uri
req["headers"] = ngx.req.get_headers()
req["time"] = ngx.req.start_time()
req["method"] = ngx.req.get_method()
req["get_args"] = ngx.req.get_uri_args()


req["post_args"] = ngx.req.get_post_args()
req["body"] = ngx.var.request_body

content_type = getval(ngx.var.CONTENT_TYPE, "")


resp["headers"] = ngx.resp.get_headers()
resp["status"] = ngx.status
resp["duration"] = ngx.var.upstream_response_time
resp["time"] = ngx.now()
resp["body"] = ngx.var.response_body

ngx.log(ngx.CRIT, json.encode(data));

Then, refer to it from the nginx configuration:

location / {
    proxy_pass http://127.0.0.1:8081/;
    log_by_lua_file lua/request_logger.lua;
}

To gather the body, a block is needed in nginx main.conf:

#we must declare variables first, we cannot create vars in lua
set $response_body '';

body_filter_by_lua_block {
    -- arg[1] contains a chunk of response content
    local resp_body = string.sub(ngx.arg[1], 1, 1000)
    ngx.ctx.buffered = string.sub((ngx.ctx.buffered or "") .. resp_body, 1, 1000)
    -- arg[2] is true if this is the last chunk
    if ngx.arg[2] then
      ngx.var.response_body = ngx.ctx.buffered
    end
}

The post linked above details this a lot better.

nginx: Log complete request / response with all headers?

3 Answers3

Linked