68

Apparently, I shouldn't have spent sleepless night trying to debug an application. I wanted to restart my nginx and discovered that its config file is empty. I don't remember truncating it, but fat fingers and reduced attention probably played their part.

I don't have backup of that config file. I know I should have made it.

Good for me, current nginx daemon is still running. Is there a way to dump its configuration to a config file that it'll understand later?

Sergio Tulentsev
  • 783
  • 1
  • 8
  • 9

5 Answers5

76

You need a gdb installed to dump memory regions of running process.

# Set pid of nginx master process here
pid=8192

# generate gdb commands from the process's memory mappings using awk
cat /proc/$pid/maps | awk '$6 !~ "^/" {split ($1,addrs,"-"); print "dump memory mem_" addrs[1] " 0x" addrs[1] " 0x" addrs[2] ;}END{print "quit"}' > gdb-commands

# use gdb with the -x option to dump these memory regions to mem_* files
gdb -p $pid -x gdb-commands

# look for some (any) nginx.conf text
grep worker_connections mem_*
grep server_name mem_*

You should get something like "Binary file mem_086cb000 matches". Open this file in editor, search for config (e.g. "worker_connections" directive), copy&paste. Profit!

Update: This method isn't entirely reliable. It's based on assumption that nginx process will read configuration and don't overwrite/reuse this memory area later. Master nginx process gives us best chances for that I guess.

kupson
  • 3,388
  • 18
  • 18
  • 4
    Thanks, but this is too hardcore for me. I will rewrite the config file from scratch :) – Sergio Tulentsev Feb 18 '12 at 14:03
  • 1
    An mmap'd file implies an ope filehandle. See for an easier way to recover one: http://serverfault.com/questions/45237/link-to-a-specific-inode – Jeff Ferland Mar 31 '12 at 19:03
  • @JeffFerland nginx don't keep open fd for configuration file. – kupson Apr 01 '12 at 17:11
  • Yeah, most applications don't... thought I edited my comment after to include mention. Oops. – Jeff Ferland Apr 01 '12 at 17:47
  • One more thank you! :) Recovered a 2nd half of config file, now grepping on raw /dev/sda ... – zed_0xff Dec 12 '12 at 14:39
  • 1
    You can use the GNU binutils' strings(1) command to simply do e.g. `strings mem_* | grep worker_connections` instead. – Josip Rodin Nov 26 '15 at 10:59
  • Nice, worked like a charm! after dumping and finding the lines I used the methods from [here](http://unix.stackexchange.com/questions/47407/cat-line-x-to-line-y-on-a-huge-file) to save the config lines into a new file. – sepehr Oct 10 '16 at 18:46
  • 1
    This feels so hardcore! I'm officially a hacker now. Reading out memory to find your configs back. Thanks for explaining @kupson – adriaan Jan 14 '19 at 11:23
  • 1
    This might be the coolest thing I've ever seen... Thank you! I was able to `grep server_name mem_*` which spit out a file and then just searched for `location /` in nano and got the entire file back. – Sam Mearns Apr 12 '19 at 20:44
  • 1
    I can't believe this actually worked. That'll teach me to improperly copy a symlink and modify it. – RaisinBranCrunch Aug 04 '20 at 18:37
  • 1
    This just saved my bacon. Highly appreciated, Sir! – Eugen Rieck Aug 28 '20 at 02:44
  • You sir, are a legend. – Lee Jan 11 '22 at 18:54
  • To make it even easier: I wrote an awk script to automatically extract intact config blocks from these dumps, and posted in another answer: https://serverfault.com/a/1090838/86724 – cincodenada Jan 23 '22 at 00:41
  • Consider it thanks from one more butt saved, I accidentally overwrote my slightly-annoying-to-reproduce config, recovered it with this, and then spent more than the time saved twiddling with Awk :P Hopefully others find it useful! – cincodenada Jan 23 '22 at 00:43
37

This will not help on this request, but might help other reaching here for the same reason. Newer nginx versions have the -T option to dump the nginx config read from all nginx config files, not from memory:

nginx -T

This can be useful to confirm that a config file is being read, to compare with other server or search for configs.

Again, this will not dump the config from the running process, only what a new process would load.

higuita
  • 1,093
  • 9
  • 13
2

The ngx_conf_t is a type of a structure used for configuration parsing. It only exists during configuration parsing, and obviously you can't access it after configuration parsing is complete.

zxc
  • 21
  • 1
  • 3
    It's 'obviously' inaccessible just because apparently there's no such facility implemented in nginx; other programs have such facilities, such as `postconf -n` for Postfix or `exim -bP` for Exim or (the badly named) `testparm -v` for Samba, etc. – Josip Rodin Nov 26 '15 at 10:54
1

This answer is a small but I think useful addition to the brilliant top answer, so follow those directions first. Then you can use this script to more easily find your config.

Once you have the memory dumps from the top answer, you can use this slightly-overengineered Awk script to extract anything that looks like a top-level brace-enclosed configuration block.

#!/usr/bin/awk -f
### Setup
# We're searching for a keyword followed by an open brace to start
# And the a close brace at the start of a line to end
# Also include commented sections, cause otherwise they look funny
BEGIN {
  start="[[:alpha:]_#]+ \\{$";
  end="^#?}" 
}

# Shortcut to extract a regex pattern from $0
function extract(ere) { return substr($0, match($0, ere), RLENGTH) }

# Check for end conditions first
# This way we end the section before we print below

# For the primary end condition, print out the matched bit
$0 ~ end { print extract(end); go=0}
# And a safety stop: bail on any non-printable lower-ASCII characters
/[\x00-\x08\x0e-\x19]/ { go=0 }

# If we're in a section, print the line!
go {print}

# Otherwise, check for our starting condition
# If we find it, print just that bit and turn on our flag
!go && $0 ~ start {
  go=1;
  print "### Extracted from memory dump:";
  print extract(start)
}

Save that to extract.awk and then run awk -f extract.awk mem_*, or if you prefer one-liners, here you go:

awk 'BEGIN { start="[[:alpha:]_#]+ \\{$"; end="^#?}" } function extract(ere) { return substr($0, match($0, ere), RLENGTH) } $0 ~ end {print extract(end); go=0} /[\x00-\x08\x0e-\x19]/ { go=0 } go {print} !go && $0 ~ start { go=1; print "### Extracted from memory dump:"; print extract(start)}' mem_*

This script should dump a list of top-level config sections that you can then look through to recover the ones you need, without digging through a bunch of other memory noise.

Note: Awk may complain about malformed characters on STDERR, you can just ignore it. If you have recent GNU awk you can add the -b flag indicate that you expect binary data, which will silence the warning.

But why?

Yeah, you can just grep through those dumps, or open them in an editor and search for blocks, but there will be small pieces of your config scattered throughout the memory map, so it can be annoying to dig through. strings loses whitespace, and you can grep for things like a brace followed by a newline...but we have tools to help us do this. And if you're already copy-pasting scripts from ServerFault, you might as well do one more to make your life easier.

cincodenada
  • 192
  • 6
1

the ideal way is to look for the ngx_conf_t struct from nginx process image.

It is defined here

http://trac.nginx.org/nginx/browser/nginx/trunk/src/core/ngx_conf_file.h#L166

My C & gdb sucks so home someone else could come up with a solution.

est
  • 171
  • 1
  • 8