5

I'm responsible for a nginx server that redirects and splits requests. At our production engine we run nginx 1.2.1 and on a test machine we run 1.4.1. The configuration is the same but on the production environment I always get a 500 Internal Server Error"but on the test environment all runs fine. I always checked the access.log and the error.log of Nginx. But there is nothing to worry about.

How can I analyse the error and do some further investigation on the cause of this error?

Rex
  • 7,815
  • 3
  • 28
  • 44
shylynx
  • 161
  • 1
  • 1
  • 7
  • Have you tried to run nginx in debug mode? – ALex_hha Mar 05 '14 at 11:24
  • Tried it, but nothing spectacular except the line "2014/03/05 11:18:49 [debug] 19796#0: *6 HTTP/1.1 500 Internal Server Error" – shylynx Mar 05 '14 at 12:21
  • Are you sure that nginx compiled with --with-debug configure argument? – ALex_hha Mar 05 '14 at 13:54
  • I haven't compiled it. I used the debug directive to show more verbose output. – shylynx Mar 05 '14 at 13:55
  • It's not the same. To activate debugging log you have to compile nginx with --with-debug configure option and set debug level in error_log directive. You can get more info at http://wiki.nginx.org/Debugging – ALex_hha Mar 05 '14 at 13:58
  • I assume that its compiled with debug, because output is more verbose if I use the the debug directive. – shylynx Mar 05 '14 at 14:19
  • Did you see "--with-debug" in output of the "# nginx -V" – ALex_hha Mar 05 '14 at 14:21
  • Yes, contains the option. – shylynx Mar 05 '14 at 14:25
  • Could you add full debug log with 500 error to the question? – ALex_hha Mar 05 '14 at 15:43
  • You said that ngnix redirects and splits the traffic. Where the traffic goes? Maybe the backends (uwsgi?) provide this 500's? Please provide more information: what kind of backend do you use and logs from it. – neutrinus Mar 05 '14 at 09:20
  • The backends of the production and the test environment are the same. Also the configuration of nginx is the same. Only the nginx server instances are different (1.2.1 on production and 1.4.1 on test). – shylynx Mar 05 '14 at 12:03

4 Answers4

2

The first thing to ask yourself: what is emitting the 500 response. Looking at the response headers and the page style will tell you a lot about where it came from. Eg. Is there a X-Powered-By header in the response? If so, it wouldn’t have come from Apache (for example).

A Tomcat error page looks very different to an Nginx page, compared to Apache, etc. This is why I coach people at work to give me a good screen-shot.

Also, if you see logs of 500, but nothing in the error log, then it most likely came from the backend, and you should look there.

Also, why are your test and production nginx versions different? You haven’t said anything about using the same version in production as you do in test.

Beware changes in default behaviour in different versions of software. I was reminded of this a couple of time recently migrating to Apache 2.4 from 2.2.

Finally, you say the backend is the same (really, as in the very same instance?) for test and prod, but that does t necessarily mean a request would be processed the same (eg. different host header or SNI server name being passed)

Hope this helps you master reverse-proxy debugging.

Cameron Kerr
  • 3,919
  • 18
  • 24
1

Here is the nginx checklist I use when diagnosing errors.

  1. verify your nginx configuration using

    sudo nginx -t

    it is a very basic step but one that should always be done first.

  2. verify nginx is running

    sudo service nginx status

  3. Verify the log files specified in your site configuration

    find /etc/nginx -name '*.conf' | xargs grep -i log

  4. If you are getting a 500 error, you should be seeing an entry in your error log related to the error that will give you a hint as to why the error is happening. If you do not see an error message in your error log, you have an error log configuration problem and you will want to verify the timestamps on the error log file to make sure it is being updated.

2ps
  • 1,076
  • 8
  • 11
-1

You can check external connections from your nginx config (Proxy, FCGI ...) and check for their logs.

dima.h
  • 145
  • 1
  • 6
-1

To capture any external requests/errors you might get and cause a 500 error: You can also run your address you are trying to access on a chrome browser while having open the chrome://net-internals page at the "Events" menu.

There you can analyze more external requests/responses (dns info, headers sent etc.)

Hskdopi
  • 129
  • 6