Nginx drops connections

Question

I'm having a setup where I use the linode nodebalancer (loadbalancer) for my nginx/php5-fpm servers. This balancer passive checks. These passive checks, check the status code of requests. If there are too many 5XX status codes the node (vps) is marked offline by the loadbalancer

The nodebalancer is putting my servers offline in a random way. When contacting linode support they came to the conclusion that there are no 500 errors, but connections are dropped (or timeout).

I can't find anything in my nginx logs. Is there any way to debug this problem and see what connections have been timed-out dropped by nginx?

EDIT I can see a lot of 408 requests from the same IP/user agent. They come in by bulk.. Is this suspicious? How would you handle this situation? Snapshot from access.log

69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Thanks!

score 0 · Answer 1 · answered Apr 22 '14 at 15:19

0

Would be nice to know what HTTP code are you getting. I suspect it should be 408 (Request Timeout), so nodebalancer will think the server is gone and will mark it offline.

What you should try:

Increase the timeouts
Use TCP mode instead HTTP mode

Warning: Using TCP mode, cookie handling if OFF.

answered Apr 22 '14 at 15:19

boris quiroz

1,140
1
7
18

Can you give some more info on Use TCP mode instead HTTP mode ? Is this the nodebalancer check? What are the pros/cons? – Apr 22 '14 at 16:21
Yes, that's in the nodebalancer check. Pros: you'll workaround this problem. Cons: you won't be able to use X-Fowarded-For (among other Pros/Cons). Check on Linode's documentation about it. – boris quiroz Apr 22 '14 at 18:11
What timeouts should be increased for this? Linode tells me 408 connections are not counted for passive checks, but that's the only weird thing I see in the access logs (no 5xx errors) – Apr 22 '14 at 18:29
The values you'll need to increase are: Check Interval, Check Timeout and Check Attempts. Also, which Health Check Type are you using? Could you please provide this values? – boris quiroz Apr 23 '14 at 13:49

Nginx drops connections

1 Answers1