2

I have quite heavy loaded web server using:

Ubuntu server
nginx
php-fpm + apc

Yesterday something wierd happened with my server. It crashed and stopped responding and after I rebooted it, web pages started loading very-very slowly, giving "request timed out" in most cases.

I've checked /var/log/syslog and saw a lot of messages like: TCP: Possible SYN flooding on port 80. Sending cookies.

Page takes about 2 minutes to load locally:

time wget -O /dev/null mysite.net
--2012-12-21 13:17:15--  http://mysite.net/
Resolving ficbook.net... 85.254.49.180
Connecting to mysite.net|85.254.49.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1311 (1.3K) [text/html]
Saving to: `/dev/null'

100%[========================================================================================================>] 1,311       --.-K/s   in 0s      

2012-12-21 13:19:18 (181 MB/s) - `/dev/null' saved [1311/1311]


real    2m2.438s
user    0m0.000s
sys 0m0.000s

I am not usre if it really is a SYN Flood attack. If so, why cookies are not helping? Here is the info from netstat: 

netstat -tuna | grep :80 | grep SYN_RECV
tcp        0      0 85.254.49.180:80        92.37.173.66:3214       SYN_RECV   
tcp        0      0 85.254.49.180:80        81.26.91.4:49471        SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4940      SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.140.242:23259    SYN_RECV   
tcp        0      0 85.254.49.180:80        94.139.229.219:49827    SYN_RECV   
tcp        0      0 85.254.49.180:80        95.67.233.125:51267     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.2.69:7051        SYN_RECV   
tcp        0      0 85.254.49.180:80        95.67.239.40:54497      SYN_RECV   
tcp        0      0 85.254.49.180:80        195.91.229.193:58981    SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4925      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:59086      SYN_RECV   
tcp        0      0 85.254.49.180:80        92.113.26.124:3887      SYN_RECV   
tcp        0      0 85.254.49.180:80        77.34.83.254:26963      SYN_RECV   
tcp        0      0 85.254.49.180:80        195.208.64.130:3542     SYN_RECV   
tcp        0      0 85.254.49.180:80        81.26.91.4:49480        SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53130     SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4939      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:41696      SYN_RECV   
tcp        0      0 85.254.49.180:80        178.45.39.169:41758     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.37:51534     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.9.197:8249       SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:3531       SYN_RECV   
tcp        0      0 85.254.49.180:80        178.34.206.52:3409      SYN_RECV   
tcp        0      0 85.254.49.180:80        193.188.254.93:50317    SYN_RECV   
tcp        0      0 85.254.49.180:80        217.66.152.162:8883     SYN_RECV   
tcp        0      0 85.254.49.180:80        109.198.235.10:56382    SYN_RECV   
tcp        0      0 85.254.49.180:80        95.53.159.39:2256       SYN_RECV   
tcp        0      0 85.254.49.180:80        188.232.13.175:49819    SYN_RECV   
tcp        0      0 85.254.49.180:80        88.203.2.27:64080       SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.64.52:12382     SYN_RECV   
tcp        0      0 85.254.49.180:80        92.124.76.189:3416      SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:30532      SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53131     SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.123.1:44943      SYN_RECV   
tcp        0      0 85.254.49.180:80        176.51.255.3:1642       SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.165.112:56906     SYN_RECV   
tcp        0      0 85.254.49.180:80        88.203.2.27:64081       SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.37:51533     SYN_RECV   
tcp        0      0 85.254.49.180:80        176.51.211.131:1699     SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:22233      SYN_RECV   
tcp        0      0 85.254.49.180:80        211.167.112.18:58353    SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.32:38640     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.144.185.150:64421   SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4928      SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4927      SYN_RECV   
tcp        0      0 85.254.49.180:80        94.153.254.218:1084     SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:30384      SYN_RECV   
tcp        0      0 85.254.49.180:80        46.201.3.189:51032      SYN_RECV   
tcp        0      0 85.254.49.180:80        109.187.107.41:50565    SYN_RECV   
tcp        0      0 85.254.49.180:80        91.146.60.86:49266      SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53134     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.238.25:2515       SYN_RECV   
tcp        0      0 85.254.49.180:80        176.102.16.8:54291      SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4918      SYN_RECV   
tcp        0      0 85.254.49.180:80        95.153.164.165:26752    SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.239.76:46519      SYN_RECV   
tcp        0      0 85.254.49.180:80        94.139.229.219:49826    SYN_RECV   
tcp        0      0 85.254.49.180:80        188.239.193.48:49418    SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4919      SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.32:38639     SYN_RECV   
tcp        0      0 85.254.49.180:80        95.67.233.125:51266     SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:59092     SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.136.21:44804     SYN_RECV   
tcp        0      0 85.254.49.180:80        95.109.193.247:1206     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.112.11.130:2714     SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4941      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:52640      SYN_RECV   
tcp        0      0 85.254.49.180:80        37.79.93.27:64801       SYN_RECV   
tcp        0      0 85.254.49.180:80        91.203.96.76:45132      SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.238.25:2513       SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:60092     SYN_RECV   
tcp        0      0 85.254.49.180:80        188.239.193.48:49416    SYN_RECV   
tcp        0      0 85.254.49.180:80        178.130.42.68:60373     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.239.243.181:58110    SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53128     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.9.197:18870      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:53380      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.135.63.40:58845      SYN_RECV   
tcp        0      0 85.254.49.180:80        80.239.243.110:52234    SYN_RECV   
tcp        0      0 85.254.49.180:80        46.201.3.189:51028      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:53457      SYN_RECV   
tcp        0      0 85.254.49.180:80        85.235.176.138:12101    SYN_RECV   
tcp        0      0 85.254.49.180:80        109.187.107.41:50567    SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.48.29:4172       SYN_RECV   
tcp        0      0 85.254.49.180:80        188.232.13.175:49820    SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:6651       SYN_RECV   
tcp        0      0 85.254.49.180:80        91.198.143.6:45591      SYN_RECV   
tcp        0      0 85.254.49.180:80        85.235.176.138:50667    SYN_RECV   
tcp        0      0 85.254.49.180:80        176.209.98.72:53653     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.239.71:49701      SYN_RECV   
tcp        0      0 85.254.49.180:80        188.232.13.175:49817    SYN_RECV   
tcp        0      0 85.254.49.180:80        188.239.193.48:49417    SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:54175      SYN_RECV   
tcp        0      0 85.254.49.180:80        61.147.79.111:51039     SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:58854      SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53135     SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4938      SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4942      SYN_RECV   
tcp        0      0 85.254.49.180:80        176.209.98.72:53662     SYN_RECV   
tcp        0      0 85.254.49.180:80        2.74.51.158:1092        SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.140.242:48178    SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.129.42:29549     SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:28428      SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:50983     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.64.52:12381     SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:55459     SYN_RECV   
tcp        0      0 85.254.49.180:80        84.244.12.209:64975     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.2.121:10768      SYN_RECV   
tcp        0      0 85.254.49.180:80        84.240.248.206:3494     SYN_RECV   
tcp        0      0 85.254.49.180:80        195.91.229.193:52428    SYN_RECV   
tcp        0      0 85.254.49.180:80        95.109.193.247:1202     SYN_RECV   
tcp        0      0 85.254.49.180:80        79.105.204.56:56822     SYN_RECV   
tcp        0      0 85.254.49.180:80        85.15.184.141:56335     SYN_RECV   
tcp        0      0 85.254.49.180:80        164.177.225.31:50584    SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.238.25:2511       SYN_RECV   
tcp        0      0 85.254.49.180:80        84.240.248.206:3493     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.239.76:26950      SYN_RECV   
tcp        0      0 85.254.49.180:80        84.240.248.206:3495     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.144.185.150:58141   SYN_RECV   
tcp        0      0 85.254.49.180:80        178.215.97.15:13346     SYN_RECV

I've tried disabling syn cookies, but it had no effect. Seems, that server is limiting connection count, if you look at metrics "hits per minute" it looks like this:

enter image description here

Esterday, before crash, everyting worked fine. I would appriciate any info or advice on what could be the problem or how to diagnose it.

UPDATE

I am pretty sure it is not an attack. When I restart nginx - everything works for few hours and then, again syslog is full of:

Possible SYN flooding on port 80
Possible SYN flooding on port 9000

And nginx error log first get a lot of 104 errors:

2013/01/08 20:28:24 [error] 959#0: *2387458 recv() failed (104: Connection reset by peer) while reading response header from upstream

And then 110:

2013/01/08 21:27:19 [error] 30349#0: *760749 upstream timed out (110: Connection timed out) while connecting to upstream

This happens in evenings, when load reaches certain amount of hist (about 800 per second) and something goes wrong.

Turning syn cookies off and adjusting backlog give no effect.

Internet is full of similar claims, but no real answer can be found. Please help!

Temnovit
  • 1,107
  • 6
  • 19
  • 27
  • 1
    You need to figure out how this attack is hurting you so that you can figure out what to do about it. Are any of your cores maxed? Is your bandwidth maxed? – David Schwartz Dec 21 '12 at 16:23

3 Answers3

1

It sounds like there's something wrong with your upstream server that makes nginx appear very slow.

When nginx is appearing slow, do requests take a long time only if they're proxied over to your php-fpm + apc? Have you tried defining a non-proxied location, and seeing whether that exhibits any problems?

Is your php-fpm + apc running out of memory, out of connections / file-descriptors or out of worker threads / processes? Are you using OpenVZ? Or any other potentially broken-by-design kernel-level virtualisation? If not, do you have any other process or memory limits that you might be hitting? You can check for limits by doing an su into the user which runs your php-fpm / apc, and running limit in tcsh.

You should probably post your entire configs for nginx and php-fpm + apc, otherwise, it's a big guessing game. I'm not a php guy, but my educated guess is that you have some kind of connection or worker thread limit on your php-fpm+apc side, which your nginx proxy is exceeding.

Also, I see you have a nice graph of how things went downhill one day all of a sudden; did you make any recent changes or upgrades within a couple of days before that happened?

cnst
  • 12,948
  • 7
  • 51
  • 75
  • Also, one avenue I would go regarding this is to make the web-site as cache-friendly as possible (there are all sorts of ways to ensure a lot of stuff can be generalised between visitors and cached efficiently), and doing some serious caching within nginx. – cnst Jan 17 '13 at 01:03
0

Is it possible to rate limit the incoming connections using IPtables or nginx ?

Garfield Carneiro
  • 101
  • 4
0

I strongly recommend you to setup some default Iptables chains. Try to drop SYN floodings and log all dropped packets. You even don't know what the heck is happening on your network layer.

  • Stop flooded attacks, drop the packages
  • Log all dropped packets
  • Please, update us on what happens with this

Finally, it's simple for somebody who has experience with network analysis to detect your server crash and to stop the attack immediately.

mate64
  • 1,641
  • 4
  • 18
  • 29