0

We are having connection stall issues on many of our http servers. As a test I have created a script that connects to the webserver using curl and tries do download a txt file of about 3mb size.

Every now and then the speed goes down to 0 and the connection hangs.

When I disable iptables the problem resolves and everything works ok. 

 iptables -vL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   28  2080 ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere
    0     0 ACCEPT     all  --  lo     any     anywhere             anywhere
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere            state NEW tcp dpt:ssh
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere            state NEW tcp dpt:http
   17  1924 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 15 packets, 1564 bytes)
 pkts bytes target     prot opt in     out     source               destination

There is couple of routers and firewalls in between.

Any ideas what this could cause?

Ah. I use Centos 6.3 Kernel 2.6.32-279.19.1.el6.x86_64 Iptables version is: iptables-1.4.7-5.1.el6_2.x86_64

Drop message

IPTables-Dropped: IN=eth1 OUT= MAC=00:50:56:a6:00:9e:00:16:9c:ff:20:00:08:00 SRC=xxx DST=xxx LEN=80 TOS=0x00 PREC=0x00 TTL=62 ID=55755 DF PROTO=TCP SPT=34787 DPT=80 WINDOW=1539 RES=0x00 ACK URGP=0
Chris
  • 597
  • 1
  • 6
  • 18
  • Anything in the nat table `iptables -vL -t nat`? Also, how do you disable iptables? – Khaled Jan 16 '13 at 08:32
  • Nothing in the nat table as I don't do any nat. I disable iptables by shuting down the service . (service iptables stop) – Chris Jan 16 '13 at 09:17
  • Are these the iptables settings on the web server machine? – David Schwartz Jan 16 '13 at 09:31
  • yes. it somehow fixed the issue by setting tcp_sack to 0 on the webserver. Not sure if this is whise or not or what the reason is that it works like that – Chris Jan 16 '13 at 09:55

1 Answers1

0

Most likely, you are running out of connection tracking entries. Raise net.nf_conntrack_max. If this is due to an attack, you may need to raise it quite a bit and decrease some settings like net.netfilter.nf_conntrack_tcp_timeout_syn_recv.

You could just rewrite your rules without connection tracking -- you don't seem to need it. That might reduce your flexibility in the future though.

David Schwartz
  • 31,215
  • 2
  • 53
  • 82
  • Hi. I was able to fix it by disabling tcp_sack. not sure what this does or if this is a whise idea. maybe this hint tells you more than me :) – Chris Jan 16 '13 at 09:44
  • @Chris: Are you sure that's the only change you made? That would suggest you have some kind of ancient firewall in the path or are under a [sack attack](http://serverfault.com/a/11002/91987). – David Schwartz Jan 16 '13 at 09:54
  • yes. I can watch it live how the connection drops as soon as I enable this again... I am not aware on the firewalls in between but I have a meeting tomorrow with one of the network guys. anything I could ask them? By the way. This happens to all our webservers that are in that specific network zone – Chris Jan 16 '13 at 09:58
  • Ask them if there's anything unusual about the way that zone is connected to other networks. Report your findings and see if they believe you. ;) – David Schwartz Jan 16 '13 at 10:43