Linux fails to interpret ACK, keeps resending SYN+ACK

Question

The following is a wireshark dump of the problem occurring, IP-adresses replaced with 'client' and 'server':

4414.229553  client -> server TCP 62464 > http [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=3 TSV=116730231 TSER=0
4414.229633 server -> client  TCP http > 62464 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=2406364374 TSER=116730231 WS=6
4414.263330  client -> server TCP 62464 > http [ACK] Seq=1 Ack=1 Win=524280 Len=0 TSV=116730231 TSER=2406364374
4418.812859 server -> client  TCP http > 62464 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=2406365520 TSER=116730231 WS=6
4418.892176  client -> server TCP [TCP Dup ACK 778#1] 62464 > http [ACK] Seq=1 Ack=1 Win=524280 Len=0 TSV=116730278 TSER=2406365520
4424.812864 server -> client  TCP http > 62464 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=2406367020 TSER=116730278 WS=6
4424.891240  client -> server TCP [TCP Dup ACK 778#2] 62464 > http [ACK] Seq=1 Ack=1 Win=524280 Len=0 TSV=116730337 TSER=2406367020

So the normal SYN, SYN+ACK, ACK sequence seems to occur, except that the server doesn't seem to interpret the ACK. Instead it keeps resending the SYN+ACK, which the client dutifully keeps responding to with a duplicate of the previous ACK. I don't see how this could happen.

I noticed the problem because iptables connection tracking considers these connections established and keeps them in memory for a full 120-hour timeout. I have some firewall rules to prevent large numbers of concurrent connections, the limits of which people were hitting, without actually having that many connections active. The netstat command doesn't show these phantom connections.

Other info:

The server is a standard debian lenny system with a stock kernel:

Linux tb 2.6.26-2-686 #1 SMP Wed Aug 19 06:06:52 UTC 2009 i686 GNU/Linux

running:

Apache/2.2.9 (Debian) mod_ssl/2.2.9 OpenSSL/0.9.8g

I don't have all the info on the client (I can't reproduce locally), but it's a Mac running the Chrome browser.

I don't have any firewall rules messing with ACK packets. Basically I only filter SYN packets, all other TCP packets are allowed through. So I don't actually use the connection tracking for firewalling, other than counting concurrent connections and some graphing of TCP-Established packets compared to other packet types.

Edit: my iptables-rules pertaining to TCP port 80:

iptables -P INPUT ACCEPT
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 50 -j LOGDROP-CONN
iptables -A INPUT -p tcp --syn -m multiport --dports 80,443 -j ACCEPT
iptables -A INPUT -p tcp --syn -j REJECT --reject-with tcp-reset
iptables -A LOGDROP-CONN -m limit --limit 1/minute --limit-burst 1 -j LOG --log-prefix "ConConn "
iptables -A LOGDROP-CONN -j DROP

Edit 2: another dump, this time using tcpdump -vv:

16:05:52.999525 IP (tos 0x0, ttl 55, id 46466, offset 0, flags [DF], proto TCP (6), length 64) client.50538 > server.www: S, cksum 0x4429 (correct), 38417001:38417001(0) win 65535 <mss 1452,nop,wscale 3,nop,nop,timestamp 117224762 0,sackOK,eol>
16:05:52.999580 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) server.www > client.50538: S, cksum 0xa2ab (correct), 3062713115:3062713115(0) ack 38417002 win 5792 <mss 1460,sackOK,timestamp 2418739698 117224762,nop,wscale 6>
16:05:53.321788 IP (tos 0x0, ttl 55, id 24299, offset 0, flags [DF], proto TCP (6), length 52) client.50538 > server.www: ., cksum 0xe813 (correct), 1:1(0) ack 1 win 65535 <nop,nop,timestamp 117224765 2418739698>
16:05:56.252697 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) server.www > client.50538: S, cksum 0x9f7a (correct), 3062713115:3062713115(0) ack 38417002 win 5792 <mss 1460,sackOK,timestamp 2418740512 117224765,nop,wscale 6>
16:05:56.277250 IP (tos 0x0, ttl 55, id 15533, offset 0, flags [DF], proto TCP (6), length 52) client.50538 > server.www: ., cksum 0xe4c4 (correct), 1:1(0) ack 1 win 65535 <nop,nop,timestamp 117224798 2418740512>

Server side. It seems that the problem is on the server side as well, though it could of course be due to some strange packets or sequence coming from the client. — Bart Noordervliet, Jul 21 '11 at 11:52
Evidently your server isn't getting the ACKs because the repeat SYN,ACKs are coming 5 seconds later. Can you provide your iptables rules - you might think you're only filtering SYNs but it would be helpful to see for sure... — PP., Jul 21 '11 at 14:18
(on the first capture) what in the world is the server's last SYNACK doing with the TSER of the previous ACK packet? Something screwy going on with the TCP stack, there - if the ACK was lost somewhere, causing a duplicate SYNACK to be sent, it should have the TSER of the original SYN, not the second ACK - putting the second ACK's TSV in there means it was seen? — Shane Madden, Jul 21 '11 at 15:54
I agree, it's very strange. I'm trying to arrange to run a test without any firewall rules loaded, as per the suggestion offered below. That'll allow me to figure out whether this is a kernel bug, iptables bug or connection-tracking bug (or still regular operator-error, but I've begun to doubt that). — Bart Noordervliet, Jul 22 '11 at 13:26
I have ran into the same issue today. Seems that ACK is never received, but repeated SYN,ACK is with wrong TSER. System: Ubuntu 11.04 Desktop -> Ubuntu 10.04 Server, Dell hardware. Have You found a solution? — Coder, Aug 25 '11 at 14:47

score 2 · Answer 1 · answered Jul 21 '11 at 15:50

I'd brute force it. First I'd try if it works with iptables stopped. If it does, then it's something in the iptables.

Then I'd add rules one by one and watch which one causes connection failure. Then I'd play with that rule until it did what I wanted without totally disrupting the traffic.

If it doesn't work with iptables stopped, then it would start to be really strange.

score 0 · Answer 2 · answered Jul 22 '11 at 20:57

OK, I don't have a full answer, but I've learned a lot since this issue first came up. I'll share my insights here.

Firstly, the problem was triggered by Google Chrome opening a number of sockets (6 in my tests) to any website you open with it. It does this to kickstart downloading the various elements of a website in parallel. For simple websites that don't have many items, like mine, some of these pre-opened sockets will sit idle. I've read that other modern browsers do this as well.
The client triggering the connection-tracking problem probably has a broken home-router or something, because the idle sockets tend to disappear without any FIN or RST packets being sent to tear them down.
The repeating of the SYN+ACK packets seems to be normal TCP behaviour, I've been able to witness it on various third-party webservers by running a packet dump and then doing something like telnet www.website.com 80 and not sending any data. It could still be a Linux peculiarity though, as the servers I tested may well all run Linux. This behaviour also happened without any iptables-rules loaded, so it's really kernel-related.
After sending a few SYN+ACK packets without response, the Linux kernel will tear down its side of the connection. iptables connection-tracking doesn't seem to share this logic, so there the connection will remain in the ESTABLISHED state until it times out. The default timeout of this is 5 days (!). I'm planning to reduce this to something more sensible such as a few hours.
Repeating the SYN+ACKs after receiving ACK is not completely default behaviour, since I didn't see it when I opened a listening port with nc -l ### and connected to that. So it might be a certain TCP option Apache sets on its listening socket. Or it might be something entirely unique to Apache. Most other daemons announce themselves immediately after connection, so they are no valid test cases for this. The connection needs to do a 3-way handshake and then immediately fall idle.

Thanks for the insights given above though, they have been helpful getting to the bottom of this.

Linux fails to interpret ACK, keeps resending SYN+ACK

2 Answers2

Linked