I am having the weirdest issue: a web server which sometimes, only on very specific requests, will send a RST to the client after having sent the FIN datagram. First, a description of the setup:
- Server: VM guest Ubuntu 12.04.1 LTS in a Win7 x64 host, bridged mode, ufw is disabled on the host
- Client: iOS simulator in a Mountain Lion (hackintosh) VM guest on a Win7 x64 host, bridged mode.
- Both are on the same LAN. One is connected to the router via ethernet, the other via WiFi.
The HTTP logs on the server indicate that the cliens sometimes issues multipe subsequent identical requests. The client is simply re-trying after receiving an RST packet from the server.
I am attaching several tcpdumps:
- Good1 is the server-side tcpdump of a good session ("good" meaning no RST was generated).
- Good3 is another sever-side tcpdump of a good session.
The difference between Good1 and Good3 is the order in which ACK's were sent from the server, ACK'ing the client's request. The request arives in 2 segements. One for the HTTP headers and the second for a body containing an empty JSON object: "{}".
In Good1, the server ACKs both request after the arrival of the second request. In Good3, the server ACKs each request as soon as the request arrives. Not that it should make a difference.
- Bad1 is a dump, both client- and server-side, of a bad session.
- Bad2 is another bad session, this time server-side only.
Note that in all "bad" sessions, the server ACKs each request immediately after having received it. The situation is the same in a few other bad sessions, which are not included. Curiously, the same behaviour is exhibited in "Good3", making this observation somewhat meaningless. I don't see how it could matter.
Bottom line is that I can not pinpoint a significant difference between good and bad sessions.
My question is: why are those RST's being generated?
Or at least: How can I further debug this, or provide more helpful info here?
Edit Two new facts that I have learned:
- Section 4.2.2.13 of the RFC (1122) (and Wikipedia, in the article "TCP", under "Connection Termination") says that a TCP application on one host may close the connection before it has read all of the data in its socket buffer, and in such a case the TCP on the host will sent a RST to the other side, to let it know that not all the data it has sent has been read.
I'm not sure I completely understand this, since closing my side of the connection still allows me to read, no? It also means that I can't write any more.
I am not sure this is relevant, though, since I see a RST after FIN.
- There are multiple complaints of this happening with wsgiref (Python's dev-mode HTTP server), which is exactly what I'm using.
I'll keep updating as I find out more.
Thanks!
Good1 -- Server Side
13:28:2.308319 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [S], seq 94268074, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 943308864 ecr 0,sackOK,eol], length 0 13:28:2.308336 IP 192.168.1.132.5000 > 192.168.1.51.51479: Flags [S.], seq 1726304574, ack 94268075, win 14480, options [mss 1460,sackOK,TS val 326480982 ecr 943308864,nop,wscale 3], length 0 13:28:2.309750 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [.], ack 1, win 8235, options [nop,nop,TS val 943308865 ecr 326480982], length 0 13:28:2.310744 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [P.], seq 1:351, ack 1, win 8235, options [nop,nop,TS val 943308865 ecr 326480982], length 350 13:28:2.310766 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [P.], seq 351:353, ack 1, win 8235, options [nop,nop,TS val 943308865 ecr 326480982], length 2 13:28:2.310841 IP 192.168.1.132.5000 > 192.168.1.51.51479: Flags [.], ack 351, win 1944, options [nop,nop,TS val 326480983 ecr 943308865], length 0 13:28:2.310918 IP 192.168.1.132.5000 > 192.168.1.51.51479: Flags [.], ack 353, win 1944, options [nop,nop,TS val 326480983 ecr 943308865], length 0 13:28:2.315931 IP 192.168.1.132.5000 > 192.168.1.51.51479: Flags [P.], seq 1:18, ack 353, win 1944, options [nop,nop,TS val 326480984 ecr 943308865], length 17 13:28:2.316107 IP 192.168.1.132.5000 > 192.168.1.51.51479: Flags [FP.], seq 18:684, ack 353, win 1944, options [nop,nop,TS val 326480984 ecr 943308865], length 666 13:28:2.317651 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [.], ack 18, win 8234, options [nop,nop,TS val 943308872 ecr 326480984], length 0 13:28:2.318288 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [.], ack 685, win 8192, options [nop,nop,TS val 943308872 ecr 326480984], length 0 13:28:2.318640 IP 192.168.1.51.51479 > 192.168.1.132.5000: Flags [F.], seq 353, ack 685, win 8192, options [nop,nop,TS val 943308872 ecr 326480984], length 0 13:28:2.318651 IP 192.168.1.132.5000 > 192.168.1.51.51479: Flags [.], ack 354, win 1944, options [nop,nop,TS val 326480985 ecr 943308872], length 0
Good3 -- Server Side
13:28:3.311143 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [S], seq 1982901126, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 943309853 ecr 0,sackOK,eol], length 0 13:28:3.311155 IP 192.168.1.132.5000 > 192.168.1.51.51486: Flags [S.], seq 2245063571, ack 1982901127, win 14480, options [mss 1460,sackOK,TS val 326481233 ecr 943309853,nop,wscale 3], length 0 13:28:3.312671 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [.], ack 1, win 8235, options [nop,nop,TS val 943309854 ecr 326481233], length 0 13:28:3.313330 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [P.], seq 1:351, ack 1, win 8235, options [nop,nop,TS val 943309855 ecr 326481233], length 350 13:28:3.313337 IP 192.168.1.132.5000 > 192.168.1.51.51486: Flags [.], ack 351, win 1944, options [nop,nop,TS val 326481234 ecr 943309855], length 0 13:28:3.313342 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [P.], seq 351:353, ack 1, win 8235, options [nop,nop,TS val 943309855 ecr 326481233], length 2 13:28:3.313346 IP 192.168.1.132.5000 > 192.168.1.51.51486: Flags [.], ack 353, win 1944, options [nop,nop,TS val 326481234 ecr 943309855], length 0 13:28:3.327942 IP 192.168.1.132.5000 > 192.168.1.51.51486: Flags [P.], seq 1:18, ack 353, win 1944, options [nop,nop,TS val 326481237 ecr 943309855], length 17 13:28:3.328253 IP 192.168.1.132.5000 > 192.168.1.51.51486: Flags [FP.], seq 18:684, ack 353, win 1944, options [nop,nop,TS val 326481237 ecr 943309855], length 666 13:28:3.329076 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [.], ack 18, win 8234, options [nop,nop,TS val 943309868 ecr 326481237], length 0 13:28:3.329688 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [.], ack 685, win 8192, options [nop,nop,TS val 943309868 ecr 326481237], length 0 13:28:3.330361 IP 192.168.1.51.51486 > 192.168.1.132.5000: Flags [F.], seq 353, ack 685, win 8192, options [nop,nop,TS val 943309869 ecr 326481237], length 0 13:28:3.330370 IP 192.168.1.132.5000 > 192.168.1.51.51486: Flags [.], ack 354, win 1944, options [nop,nop,TS val 326481238 ecr 943309869], length 0
Bad1 -- Server Side
13:28:1.311876 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [S], seq 920400580, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 943307883 ecr 0,sackOK,eol], length 0 13:28:1.311896 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [S.], seq 3103085782, ack 920400581, win 14480, options [mss 1460,sackOK,TS val 326480733 ecr 943307883,nop,wscale 3], length 0 13:28:1.313509 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [.], ack 1, win 8235, options [nop,nop,TS val 943307884 ecr 326480733], length 0 13:28:1.315614 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [P.], seq 1:351, ack 1, win 8235, options [nop,nop,TS val 943307886 ecr 326480733], length 350 13:28:1.315727 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [.], ack 351, win 1944, options [nop,nop,TS val 326480734 ecr 943307886], length 0 13:28:1.316229 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [P.], seq 351:353, ack 1, win 8235, options [nop,nop,TS val 943307886 ecr 326480733], length 2 13:28:1.316242 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [.], ack 353, win 1944, options [nop,nop,TS val 326480734 ecr 943307886], length 0 13:28:1.321019 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [P.], seq 1:18, ack 353, win 1944, options [nop,nop,TS val 326480735 ecr 943307886], length 17 13:28:1.321294 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [FP.], seq 18:684, ack 353, win 1944, options [nop,nop,TS val 326480736 ecr 943307886], length 666 13:28:1.321386 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [R.], seq 685, ack 353, win 1944, options [nop,nop,TS val 326480736 ecr 943307886], length 0 13:28:1.322727 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [.], ack 18, win 8234, options [nop,nop,TS val 943307891 ecr 326480735], length 0 13:28:1.322733 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [R], seq 3103085800, win 0, length 0 13:28:1.323221 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [.], ack 685, win 8192, options [nop,nop,TS val 943307892 ecr 326480736], length 0 13:28:1.323231 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [R], seq 3103086467, win 0, length 0
Bad1 -- Client Side
t:11.374654 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [S], seq 920400580, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 943307883 ecr 0,sackOK,eol], length 0 t:11.375764 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [S.], seq 3103085782, ack 920400581, win 14480, options [mss 1460,sackOK,TS val 326480733 ecr 943307883,nop,wscale 3], length 0 t:11.376352 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [.], ack 1, win 8235, options [nop,nop,TS val 943307884 ecr 326480733], length 0 t:11.378252 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [P.], seq 1:351, ack 1, win 8235, options [nop,nop,TS val 943307886 ecr 326480733], length 350 t:11.379027 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [P.], seq 351:353, ack 1, win 8235, options [nop,nop,TS val 943307886 ecr 326480733], length 2 t:11.379732 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [.], ack 351, win 1944, options [nop,nop,TS val 326480734 ecr 943307886], length 0 t:11.380592 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [.], ack 353, win 1944, options [nop,nop,TS val 326480734 ecr 943307886], length 0 t:11.384968 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [P.], seq 1:18, ack 353, win 1944, options [nop,nop,TS val 326480735 ecr 943307886], length 17 t:11.385044 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [.], ack 18, win 8234, options [nop,nop,TS val 943307891 ecr 326480735], length 0 t:11.385586 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [FP.], seq 18:684, ack 353, win 1944, options [nop,nop,TS val 326480736 ecr 943307886], length 666 t:11.385743 IP 192.168.1.51.51472 > 192.168.1.132.5000: Flags [.], ack 685, win 8192, options [nop,nop,TS val 943307892 ecr 326480736], length 0 t:11.385966 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [R.], seq 685, ack 353, win 1944, options [nop,nop,TS val 326480736 ecr 943307886], length 0 t:11.387343 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [R], seq 3103085800, win 0, length 0 t:11.387344 IP 192.168.1.132.5000 > 192.168.1.51.51472: Flags [R], seq 3103086467, win 0, length 0
Bad2 -- Server Side
13:28:1.319185 IP 192.168.1.51.51473 > 192.168.1.132.5000: Flags [S], seq 1631526992, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 943307889 ecr 0,sackOK,eol], length 0 13:28:1.319197 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [S.], seq 2524685719, ack 1631526993, win 14480, options [mss 1460,sackOK,TS val 326480735 ecr 943307889,nop,wscale 3], length 0 13:28:1.320692 IP 192.168.1.51.51473 > 192.168.1.132.5000: Flags [.], ack 1, win 8235, options [nop,nop,TS val 943307890 ecr 326480735], length 0 13:28:1.322219 IP 192.168.1.51.51473 > 192.168.1.132.5000: Flags [P.], seq 1:351, ack 1, win 8235, options [nop,nop,TS val 943307890 ecr 326480735], length 350 13:28:1.322336 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [.], ack 351, win 1944, options [nop,nop,TS val 326480736 ecr 943307890], length 0 13:28:1.322689 IP 192.168.1.51.51473 > 192.168.1.132.5000: Flags [P.], seq 351:353, ack 1, win 8235, options [nop,nop,TS val 943307890 ecr 326480735], length 2 13:28:1.322700 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [.], ack 353, win 1944, options [nop,nop,TS val 326480736 ecr 943307890], length 0 13:28:1.326307 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [P.], seq 1:18, ack 353, win 1944, options [nop,nop,TS val 326480737 ecr 943307890], length 17 13:28:1.326614 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [FP.], seq 18:684, ack 353, win 1944, options [nop,nop,TS val 326480737 ecr 943307890], length 666 13:28:1.326710 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [R.], seq 685, ack 353, win 1944, options [nop,nop,TS val 326480737 ecr 943307890], length 0 13:28:1.328499 IP 192.168.1.51.51473 > 192.168.1.132.5000: Flags [.], ack 18, win 8234, options [nop,nop,TS val 943307896 ecr 326480737], length 0 13:28:1.328509 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [R], seq 2524685737, win 0, length 0 13:28:1.328514 IP 192.168.1.51.51473 > 192.168.1.132.5000: Flags [.], ack 685, win 8192, options [nop,nop,TS val 943307896 ecr 326480737], length 0 13:28:1.328517 IP 192.168.1.132.5000 > 192.168.1.51.51473: Flags [R], seq 2524686404, win 0, length 0
