Connection remains flagged as ESTABLISHED even if host is unconnected

6

2

I am working on a Linux embedded distribution and I need to monitor incoming and outcoming connections (if they are still actives etc.) and I am experiencing issues because I noticed that, even if the remote host turn off the connection, I can see the connection as ESTABLISHED through netstat.

I have two different versions of this Linux embedded system: one which uses an old 2.6.34 kernel and it works fine, while the one which is failing uses a kernel 3.18.18.

Looking for more information I suspect I had to set/change TCP timeouts, so I configured them through syctl.conf but doesn't change anything. The system which uses 2.6.34 kernel doesn't have TCP timeouts configured, so it makes me think that TCP are not the root of my problem.

Do you have an idea about what could be the reason of the fact that the connection remain as ESTABLISHED?

simo-zz

Posted 2016-01-05T08:29:05.300

Reputation: 191

This is just the way TCP works. A connection can remain established for days or even weeks without a single packet being exchanged. If you don't send, there's no way to know what will happen if you do send. Your question is too vague to give you a helpful answer to though because you don't explain your outer problem, just ask for help with a proposed solution. Are you writing application code that actually implements the TCP endpoint? If so, what protocol does it use on top of TCP? Or are you trying to monitor some other application's connections? Or what? – David Schwartz – 2016-01-05T09:15:57.000

I am trying to monitor some other application's connections. If I was writing code I would have specified it. I have a remote host (Windows client) which connects to my Linux system (server) and once the client is connected it starts to retrieve data from server through TCP protocol. If the client disconnects, the connection on the server does not disappear. It remains as established. The same softwares are used on a old Linux system (as specified in the question) and it works all fine. – simo-zz – 2016-01-05T09:43:44.357

That's all normal. The disconnection should be detected as soon as the connected side tries to send data. Obviously, a disconnected device can't send any data, so there's no way to magically know that it's disconnected until it fails to respond to a query. – David Schwartz – 2016-01-05T09:45:01.017

So the TCP timeout should work for this, right ? What could be the reasons the TCP timeout I set does not work ? – simo-zz – 2016-01-05T09:49:34.187

1The timeout only works if the connected side is trying to send data. Otherwise, there's nothing to time out. A TCP connection can remain established for days without a single packet being sent if neither side tries to send any data. In general, the only way to detect a disconnected device is to try to send data to it. If you don't ever send any data, you're simply not assured of detecting it. What application protocol (on top of TCP) are we talking about here? – David Schwartz – 2016-01-05T09:50:43.930

OK. I understand that the problem could be on the server side application and I should review that. – simo-zz – 2016-01-05T09:56:04.537

There probably isn't a problem. Nothing you've said suggests that anything is wrong. – David Schwartz – 2016-01-05T09:56:59.620

Just a last question. TCP keep alive should not do this ? – simo-zz – 2016-01-05T09:57:26.800

TCP keep alive is used for protocols not designed to work on top of TCP that still need to detect a disconnection without sending data. Most likely you're dealing with some protocol designed to work on top of TCP, so keep alives are not needed. (You're not the application endpoint anyway, right?) – David Schwartz – 2016-01-05T09:58:25.987

The server is a GNSS modem which I use to send and receive GPS data. The GPS data is retrieved from a GPS board and sent to the clients through a GPS caster application. – simo-zz – 2016-01-05T10:23:17.070

1Well then the sending of data should cause the connection loss to be detected. The application on the other side needs to send data periodically too, or detect the absence of received data. It has to do something if it needs to detect connection loss. – David Schwartz – 2016-01-05T10:23:54.107

Answers

2

If a host disconnects from the other and you still see its connection as ESTABLISHED, it's probably related to the fact it's not honoring the TCP protocol and not closing the connection cleanly.

The netstat output is an interpreter of the current state of TCP connections. If a client wants to disconnect/close the socket that has previously been open and established, they should notify this to the remote system. This is done sending the FIN request to the other node (more info here), in this case, the server.

It they fail to do so, the client indeed disconnects, but the remote server keeps thinking that the client is still connected and thus keeping their state as ESTABLISHED, and that's where the tcp_keepalive_time parameter join the equation. As no further packets will be received, the kernel will wait the specified time to this parameter to time-out the connection and forcibly close it.

You can debug this issue using the tcpdump tool. You could simply listen to the connection from the client host on the server side and check if it fails to send the FIN request.

tcpdump host X.X.X.X and port Y

nKn

Posted 2016-01-05T08:29:05.300

Reputation: 4 960

OK. But I set the parameter tcp_keepalive_time many times, so there should be something else. – simo-zz – 2016-01-05T09:26:22.933

Did you run sysctl -p after modifying the file? – nKn – 2016-01-05T09:29:54.103

Yes, I also executed this command. It's all as I have configured. – simo-zz – 2016-01-05T09:36:04.080

Then either the client has been not closed, or for some reason there still are packets being received on the server side. Note that ACK are not considered "idle-breaking" packets in this case. Note as well that any modifications made to the file do not affect any previously ESTABLISHED sockets, so it just affects newly established ones. – nKn – 2016-01-05T09:39:37.037

Asked: 2016-01-05T08:29:05.300

Viewed: 10 436 times

Active: 2018-12-11T14:36:54.373