What is it happening if a port is constantly (since one or more days) in FIN_WAIT1?

1

I've a problem when I try to communicate with a server.

If I use the command line:

netstat -np 10.aaa.bbb.12

where 10.aaa.bbb.12 indicates the server address; I obtain an extract with the following results:

    tcp        0     87 10.xxx.yyy.4:59438          10.aaa.bbb.12:7955          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:36100          10.aaa.bbb.12:7952          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:59422          10.aaa.bbb.12:7955          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:41678          10.aaa.bbb.12:7951          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:60999          10.aaa.bbb.12:7953          FIN_WAIT1   -                   
    tcp        0     86 10.xxx.yyy.4:59456          10.aaa.bbb.12:7955          ESTABLISHED 21203/sender   
    tcp        0     87 10.xxx.yyy.4:41694          10.aaa.bbb.12:7951          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:36084          10.aaa.bbb.12:7952          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:60966          10.aaa.bbb.12:7953          FIN_WAIT1   -                   
    tcp        0     87 10.xxx.yyy.4:41711          10.aaa.bbb.12:7951          FIN_WAIT1   -                   
    tcp        0     86 10.xxx.yyy.4:32783          10.aaa.bbb.12:7953          ESTABLISHED 21269/sender   
    tcp        0     87 10.xxx.yyy.4:60983          10.aaa.bbb.12:7953          FIN_WAIT1   -                   
    tcp        0     86 10.xxx.yyy.4:41728          10.aaa.bbb.12:7951          ESTABLISHED 21225/sender   
    tcp        0     86 10.xxx.yyy.4:36118          10.aaa.bbb.12:7952          ESTABLISHED 21247/sender

The ports in FIN_WAIT1 have this state since one day. I don't understand why.

All the server ports in above indicated state accepts connection and it seems the server accept commands, but the server doesn't reply as we expect. All replyes waited from the server fall into timeouts.

I may verify the connection with the server using this command line:

nc 10.150.224.12 7955 -w 10 <ts.txt

Where the file ts.txt contains a command that requires a specific reply known to us.

Sir Jo Black

Posted 2019-03-05T10:39:44.490

Reputation: 113

What are the server/client applications? Are you writing one or both? Sockets in this state are still open in one direction - IIRC the server-to-client direction has had shutdown() called, but the client-to-server direction is still active... Have you called shutdown(fd, SHUT_RDWR) and close(fd) on the socket from both ends? Is the client application still running? The server application does not need to stay running for sockets to remain in this state.

– Attie – 2019-03-05T10:55:55.000

Steps to temporarily resolve the issue are outlined here: https://serverfault.com/a/637203/405400... however this won't help in the long run if your application is poorly written.

– Attie – 2019-03-05T10:57:25.450

Can you run netstat -tnp on 10.aaa.bbb.12 and see what is using those ports? I suspect the process is still running, or is accidentally suspended (^Z) or similar. – Attie – 2019-03-05T11:00:03.167

Answers

3

This seems to occur when the server has an active connection with a client and wants to shut down the TCP connection (probably in response to a normal application layer "exit"). The server sends the client a packet with a "FIN" bit set. At this point, the server is in FIN_WAIT_1 state. When the client gets the FIN packet it goes into the CLOSE_WAIT state and sends an acknowledgment packet back to the server, which then will go into the FIN_WAIT_2 state.

This means that your clients are dropping off without an orderly shutdown of their TCP sockets.

The best solution is to correct the application.

An immediate and temporary resolution is to run this ServerFault script:

# record what tcp_max_orphans's current value
original_value=$(cat /proc/sys/net/ipv4/tcp_max_orphans)

#set the tcp_max_orphans to 0 temporarily
echo 0 > /proc/sys/net/ipv4/tcp_max_orphans

# watch /var/log/messages
# it will split out "kernel: TCP: too many of orphaned sockets"
# it won't take long for the connections to be killed

# restore the value of tcp_max_orphans whatever it was before. 
echo $original_value > /proc/sys/net/ipv4/tcp_max_orphans

# verify with 
netstat -an|grep FIN_WAIT1

You might also set tcp_orphan_retries found at /proc/sys/net/ipv4/tcp_orphan_retries, to reduce the time for these sockets to stay in this state. Beware that the value of 0 actually means 8.

But again, the best solution is to correct the application to close properly.

harrymc

Posted 2019-03-05T10:39:44.490

Reputation: 306 093

Unlikely I may not try the script because I haven't root right! :( – Sir Jo Black – 2019-03-05T11:32:00.180

1You could maybe ask an administrator to do that and set tcp_orphan_retries to a lower value than 8. But someone should really fix the application. – harrymc – 2019-03-05T11:42:26.580

Our client applications seems not be affected to this kind of problems. BTW me and other technicians are searching for any socket-closure issues. – Sir Jo Black – 2019-03-05T11:49:06.623

What do it happen of the connection if a connected client is closed killing the pid by means of the kill -KILL command? – Sir Jo Black – 2019-03-05T11:51:30.043

Asked: 2019-03-05T10:39:44.490

Viewed: 344 times

Active: 2019-03-05T11:03:48.677