0

I am using mosquitto as a broker to exchange data between clients using the MQTT protocol. But recently I have the following phenomenon.

My mosquitto sometimes connects very hard and sends it too. After checking everything, mosquitto is still running normally, CPU and memory don't spike. However, when checking with netstat, I see FIN_WAIT1 queue with up to 2000 connections. And that caused the MQTT connection to be affected and I specifically checked there was no sign of being attacked from a specific IP, each connection in it was from a different IP, I think that is when my client disconnect. Because FIN_WAIT1 appears when the client sends a packet to the server to disconnect.

Is it because of the buffer overflow of TCP port ?

My client volume is always around 10000. My configuration is 4 core / 4gb RAM.

berndbausch
  • 973
  • 7
  • 11
Nguyen Ba Hien
  • 1
  • You may want to explain what you mean by "connects very hard and sends it too". – berndbausch Mar 02 '21 at 00:47
  • sorry, I mean, when the FIN_WAIT queue is high, my clients take a long time to connect to the server, and sending the messages takes a lot of time. Like when the network connection is not stable. Only when the FIN_WAIT1 queue is very low will connections be established quickly. – Nguyen Ba Hien Mar 03 '21 at 01:11

1 Answers1

0

I think you need to modify some sysctl values. The linux kernel is default tuned to be good with about anything, and when you do a service with some load you might need to tweak stuff, especially network buffers etc.

Have a look at changing /proc/sys/net/ipv4/tcp_max_orphans to something lower than the defaults to get them killed faster (look at the 3.3.11. tcp_max_orphans section here), and possible do more modifications such as per this guide.

Fredrik
  • 528
  • 2
  • 10
  • Thanks for answer. Currently working by changing the "Tcp_max_orphans" value lower but I get a warning from the system "TCP: too many orphaned sockets". Is that worrying? – Nguyen Ba Hien Mar 03 '21 at 01:09
  • Not really, you are modifying this since you have too many FIN1 connections, so the warning is valid, but this is how you mitigate it. I would suggest you look on the client side instead, why are they dropping the connection without closing it, or what is happening there. You shouldn't need to have them in the FIN state, they should close themself instead of the server doing it. – Fredrik Mar 03 '21 at 07:19
  • Since the server is currently being used as the MQTT packet exchange, this is a protocol for IoT applications so users can suddenly lose connection without notifying the server in time. But before now I have not seen such a phenomenon in over a year. Or could it be due to a server attack? Thanks – Nguyen Ba Hien Mar 03 '21 at 08:58