I have a node in our cluster which gets lots of "nf_conntrack: table full, dropping packet" messages in the syslog. I checked the nf_conntrack_count and it was running right up against the nf_conntrack_max. Looking into the table, I saw most of the entries were DNS requests, so I added these rules to the "raw" netfilter table.
$ sudo iptables -t raw -vnL
Chain PREROUTING (policy ACCEPT 146M packets, 19G bytes)
pkts bytes target prot opt in out source destination
33M 4144M CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53 CT notrack
33M 2805M CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 CT notrack
Chain OUTPUT (policy ACCEPT 73M packets, 8311M bytes)
pkts bytes target prot opt in out source destination
10785 882K CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 CT notrack
0 0 CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53 CT notrack
That left the count hovering around 13000 or so, and nf_conntrack_max is set to 65535. However I still keep getting the dropped packet messages. Most of the rest of the packets are UDP, and I set the nf_conntrack_udp_timeout as low as 1 second, leaving the nf_conntrack_count around 1000. However I still get the dropped packed message.
From here, if I raise the max, it will stop the dropped packet messages, however I don't see why that is necessary.
I am running docker, and there is an elasticsearch container (this problem seems to happen on whichever node is running elasticsearch). Not sure if it is relevant, but the node has 48 cores.
$ uname -a
Linux qtausc-pphd0128 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
So why is it dropping packets when the count is far less than the max?