I see a lot of these messages in /var/log/messages of my Linux server
kernel: nf_conntrack: table full, dropping packet.
kernel: __ratelimit: 15812 callbacks suppresse
while my server is under DoS attack but the memory is not still saturated. I am wondering what is the significance of the message and how to counter possible security implications.
The message means your connection tracking table is full. There are no security implications other than DoS. You can partially mitigate this by increasing the maximum number of connections being tracked, reducing the tracking timeouts or by disabling connection tracking altogether, which is doable on server, but not on a NAT router, because the latter will cease to function.
sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=54000
sysctl -w net.netfilter.nf_conntrack_generic_timeout=120
sysctl -w net.ipv4.netfilter.ip_conntrack_max=<more than currently set>
Here is a great article dealing with the subject. Essentially it just means that the table that IP tables (through APF or any other solution) uses to store persistant connections is full. At the expense of memory you can increase it.
I would suggest also setting your firewall rules to reject rather than drop
One temporary, fix if you need to keep your iptables NAT rules is:
linux:~# sysctl -w net.netfilter.nf_conntrack_max=131072
I say temporary, because raising the nf_conntrack_max doesn't guarantee, things will get smoothly from now on. However on many not so heavily traffic loaded servers just raising the net.netfilter.nf_conntrack_max=131072 to a high enough value will be enough to resolve the hassle.
– Increasing the size of nf_conntrack hash-table
The Hash table hashsize value, which stores lists of conntrack-entries should be increased propertionally, whenever net.netfilter.nf_conntrack_max is raised.
linux:~# echo 32768 > /sys/module/nf_conntrack/parameters/hashsize
The rule to calculate the right value to set is:
hashsize = nf_conntrack_max / 4
– To permanently store the made changes ;a) put into /etc/sysctl.conf:
linux:~# echo 'net.netfilter.nf_conntrack_count = 131072' >> /etc/sysctl.conf
linux:~# /sbin/sysct -p
b) put in /etc/rc.local (before the exit 0 line):
echo 32768 > /sys/module/nf_conntrack/parameters/hashsize
Note: Be careful with this variable, according to my experience raising it to too high value (especially on XEN patched kernels) could freeze the system. Also raising the value to a too high number can freeze a regular Linux server running on old hardware.
– For the diagnosis of nf_conntrack stuff there is ;
/proc/sys/net/netfilter kernel memory stored directory. There you can find some values dynamically stored which gives info concerning nf_conntrack operations in "real time":
linux:~# cd /proc/sys/net/netfilter linux:/proc/sys/net/netfilter# ls
-al nf_log/ total 0 dr-xr-xr-x 0 root root 0 Mar 23 23:02 ./ dr-xr-xr-x 0 root root 0 Mar 23 23:02 ../
-rw-r--r-- 1 root root 0 Mar 23 23:02 0
-rw-r--r-- 1 root root 0 Mar 23 23:02 1
-rw-r--r-- 1 root root 0 Mar 23 23:02 10
-rw-r--r-- 1 root root 0 Mar 23 23:02 11
-rw-r--r-- 1 root root 0 Mar 23 23:02 12
-rw-r--r-- 1 root root 0 Mar 23 23:02 2
-rw-r--r-- 1 root root 0 Mar 23 23:02 3
-rw-r--r-- 1 root root 0 Mar 23 23:02 4
-rw-r--r-- 1 root root 0 Mar 23 23:02 5
-rw-r--r-- 1 root root 0 Mar 23 23:02 6
-rw-r--r-- 1 root root 0 Mar 23 23:02 7
-rw-r--r-- 1 root root 0 Mar 23 23:02 8
-rw-r--r-- 1 root root 0 Mar 23 23:02 9
IV. Decreasing other nf_conntrack NAT time-out values to prevent server against DoS attacks
Generally, the default value for nf_conntrack_* time-outs are (unnecessery) large.
Therefore, for large flows of traffic even if you increase nf_conntrack_max, still shorty you can get a nf_conntrack overflow table resulting in dropping server connections. To make this not happen, check and decrease the other nf_conntrack timeout connection tracking values:
linux:~# sysctl -a | grep conntrack | grep timeout
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_events_retry_timeout = 15
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
All the timeouts are in seconds. net.netfilter.nf_conntrack_generic_timeout as you see is quite high – 600 secs = (10 minutes).
This kind of value means any NAT-ted connection not responding can stay hanging for 10 minutes!
The value net.netfilter.nf_conntrack_tcp_timeout_established = 432000 is quite high too (5 days!)
If this values, are not lowered the server will be an easy target for anyone who would like to flood it with excessive connections, once this happens the server will quick reach even the raised up value for net.nf_conntrack_max and the initial connection dropping will re-occur again …
With all said, to prevent the server from malicious users, situated behind the NAT plaguing you with Denial of Service attacks:
Lower net.ipv4.netfilter.ip_conntrack_generic_timeout to 60 – 120 seconds and net.ipv4.netfilter.ip_conntrack_tcp_timeout_established to something like 54000
linux:~# sysctl -w net.ipv4.netfilter.ip_conntrack_generic_timeout = 120
linux:~# sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 54000
This timeout should work fine on the router without creating interruptions for regular NAT users. After changing the values and monitoring for at least few days make the changes permanent by adding them to /etc/sysctl.conf
linux:~# echo 'net.ipv4.netfilter.ip_conntrack_generic_timeout = 120' >> /etc/sysctl.conf
linux:~# echo 'net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 54000' >> /etc/sysctl.conf
according to the awesome article at http://pc-freak.net/blog/resolving-nf_conntrack-table-full-dropping-packet-flood-message-in-dmesg-linux-kernel-log/
I found this link on the web which gives an in-depth answer: https://discuss.aerospike.com/t/how-to-handle-net-netfilter-nf-conntrack-count-max-exceeded/5051 Here's the relevant bits in case the link becomes stale:
Impermanent changes:
$ for i in close_wait fin_wait time_wait ; do sysctl -w net.netfilter.nf_conntrack_tcp_timeout_${i}=60 ; done
$ sysctl -w net.netfilter.nf_conntrack_max=524288
Revert change if it doesn't help. You can look at /var/log/syslog for errors:
$ for i in close_wait fin_wait time_wait ; do sysctl -w net.netfilter.nf_conntrack_tcp_timeout_${i}=120 ; done
$ sysctl -w net.netfilter.nf_conntrack_max=262144
Make change permanent:
$ cat <<EOF > /etc/sysctl.conf
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_max = 524288
EOF
