how to decently configure net.ipv4.neigh.default.gc_thresh_n

Question

We recently had a storm of kernel: neighbour: arp_cache: neighbor table overflow! errors on some nodes of our kubernetes cluster.

After research we've figured out that we needed to bump our servers net.ipv4.neigh.default.gc_thresh{1,2,3} to match the needs of some of the new applications that have recently been added to our cluster.

While doing some research I was not able to find any decent way to calculate what would be the best suited values to put there.

How does one figures out what should be set there?

score 2 · Answer 1 · answered Feb 11 '19 at 20:20

Let's look at the documentation:

neigh/default/gc_thresh1 - INTEGER
    Minimum number of entries to keep.  Garbage collector will not
    purge entries if there are fewer than this number.
    Default: 128

neigh/default/gc_thresh2 - INTEGER
    Threshold when garbage collector becomes more aggressive about
    purging entries. Entries older than 5 seconds will be cleared
    when over this number.
    Default: 512

neigh/default/gc_thresh3 - INTEGER
    Maximum number of non-PERMANENT neighbor entries allowed.  Increase
    this when using large numbers of interfaces and when communicating
    with large numbers of directly-connected peers.
    Default: 1024

In order to overflow the neighbor table, you have to have more than gc_thresh3 neighbor table entries. In this case, Kubernetes pods, as each pod has its own network namespace with a unique interface and unique MAC address.

That's a lot of containers!

How you tune these values depends on the workload you expect to serve. Was it a one-off? Do nothing. Not sure what to do? Double everything and wait to see what happens. Do you know you have 5000 pods? Set the values so that you don't run out of space.

thanks for the reply. My interpretation of the documentation is that in my case the number of entries would not be limited to the amount of pods: `Increase this when using large numbers of interfaces and when communicating with large numbers of directly-connected peers.` the directly connected peers part has me thinking that each of the pods that establish or receive a lot of connections don't count as 1 in there. — vinni_f, Feb 11 '19 at 21:43
@vinni_f This table is storing MAC addresses, not tracking connections. — Michael Hampton, Feb 11 '19 at 21:50
from [here](http://linux-ip.net/html/tools-ip-neighbor.html) and by looking at the result from `arp -an` it seem to track connections to the MAC addresses. The text from `neigh/default/gc_thresh3` that you pasted from the doc in your answer also leads me to think it also track connections. Can you expand a bit more on that — vinni_f, Feb 11 '19 at 21:57

score 2 · Accepted Answer · answered Feb 14 '19 at 02:23

Here is what I ended up doing:

Collected data from all the servers with that:

for n in ip netns; do echo -en "$n: "; sudo ip netns exec $n arp -an|wc -l; done 2>&1 | grep -v 'No such file or directory' > /tmp/arp.log

count=0for i in cat /tmp/arp.log|awk '{ print $2 }'; do count=expr $count + $i; done

echo $count

rm -f /tmp/arp.log

Figured out my top value, doubled that and that number is my new gc_thresh3.

NOTE: Something that is not accounted for in the last commands is that some containers are using the host network so that means that the actual amount of entries in the table is probably lower. Since the hit on the RAM is insignificant I didn't care about that.

how to decently configure net.ipv4.neigh.default.gc_thresh_n

2 Answers2