2

Check the NIC ring buffer:

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:        4096
RX Mini:      0
RX Jumbo:     0
TX:        4096
Current hardware settings:
RX:         256
RX Mini:      0
RX Jumbo:     0
TX:         256

One can set "RX/TX" up to the limit shown in the "Pre-set maximums" like:

# ethtool -G eth0 rx 4096 rx 4096

Question is: by default;, why are these set so low (in every server I have, all of them are at 256) instead of a higher value, or their Hardware max capabilities? Is there any drawbacks (if yes, wich?) increasing these values?

CrazyRabbit
  • 119
  • 1
  • 7

1 Answers1

2

First, numbers you set aren't in bytes as many people think, they are in descriptors (and the descriptor size is hardware-dependent). So, when you increase ring length you request more memory to be allocated in kernel for these descriptors. In general, you want this kernel memory in L1 cache so that interrupt processing will be as fast as possible. Increasing ring size makes this less probable and in some cases completely impossible.

Next thing is interrupt coalescing - in general, when you increase ring buffer size, the NIC will adjust its low/high marks appropriately and will trigger interrupt when more data is buffered (that is less often). Time required by kernel to process these larger amounts of data during interrupt processing would also increase as the result.

All of the above results in a simple bucket effect - with a larger ring packet drop probability decreases and network latency increases. This may be perfectly fine if you're streaming large files over TCP and may be completely undesirable if you're a low-latency small packet application (i.e. gaming and such).

Default numbers you see are reasonable trade-off between the two.

Peter Zhabin
  • 2,276
  • 8
  • 10
  • Thanks for this explanation. I'm using an application to serve files over TCP and was having fifo erros so based on this https://serverfault.com/questions/650596/meaning-of-rx-queue-csum-err-and-rx-fifo-errors i've increased it. Is there any way, other than trial and error, to increase this to a "good value" between the 256 default and the 4096 max accepted value? – CrazyRabbit Sep 24 '21 at 17:46
  • 1
    @user1773988, for a specific workload for which packet size is known we use `iperf` with appropriate parameters to emulate load and see performance and system CPU load (which is also affected by this setting) to find a sweet spot. We once had to set these on a bunch of Linux firewalls in a role of poor man's BRAS in an ISP environment and that's where trial and error is the only method.. – Peter Zhabin Sep 24 '21 at 17:55
  • Thanks for you input. Subsidiary question; can these be decreased bellow 256? Let's say a Redis server, small packets, dedicated NIC, havig rx and tx set to 128? – CrazyRabbit Sep 24 '21 at 18:06
  • 1
    @user1773988 this can be lowered up to 48 AFAIR, but packet drop probabilities will increase esp in case of an unexpected pps bursts. Redis is a TCP application and drops are not going to do any good. I don't have first-hand experience with low latency stuff, just from ISP perspective, so we were almost always rising these values but to a certain point based on 24h monitoring stats. – Peter Zhabin Sep 24 '21 at 18:30