How to properly benchmark latency over LAN?

Question

I'm trying to estimate latency over LAN. To do so obvious idea was to use ping. So here it goes:

64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.241 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.190 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.177 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.190 ms

Okay well but there's a firewall so maybe 1 second is interrupting conntrack in firewall, lets try smaller interval ping -i 0.05:

64 bytes from 192.168.1.1: icmp_seq=27 ttl=64 time=0.119 ms
64 bytes from 192.168.1.1: icmp_seq=28 ttl=64 time=0.106 ms
64 bytes from 192.168.1.1: icmp_seq=29 ttl=64 time=0.104 ms
64 bytes from 192.168.1.1: icmp_seq=30 ttl=64 time=0.109 ms

It looks signigicantly faster. So lets take even smaller interval ping -i 0.005:

64 bytes from 192.168.1.1: icmp_seq=2139 ttl=64 time=0.059 ms
64 bytes from 192.168.1.1: icmp_seq=2140 ttl=64 time=0.073 ms
64 bytes from 192.168.1.1: icmp_seq=2141 ttl=64 time=0.056 ms
64 bytes from 192.168.1.1: icmp_seq=2142 ttl=64 time=0.068 ms

At this point I collected big log of pings, then calculated average and it turned out to be: 0.068 ms. When I repeated test for 8k jumbo frames ping -M do -s 8000. it settled around 0.220 ms.

So the question is - what is actual latency? Why decreasing interval decreases ping response time as well?

I'm more interested in big-throughput latency than occasional one because it's supposed to route NFSv4, so probably 0.220 is realistic one but I'd like someone to confirm.

Answer 1

If you're looking for a better tool than ping, look at the open source Flent tool. Flent, the "FLExible Network Tester", is a throughput-with-latency test tool created by some very smart networking researchers who spent many years looking at how network latency degrades under load (particularly the bufferbloat problem).

Most people measuring latency on their network don't really care as much about latency between small once-per-second packets on an otherwise idle network, they care more about the latency involved in a large packet flow. That's what Flent tests.