What explains the effect of the NB parameter when using the HPL benchmark

Question

I executed the HPL benchmark on different homogeneous 8-node clusters of single board computers.

It is obvious (see figure 1) from the results that the NB parameter has a strong impact on the performance results.

Figure 1: HPL results of different homogeneous 8-node clusters of single board computers

Why has for some clusters a NB value like 64MB a positive effect and for others it has a negative effect? The same is for a NB value like 192 MB or 256 MB.

I see the results, but I cannot explain why they are this way.

All I can say is that the CPU is for all clusters the bottleneck because all cores of the nodes are 100% utilized when executing the benchmark.

Update 1: Maybe the superuser cummunity is not optimal for this question. The visibility is quite low...