7

I'm trying to maximize the NFS/TCP bandwidth between a Linux client and a single NFS server over a 10Gbps link. Although each NFS mount has its own device (major/minor), everything comes down to a single TCP connection and this is a bottleneck.

Using iperf and multiple TCP connections, I can reach ~8Gbps, but NFS is stuck on a single TCP stream and limited to ~2Gbps (no tcp tuning yet).

On Solaris, there's the system setting rpcmod:clnt_max_conns which allow to set how many concurrent TCP connection the system is able to use to a single NFS server.

Is there a Linux equivalent or a way to do so ?

As a workaround, I know I could set many IP on the NFS server, and then load-balance the mountpoints on them but that wouldn't be optimal.

BenMorel
  • 4,215
  • 10
  • 53
  • 81
Benoît
  • 1,331
  • 3
  • 11
  • 23
  • 1
    Have you looked into the throughput from your disks? 8gbps is A LOT to ask of an i/o stack. – Daniel Widrick Sep 26 '13 at 08:10
  • That's not issue, the NAS server is high performance storage appliance definitely able to sustain this. – Benoît Sep 26 '13 at 10:43
  • What distro are you using on the server? How much RAM? number of cores? How many disks? type of disks? average iops per disk that you are seeing? operations in queue on the disks? time in queue per op on the disk? version of nfs? number of clients connecting to the server? There is a lot that goes into perf tuning and it is important to step back and look at the big picture first, and then narrow down. If you have done this, it helps to post numbers and stats you have already gathered. Even the clients may have something to do with it (eg, buffer size, mount options, dirty pages, etc). – MaQleod Sep 27 '13 at 03:06
  • I am not looking to fine-grained tune the system. I can see that a TCP connection doesn't go above 250MB/s but using netcat with 8 concurrent TCP streams, I can reach 800MB/s. That's why I am looking for a way to run multiple TCP streams to the same NFS server. And because the NFS servers are proprietary, it's going to be very hard to tune them, so I'd rather use a simple trick like multiple connections. – Benoît Sep 27 '13 at 16:09
  • Multiple connections is going to be the only way to do what you're looking for. A single stream should max out around 350MB/s per thread on a fine-tuned nfs server. In any case, if we don't know what distro or nfs version you're using, it is hard to give a good recommendation. – MaQleod Sep 27 '13 at 19:26
  • see section 5.6 http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html - it talks about increasing the number of nfsd instances. – MaQleod Sep 27 '13 at 21:26
  • RedHat 6.4, NFSv3. I know how to tune TCP, this will come later, but there's no way I can run a single stream at 10Gbps. Increasing nfsd is meant for server, not for client. I guess I'll have to use many IP then. I am not looking into ideas about NFS tuning, I was asking if it's possible to have more than 1 TCP connection from a Linux client to a server, as it's possible on Solaris. – Benoît Sep 28 '13 at 18:26
  • I am facing a similar issue: Storage subsystem can sustain 2500 megabytes/second locally, but NFS is not keeping up. What did you ultimately do here? – Nemo Apr 17 '14 at 00:46
  • I'm lacking the time to give a full answer, so I just throw http://multipath-tcp.org/pmwiki.php/Main/HomePage into the ring. It's GNU/Linux' system-level multi-connection solution and can encapsulate any tcp traffic. – bjanssen May 04 '15 at 05:40

3 Answers3

1

Starting from linux kernel 5.3 a new mount option "nconnect=X" is available, where X specifies the number of TCP connections to the server to use.

kofemann
  • 4,308
  • 1
  • 21
  • 27
1

You could switch back to UDP and use jumbo packets (4k datagrams for instance). Having your whole network support jumbo frames might help a lot, so that you don't have IP packets reassembly on each side.

Xavier Nicollet
  • 600
  • 3
  • 10
0

If you're using NFS v4.1 or greater you may be able to avail of pNFS (parallel NFS). As per the following release info from Red Hat, it's available in RHEL 6.4, but I haven't seen any updates indicating it's availability in other Linux distros.

https://www.redhat.com/about/news/press-archive/2013/2/red-hat-announces-general-availability-of-next-minor-release-of-red-hat-enterprise-linux-6

pNFS is defined in RFC5661 from 2010. You may want to investigate whether or not your storage appliance fully supports pNFS.

Ruairí N.
  • 645
  • 4
  • 8