8

There is a lot of confusing information out there on pNFS, and I have some very simple questions:

  • How can I check if it is running/enabled/active on my system?
  • Is pNFS widely available today or still experimental? In particular what is its status on Ubuntu 18.04? Does nfs-kernel-server use pNFS?
  • What is the difference/relationship between NFSv4.1 and pNFS?
  • Is pNFS an optional part of NFSv4.1 or is it always active when using NFSv4.1?
  • Is pNFS also a part of NFSv4.2 or just 4.1?
  • pNFS is so called because it enables parallel access from client to server. I assume this means that without pNFS a client only has a single NFS process running that connects to the server by a single TCP connection. Is this correct?
isarandi
  • 331
  • 1
  • 11
  • 2
    I don’t think anyone uses pNFS. What’s your use case? – ewwhite Sep 26 '19 at 12:02
  • Use case is a storage server with about 20 clients. Some clients are compute servers with many parallel processes accessing the storage. I thought pNFS could help with establishing multiple parallel connections to the server, as we already have multiple parallel Ethernet connections to the storage server by bonding. The performance is mostly sufficient, except for some times. I'm simply exploring options and trying to understand possible directions for scaling later. We also have different bottlenecks at the moment (server disk io), but again, I'd like to understand the NFS space a bit better. – isarandi Sep 26 '19 at 12:07
  • And your comment is exactly what I am looking for: the status in practice right now. It's easy to find powerpoint slides from 5 years ago and RedHat docs talking about it, but it's very hard to find the practical status. – isarandi Sep 26 '19 at 12:09
  • 1
    Try to identify your bottleneck. You can increase NFS daemons, add bandwidth, etc. – ewwhite Sep 26 '19 at 12:31
  • The true bottleneck used to be server disk access. It was greatly helped by using cachefilesd as the clients usually work on the same 10-100 GB without changing it, so that can be easily cached. The new bottleneck seems to be metadata read (I guess mtime for checking if the local cache is still valid). atop already shows 100% disk utilization for about 300 reads/second, with ~20 KB/read each (From a 12 disk RAID6 with controller-based CacheCade SSD caching). Somehow separating inode attributes like mtime from the data onto a fast device could probably improve it. Not sure if possible with XFS. – isarandi Sep 26 '19 at 12:46
  • 2
    @isarandi You don't need pNFS for parallel processing of your multiple concurrent NFS requests. Even NFSv3 will work this way just fine. – BaronSamedi1958 Sep 26 '19 at 12:46

1 Answers1

6

pNFS is an optional part of NFSv4.1 spec. IOW, you need nfs server and client to talk nfsv4.1+ to make use of it. As this is an option functionality, your client and server should support it. Linux kernel support pNFS since version 3.9 and with every new release it's get more mature.

In general, pNFS allows to spread request for a single file to multiple so called data-servers. For example, an application issues 16MB read, but the NFS server sends 2MB reads to 8 different servers that have required blocks. There are several server implementations that support pnfs:

  • Linux server; supports only block layout
  • Gangesha nfs based implementations
  • NetApp 8.1+
  • HammerSpace
  • dCache (the project where I am involved)
  • FreeBSD

I can't talk about other systems, but we are moving terabytes of data per day on systems with hundreds of pNFS data-servers exposed as a single NFS server.

Check your NFS server for pNFS capability. On the client side, ensure that the server is mounted with nfsv4.1 or above. If the server is pNFS capable, then you should see the appropriate layout driver loaded as a kernel module:

$ lsmod | grep layout
nfs_layout_nfsv41_files    36864  0
nfs_layout_flexfiles    53248  0
nfsv4                 708608  11 nfs_layout_flexfiles,nfs_layout_nfsv41_files
nfs                   323584  3 nfsv4,nfs_layout_flexfiles,nfs_layout_nfsv41_files
sunrpc                454656  22 nfsv4,auth_rpcgss,nfs_layout_flexfiles,lockd,nfs_layout_nfsv41_files,rpcsec_gss_krb5,nfs
$

as well as the corresponding kernel message:

$ dmesg  | grep Layout
[41827.049921] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
$

The Layout type specifies how clients talk to various data server. Check the Storage Protocols section in RFC 5661.

Community
  • 1
kofemann
  • 4,308
  • 1
  • 21
  • 27