1

This problem has been driving me crazy.

I have an NFS server with NFS shares mounted on various clients. However, whenever I have to reboot the NFS server, I invariably end up with a bunch of "Stale file handle" errors on the mounts across all my clients which forces me to have to manually unmount and remount my NFS shares on the clients.

I have checked my exports on my NFS server with cat /etc/exports and I am passing the same fsid for each NFS export across reboots.

My questions:

  1. How does Industry handle this problem? I have a hard time imagining sysadmins going in and manually unmounting/remounting each client or simply restarting connected clients en masse. Or is that really how this is handled? (Aside from the standard, "We never have a downtime and we never have to restart NFS servers.")
  2. Why does this happen? Is this because, even though the fsid may be the same, the NFS server recalculates file handles which may not be the same across reboots?
  3. Is there something I should be doing better in my mount config to prevent this?

/etc/fstab:

[NFSserver]:/mnt/backup /mnt/backup nfs bg,nfsvers=3,tcp 0 0

Per this post, it was suggested to add hard and intr mount options, but that doesn't seem to have made any difference.

  1. If all else fails, should I just fall back to using a bash script to monitor the mount/directory for a stale file error and have it perform a umount/mount cycle?

Thanks in advance.

-TorqueWrench

TorqueWrench
  • 13
  • 3

2 Answers2

2

You're using NFS version 3, which needs several helper services in addition to the main NFS service in port 2049. One of these is rpc.statd, which has a key role in detecting reboots and recovering/clearing NFS locks after a reboot.

These helper services may be located in random ports, and they are discovered by contacting the RPC port mapper (usually a process named rpcbind on modern Linuxes). In modern networks with firewalls, such behavior can make things difficult: even though you may find them in deterministic-looking ports after a reboot, they may get allocated to quite different port numbers if/when you restart NFS services.

Fortunately, on many modern Unix-like systems, you can lock down the port numbers of the NFS lock manager (historically rpc.lockd, nowadays usually implemented in-kernel), rpc.statd and rpc.mountd. This is essential if you want to pass NFSv3 through firewalls with any sort of reliability.

For RHEL and related distributions, you can lock down the NFS helper port numbers by adding these lines in /etc/sysconfig/network:

LOCKD_TCPPORT=4045
LOCKD_UDPPORT=4045
STATD_PORT=4046
MOUNTD_PORT=4047

For Debian and related distributions, you might add this line to /etc/modprobe.d/nfs.conf:

options lockd nlm_udpport=4045 nlm_tcpport=4045

... and this line in /etc/default/nfs-common:

STATDOPTS="-p 4046"

... and this line in /etc/default/nfs-kernel-server:

RPCMOUNTDOPTS="-p 4047" # you may want to add a --manage-gids option here

(You can use different port numbers if you wish, but 4045 is the default port for NFSv3 lock manager in Solaris and hard-coded for the same in HP-UX 11.31.)

But there is another pitfall in the NFSv3 protocol. Although you can successfully mount a NFS share using just IP addresses, the NFSv3 lock protocol internally uses hostnames. Both the client and server must know each other by the correct names, otherwise the NFS file locking and lock recovery after a reboot won't work. And the "correct name" for each system is the name reported by uname -n.

So, if uname -n returns server.example on the server and respectively client.example on the client, then you should make sure those exact names will resolve to the IP addresses the hosts need to use for NFS. In other words, the server must be able to contact the client's rpc.statd using the name client.example and vice versa.

If you don't, everything may seem to work well at first... but when either end reboots, you may get those Stale file handle errors.

telcoM
  • 4,153
  • 12
  • 23
  • Thank you! What if I just mount by IP address (instead of hostname), does this still apply (since I am using IP and not hostname)? Additional question: I am using static ports. The STATD_PORT certainly does sound like the most likely problem, [which I have set on port 32766](https://forums.engineerworkshop.com/t/running-nfs-behind-a-firewall-setting-up-static-nfs-ports-on-unraid/148) Interestingly, I [don't see it in the verbose output when I mount it](https://forums.engineerworkshop.com/t/experiments-in-nfs-troubleshooting/153). Does that mean it's not being used by the client? – TorqueWrench May 18 '20 at 14:55
  • Yes, it does apply even if you use mount by IP address... because *the lock protocol will always use hostnames*, as in "I am $(uname -n) and want to lock this file". The connections to `lockd` and `statd` ports will happen later, when they are actually needed. Please see `man rpc.statd` for details. – telcoM May 19 '20 at 15:29
1

In addition to the excellent answer from @telcoM, I would like to suggest two other possible solutions:

  • mount nfs with the noac option (beware that this will cause a performance loss when issuing ls on large directory or stat on many files)

  • use NFS v4.1 (v4.0 had some bugs leading to "stale file handling", so be sure to select the v4.1 protocol).

shodanshok
  • 44,038
  • 6
  • 98
  • 162