I run two nfs servers with user home directories and other shares on them. The servers offer kerberized nfsv4 mounts as well as old v3 ones.
Until recently I was able to nfs4-mount a krb5-exported share from one server on the second without any problems. If I try now I get an permission denied and in the nfs server log I always see this error shown below. The filesystems with userdata on the nfs servers are located in /export/user1 on host server1.uni-ko.de and /export/user2 on host server2.uni-ko.de (hostnames changed). The path /export/ is used as common nfs4 export root marked by fsid=0.
## run on host server1.uni-ko.de
# mount -t nfs4 -o sec=krb5 server2.uni-ko.de:/user2 /mnt
mount.nfs4: access denied by server while mounting server2.uni-ko.de:/user2
and in the nfs servers log I see this error:
refused mount request from server1.uni-ko.de for /user2 (/): not exported
The same happens if I try to locally mount a share on the same host via nfs4 like this:
## on host server1.uni-ko.de
# mount -t nfs4 -o sec=krb5 server1.uni-ko.de:/user1 /mnt
mount.nfs4: access denied by server while mounting server1.uni-ko.de:/user1
A nfs3 mount (mount server1.uni-ko.de:/export/user1 /mnt) is possible without any problems.
The /etc/exports file on host server2.uni-ko.de looks like this:
/export gss/krb5(rw,fsid=0,nohide,no_subtree_check,no_root_squash,async,crossmnt) \
gss/krb5i(rw,fsid=0,nohide,no_subtree_check,no_root_squash,async,crossmnt) \
gss/krb5p(rw,fsid=0,nohide,no_subtree_check,no_root_squash,async,crossmnt)
# NFS V3 exports via NIS netgroups
/export/user2 @nfsv3client(rw,async,no_subtree_check,no_subtree_check,fsid=2000)
exportfs -v shows the shares are (krb5-nfs4) exported:
# exportfs -v
....
/export gss/krb5(rw,async,wdelay,nohide,crossmnt,no_subtree_check,fsid=0,sec=sys,secure,no_root_squash,no_all_squash)
/export gss/krb5i(rw,async,wdelay,nohide,crossmnt,no_subtree_check,fsid=0,sec=sys,secure,no_root_squash,no_all_squash)
/export gss/krb5p(rw,async,wdelay,nohide,crossmnt,no_subtree_check,fsid=0,sec=sys,secure,no_root_squash,no_all_squash)
/export/user2 @nfsv3client(rw,async,wdelay,hide,no_subtree_check,fsid=2000,sec=sys,secure,root_squash,no_all_squash)
Things get even more strange since from a default nfs4 client I am still able to nfs4 mount the given directory without any problems exactly as described above and both systems share the same OS in der very same version and patch level which is SuSE SLES15SP3 with the latest patches installed.
On the nfs servers there is a firewall running opening the statically assigned ports needed for NFS like mountd,statd,lockd as well as ports 111 and 2049 (all opened for tcp and udp). This is what I changed recently. Before this change nfs server ports were not static but set by defaults, but I cannot see how this could lead to the strange mount behavior described. I also disabled the firewall completely and tested again with the same result.
Behind the scenes there is a kerberos server running keeping all the principals needed. For the nfs servers and all nfs4 clients there are "nfs" and "host" principals available and each server and client has a /etc/krb5.keytab containing the exported "nfs" and "host" principal for this host. Of course user principals are also stored in the kerberos server. This all worked flawlessly for years and still does, except for the described problem which is new.