I have enabled the HDFS NFSv3 gateway on our HDFS cluster through official documentation. Everything works well except for one Ubuntu 16.04 server machine. The following is the kernel, mount and machine's sysctl -a output information.
root@Linux:~$ uname -a
Linux xxx-server-001 4.15.0-46-generic #49~16.04.1-Ubuntu SMP Tue Feb 12 17:45:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@Linux:~$ mount | grep hdfs
10.30.200.100:/ on /hdfs type nfs (rw,relatime,sync,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,noacl,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.30.200.100,mountvers=3,mountport=4242,mountproto=tcp,local_lock=all,addr=10.30.200.100)
root@Linux:~$ sysctl -a | grep nfs
fs.nfs.idmap_cache_timeout = 2
fs.nfs.nfs_callback_tcpport = 0
fs.nfs.nfs_congestion_kb = 259136
fs.nfs.nfs_mountpoint_timeout = 500
fs.nfs.nlm_grace_period = 0
fs.nfs.nlm_tcpport = 0
fs.nfs.nlm_timeout = 10
fs.nfs.nlm_udpport = 0
fs.nfs.nsm_local_state = 3
fs.nfs.nsm_use_hostnames = 0
sunrpc.nfs_debug = 0xffff
sunrpc.nfsd_debug = 0x0000
The symptoms are the following:
It could
ls /hdfsfolders with very few files in it without error, but it failed withInput/output errorwhen the folder it tried to read from contains many files (more than 100 or so).When enabled the NFS debugging information through
sudo rpcdebug -m nfs -c allon the machine, I observed the following error logs indmesgwhen I hit theInput/ouput errorthroughlsas the following. I have checked the source code here, and it looks like some buffer overflows issue. Does that mean it is a kernel bug for NFS?
[2538707.003904] NFS: dentry_delete(1232344325/sss.123.txt, 4808cc)
[2538707.003907] NFS: decode_fattr3 prematurely hit the end of our receive buffer. Remaining buffer length is 0 words.
[2538707.003914] NFS: readdir(b200/095900) returns -5
When using other laptops or servers to mount the HDFS NFS gateway through
sudo mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync 10.30.200.100:/ /hdfs, it does not have any issue. This means it is probably not the issue on the NFS gateway server itself. However, I have tried installing the4.15.0-46-generickernel on my own laptop but I could not reproduce this issue.This issue is not constantly reproducible, and sometimes it will work in the second or third time of retry after the gateway is just mounted. However, the failure rate would be 90%+ so it is still not usable.
Please let me know if there is any direction I could debug for this weird situation. Thanks in advance!
