How to identify which NFS client generates disk IO?

Question

Our pool server disk is 100% busy.

I checked with iotop and determined that nfsd is the top process which consumes disk IO.

I need to narrow that down further and want to determine which of the NFS clients using the server is/are responsible for this disk IO bottleneck. How do I proceed?

iftop will show you which client generates most of the network traffic. Very likely it will be the same client which generates the IO load. — kofemann, Apr 10 '17 at 06:28
inspired by your question, I have build a **nfstop** to monitor shuch activity https://github.com/kofemann/nfstop — kofemann, Apr 18 '17 at 12:29

13dimitar · Accepted Answer · 2017-04-07T12:03:46.493

5

iotop and then o - you will see which process reads and/or writes and how much to the HDD.

Check the pid of that process and do netstat -entp | grep <pid> - that way you will see established tcp connection and from which address it's coming. Use enp to check for both tcp and udp sessions.

You can also do a netstat -anp | grep 2049 - that way getting an ip address and pid, then correlate the pid to the one from iotop.

edited Apr 07 '17 at 12:03

answered Apr 07 '17 at 10:17

13dimitar

2,360
1
12
15

It couldn't find anything when I run netstat -entp | grep . But I have a process with this pid – Gani Rakhmatov Apr 07 '17 at 11:59
Then try `netstat -encp | grep ` - the `c` switch will show continuous output. – 13dimitar Apr 07 '17 at 12:01
I've tried with `netstat -anp | grep 2049` command, but it doesn't help. – Gani Rakhmatov Apr 07 '17 at 13:01
`netstat -anp` unfortunately doesn't show the PID for processes that use NFS. `iotop` doesn't distinguish between NFS I/O and other I/O. – Stefan Lasiewski Nov 14 '19 at 21:18

score 2 · Answer 2 · answered Apr 14 '17 at 12:57

Usually the client using most IO will also doing most network traffic, so what I do is: dump all traffic for a few seconds, and then create a sorted list of the hosts (limited to the nfs hosts) that used most traffic:

tcpdump > dump.cap  # (30 secs should be enought), press ctr+ c
grep -o "<something iding an nfs client>" dump.cap | sort | uniq -c | sort -n

How to identify which NFS client generates disk IO?

2 Answers2