NFS mount subsequent reads slower than after mounting

Question

Let me explain the scenario first. I know it may sound ridiculous to have this setup, but once I explain what I'm seen if someone who knows more can explain why it is occurring, I would really appreciate it.

I have 2 sites. They are both at opposite ends of the continental US. These servers are connected via a 10Gb point to point link which roughly has a latency of about ~67ms. I have an NFS share in one end and a server that needs to mount this NFS share at the other. On initial mount, running time tree gives an output as such:

[root@server01 ~]# time tree /usr/local/ftppub/production/biz_12345
/usr/local/ftppub/production/biz_12345
└── incoming
    ├── inbound
....
    ├── test
    │   └── OVR_DS_MASTER.csv
    ├── test1.csv
    ├── test2.csv
    └── to_be_deleted
        └── payments.csv

23 directories, 2229 files

real    0m9.636s
user    0m0.013s
sys 0m0.047s
[root@server01 ~]#

Notice the real time, it is about 10 seconds.

But when I run this same query the second and any subsequent time, the time is 12 times the amount:

[root@server01 ~]# time tree /usr/local/ftppub/production/biz_12345
/usr/local/ftppub/production/biz_12345
└── incoming
    ├── inbound
....
    ├── test
    │   └── OVR_DS_MASTER.csv
    ├── test1.csv
    ├── test2.csv
    └── to_be_deleted
        └── payments.csv

23 directories, 2229 files

real    1m55.544s
user    0m0.026s
sys 0m0.137s
[root@server01 ~]#

cat /pro/mounts gives the following settings for this mount:

1.2.3.4:/FTP /usr/local/ftppub nfs rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1.2.3.4,mountvers=3,mountport=2049,mountproto=udp,local_lock=none,addr=1.2.3.4 0 0

These are the default mounts generated by the server when mounting this NFS share. I have tried playing around with these parameters, but instead of making things better, access to this NFS share suffers. What doesn't compute is how come the first read of this folder structure is 12 times faster than all subsequent reads? I would have thought the issue was with tree, but testing the same with du gives the same result. The read from the first du command is about ~10 seconds, all subsequent du commands to the same directory yields an output of almost 2 minutes. This happens for any new directory I try to read for the first time after I mount NFS, any subsequent reads from that same directory slow down by ~12 times.

NFS mount subsequent reads slower than after mounting

0 Answers0