Large file copy from NFS to local disk performance drop

Question

I'm trying to copy a 200GB file from an NFS mount to a local disk. The local disk is an XFS filesystem on a LVM on top of a RAID 5 system (hardware RAID controller).

I'm using rsync to monitor the transfer speed. At the beginning, the IO speed is about 200MB/s, stable for the first 18GB. But then the performance drops by a factor of 10-20 and never recovers to the initial rate. Sometimes it reaches about 50-100MB/s, but just for a few seconds, and then the process seems to hang for a bit.

At the same time all file-stat operations on the target filesystem are blocking for a long time (minutes). Also, interrupting the copy process blocks for several minutes. A subsequent delete of the partly copied file takes also several minutes.

Any ideas what could be causing this?

Answer 1

Thanks to jet's comment, I looked into the caching of disk IO under Linux. It turns out, as the system has a lot of RAM (48GB) which is almost free, a lot is used for the I/O cache.

I monitored /proc/meminfo and looked for the fields 'Dirty' (dirty I/O cache pages) and 'Writeback' (written dirty pages to disk).

watch -n1 -- "grep -E Dirty|Writeback /proc/meminfo"

Shows that Dirty increases to about 18GB, than goes down again, while Writeback increases, once Dirty pages have been written, it grows again etc. Monitoring the disk io using iostat one sees that during caching the disk does not write, only while 'Writeback' decreases data is written to the disk. The write speed is about 15MB/s.

So the first 18GB are fast, written to the cache but than, when the actual writing occurs it slows down and while writing the disk I/O is saturated and blocking until all dirty pages are send to the disk.