1

I'm currently copying large directory tree from an external HDD to to internal storage of my server. The performance of the copy operation is basically quite good, i.e. 35MB/s with the external HDD connected via USB2.

However, I only get this performance for about 30s and then the copy process does not get any CPU time. This means, rsync (and also cp) does not copy any data, does not update its stdout, or react on SIG-TERMs. After about 1 minute, it gets scheduled again and copies with 35MB/s again for 30s, and then nothing again and so forth.

Does anyone know what could happen here or name me any tool how I could investigate what is happening? I already increased the nice value of the rsync process to 19, but this does not change anything. During the idle time of rsync, the whole system is idle (no other IO, no other process that uses the CPU. Basically, the whole system is idle). During the active times, the rsync process has three threads that switch between 'running', 'uninterruptible sleep, and 'interruptible sleep'. During the non-active time, all threads are either have an 'uninterruptible sleep' or 'interruptible sleep' state. Because of this, I think that rsync has to wait for some long-lasting IO operation. But how could I determine what operation takes that much time?

The system is running on a 5.4.34-1-pve linux kernel and the internal storage is a LUKS encrypted ZFS file system on LVM. The LVM sits on top of a mdadm RAID 5. The command I used to start rsync is: rsync -ah --no-compress --progess

Thanks for any suggestion!

PraMiD
  • 11
  • 2
  • How many files, and what is the distribution of their size? Many thousands of small files is typically miserably slow, doing a lot of metadata I/O. – John Mahowald Jun 16 '20 at 03:32
  • The files all have a size of 2-10 GB in a flat tree. It's a bunch of folders that contain the files directly (less than 10 files per folder and O copied about 20 folders). Thus, the file size should not be the problem. – PraMiD Jun 17 '20 at 05:12

1 Answers1

0

We figured out that the problem were the caches used by ZFS. The problem is that we did not create additional read and write caches. As a result, ZFS uses the default caches are placed in/next to the zpool containing the data. Thus, for any read operation the data is written read from the RAID, written to the cache (which also goes to the RAID), and then provided to the user. Similar for writes. As the RAID is encrypted, every read and write tiggers multiple encryptions/decryptions and RAID syncs which leads to the performance drops after some time after the RAM buffers are full.

We solved this problem by adding additional read/write caches for ZFS (L2ARC and ZIL SLOG in ZFS) - in our case two additional small HDDs. Even if the caches are placed on HDDs, we get much higher read and write speeds and no performance drops.

PraMiD
  • 11
  • 2