Why do file copy operations in Linux get slower over time?

Question

I have 12 ~200 MB files. Copying one of them to another drive takes 20 seconds. When I try to copy all of them at once, it first says it's going to take 5 minutes, then it keeps slowing down into the hours and days.

I'm copying the files through Thunar in XFCE, but I remember having similar problems using the command line. Why does this occur and what can I do to remedy it, other than just copying the files one by one?

Answer 1

This is a known issue with the Linux kernel itself that has been known for ages, no one was tackled the issue yet.

there is currently no fix or known work-around, aside from using Windows which increases in speed and then normalizes. I find this bug to be a very high-priority issue, but the programmers do not feel the same way. It is this bug alone that is starting to drive me away from Linux.

Refs:

• Bug 12309 - Large I/O operations result in poor interactive performance and high iowait times
• USB file transfer causes system freezes; ops take hours instead of minutes

Answer 2

You should try rsync to monitor your transfer speed. For example:

rsync -PSauv sourcedir/* destdir/

If you actually see a drop in transfer speed, as opposed to files that are just big, then you need to look at a few other things.

1. If your disk is I/O-bound, you will have problems. You often see this when writing heavily to a flash drive or an external USB device.

2. Test both your disks with hdparm -Tt <device> to see if one is dramatically slower than it should be, or dramatically slower than the other device which can lead to I/O issues.

3. Check your dmesg output and your SMART status to make sure you don't have any obvious disk errors. You might also consider badblocks -n if you suspect disk errors on a non-SMART drive.

There are lots of reasons why a disk can thrash or perform sub-optimally, but until you can measure the problem, you won't know what the problem is. :)

Answer 3

Remember that also the file copy process is somehow cached by Linux. If you take a single file (even if a 200MB one) and start copying it to another place, Linux caches, with great probability, the whole file in memory and writes it in its new location. Then, if you choose to copy that file again and with it a bunch of other files, then the first file is already cached and then probably skips the whole "read part", while the others are first read from and then written to their sources/destinations. This feature can even lead to infinite transfer speeds and similar oddities, so don't trust any measurement that could be invalidated by previous chaching mechanisms.

Answer 4

Burn a copy of SliTaz onto a CD/USB.

It is designed for use on really ancient machines and runs really fast since it's in RAM. You don't need to do an install, just run it live. If you have a USB mounted, you will be able to see it, and then copy what you need to your computer. The transfer rates I have found to be stable and fast.