18
6
I routinely have to copy the contents of a folder on a network file system to my local computer. There are many files (1000s) on the remote folder that are all relatively small but due to network overhead a regular copy cp remote_folder/* ~/local_folder/
takes a very long time (10 mins).
I believe it's because the files are being copied sequentially – each file waits until the previous is finished before the copy begins.
What's the simplest way to increase the speed of this copy? (I assume it is to perform the copy in parallel.)
Zipping the files before copying will not necessarily speed things up because they may be all saved on different disks on different servers.
http://serverfault.com/questions/152331/parallel-file-copy – Ciro Santilli 新疆改造中心法轮功六四事件 – 2015-08-03T09:24:18.507
Zipping the files before copying will speed things up massively because there will not need to be any more "did you get that file", "yes, I did", "here's the next one", "okay", ... It's those "turnarounds" that slow you down. – David Schwartz – 2013-01-15T17:19:51.643
It's probably disk speed, rather than network speed, that is your limiting factor, and if that is the case then doing this per file in parallel will make the operation slower, not faster, because you will force the disk to constantly seek back and forth between files. – Joel Coehoorn – 2013-01-15T17:20:51.260
While zipping might not be a good idea (running compression algo over 1000s of files might take a little while), tar might be viable. – Rob – 2013-01-15T17:27:50.193
@JoelCoehoorn still, there are cases when this is not the case: e.g. multiple spindles + small files (or simply random reads). In this scenario, "parallel cp" would help. – CAFxX – 2013-04-13T06:37:25.137