Copy hundreds of thousands files Mac to NAS

I'm trying to transfer my Aperture library to the NAS. It contains few hundred of thousands of files (quite small ones).

After OS X finishes copying the big files, the estimated time of the copy operation goes from 2 hours to 4–5 days (and growing).

I know that a solution could be to tar them and copy the tarball, but the problem is that the library I want to copy is about 80GB and I have only 10GB left on my Mac.

Any ideas how to copy them in such conditions?

OgreSwamp

Posted 2013-08-22T20:27:54.303

Reputation: 173

Answers

The problem probably is that you are working with many small files, and most copying tools have an overhead per file. It's possible to tar the files up, pipe the data over to your destination, and unpack the archive on the other end, without ever actually creating a tarball anywhere on the system.

tar cf - "~/Pictures/Aperture Library" | (cd "/Volumes/NAS/" && tar xf -)

The downside of this is that you won't have any kind of progress meter, but it should be significantly faster than a more conventional copy.

evilsoup

Posted 2013-08-22T20:27:54.303

Reputation: 10 085

Awesome, I thought that it is possible to create a pipe with tar. – OgreSwamp – 2013-08-22T22:04:20.663

4You could pipe into pv to get an indication of what's going on. – slhck – 2013-08-22T22:11:14.967

You can use the rsync command line tool to do a one-way synchronization of your files. The way rsync works makes it easy to interrupt the process if you don't have the patience for it—you can easily resume it later on. Only the files that are still to be transferred will be copied.

You need to open Terminal.app, and then call rsync like so:

rsync -avh --progress "~/Pictures/Aperture Library" "/Volumes/NAS/"

Here, the first path is pointing to your library, which by default should be under Pictures. If you're unsure, you can drag and drop the library to the terminal command line and it will fill the path automatically for you. The same goes for the path of your NAS.

The -a option enables the archive mode, which sets some defaults, including recursive copying. -v will make the command more verbose. -h turns on human-readable file sizes.

Rsync will show you a progress meter. If you want to cancel the process, press CtrlC. You can call the command again to have rsync continue.

slhck

Posted 2013-08-22T20:27:54.303

Reputation: 182 472

Late addition to this, but what seemed to have helped me is using the -W option which copies the whole file at once (I also had the issue of having lots of smaller files) – Jann – 2018-08-13T07:23:22.883

Thanks. But as far as I know the only advantage will be that I can continue rsync later. But I don't feel like ready to wait 4 days to copy 80GB. – OgreSwamp – 2013-08-22T21:54:28.807

May be there is a solution for "tar" to network, not to my SSD with 10GB left. If I can copy folder as one file - it will be copied in couple hours. – OgreSwamp – 2013-08-22T21:59:12.110

I think the overhead when using Finder is bigger. Also the calculation could be quite off. evilsoup's solution is nice though. – slhck – 2013-08-22T22:12:58.957

The problem are small files. I've tried rsync and it still spend 1 second to copy about 10-20 files (even if their size is around 1K). That really slowed down the copy process. Hope the tar pipe will help to cpu it (there is no progress indicator, but it looks like it copies faster (I've checked what was copied on NAS already) – OgreSwamp – 2013-08-22T23:18:31.843

Try the MHISoft fastcopy opensource. It copy files and directories recursively. https://github.com/mhisoft/fastcopy/releases

Tony

Posted 2013-08-22T20:27:54.303

Reputation: 111

if external device supports full ssh shell, you can do:

tar cf - /your/dir|ssh -C 'tar xf - -C /directory/where/you/unpack'

it is not recommended use -c option for tar because you have many files. printing him cause speed degradation. flag -C for ssh is needed for compression on the fly on network. small piece of data cause small data to crypt/decrypt, this increase speed. but if it is still to slow, you can change crypt chiper from 3des to twofish or if it's possible to 1des. last is not secure but fastest.

otherwise you can use rsync, but rsync should be installed on external device. see man. rsync can transport data by rsync server or ssh.

Znik

Posted 2013-08-22T20:27:54.303

Reputation: 259