How to copy 700K .jpgs from linux to windows?

Question

I have 700K+ .jpg files on my linux server in a two-level structure, with 6000+ directories at level 1 and the individual .jpg files distributed amongst those directories on level 2. These files take up 16GB according to du -ch | grep total. There is 3.5GB free space on the disk.

I'm looking for a sensible way to copy these files to a windows machine, and to update the windows copy at regular intervals with new files from the linux server.

I've tried FileZilla, but it only managed ~100K files in an hour, with the load avg on the linux server at around 2. That's both too slow and taking too many resources. With 10 connections FileZilla only managed ~150KB/s on a 100Mbps line.

I'm hoping it's possible to use tar of individual directories in some fashion to get "bigger chunks" while not filling up the server disk..?

Answer 1

I can think of two ways to do this that should work for you:

Manual Solution

Use rsync over SMB file sharing. Share a folder from your Windows machine, and then mount it under Linux using SMBfs. You can then use rsync to transfer files across:

rsync -av /path/to/source/directory /path/to/mounted/destination/folder

The great thing about rsync is that if the file transfer is interrupted, you can run the rsync command again and it will quickly skip over any files that have already been transferred.

Automatic Solution

Use BitTorrent Sync. Install BitTorrent Sync on both your Windows and Linux machines. You can then add your folder of images on the Linux machine, and share it with your Windows machine by copying the folder key across. BitTorrent Sync will automatically ensure that the two locations are kept in sync.