Fastest way to copy large file directory over slow internet

Question

I need to copy a directory from one of our servers several hundred miles away. Both sites where I am and where the files are have business cable. The cable is 5 Mbps(Down) / 768 Kbps(Up). This poses a small problem considering I need to copy about 85GB of data.

I used a file transfer calculator and it said it would take 11 days and some change.

I was wondering if there is any way to leverage something like DeDup to speed up the transfer.

The server I would be copying from is Windows 2003 R2, my box is Windows 7 Pro x64.

I was going to transfer the data to my iSCSI target which is running Ubuntu Linux, with ZFS and SCST. Would the dedup being performed by ZFS aid in the transfer so does the data need to be DeDup'd on the sending end?

Are you enable to employ a human at both ends? Not necessarily at the same time. — mdpc, Sep 23 '11 at 22:26

score 19 · Accepted Answer · answered Sep 23 '11 at 22:33

19

Buy an external hard drive. Copy data at site A. Mail it to site B. Copy to machine at site B.

Don't underestimate the power of the postal service.

answered Sep 23 '11 at 22:33

MDMarra

100,183
32
195
326

Sneakernet4Life! Since the USPS is going TU soon... – Holocryptic Sep 23 '11 at 22:40
1

That idea had seriously crossed my mind. – ianc1215 Sep 23 '11 at 23:09
1

Or [carrier pigeon](http://arstechnica.com/tech-policy/news/2010/09/carrier-pigeons-beat-rural-internet-upload-speeds.ars). – Shane Madden Sep 24 '11 at 00:17
A couple of USB sticks is probably cheaper to buy, and would *certainly* be cheaper to ship. – womble Sep 24 '11 at 00:57
@womble I think that is the best idea. – ianc1215 Sep 24 '11 at 01:38
1

I can't believe no one's chimed in with this famous quote: http://en.wikipedia.org/wiki/Andrew_S._Tanenbaum#Quotes – Ward - Reinstate Monica Sep 24 '11 at 04:01
2

I hear the latency is horrendous, though. – mkomitee Sep 27 '11 at 13:26
"Never underestimate the bandwidth of a minivan filled with hard disks" ? – voretaq7 Sep 27 '11 at 17:58

score 5 · Answer 2 · edited Apr 13 '17 at 12:14

5

Is there duplicate data?

If not, then a dedup will help not at all; if there is, it will only be deduplicated over the wire if you transfer it with zfs send -D - since your destination is Windows, then that's probably a no.

Don't forget good old fashioned compression.

edited Apr 13 '17 at 12:14

Community

1

answered Sep 23 '11 at 22:28

Shane Madden

112,982
12
174
248

There is a lot of data that is redundant, they are a bunch of PDF file and AutoCAD DWG files. We found out that compressing the data in certain ways corrupts the DWG files. If I was going to compression would it be better to use something like bzip2 instead of zip. I was told bzip2 is "lossless" compression whatever that means. – ianc1215 Sep 23 '11 at 23:08
7

All normal computer compression is lossless (zip, gzip, bzip2, rar, etc). It means the data which goes in is the same when it comes out. It won't corrupt your DWG files unless you have hardware failure or hit a program bug or it gets corrupted while copying. Lossy compression is where some of the data is thrown away on the grounds that we probably won't notice, and it applies to music, pictures and video, typically. (i.e. what goes in isn't what comes out, but it sounds or looks pretty much the same). In other words, you should use compression, but fix why it corrupts things first. – TessellatingHeckler Sep 23 '11 at 23:17

score 0 · Answer 3 · answered Oct 10 '11 at 10:04

For Directory with lots of redundant data, you can try DZO Compressor. Deduplication makes sense. DZO performs Deduplication as well as Lossless Compression. The product is still in beta phase. As per initial results on Data with duplicates, it performs 2-3 times better than winzip, winrar and others. See their website http://essensolabs.com/ for details or download the beta from below link. http://essensolabs.com/invitation-trial.html

If the data doesn't contain duplicates, then it gives the same compression as 7zip as it uses lzma for lossless compression.

Above can only help in reducing the size and thus lesser time for transfer, but it may still be in days !!!

Fastest way to copy large file directory over slow internet

3 Answers3