1

Our latest project at work involves sending 100GB+ files from our office in Dallas to our client in Australia. I've been asked to put together the fastest, most reliable way of transferring these files. Usually this would be no sweat, but when they said 100GB I decided I needed to ask for advice. I'm also in a time crunch, so I have about 24 hours to put together a professional solution (no major software development).

It must be screaming fast, and it must have tolerance to internet service interruptions on both the upload and download side (upload resume and download resume). It also needs to be as turn-key as possible on the client download side. A GUI interface tool would be great.

My first thoughts were to use an Amazon S3 account with a 3rd party download/upload manager, but I'm not sure how fast Amazon's S3 bandwidth limitations are. Is EC2 faster? Also Amazon's website tended to be catering to software programmers - are there any reliable pre-rolled applications that support multipart uploads and downloads?

Scott Pack
  • 14,717
  • 10
  • 51
  • 83
uberdanzik
  • 131
  • 2
  • 5
  • is that 100 GB each or 100 gb in total? if it doesn't need to be fast, also consider sending bluray discs maybe. Sending files is one thing, receiving is another thing. we don't have very fast link here. we used to send big files with yousendit but not in that order. with our local photographer it's faster to pick up dvd than send gbs of data. it's also faster to have backup tapes picked up here, although that's not the only reason. – imel96 Jun 14 '13 at 02:12
  • fedex. But also what is your BW at both sites? and what are your time requirements? – Doon Jun 14 '13 at 02:51
  • The total package size is 100GB, but our directory structure is extremely complex and tough to break apart (hundreds of folders 10-20 levels deep) I'd rather not rely on the client to re-assemble. I may go with rar files for this reason. Yes, I proposed the FEDEX solution, but it was denied by management, they want to establish a one-stop digital download solution for smaller files as well. – uberdanzik Jun 14 '13 at 04:14
  • 1
    This type of question has been asked and answered several times. Have you consulted the archives of ServerFault or SuperUser SE before you wrote the question? – mdpc Jun 14 '13 at 04:26
  • 1
    Internet in Australia sucks - if they are in a normal office they may only have a <20mbps ADSL2+ downlink, maybe <10mb... If so you're wasting time optimizing as it will take many, many, many hours to download at the other end – Mark Henderson Jun 14 '13 at 05:50
  • @MarkHenderson is right. You have to consider the narrowest stage of the pipe first. If you've got 100GB total, but the deltas aren't too much, then seeding via shipping a portable HD (maybe two copies) and then setting up a little-and-often sync is your best bet; otherwise it could take days. – SmallClanger Jun 14 '13 at 07:04

1 Answers1

2

The limiting factor is going to be the size of the pipe between you and your client. There is no technology that is going to be faster than another. Any modern file transfer mechanism will be able to saturate whatever links you have.

The first thing to do to cut down the transfer time is to efficiently compress your data. Experiment with different compression formats like rar or 7zip to see which is most effective.

Since all transfer methods are fairly equivalent in terms of speed, concentrate on restartability. SCP or SFTP have good restart capabilities and are easily scriptable. If the files ultimately need to end up at multiple offices of your client then BitTorrent is a good option because their offices can share pieces between themselves.

However:

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.

It's likely that the fastest way in terms of elapsed time is to use a USB hard drive or equivalent and just mail it or send it via courier. I suggest you do some tests with your client to see what kind of throughput you get, them do the math to see if a station wagon full of tapes hurtling down the highway is faster. (Or a USB hard drive on a jet, whichever you prefer.)

longneck
  • 22,793
  • 4
  • 50
  • 84
  • Thanks! I'll definitely look into the compression methods. I love the station wagon analogy, but it's expensive, and apparently customs sometimes likes to hold on to packages for a few days. They also want to establish a one-stop pipeline for smaller files. – uberdanzik Jun 14 '13 at 04:20
  • There is an inherent overhead in using 'secure' protocols which may impact the overall transfer speed as the encryption is done in real time. If the data does need protection consider encrypting the data before you send it and use standard (un-encrypted) data transfer protocols. If you have multiple offices with independent intermet connections, maybe split the file and send parts to each site using the internal network and let them get dragged over each sites distinct link for combining at their destination? If you're going to encrypt, compress FIRST. Compressing encrypted data isn't good. – FreudianSlip Jun 14 '13 at 06:44