19
1
The file is around 170GB. I wouldn't upload it on the FTP, and download it. It is not reliable, sometime, connections drop, and it will have big trouble. So, is there any better way to do so? Anyone suggest? Thanks.
P.S.:two computers are not in same network.
Ted Wong
Posted 2013-03-13T09:20:18.717
Reputation: 903
26
Wierdly enough bitorrent might work pretty well here, assuming office policies allow it- it breaks up the file for you, checks if its correct and if not redownloads it. You're probably going to want to run your own tracker, but many bitorrent clients do that anyway, and if possible use webseeds to speed things up even more - burnbit makes this easy.
As an alternative I'd also suggest doing the oldschool pirate thing, and splitting up the files and using a parity file, then transferring it by any means you have, FTP or web server. If you use a webserver, downloads can be continued - with something like wget, and the parity archive would allow you to rebuild the file with a few broken pieces.
Naturally consider encrypting the files or file chunks, if the data is of a sensitive nature as well.
A more recent option may be bitorrent sync - It runs on everything but the toaster (Unless your toaster run windows, linux on x86, PPC or ARM), and handles most of the grunt work for you. It uses the underlying bitorrent protocol but is a lot simpler to use.
Journeyman Geek
Posted 2013-03-13T09:20:18.717
Reputation: 119 122
+1 for toaster. U saved the toaster from toasting big chunks – Pathfinder – 2016-05-27T08:01:52.577
1additionally if you need to copy to multiple hosts you'll decrease the time significantly. – Colton – 2013-03-13T18:02:50.513
20
Take a look at robocopy, it supports restarting, and in general is a lot more stable than other options.
Adam Caudill
Posted 2013-03-13T09:20:18.717
Reputation: 551
robocopy rules ! – wim – 2013-03-14T05:33:32.820
4
WinZip can produce lots of (relatively) little files (originally designed to archive to floppy disks, but you can choose 1GB now). Then non-resumable FTP (as the Microsoft option is) is OK. Finally, WinZip will retrieve the file at the other end.
Mark Hurd
Posted 2013-03-13T09:20:18.717
Reputation: 379
3I think 7-zip is much better than WinZip. – Alvin Wong – 2013-03-14T00:31:28.260
@AlvinWong Perhaps; WinZip was first and still pretty good. I just haven't used 7-zip enough. – Mark Hurd – 2013-03-14T01:59:22.970
1something being oldest is not a good reason to use it in the IT world... – wim – 2013-03-14T05:36:47.900
@wim I haven't needed to use anything else. If WinZip is so wrong downvote me, preferably giving reasons why it is actually crap for this purpose. If 7-zip is so right, add it as an answer. – Mark Hurd – 2013-03-14T06:20:59.663
Well, you're not wrong so no downvote.. but Alvin has a point 7-zip capabilities are a superset of winzip, and it's free.. – wim – 2013-03-14T06:39:28.310
4
Robocopy and BitTorrent have already been suggested and sound like a good idea. Other options that may work better in a restrictive network environment where you cannot e.g. create the SMB connection that seems to be required for RoboCopy:
FTP. I know you don't like it, but with a good server and client, it should work well. Create an FTP server on either source or recipient, make sure that it supports encrypted connections (to prevent transparent proxies etc. from interfering) and files > 4 GB. Then upload/download the file using a good FTP client (making sure to use binary mode). FTP supports connection resuming, so if the connection drops, just resume. A current copy of wget should be fine.
You can do the same with HTTP: Set up a HTTP(S) server supporting large files, and download it with a current copy of wget.
Otherwise, there are rsync binaries for Windows and numerous proprietary rsync-like programs that you could use. Especially if you expect that the file will need to be updated and only small portions of the file will change, you may want to look into that direction.
Remember that at 8 MBit/s (1 MByte/s), it will take you two days to transfer the file. Unless you have really fast connection, sending a physical hard drive with a copy of the file(s) may be faster.
Jan Schejbal
Posted 2013-03-13T09:20:18.717
Reputation: 1 014
1Send the hard drive by Fedex. In the end it will cost less and be faster. That is how most larger corporations move backups and such. – Pow-Ian – 2013-03-13T19:06:49.713
1Using the file transfer protocol for file transfer! Who'd a thunk it ??? – wim – 2013-03-14T05:37:29.567
@Pow-Ian: I did not want to advertise a specific company, but FedEx really lives up to their original Federal Express name. I handed them a (pretty sizeable) package weighing around 10-12 kg at noon on Saturday in Germany, didn't even pick their fastest service, and got it delivered to Belgium by 10:35 AM on Monday for 29 EUR. If I have anything that needs to get somewhere fast, I will definitely use FedEx again. – Jan Schejbal – 2013-03-15T14:48:10.893
@Jan Schejbal I say call a spade a spade. I know FedEx is the fastest so i don't mind telling others about my experiences. For full disclosure I have seen their facilities and toured their Memphis TN USA air port processing facility. They can literally ship things next day from NY, NY to Japan. – Pow-Ian – 2013-03-15T14:51:24.227
1
Some things that come to my mind are private P2P Networks (uTorrent, DC++) or a tiny http server with wget
Midhat
Posted 2013-03-13T09:20:18.717
Reputation: 464
0
I would sugest to use some tool (windows) based on rsync http://en.wikipedia.org/wiki/Rsync
Josef Procházka
Posted 2013-03-13T09:20:18.717
Reputation: 101
4one line sentences, on which a link supposedly answers the question are a no-no in SuperUser. please elaborate on your answer. – Lorenzo Von Matterhorn – 2013-03-13T15:59:53.770
10An external USB HD? – mouviciel – 2013-03-13T09:22:17.447
@mouviciel yes, it is. But my environment not allow me to do so, I updated the question, the only way is via network. – Ted Wong – 2013-03-13T09:23:55.247
9http://en.wikipedia.org/wiki/Sneakernet is a network. – ta.speot.is – 2013-03-13T09:27:47.483
4Break it up into smaller pieces, then you only have to worry about one small piece at a time. Pro tip: BitTorrent does exactly this. – ta.speot.is – 2013-03-13T09:29:12.180
1
I use SyncApp from BitTorrent to keep my data in sync (~5TB, one file is 300GB). SyncApp has a Windows and Mac application with GUI and a Linux application with a webinterface (perfect for servers). At the moment there is only a closed beta available. You can register for an invite here: http://labs.bittorrent.com/experiments/sync.html - Screenshot: http://i1-news.softpedia-static.com/images/news-700/BitTorrent-Sync-Is-a-P2P-Dropbox.png
– fnkr – 2013-03-13T10:08:55.5204I would use Skype but im not brave enough to put it as an answer :) – Robert Niestroj – 2013-03-13T11:31:15.083
1FedEx has better bandwidth than the internet – zzzzBov – 2013-03-13T19:39:00.823
Do you only care about performance but not security? – Alvin Wong – 2013-03-14T00:41:14.153
1
Consider RFC 1149
– Alvin Wong – 2013-03-14T00:57:29.083For a 170GB file, there's a high possibility that checksums in IP and TCP will miss some transmission errors, so you should use protocols that actively checks the data. Bittorrent is a good one. – billc.cn – 2013-03-14T01:04:56.993
Yes, thats why I use the SyncApp from BitTorrent. – fnkr – 2013-03-14T14:27:30.323
4Why is this flagged as a duplicate of questions asking how to e-mail 2-30 MB of data? There are a few orders of magnitude between "send 17 MB" and "send 170 GB", and the solutions are completely different. Dropbox/filehosters are not really a feasible suggestion for 170 GB... – Jan Schejbal – 2013-03-15T14:51:50.733