13

I've got an old windows 2000 box in a remote location with a T1 connection and a vpn to my location. I normally use smb mounts to transfer files but now it's time to decommission the server and copy it's backups to my location. I have about 40 gigabytes (compressed) to copy. I'm prepared for it to take a long time, but I have a few caveats.

  • I need to limit the bandwidth so terminal service connections to the site are not affected
  • I want to be able to resume a partial transfer

There are a few small files and several large files (10-20 gigabytes). I'm familiar with rsync on *nix platforms but have had bad luck with windows and I don't know that it will really keep partially transfered files. What do you use?

reconbot
  • 2,435
  • 3
  • 25
  • 30
  • 1
    Robocopy seems to be a good solution, I'll have to try it sometime soon. I'm currently 53.7% on a private torrent. I chose uTorrent over robocopy because of it's scheduler which will allow me to transfer unlimited during off hours and with a decent limit when it matters. Checksumming doesn't hurt either. I ended up using Jeff's instructions http://www.codinghorror.com/blog/archives/001272.html though I'm sure I'd want more control on a public torrent. – reconbot Jul 28 '09 at 13:22

6 Answers6

19

robocopy.exe has a switch called inter-packet gap, allowing you to insert a time window in between the packets of your copy, and thereby reduce the impact on the channel.

It's not exactly "use no more than 30% of the available bandwidth", but you can acheive the same effect with a little math. You can always specify some number of milliseconds and let it run for a bit, then CTRL+C to interrupt, adjust your command as needed, then resume. I've done just this when I didn't want to overload the WAN during the business day with massive replications.

robocopy has another switch /z allowing for "resumeable" transfers, so if the transfer is interrupted you can pick up where you left off, and don't need to shift the whole 40 GB again.

I'm not sure how granular the resume bit is, because I've never really tested for example a single 40GB ZIP file. Test something smaller first ;-)

There are some nice GUIs for robocopy which can assist with the syntax, but anyone with a Linux background will grok it easily. Grab the latest versions from a copy of Windows 2003 or later. Otherwise you'll find it as a Windows 2000 Resource Kit Tool download.

In the Wikipedia entry for robocopy, someone noted that the penalty for restartable copying (the /z switch) is 6x slower performance (see Known Flaws).

nray
  • 1,540
  • 17
  • 23
  • You're doing it wrong - robocopy has been a tool of choice for years now, tested on any number of different networks. The "robo" is supposed to mean "robust". – nray Jul 27 '09 at 19:00
  • +! for Robocopy. Used it several times like this and never had an issue. – KPWINC Jul 27 '09 at 19:44
  • +1 for a well-known, well-used tool. Strange as it sounds, it's sometimes easier to plunk robocopy down into a (ick) batch file and have it do repeated copies this way via System Tasks. – Avery Payne Jul 27 '09 at 21:02
  • 1
    On my local LAN, I found that an IPG value of 50 ms got me down to around 10Mbit/sec, which was the desired value. – trp Oct 19 '12 at 22:03
  • Also, I found that I could not practically get higher throughput with IPG values less than about 25ms. – trp Oct 22 '12 at 15:50
5

40 GB of files are going to take more than 60 hours to copy over a T-1. It might be easier to mail an external hard drive in both directions. You can have someone plug it into the server, copy/encrypt your files and then have them mail it back to you.

Carl C
  • 1,038
  • 3
  • 10
  • 19
  • looks like about 60 hours according to my math: 40GB * 1024(MB/GB) * 8(bit/byte) / 1.5mbit / 3600(sec/hr) = ~60 hours. Of course there will probably be overhead and stuff, so will take a little longer, so maybe say 70 or 80 to be safe? But 1.5mbit is pretty slow, even a really old PC should have no trouble feeding data at that rate. – davr Jul 27 '09 at 20:41
  • 2
    "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." —Tanenbaum, Andrew S. (1996) – nray Jul 27 '09 at 22:41
  • 2
    The utility of spare hard drives is lost on my office. I don't mind waiting. – reconbot Jul 28 '09 at 13:16
  • 40GiB aint that big for a AAA Title nowadays. You don't have FTTH yet?! You'd have finished the steam download before getting back from best buy (not to mention sucking the 25GiB day one patch). – mbx Mar 02 '16 at 15:09
  • Nowadays when? 2016? You know that post was done in 2009? – JasonXA Feb 23 '17 at 10:20
4

You could try setting up FTP on the 2000 box, and using a FTP client at the other end, like filezilla to deal with partial copies.

DanBig
  • 11,393
  • 1
  • 28
  • 53
3

You say you've used rsync on *nix, there are several ports of rsync to windows you might look into. For example:

Personally I've found rsync pretty reliable, and it can handle interruptions in the middle of transfers and so on. Rsync also has a flag to limit transfers to an exact kbit/sec transfer speed, so that would seem to work for you as well.

davr
  • 1,729
  • 3
  • 14
  • 24
1

I use Robocopy to back up all of my servers as well as a live Mailserver backup. Once it is set up right, its truly set it and forget it. Best thing that ever happened to my IT life. Just putting in my 32 cents (allowing for inflation)

1

I use KillCopy for this purpose. I find it very convenient.
It has separate speed and other settings profiles for local and network copy.
It is also able to resume when a transfer of a large file (or files) is interrupted in case of bad connection, or even when the computer hosting the program crashes.

Roland Pihlakas
  • 271
  • 3
  • 7