10

I need to transfer a number of files files over a low-quality broadband link to a server. The files are large, and take approximately 30 minutes to transfer per file.

I use scp, but it sometimes hangs -- the transfer doesn't fail with an error, it keeps running, but no further data is transferred.

So, I'm looking for a "failsafe" upload solution, one that will work even if the link fails for a few minutes or is otherwise unreliable.

My idea is:

  1. split big file in small parts
  2. upload parts, with timeout and retry if fail

Questions:

  • is there a ready-to-run tool that implements this idea? (no specific need for scp. could be ftp or anything else)
  • is there a way to detect when scp hangs? ((that is, it is still running, but does not transfer data)
womble
  • 95,029
  • 29
  • 173
  • 228
user1219721
  • 467
  • 1
  • 6
  • 15

1 Answers1

16

You can use rsync to copy your file from one computer to the other. rsync can use ssh as its underlying transport. Combine rsync --partial with a script such as this one to try again in case of network failure, and you should be able to move your files even in the face of network errors.

Another way to do it would be to mount the remote filesystem on your local computer with sshfs -o reconnect, and then just cp the file(s). sshfs/Linux will take care of the rest. Based on some testing I did today, this seems to be much slower than rsync.

Finally, you can set up a VPN between the two machines. This involves the most work, and either of the above solutions are much easier, but it would solve the problem of a flaky network connection. And as some people have noted, this can also be flaky. It can work if the VPN is very aggressive about reestablishing connections, as OpenVPN is, but the above solutions are much better.

Community
  • 1
Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • 1
    In order to make this work, you'll need to use the `--partial` flag (or something that implies `--partial` like `--inplace`) otherwise whenever the transfer is interrupted, the partial transfer will be deleted and you'll have to start again. – womble Jul 15 '12 at 02:04
  • Rsync definitely the way to go... Or robocopy for windows. – Brian Adkins Jul 15 '12 at 02:46
  • How exactly would a VPN help with a flakey connection? – EEAA Jul 15 '12 at 02:57
  • The VPN software (e.g. OpenVPN) can reconnect automatically in the background, transparently to any applications that are running over the VPN. – Michael Hampton Jul 15 '12 at 02:58
  • @ErikA My thoughts precisely - VPN over flakey link = flakey VPN that's always having to restart. It only adds a second level to the problem which is that the transport software must be able to restart file transfer exactly where it was interrupted. – Fiasco Labs Jul 15 '12 at 03:00
  • I didn't say it was the best solution, or even a _good_ solution, just that it is _a_ solution. :) – Michael Hampton Jul 15 '12 at 03:01
  • @MichaelHampton, I see where you're going with this and the auto-link reestablishment while the transfer program retries (if timeouts aren't exceeded) might actually help if the overhead doesn't exacerbate the weak link issues. Uggh! been there, no fun. – Fiasco Labs Jul 15 '12 at 03:04
  • I find it's best to not have a flaky network in the first place. Then again, we have to live in the real world... – Michael Hampton Jul 15 '12 at 03:08
  • It's actually not a solution at all, at least not for this problem. Your statement that it "solves the flaky connection problem" is patently false. VPN does nothing of the sort. With VPN, applications still need to be aware of the situation, and restart when the connection comes back up. VPN does nothing but add complexity and overhead. Your rsync recommendation is good. Simple, robust, easy to troubleshoot. I'd recommend that the OP just use rsync. – EEAA Jul 15 '12 at 03:54
  • rsync sometimes hangs too... and when it doesn't, on a large filesystem it often doesn't reach the next file which needs transfering ("receiving incremental file list") before the connection is lost and it retries. – Michael Mar 27 '17 at 06:11