How to safely upload many GB of data to Amazon S3?

0

1

I am backing up my personal photo/video library to Amazon S3 (several hundred GB).

I am currently using Cyberduck on my Mac to drag & drop, but I've already had a few fail without a clear retry/resume behavior.

I'd like to find a way to upload a directory with confidence that they are all transferred completely, and with a clear way to retry/resume any that failed.

What would you recommend? A command line approach is fine.

Andrew

Posted 2015-11-29T07:01:53.133

Reputation: 11 982

If you're on a Mac, is there a reason you can't just mount the SSH connection as a disk and run Time Machine against that disk? (I don't use a Mac myself but I don't see a reason why this wouldn't work, and AFAIK Time Machine is incremental which saves you bandwidth) – pzkpfw – 2015-11-30T12:43:37.317

woah, that's kind of a crazy idea. Never thought of using S3 with TM. However, I don't need TM and would rather do a one time full backup. I mentioned that I'm on a Mac, but I'm curious if there is something that would work with Linux too. For example, would rsync work for this? I'm not familiar with it enough to know which flags to use, but it seems like a reliable tool. – Andrew – 2015-11-30T23:22:17.353

1rsync should work and is easy to use, just google some examples. There's also duplicity which wraps around rsync to create a time machine-like backup solution, works well. See https://easyengine.io/tutorials/backups/duplicity-amazon-s3/ – pzkpfw – 2015-12-01T07:17:07.430

Answers

1

According to your problem you can try Bucket Explorer

Features of Bucket Explorer

  • You can browse buckets and the files stored at Amazon S3.
  • Upload and download files, to and from Amazon S3 buckets.
  • Bucket Explorer works on every OS where Java is supported.

Bucket Explorer comes with Multipart upload feature.This allows you to upload a single object as a set of parts. In upload process, each part is a portion of the object's data, and each part can be uploaded independently. Multipart Upload offers many benefits over simple upload, especially when object size is very large.

Hussain7

Posted 2015-11-29T07:01:53.133

Reputation: 147

Asked: 2015-11-29T07:01:53.133

Viewed: 285 times

Active: 2016-01-02T07:04:39.050