3

I'm looking to backup a few terabytes of data in our S3 buckets to Amazon Glacier. I've seen the options for "automatic archival" of S3 -> Glacier, but I'd rather it just be a simple one time dump (and the files can't be removed from S3). Looking at:

http://aws.amazon.com/glacier/#importexport

There's a nice easy-to-use API, but it seems like a tool for this should already exist. Are there any tools for easily moving data from S3 to Glacier?

EEAA
  • 108,414
  • 18
  • 172
  • 242
Ralphleon
  • 131
  • 5
  • No reason you couldn't use the automated archival for this. Turn it on, wait for it to finish, turn it off. – ceejayoz Nov 16 '12 at 18:39
  • All the talk of "moving the files" freaked me out a bit, definitely want to keep them in place. – Ralphleon Nov 16 '12 at 18:45
  • Automated archival will move them from S3 storage to Glacier storage. If you need them to exist in both simultaneously you'll need something else. – ceejayoz Nov 16 '12 at 18:47
  • Yes exactly. We still want the data in S3 -- we want to use glacier as a backup to the storage. – Ralphleon Nov 16 '12 at 19:11

1 Answers1

2

I see some useful comments are already here but will throw in just one comment: I would frankly go with putting your objects into S3 and then simply create a very short "Transition to Glacier" policy, like a few hours or 1 day old. You can access Glacier directly through the API, but the reason I would go this route is that you will still see your objects listed within S3, but you'll see in the "Storage Class" column that after a few days everything will have been moved into Glacier. Then, when/if you need to retrieve something out of Glacier, you can make the retrieval request (called "initiate restore" in the new ACTIONS menu of S3's web GUI) it's all right there for you to see and work with.

I'm a huge fan of the CLI for much of my AWS work, but I think until there's a really robust and usable CLI tool for Glacier, this seems easier to work with.

Neal Magee
  • 329
  • 1
  • 7
  • Just to add a note here, AWS updated their pricing structure this past year such that if you move objects from S3 into Glacier and then delete them in < 90 days, you incur a charge. See http://aws.amazon.com/s3/faqs/ – Neal Magee Sep 24 '13 at 20:47
  • Does command line aws s3 sync still work when the object has been migrated to glacier? Can it still compare treemaps when the object has been migrated? – sal Jan 14 '14 at 21:09
  • Yes, it would seem, the minimum time charged for Glacier storage is 3 months. Considering that nominally Glacier costs about ⅓ of S3, if you fully replace the backups more often than once a month, using Glacier makes no sense. It is more expensive and you have to wait hours to get your backups in case you ever need them. – Mikhail T. Jun 24 '14 at 04:36