7

I'm looking at rsnapshot and S3 to backup my server. But since rsnapshot can't directly upload to S3, and I'm not sure if S3 preserves file permissions, I'm thinking of archiving the backup folders created by rsnapshot and uploading them to S3.

Is that feasible? Or is there a better option?

womble
  • 95,029
  • 29
  • 173
  • 228
john2x
  • 173
  • 1
  • 6

2 Answers2

6

Have you tried, http://duplicity.nongnu.org/ ? It supports a lot of things to backup on including Amazon S3. From the features page,

Currently local file storage, scp/ssh, ftp, rsync, HSI, WebDAV, Tahoe-LAFS, 
and Amazon S3 are supported, and others shouldn't be difficult to add.

rsnapshot isn't an actively developed project any more (last release 1.2.0 was in 2005). Duplicity is actively developed (May 2012 release) and you may want to take a look at that.

UPDATE 1: Duplicity supports full UNIX permissions -- http://duplicity.nongnu.org/duplicity.1.html

Duplicity incrementally backs up files and directory by encrypting tar-format volumes with GnuPG and uploading them to a remote (or local) file server. See URL FORMAT for a list all supported backends and how to address them. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Currently duplicity supports deleted files, full Unix permissions, uid/gid, directories, symbolic links, fifos, etc., but not hard links.

s3 has object and bucket policies for read/write/update access that you configure for users and groups.

UPDATE 2: rsnapshot has a active github community and releases are frequent https://github.com/rsnapshot/rsnapshot

Chida
  • 2,471
  • 1
  • 16
  • 29
  • I've taken a look at duplicity as well, but I was concerned that when my files are placed on S3, the permissions aren't preserved. Which is why I thought of "tar"ing up the rsnapshot folders. I'll go with duplicity if I can confirm that S3 preserves permissions. – john2x Aug 14 '12 at 06:47
  • You can also use s3cmd which preserves file attributes with the -p option. – Chida Aug 14 '12 at 07:02
  • Thanks for the update. I'm confused, since mgorven says below that S3 has no concept of *nix permissions. – john2x Aug 14 '12 at 07:34
  • He is right. S3 has the *nix permissions implemented as bucket and object policies for read/write/update for users and groups. You have to define a bucket polivy fo the entire bucket. I believe duplicity and s3cmd when then mention preserve file system attributes, it probably means it does within the archive. I haven't tried it myself so I cannot comment on it. – Chida Aug 14 '12 at 07:38
  • s3cmd 'seem' to support it and someone has asked a very identical question here. Please see the response -- http://comments.gmane.org/gmane.comp.file-systems.s3.s3tools/158 – Chida Aug 14 '12 at 07:42
  • 1
    Just tried out duplicity, and it does indeed preserve file permissions when backing up to S3 and restoring from it. Thanks! – john2x Aug 14 '12 at 08:29
  • 1
    "rsnapshot isn't an actively developed project anymore" - what? https://github.com/rsnapshot/rsnapshot/releases – developerbmw Dec 14 '15 at 21:37
4

For S3 you really want a few big files rather than lots of small files, and so rsnapshot's scheme isn't a great fit for this (at least not for uploading directly). S3 also has no concept of *nix permissions or ownership. You want to find a backup system which creates archive files of some sort -- these store permissions and ownership internally, and create a few big files which can be uploaded to S3 and managed sensibly.

mgorven
  • 30,036
  • 7
  • 76
  • 121
  • 2
    to expand. The way amazon deals with costs on a per "put" basis makes it expensive to have very many small files. – Sirex Aug 14 '12 at 07:06