How do I backup 50GB+ of files on a linux box to a remote sftp account

Question

How do I backup 50GB+ of files on a linux box to a remote sftp account.

My requirements

• Store several versions of the backups.
• Encrypted before sent to remote server.
• Disk and bandwidth efficent, think incremental.
• Using simple linux commands (preferred support by Busybox on a Synology Diskstation)

My solution

Currently I'm using rsnapshot to make one local backup every day. Then storing weekly, monthly backups in the rsnapshot way. Then doing a full backup every month with tar, which I encrypt with gpg2 before sending to my friend.

Then I do an incremental level 1 backup using the snapshot file from the full backup.

But unfortunately this doesn't really work. Because each time rsnapshot is executed, my files get new dates and maybe other stuff. And it looks like tar only uses date to verify if the file has changed or not.

Questions

• Can I solve my backup problem with this solution?
  • One way to do it, is to backup the original files instead of the rsnapshot/rsync copy. But currently I don't really trust tar, maybe it will miss something else. I would like something that does a checksum or some other more performance friendly magic.
• Should I use some other software?

Code example

This is a simplified version of my script, that illustrates the problem.

cd ~
mkdir backup-test
cd backup-test

mkdir data
mkdir rsync-copy
mkdir backup

# Create 10 10mb files
for i in 0 1 2 3 4 5 6 7 8 9 
do
    dd if=/dev/urandom of=data/file$i.txt bs=1048576 count=10
done

# Simulate rsnapshot of my files.
rsync data/* rsync-copy/

# Make full backup
/opt/bin/tar c -g backup/full.snar rsync-copy --file backup/full.tar

# Simulate rsnapshot of my files again. No changes in my files.
rsync data/* rsync-copy/

# Make incremental backup, this will create a new full copy.
cp backup/full.snar backup/inc.snar
/opt/bin/tar c -g backup/inc.snar rsync-copy --file backup/inc.tar

# Make incremental backup, this will not create a new full copy.
# Because we haven't executed rsync, which sets dates and stuff.
cp backup/inc.snar backup/inc2.snar
/opt/bin/tar c -g backup/inc2.snar rsync-copy --file backup/inc2.tar

Answer 1

It seems to me that you're happy with rsnapshot but not with tar. If so, the solution is to replace tar with something that preserves the 'only changes' of rsnapshot.

Out of my head, I can thing of two easy options:

1. after rsnapshot, find the differences between the last and previous snapshots to store only those to a tar file. It could be simply diff, with some flags tell it not to bother checking the content to provide the exact difference. The list of changed files should be enough. Another way would be to write some script to walk both snapshots and report new files and those with different inode numbers.

2. use rdiff-backup, which uses the rsync algorithm but stores differences instead of transmitting them.