How do I backup 50GB+ of files on a linux box to a remote sftp account.
My requirements
- Store several versions of the backups.
- Encrypted before sent to remote server.
- Disk and bandwidth efficent, think incremental.
- Using simple linux commands (preferred support by Busybox on a Synology Diskstation)
My solution
Currently I'm using rsnapshot to make one local backup every day. Then storing weekly, monthly backups in the rsnapshot way. Then doing a full backup every month with tar, which I encrypt with gpg2 before sending to my friend.
Then I do an incremental level 1 backup using the snapshot file from the full backup.
But unfortunately this doesn't really work. Because each time rsnapshot is executed, my files get new dates and maybe other stuff. And it looks like tar only uses date to verify if the file has changed or not.
Questions
- Can I solve my backup problem with this solution?
- One way to do it, is to backup the original files instead of the rsnapshot/rsync copy. But currently I don't really trust tar, maybe it will miss something else. I would like something that does a checksum or some other more performance friendly magic.
- Should I use some other software?
Code example
This is a simplified version of my script, that illustrates the problem.
cd ~
mkdir backup-test
cd backup-test
mkdir data
mkdir rsync-copy
mkdir backup
# Create 10 10mb files
for i in 0 1 2 3 4 5 6 7 8 9
do
dd if=/dev/urandom of=data/file$i.txt bs=1048576 count=10
done
# Simulate rsnapshot of my files.
rsync data/* rsync-copy/
# Make full backup
/opt/bin/tar c -g backup/full.snar rsync-copy --file backup/full.tar
# Simulate rsnapshot of my files again. No changes in my files.
rsync data/* rsync-copy/
# Make incremental backup, this will create a new full copy.
cp backup/full.snar backup/inc.snar
/opt/bin/tar c -g backup/inc.snar rsync-copy --file backup/inc.tar
# Make incremental backup, this will not create a new full copy.
# Because we haven't executed rsync, which sets dates and stuff.
cp backup/inc.snar backup/inc2.snar
/opt/bin/tar c -g backup/inc2.snar rsync-copy --file backup/inc2.tar