Splitting tar archives and uploading them one at a time

Question

I want to backup my systems to split tar archives with a script uploading them one by one. It has to create the split archive and then run a script. The script uploads the part and deletes it. This is to ensure backups do not use so much space on the system. I could create the split archives and upload them, but I'd need 50% free space. So I need to create them one at a time. I am looking for advice on the best approach. I have a couple in mind, you can suggest a better one.

Approach one: Split the archives with tar itself, and use --new-volume-script. The problem with this is that I have to calculate how big the backup is going to be. Tar seems to require specific directions for how many parts are going to exist and how big they have to be. This means my script would have to calculate this and generate the parameters to tar.

tar -c -M -L 102400 --file=disk1.tar --file=disk2.tar --file=disk3.tar largefile.tgz

This creates three 100Mb files for each part. If there is a way to do this dynamically with tar automatically naming the files and creating as many as it needs, I would like to know because it would make this approach workable.

Approach two: write my own script that behaves like split. The output from tar is given to it on stdin, and it uploads the files and makes tar wait. This would be the easiest solution.

Answer 1

This solution doesn't use tar, but you may be able to make it work with afio. All the logic to split the archive is build-in, with the option to run scripts after each volume split:

cd /path/to/files -print | \
   afio -oxv -s 1g -H rotate.sh backup-`date -Imin`-Vol%V.afio

And rotate.sh is your script to upload and delete each archive file. This generates archives:

backup-2014-11-29T18:04-0500-Vol1.afio
backup-2014-11-29T18:04-0500-Vol2.afio
backup-2014-11-29T18:04-0500-Vol3.afio
...

And runs rotate.sh after each volume is complete.

Other options:

-o              # create an archive
-x              # perserve ownership suid/sgid
-v              # verbose
-s 1g           # split archives after 1g
-H rotate.sh    # run this script after each 'tape change'
-Z -P xz        # Compress, and use xz instead of gzip
 # Also, %V, below, inserts the volume number into the file name
 backup-`date -Imin`-Vol%V.afio               

Other afio aspects: It is similar to cpio, except that it is specifically geared towards scripted backups. Also it is safer for compressed archives because it stores each file compressed individually, instead of compressing the whole stream. This way a data corruption after compression only affects one file, rather than the whole archive. In the same way, it can also gpg encrypt each file as it is stored, which is great for cloud storage.