1

Environment: Ubuntu 16.04 in Azure

I'm attempting to backup specific folders. I'm using a simple tar command with no compression. My objective is to keep the file for a month, adding changes incrementally once per day, then compress tar and start a new one once per month.

My problem is this: Local backups make no sense to me because if the local filesystem dies or errors, both the original and backup die. (seems obvious, but I state for clarity)

I have tried backing up to a separate Azure storage container using an SMB connection, and also to another identical Ubuntu machine over NFS.

I tried the second option because I read this article: tar incremental backup is backing everything up, every time when used on the Dropbox directory

So, no matter what I do, the tar seems to ignore the -u flag when issuing the following command:

cd /savelocation; sudo /bin/tar --ignore-failed-read -up -f /savelocation/backupfoldername.tar /var/www/foldertobackup

Instead of getting a few megs extra, which would represent the files added or changed, I'm getting a file that's 44GB which is twice the size of the original.

Any thoughts or questions are most appreciated.

ivanleoncz
  • 1,433
  • 4
  • 18
  • 32
Bruce
  • 21
  • 4

2 Answers2

0

According to my tests here.


First: I believe that the cd is not necessary in order to perform the command.

Second: you should use --listed-incremental on your command.

-g, --listed-incremental FILE

handle new GNU-format incremental backup


Seems that --listed-incremental, only works when you already have a TAR file, according to my tests and with one of the observations listed on the comments.

folder with empty files

$ ll folder
total 8,0K
-rw-rw-r-- 1 ivanleon ivanleon 9 Jun  6 11:32 file1
-rw-rw-r-- 1 ivanleon ivanleon 9 Jun  6 11:32 file2
-rw-rw-r-- 1 ivanleon ivanleon 0 Jun  6 11:27 file3

creating Tape ARchive file

$ sudo /bin/tar -cvf folder.tar folder
folder/
folder/file2
folder/file3
folder/file1

listing Tape ARchive file

$ tar tvf folder.tar 
drwxrwxr-x ivanleon/ivanleon 0 2017-06-06 11:27 folder/
-rw-rw-r-- ivanleon/ivanleon 0 2017-06-06 11:27 folder/file2
-rw-rw-r-- ivanleon/ivanleon 0 2017-06-06 11:27 folder/file3
-rw-rw-r-- ivanleon/ivanleon 0 2017-06-06 11:27 folder/file1

adding/modifying data in some files of folder

$ echo "newdata1" >> folder/file1
$ echo "newdata2" >> folder/file2

generating new Tape ARchive file (doing incremental backup)

$ sudo /bin/tar -cvf --listed-incremental folder.tar folder
folder.tar
folder/
folder/file2
folder/file3
folder/file1

listing Tape ARchive file (files were substituted)

$ tar -tvf folder.tar 
drwxrwxr-x ivanleon/ivanleon 0 2017-06-06 11:27 folder/
-rw-rw-r-- ivanleon/ivanleon 0 2017-06-06 11:27 folder/file2
-rw-rw-r-- ivanleon/ivanleon 0 2017-06-06 11:27 folder/file3
-rw-rw-r-- ivanleon/ivanleon 0 2017-06-06 11:27 folder/file1
ivanleoncz
  • 1,433
  • 4
  • 18
  • 32
  • Ok, so this one command will work for both creating it the first time and subsequent updates? – Bruce Jun 06 '17 at 15:27
  • I tried your syntax and it created a file called --listed-incremental. the -f always has to come right before the file name. However, when I moved the --listed-incremental before the -f, i get the message "/bin/tar: You must specify one of the '-Acdtrux', '--delete' or '--test-label' options". if ignores the -c. Any ideas? I DO need the --ignore-failed-read in there, can I only have one for some reason? – Bruce Jun 06 '17 at 15:48
  • 1
    With more research, I see that --listed-incremental needs to be expressed as --listed-incremental=snapshot.file, that file being the extra metadata that is stored to determine changes. This also has the effect of creating separate files on each consecutive day. Which could get complicated. I may experiment with just --incremental, which seems to only affect the original file and needs no external metadata. [link](https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html) – Bruce Jun 06 '17 at 16:20
  • I think that specifying an snapshot file, could be excellent when doing Differential Backups (I'm not an expert in backup routines with TAR), but for Incremental Backups, this command above seems to perform a great job, although I believe that there could be another approaches :). – ivanleoncz Jun 06 '17 at 18:18
0

I did some more research regarding --listed-incremental. Most articles were confusing at best, with not quite enough detail to fully understand. I then found the following article which explained very clearly:

http://paulwhippconsulting.com/blog/using-tar-for-full-and-incremental-backups/

The only thing I'd add, which I asked the author about, is that if you're testing on a small target, it can look like you're not getting a decent incremental if you make it right away, as there is significant overhead in the incremental. However, once you implement it on larger backups, it runs perfectly.

Instead of backing up to my separate Azure storage account which is connected via SMB, I made a second identical server and attached it via nfs. It's much more stable, and there are no filesystem differences that can mess things up. I'm setting up a script that will zip each backup and then copy it to the Azure storage container after each backup is completed. That saves processing time and power on the production server by performing the zip on the backup server. I hope this helps someone.

Bruce
  • 21
  • 4