Script for remove old backups

Question

I have a lot of backups, like:

backup-20130528_054504_mysql.tar.gz, 
bill_db_201305290130.BAK.bz2,
etc_20130412.tbz

This is full backups which done each day.

Before today, I have a cron task with a command line like this:

/usr/bin/find /media/backups -ctime +7 -delete

But it was the bad idea.

One server stopped do new backups (filesystem was full) more than 7 days ago, and all old backups was deleted by cron.

Probably I need a new script as replacement for this "find -ctime", which will hold some old backups anyway.

All my backup files have names {unique_prefix}YYYYMMDD...

I want something like :

one backup for each year before last year
one backup for each month of last year
4 backup for each week of previous month
and 7 backups for last 7 days.

for each {unique_prefix}.

All this files are downloaded to my server by cron via rsync, ssh , etc..

I dislike reinvent the wheel, and pretty sure that is a typical task.

Please, point me to the solution.

http://gehrcke.de/timegaps has been designed for your purpose. — Jan-Philip Gehrcke, Mar 16 '14 at 23:33
Beautiful idea, but the current implementation is scary, as by default it removes the most recent item: https://github.com/jgehrcke/timegaps/issues/7. (I haven’t been able to find a workaround in the discussion; pointers welcome.) — Flash Sheridan, May 17 '22 at 17:30

score 3 · Answer 1 · answered May 31 '13 at 18:58

A shell script can handle backups up to a certain point of complexity. You also have tools that leverage tar and rsync to accomplish even more convoluted tasks (rsnapshot, for example).

However, it looks like you need a very fine-grained backup and retention policy that a shell script might not be well suited to handle unless it is unnecessarily complicated and the other tools mentioned do not provide.

You should consider using a dedicated software for this task. There are many, but bacula and amanda come to mind.

Thank you alot. I take the point, but still like scripts more then dedicated software. — Korjavin Ivan, Jun 03 '13 at 03:38

score 2 · Accepted Answer · answered May 31 '13 at 19:08

2

I disagree with dawud and think simple backups are the best.

What you really need is not a smarter way of deleting things (though that wouldn't be a bad idea, I wrote something similar which simply keeps the last N backups), but proper monitoring of your filesystem so you'll notice it filling up well before it does and you can take action. That way you won't lose your data.

answered May 31 '13 at 19:08

Dennis Kaarsemaker

18,793
2
43
69

1

While I agree in spirit, what works for you might not be the best for OP. He's stated he is not willing to build it and wants to be pointed to the solution. That solution is a proper OTS backup suite. He's going to have a much easier time setting up Bacula or Amanda than starting from scratch. – Aaron Copley May 31 '13 at 19:41
Yes, thank you. I really prefer simple thing. But simply keeps the last N - very simple. Looks like I just need good script for retention – Korjavin Ivan Jun 03 '13 at 03:37

score 0 · Answer 3 · answered Aug 31 '22 at 11:10

You can use time based policy (dangerous if you stop producing new backups, as all eventually be deleted):

find . -type f -name 'app-*.tar.gz' -mtime +100 --delete

Keeping last X items is safe:

ls -t app-*.tar.gz | tail -n +100 | xargs --no-run-if-empty rm

The best you can get with daily/weekly/monthly/yearly strategy which is not trivial to implement. You can try luck with rsnapshot and cron.

There are libs that supports deletion of old snapshots, you need to adopt their conventions though:

Script for remove old backups

3 Answers3