What do you think about this Debian Backup strategy?

Question

I am not really a sys admin, I am mainly a developer with some unix knowledge, and because we don;t have any competent sysadmin , I have been delegated the task to implement the backup strategy.

We have many web servers, some are running Mysql Srevers, other apache, another one is running nginx, and we are using SVN as a versioning system.

We were looking forward to implement a backup strategy, and to be able to automatically restore a server .

We have though of 2 possibilities :

1. RSYNC the whole DISK everyday
2. Analyse our configuration, and normalize each installation steps, backup these information only, and use MySQL replication, and SVN to restore data, and rsync to backup only configuration options, and package list

For our point of view, the first method has the advantage to be very simple to implement, and have the disadvantage to use a lot of server resources (CPU /RAM/bandwidth)

Because we didn't want to see our servers lagging because of a backup script running, we decided to go deeper with (2).

After some reflexion, I came up with the idea that each of our servers can be divided into 5 parts

1 - Webserver Data

SVN can handle all our php/css/js/html files, we just need to have a configuration file that stores information about folders and repositories. For example : in the file /etc/backup/svn_folders.list , I would have

FOLDERNAME1    SVN Repository Address1
FOLDERNAME2    SVN Repository Address2
etc...

Then in case of a crash, we just need to parse this file, and SVN checkout .

2 - backup MySQL data with replication

We have 3 main mysql servers, I have implemented on a backup server mysql_multi, with 3 instances of mysql runnning at the same time, being each salves of the main servers. Then, every day, I

Stop slaves
mysqldump
start slave

This way, I am certain that our main MySQL servers aren't affected by the backup process. Then foreach main servers, I just need to stock these informations in a conf file /etc/backup/mysql.info serverID = ID

To recover the database, I will just have to get this serverID from the conf file, and then, rsync the corresponding dump image from the backup server to the restored server.

3 - Package list

With debian, it is easy to know the full package list installed on the system. A cron will just stock this list into /etc/backup/package.list

4 - Custom applications

Sometime, we need to install packages manually (perl, compilation, etc...) I was thinking of creating a folder /etc/backup/manual/ containing all the automated installation script , then on restore, running each script in this folder should make the trick

5 - Configuration Files

A file /etc/backup/conf_data.list with a list of all configuration directories (Ex : /etc ) , can be parsed via a cronjob, and then rsynced on the backup server.

On restoration, we need to first restore, the /etc/apt/sources.list, doing step - and 4, and then rsync the saved configuration files back to the server.

Can you please let me know what you think about this concept. Have you already implemented something like that? Have you run into issues?

Answer 1

I think you have come up with a decent plan however there are 2 errors above. rSync only backs up changes, not the entire disk and rsync should get the config files if they change regardless assuming you configure it correctly.

My opinion is you need to start off with a solid evaluation of what losing that data would cost and decide if it is worth chancing the loss based on your knowledge. If the cost is higher than your comfort zone you need to get some consulting or hire another company to implement a solid strategy. No offense but if there is a significant risk you do not want to be left holding the bag when the wheels come off the wagon. I have seen people lose their jobs over things like this. They want to go above and beyond but when something does not go right they are way over their heads. Sometimes you just have to know when to bow out. You can still have input but not shoulder the blame if it fails.

Just ny .02

Answer 2

You might want to add a regular run of aptitude-create-state-bundle into your scheme for complete system recovery. See http://man.he.net/man1/aptitude-create-state-bundle.