2

I am not really a sys admin, I am mainly a developer with some unix knowledge, and because we don;t have any competent sysadmin , I have been delegated the task to implement the backup strategy.

We have many web servers, some are running Mysql Srevers, other apache, another one is running nginx, and we are using SVN as a versioning system.

We were looking forward to implement a backup strategy, and to be able to automatically restore a server .

We have though of 2 possibilities :

  1. RSYNC the whole DISK everyday
  2. Analyse our configuration, and normalize each installation steps, backup these information only, and use MySQL replication, and SVN to restore data, and rsync to backup only configuration options, and package list

For our point of view, the first method has the advantage to be very simple to implement, and have the disadvantage to use a lot of server resources (CPU /RAM/bandwidth)

Because we didn't want to see our servers lagging because of a backup script running, we decided to go deeper with (2).

After some reflexion, I came up with the idea that each of our servers can be divided into 5 parts

1 - Webserver Data

SVN can handle all our php/css/js/html files, we just need to have a configuration file that stores information about folders and repositories. For example : in the file /etc/backup/svn_folders.list , I would have

FOLDERNAME1    SVN Repository Address1
FOLDERNAME2    SVN Repository Address2
etc...

Then in case of a crash, we just need to parse this file, and SVN checkout .

2 - backup MySQL data with replication

We have 3 main mysql servers, I have implemented on a backup server mysql_multi, with 3 instances of mysql runnning at the same time, being each salves of the main servers. Then, every day, I

Stop slaves
mysqldump
start slave

This way, I am certain that our main MySQL servers aren't affected by the backup process. Then foreach main servers, I just need to stock these informations in a conf file /etc/backup/mysql.info serverID = ID

To recover the database, I will just have to get this serverID from the conf file, and then, rsync the corresponding dump image from the backup server to the restored server.

3 - Package list

With debian, it is easy to know the full package list installed on the system. A cron will just stock this list into /etc/backup/package.list

4 - Custom applications

Sometime, we need to install packages manually (perl, compilation, etc...) I was thinking of creating a folder /etc/backup/manual/ containing all the automated installation script , then on restore, running each script in this folder should make the trick

5 - Configuration Files

A file /etc/backup/conf_data.list with a list of all configuration directories (Ex : /etc ) , can be parsed via a cronjob, and then rsynced on the backup server.

On restoration, we need to first restore, the /etc/apt/sources.list, doing step - and 4, and then rsync the saved configuration files back to the server.

Can you please let me know what you think about this concept. Have you already implemented something like that? Have you run into issues?

Ant
  • 193
  • 1
  • 9
  • How many files, how much space are you using? An rsync solution should be relatively fast, if you don't have lots of changes, and don't have lots of small files. My rsync (via dirvish) of my web servers 30-90 seconds. In any case, I would tempted to have both solutions. – Zoredache Mar 02 '12 at 03:40
  • on of our servers have a lot of small files, it is a SOAP server, and I keep all the xml calls/answers for debugging purpose (the last months). – Ant Mar 02 '12 at 03:41
  • I Didn't know dirvish, this looks to be a good product. – Ant Mar 02 '12 at 03:48
  • 1
    Since you asked about how to track config changes in a comment see this question. http://serverfault.com/questions/3852/what-tool-do-you-recommend-to-track-changes-on-a-linux-unix-server – Zoredache Mar 02 '12 at 04:20

2 Answers2

1

I think you have come up with a decent plan however there are 2 errors above. rSync only backs up changes, not the entire disk and rsync should get the config files if they change regardless assuming you configure it correctly.

My opinion is you need to start off with a solid evaluation of what losing that data would cost and decide if it is worth chancing the loss based on your knowledge. If the cost is higher than your comfort zone you need to get some consulting or hire another company to implement a solid strategy. No offense but if there is a significant risk you do not want to be left holding the bag when the wheels come off the wagon. I have seen people lose their jobs over things like this. They want to go above and beyond but when something does not go right they are way over their heads. Sometimes you just have to know when to bow out. You can still have input but not shoulder the blame if it fails.

Just ny .02

DaffyDuc
  • 512
  • 2
  • 7
  • He didn't say that would would over-write the files every day. Rsync Can be used to keep changes over time. The backup systems rshapshot/dirvish are based off this functionality. – Zoredache Mar 02 '12 at 03:52
  • @ Zoredache : Let me get some clarification about dirvish : I have a server who explodes in a datacenter , the hosting company will then give me a new server (Same IP, but maybe, different hardware). I will then install from scratch a Debian, and use gzip images from dirvish , to restore the lost filesystem. That should work this way isn't it? @DaffyDuc : You're right, about the config files : I should keep an historic of it : For example, a hacker get's into the server, uninstall some packages, and change some mandatory conf files, if I restore the last saved backup, I will be f*^^*d. – Ant Mar 02 '12 at 04:07
  • What about using SVN on /etc/ ? – Ant Mar 02 '12 at 04:09
  • 1
    @zoredache I agree however its still just going to keep changes, not transfer the entire file system. There are plenty of ways to skin this cat.@cooluhuru I see no reason why you could not use svn on etc but I have never heard of this being done. Definitely possible. – DaffyDuc Mar 02 '12 at 05:32
1

You might want to add a regular run of aptitude-create-state-bundle into your scheme for complete system recovery. See http://man.he.net/man1/aptitude-create-state-bundle.

OliCoder
  • 111
  • 2