Use git for multiple server configuration files

Question

We have migrated a lot of source code over to git and are very happy with our current solution. We would like to have our server configuration files versioned on the same system, but there are a few things that don't work the way we would like it to and I hope someone can share his experience here.

This question is similar to Using revision control for server configuration files?, but we have some special requirements that do not work with the suggestions on that question.

The current setup uses subversion for configuration files. The corresponding repository looks something like this

 / # root of repository
 +--www.domain.com/     # configuration for www
 |  \--etc/
 |     \--apache2/
 +--dev.domain.com/     # configuration for dev
 |  +--etc/
 |  \--opt/
 |     \--app1/         
 |         \--conf/     # configuration for app1 on dev
 \--staging.domain.com/ # configuration for staging

With subversion this would work just fine, because it's possible to just checkout a sub-directory of a repository. In addition you can use svn:externals to point to one common structure for several different configuration setups. We only had to deal with the .svn files in all versioned directories. Git on the other hand does not have svn:externals and sparse checkouts always require the path from the root to the actual directory to be the same.

When discussing the migration to git, I tried to write down the main Requirements for the server configuration versioning:

• we only want a single repository
• it should be possible to easily push changes to the central remote
• changesets should contain the real author

Is there a nice way to have all the configuration in one repository and only have a sub-path as working copy? Currently I am considering two approaches, but wanted to ask this question here first

1. If the .git repository is at a fixed location, e.g. somewhere in /var, we could link to the sub-path from the "target" working directory. The main problem: I would not know of a way to "link" from /etc to another directory in order to only import the contents, except symlinking single files
2. I found another alternative on this SO question, suggesting to have multiple branches in one repository. This would certainly increase complexity, but I could see us trying this way.

Using git on a single machine for configuration file management works fine, but I believe there must be someone who is using it the way we would like to use it.

Thank you
Kariem

Answer 1

I've used something like this before; this is how it worked.

Repo Setup

1. Create a git repo, "etc_files".
2. Create a branch for each machine type, e.g., "server/www", "server/dev", etc.
   • git supports slashes in branch names. This helps me keep the branches straight in my head.
   • If you have few enough machines, you could have a branch for each individual machine instead.
3. Create a branch for each piece of shared infrastructure, e.g. "modules/apache", "modules/cups", etc.
   • These branches are for holding files that are the same between all machines, like /etc/resolv.conf. These would be the files you keep in "svn:externals" repos now.

Building a New Machine

1. On a new machine, clone the git repo and check out the branch for that machine type.
   • I make this a read-only clone to prevent people from committing changes from production machines without testing.
2. Set up a cron job to automatically git pull the repo every day.

Changing Machine Branches

Changing the code in a single machine branch is simple; just git checkout the appropriate branch in your development environment, make the changes, and commit them back to the central repo. All machines in that branch will automatically get the changes the next time the cron job runs.

Changing Module Branches

Changing the code for a module branch is only slightly more tricky, as it involves two steps:

1. git checkout the appropriate module branch
2. Make your changes and commit them to the centralized server.
3. git checkout each machine branch which uses that module branch, and then merge the module branch into it. git will figure out that you've merged that module branch before and only notice the changes that have happened since that last common parent.

This method has both benefits and drawbacks. One benefit is that I can make a change to a module branch and apply it to the machine branches that need it, which letting the machine branches that don't stay with the older version until they're ready. The drawback, then, is that you have to remember to merge your module branch into each machine branch that might be using it. I use a script that traverses the commit tree and automatically does this merging for me, but can still be a pain.

---

As an alternative, newer versions of git support something called "submodules":

Submodules allow foreign repositories to be embedded within a dedicated subdirectory of the source tree, always pointed at a particular commit.

This would allow you to build something a little bit like "svn:externals" trees, which you could then update in much the same way as you do now.

Answer 2

Bit of a nub here but It Sounds like a post receive hook could do the job

Repo master is stored in /var/master
the hook clones it to /var/localclone
then copies out the host specific details
You would need to set up the .git/post-receive hook locally on each server (with appropriate settings)

It also sounds like you want something more like puppet or chef than git for this this would allow you to run git to manage the configs and modules centrally and have puppet\chef manage the deployment and verification for you

Answer 3

here is a quick work i thought of
1. Have a central repository in say /var/repo
2. Put files which are global to all domains in that directory.
3. Create branches for every subdomain /var/repo/subdomain1,/var/repo/subdomain2 etc.
4. Create a post hook where any push to the branch is merged with the master.

So when you change config of /var/repo/subdomain2 it is immediately merged with master so you can pull the repo entirely and have all config files
When you want to pull individual subdomain configs, just pull the appropriate branch.

How do you put your config files into /var/repo/* is entirely different matter (cron tabs, i can think of)

~$