Testing apache config changes

Question

What's the best way to reconfigure an apache server that's hosting a production site?

The basic workflow I use is as follows:

1. Make a test server that's as similar to production as possible
2. Reconfigure the test server so it's working the way I want it to
3. Wait until it's late at night so traffic is low
4. Apply the changes I made to the test server to the production server
5. Restart apache, with fingers crossed that it works
6. If anything goes wrong, roll back and restart

Because Apache needs things like IP addresses hard coded into its config, it's impossible (really not straightforward) to copy the config files exactly from a test server to production. So there's always a chance that things will go wrong when applying them in production. The thing that drives me completely crazy is that

A single typo in any apache config file takes down all sites hosted on that server!

How do people deal with this on large complex production servers? It seems like there has to be a way to check what a configuration will do, or at least if it's valid, without risking taking down the live site. Suggestions?

Answer 1

You can use the -t option with httpd to test the config without committing your running Apache instance to using it. If there are any errors, it will give you a non-0 exit status.

Additionally, if you're in a Linux-like environment that uses init scripts, then /etc/init.d/httpd (or sometimes /etc/init.d/apache2) will often support a 'test' argument, i.e. /etc/init.d/httpd test, which will check your config and tell you if it's good or not.

When you do have a good config, you can signal Apache to reload the config file without restarting it -- /etc/init.d/httpd reload usually. (If you aren't on a system with init scripts, I believe it's the USER1 signal.) Unless you're making changes on the level of removing existing vhosts or changing who can access what resources, this should not have any impact on current connections.

On Windows, the -t option is also available (httpd.exe -t), however I believe a restart is required to load the new configuration.

For your testing environment, you might consider a VM in an isolated, private network, with an IP hard-coded to match your production server's. This way you can more closely match your production environment, and don't have to worry about forgetting to change an IP address when you move the config into production. You could set up a second VM on that same private network with a web browser to test functionality, as well.

Answer 2

Treat your configuration as code: use a configuration management tool like Puppet to manage and deploy the apache config.

A general workflow:

Write (or download) a Puppet module to manage Apache, i.e. the Package, Service and config files. Set up your puppetmaster server to support two or three environments, say development, testing and production, and add your servers to their respective environment.

Your create a template from your current Apache config, where the values that differ between your development, testing and production environment, and between your servers, are replaced by variables. Depending on the environment that Puppet is pushing the config to, the variables will be replaced by the values you set. Puppet is designed to regularly check if the target servers (nodes) are in the desired state and if not, it places the config in the correct location and reloads Apache.

So how do you move your code from testing to production? Use version control software to manage your Puppet modules. Most people use branches that match the environments, and push new versions of the configs from branch to branch. Puppet will then see that the server does not match the new configuration and will update it.

This may sound a bit confusing if you've never thought about configuration this way, but you're already partway there, with your test server and workflow. You could start with just managing Apache, or even something less critical, and ultimately manage your entire server configuration with Puppet. Puppet can even become your documentation. Once your entire server is managed by Puppet, reinstalling it or building another server becomes a piece of cake, a matter of minutes.