Automating server deployment

Question

I find i am constantly settings up pretty much nearly identical servers and VPSs for a number of my clients and it can be very time consuming. Often the only thing that changes between each deployment is the different website that is to be served. Is there an easy way to automate all this and take the boring monotony of setting up 56 identical servers?

The servers i have deployed so far have only been Ubuntu, but it may be possible that i start to use other linux OSs or even Windows. So far i have looked at Capistrano, but it seems to be focused on writing little ruby programs to do the job with, and i have no knowledge at all

Answer 1

Puppet sounds perfect for what you're trying to do, with the caveat that as of right now, there is no support for Windows.

In your case, you would define a Server node in terms of all of the packages that are identical across the machines. Then, you define the individual hosts as nodes which inherit from Server, and set up the specific unique things for it.

Puppet is declarative - it allows you to describe of your boxes in terms of the resources each box should have. So if you want ssh - you write a class for that resource - and inside the class you can include logic about how ssh is called slightly different on FreeBSD vs Ubuntu. It also knows to use yum inside Redhat and apt-get inside Debian based distros, and ports in the BSDs. Now in your Server node, you'll just have a line like include ssh - and puppet will do the right thing and put SSH on the machine without you having to remember if that's Ubuntu or Redhat or FreeBSD.

What's nice is that all of the Server stuff lives in one place - and if at any point you add to the Server node definition, ALL machines would update their configuration accordingly.

Right now, I'm only managing three boxes using Puppet - but it's already paid off. After spending a week setting up a box we'll be using for stimulus presentation in an experiment, it turned out the graphics card driver was too old in the version of Ubuntu I put on it (8.04). I had to install the latest Ubuntu (9.04), but after that I just had to apt-get and run puppet - and everything I had spent a week setting up was restored.

Puppet does have a little bit of a learning curve, but I've successfully avoided learning Ruby - I know I'm using it, since that's what puppet is written in - but so far I've been successful in just modifying the examples in the documentation and the recipes on the wiki . Another downside is that puppet does take a bit longer to do things the first time. The upside is that everything you change across all of your machines is stored in one place - it's standard practice to keep your puppet configuration in a version control system - so you can always look back and see how you've set up servers in the past - or roll-back some unsuccessful changes.

Finally, here's a quick video that does a simple puppet demo that got me started quickly.

Answer 2

We use Cobbler and Puppet for build and configuration automation of both real and virtual machines.

Cobbler ties together DHCP, PXE boot and Kickstart to make deployment nothing more than adding a machine profile and hitting the power button. For VMs, the koan command does the (in our case) Xen magic to get the installation started -- on the dom0 I just type:

koan --system vps.fqdn --server cobbler --no-gfx

then virsh console to watch a VPS building without any interaction.

We use RHEL and have a bunch of profile set up to partition disks, configure networking and install base packages for different server classes. Cobbler supports Debian and Ubuntu breeds but I've never tried it. An aside: other interesting uses for Cobbler include running memtest ISOs and HP firmware updates.

Once our systems are built with Cobbler Puppet takes over to configure the applications, system daemons, register the box with RHN, etc. Puppet runs as a daemon which periodically checks that the systems's configuration matches the defined manifests -- you know your updates have gone to all servers. It's also a great way to be certain that a box that's been down for maintenance has the correct configuration before you return it to live service.

Puppet really is awesome. You don't need to get every aspect of your configuration under its control -- start off by having it manage something simple that you need to configure on every box (sudoers is the canonical example) and take it from there. Make sure your Puppet manifests are versioned, too; nothing's better than being easily able to roll back to a known-good configuration without having to remember what to adjust.

Answer 3

Where I am working at the moment, we have to manage the Linux portion of our server farm which is just over 300 Linux servers. This includes mostly HP Proliants, followed by IBM 3850s, some IBM blades, VMware ESX, and some KVM for our internal management servers.

cobbler

We looked at cobbler but the problem there was that cobbler is very RHEL/Red Hat specific. We need to support RHEL and SLES at the least, and Ubuntu is next.

puppet

We did consider puppet, however later decided against it as it depends on Ruby, which means that an upgrade of Ruby, could potentially break our management system.

hotwire

Hotwire is what we use (developed internally, but is open-source), and have done so for the last few years. It firstly inventories the systems that are going to be built, which means inventorying the data-center, rack, hardware, operating system, network, etc, and secondly perform the rapid build & deploy. Once the system is built, hotwire's auto-inventory keeps the inventory in-sync, while cfengine maintains them. Hotwire knows about the server hardware by talking to the SMBIOS/DMI data in the Bios via python-dmidecode.

The bonus points are that it is combines the inventory and build process into one, so there's less to manage, and the live-inventory feature is great as we know if something is not quite right.

The disadvantages are that the user-interface still needs polishing, and there are bugs here and there, but development is still hot, and bugs reported are fixed relatively fast.

cfengine

We use cfengine because other than it, and puppet, there is nothing else. It actually is a good tool, but "good" only as a function of how good your policies are - if you set dangerous policies, then a small mistake can cause alot of damage. For example, by policy, we don't "modify" files, we either replace them, or we don't. Also all files replaced have a header that makes any person editing it know that it will be replaced the next time it runs (it is run via cron hourly).

The configuration and all files pushed out by cfengine to the servers are also kept in a SCM, and using post-commit hooks, where possible, we check the syntax and if that fails, then the commit is rejected. This is easy for nice applications such as Apache, but not so easy for most enterprise applications.

Answer 4

I've had success with Puppet. Chef is a new one to emerge. For a longer list of options and a comparison chart, see the Wikipedia article, Comparison of open source configuration management software.

Answer 5

For automating installation depending on the target system:

• Debian/Ubuntu: FAI or d-i preseeding
• RedHat/Fedora: Kickstart
• Novell/openSuSE: AutoYaST
• Solaris: Jumpstart
• Windows: unattended.sourceforge.net

For the configuration management on top of that I'd suggest using puppet.

Answer 6

Another vote for Puppet here. We use it extensively to perform all server and application installation and configuration management. 200+ nodes and counting. Windows support is apparently in development, though in what state I'm not sure.

We're still looking into the initial OS bootstrap side of things, but as mentioned above Cobbler looks interesting. We're currently using a mix of PXE booting with Debian/Ubuntu preseeding, but it's hardly optimal.

Answer 7

I'm having a lot of success with Puppet, but you do have to write a lot of config.