10

We want to set up a continuous integration (CI) service for our project. That CI service should control all aspects of the project life, including deployment.

That is, we want a CI agent at each iteration to take a clean system (specifically Ubuntu Server), deploy our project there and run tests.

We probably would write CI service ourselves, from scratch, as our ecosystem is rather unusual. The question is: how to effectively provide clean slate system for each CI iteration?

Project deployment install a few Debian packages, configures several nginx sites, and also install some packages in a implementation language-specific package manager (LuaRocks), systemwide. Nothing more intrusive than that (I believe). So, it is more important to be able to set up a clean slate fast (within a minute on modern hardware, faster — better) than to ensure absolute isolation and cleanness.

I see two ways how to do what we need:

  • Either set up a jail of some kind (like lxc, maybe).
  • Or install some virtual machine that can do snapshots, and use them.

But at this point I do not have enough information to decide.

What would you advise? Any other options? Any specific tool names?

Note: The CI service would be run inside a Citrix XenServer guest. Paid solutions are OK if prices are reasonable. (Usually they are not reasonable in this field.) Remote CI solutions not OK.

Alexander Gladysh
  • 2,343
  • 7
  • 30
  • 47

4 Answers4

13

Have a look at Jenkins. It has plugins that allow you to run builds in virtual machines in VMWare or VirtualBox or in the Amazon/a Eucalyptus cloud. For virtual machines run in the cloud, you can specify an init script where you could take steps like installing packages.

Lars Kotthoff
  • 646
  • 4
  • 10
  • Thanks, but Jenkins is a CI-server and I'd like to know about means to build one ourselves. – Alexander Gladysh Aug 20 '12 at 05:27
  • 2
    @AlexanderGladysh Why do you want to reinvent the wheel? Jenkins builds can run everything from make files to plain shell scripts, plus it has a robust interface, great support, and lots of plugins. Even if your build is very complicated, it has to be a shell script somewhere which jenkins can run – TheLQ Aug 20 '12 at 13:36
  • Even if you don't want to use Jenkins, I think that something cloud-based would be the way to go for you. You could customise a machine image to provide the software you want and only add configuration and code to test when starting the instance to reduce the start-up time. – Lars Kotthoff Aug 20 '12 at 14:01
  • Anyway, Jenkins or no Jenkins, what to use to get clean-slate OS snapshots quickly? (Please, stop mentioning third-party clouds — this must be run on our own servers.) – Alexander Gladysh Aug 26 '12 at 08:37
  • Run a Eucalyptus/OpenStack/... cloud on your own servers? – Lars Kotthoff Aug 26 '12 at 16:24
5

Take a look at Chef / Vagrant. Not sure if it will be fast enough, but something to look into.

Just in case you aren't familiar with Chef, it is a server designed to allow simple code based provisioning of servers. I'd recommend taking a look at one of the webcasts. Django Chef Webcast

Vagrant, veewee, chef-solo & knife-solo stuff I think this one covers most of what you want to do.

Vagrant Blog post A blog post showing a quick setup of a server using Vagrant.

Philip Tinney
  • 266
  • 2
  • 4
2

I'd be using VM infrastructure and create a base template, or use the cloud as @LarsKotthoff suggested. Its easy to spin them up fast, and you can be sure where you started from.

However you will ultimately need something outside that VM to orchestrate what is going on (setup the machine, and kick off whatever process you want on the machine). I'd look at Jenkins for its ability to capture output of scripts for later inspection.

Although Jenkins Started as a CI server, numerous organisations I've worked with now use it for much more than that, it really is a tool to kick things off (manually or scheduled) and capture results, and validate what happened is as expected. And it's very flexible.

tenacioustechie
  • 21
  • 1
  • Thanks, but I would like to setup things right inside Xen VM instance, not outside — this is much easier to organize in my specific case. I will look into Jenkins, but I'm still looking for more narrow solution. – Alexander Gladysh Aug 20 '12 at 10:46
  • Also, please elaborate regarding "use VM infrastructure and create a base template". – Alexander Gladysh Aug 26 '12 at 08:40
2

We want to set up a continuous integration (CI) service for our project. That CI service should control all aspects of the project life, including deployment.

Jenkins has been mentioned by quite a few. You can go ground-up with scripts, but Jenkins will provide you with all the same flexibility with less overhead getting it together.

That is, we want a CI agent at each iteration to take a clean system (specifically Ubuntu Server), deploy our project there and run tests.

Project deployment install a few Debian packages, configures several nginx sites, and also install some packages in a implementation language-specific package manager (LuaRocks), systemwide. Nothing more intrusive than that (I believe). So, it is more important to be able to set up a clean slate fast (within a minute on modern hardware, faster — better) than to ensure absolute isolation and cleanness.

This is where virtual machine images come in handy. You can setup your clean image and have every machine launch a fork of that. The original image is never modified, but your testing won't know the difference. You should definitely be able to get a bootstrap image running in a minute's time.

Paid solutions are OK if prices are reasonable. (Usually they are not reasonable in this field.) Remote CI solutions not OK.

If you're looking for a running multiple machines at once, check out Nimbula (I'm being shameless; I work for them). For smaller deployments, it's free and lets you pull together your own private cloud for testing... or anything else. Between Jenkins and an orchestration file, you can spin up host servers, test clients, and everything else to automatically run the full gammut of testing code without any interaction.

Community
  • 1
Jeff Ferland
  • 20,239
  • 2
  • 61
  • 85
  • Ugh. Maybe I'm slow today, but I tried to dig through Nimbula site, but can't see the reason behind marketing speak. Do you care to point to a sane text that describes how to do what I need? (I.e. set up a guest machine template and, from script, instantiate and kill it for each CI session.) – Alexander Gladysh Aug 26 '12 at 08:46
  • @AlexanderGladysh It's handy if you're using many machines to build from / lots of CI work. It's useless if you don't have enough workload to warrant a cluster of at least 3 machines. – Jeff Ferland Aug 27 '12 at 23:10