Rolling update with puppet, ansible or fabric

Question

I have some jetty servers with a loadbalancer before them. Now I want to update my application without downtime. When one jetty is going down and no longer reachable the loadbalancer automatically removes it from the list, so this is not the issue.

The main problem is to avoid down time: so, I have to make sure only one jetty is restarting at a time - or make sure that at least N jettys are online!

At the moment I'm using a simple bash script where I need to wait manually for one jetty to came back online again before restarting the next jetty and so on.

Now bash is not very optimal for this kind of stuff and I hope there are tools better suited to make the whole task automated. E.g. I could use a simple ping URL http://jetty-number-n.com/ping which responds OK (200) if the n-th jetty is online.

How could I solve this task and with which tool?

Thanks to @ceejayoz I found rolling update for ansible. But it is still suboptimal as I would need to set a fixed timeout.

score 9 · Accepted Answer · answered Mar 26 '13 at 21:26

9

This is fairly easy with Ansible. Rough pseudo-ansible playbook:

---
  - hosts: your_server_group
    sudo: yes
    serial: 1
    tasks:
      - shell: reboot now
      - wait_for: port=22

answered Mar 26 '13 at 21:26

ceejayoz

32,469
7
81
105

2

Remember that the default timeout for `wait_for` is 300 seconds, so you may want to up it. – ceejayoz Mar 26 '13 at 21:30
1

Nice! Instead of a fixed timeout - how could I use my ping-URL to wait for the next server restart? – Karussell Mar 27 '13 at 09:55
1

@Karussell you do not want indefinite timeout because if your server never comes back up you'll be waiting forever. What you could do is update this play to raise a flag on this timeout and then bail out from deploying to the rest of the hosts if that flag is set. That way you won't serially kill all of your hosts. :) – Mxx Mar 27 '13 at 12:21
Still unsure if I understand what you mean. What I want is that it waits until the ping URL returns OK and then proceeds instead of the fixed length timeout (which can be too long but also too short) – Karussell Mar 27 '13 at 14:12
@Karussell It's not a fixed length timeout. It's checking for port 22 (SSH) to come back up. You could also target it at port 80 (webserver) or something else specific to your system. The timeout is the point at which it'll fail if the port hasn't come up. – ceejayoz Mar 27 '13 at 14:14
Ah, thanks! Still one more issue: what if jetty is up and running at this port but my app needs some initialization time? Can I specify a full ping URL instead of only the port? Looks like the docs say no: http://ansible.cc/docs/modules.html#wait-for – Karussell Mar 27 '13 at 14:19
No, but you could certainly make a custom module based on the wait_for and get_url modules to do what you want. – ceejayoz Mar 27 '13 at 14:23

score 4 · Answer 2 · answered Apr 04 '13 at 15:57

Salt has a convenient batch option.

salt -G 'os:RedHat' --batch-size 25% service.restart jetty

Restart the jetty service on 25% of RedHat servers at a time.

salt -N group1 -b 2 system.restart

Restart servers in the pre-defined "group1", 2 at a time.

score 1 · Answer 3 · answered Mar 26 '13 at 20:40

Puppet has the concepts of Services and Ordering. But puppet runs on on a single server. So it doesn't have insight into other systems that it is running on.

If you wanted to use puppet, you could have it so one master control server has a init script for managing the services of each server. So the init script would run the restart on each server and return back the status code. You could do this and chain each service restart from this one server and an ordering, chaining or notify and subscribe to move on to the next one.

You could also write a custom library for puppet, that will treat one server as the master control server, but instead of using the init scripts. It can use a custom ruby library to manage the services of each server. It's a bit more complex to set up, but could work.

Thanks for clarification! Hmmh so puppet is probably the wrong tool ... — Karussell, Mar 26 '13 at 21:10
Not necessary the wrong tool, but can be a piece to the puzzle — Ryan Gibbons, Mar 26 '13 at 21:11

score 1 · Answer 4 · answered Mar 26 '13 at 20:46

I use Puppet at work right now, but as the first poster replied, this doesn't sound like an ideal first task for Puppet. I can't speak to ansible (though I want to start trying it out).

Puppet is geared more toward rolling out changes/restarting systems simultaneously--a Puppet daemon runs on each client, and checks against a central server at a scheduled interval (default every half hour). If you want to buck the default scheduling, you usually use a different mechanism to call puppet.

If you're ok with a simultaneous restart, then restarting a service is a matter of defining a manifest with

file { 'your_war_file.war':
    ensure  => file,
    source  => "puppet:///...",
}
service { 'jetty':
    subscribe  => File['your_war_file.war'],
    hasrestart => True,
}

resource directives, then pushing out the manifest and letting Puppet do its thing. This assumes that you're ok with calling

/etc/init.d/jetty restart

which may or may not be acceptable or even possible.

You also could define your own custom resource types for this, which is definitely beyond the scope of a simple first-time puppet task.

Check out http://forge.puppetlabs.com/, however, to see if anyone else has solved a similar problem.

Thanks! But simultaneous restart will make my app unavailable for several dozend seconds, so do you know another tool/trick? — Karussell, Mar 26 '13 at 21:03

score 1 · Answer 5 · edited May 23 '17 at 12:41

The task you want to do can be done using ansible, and it definitely could be done using Puppet. I'm going to focus on how to accomplish it using puppet.

You can deploy your WAR files to jetty, using puppet, by using a normal "file" resource and a defined resource that you initialize for each application.

Puppet will downloaded from the URL you specify (on your Puppet Master). Jetty is able to redeploy WAR files without restarting, if the WAR file modification time changes. So if your manifest creates a copy in $JETTY_HOME/webapps it should be automatically picked up by Jetty. This way (or by touching the context.xml) you don't need to manually restart Jetty.

In order to download the WAR file from your Puppet Master to your Application Directory you will need the following manifest (replacing $JETTY_HOME):

define jetty::deployment($path) {

  notice("Deploying ${name} to http://$hostname:${appserver-jetty::port}/"),
  include jetty,

  file { "$JETTY_HOME/webapps/${name}.war":
    owner => 'root',
    source => $path,
  }
}

and you need to instruct your nodes to get a specific version of your application one after another (after you're sure that it has successfully started on one of the nodes). This can be done using an ENC (external node classifier) or, if you are not using ENC, by modifying your site.pp section for each node (or group of nodes):

node jetty-node1 {
   jetty::deployment { "servlet":
     path => '/srv/application/Servlet-1.2.3.war'
   }
},
node jetty-node2 {
   jetty::deployment { "servlet":
     path => '/srv/application/Servlet-2.0.1.war'
   }
}

The include jetty line is necessary only if you want to manage your Jetty configuration through puppet using eg. https://github.com/maestrodev/puppet-jetty and this defined resource needs an appserver-jetty::$port variable. This example shows the simplest way you could control exactly which version of the Application is served by which node. This way you can script the deployment, based on your health-checks and deploy on node2 only after node1 is successfully running the new version.

This example is only to demonstrate the concept.

You might want to check these additional resources for more information and ideas: On reloading the contexts: https://stackoverflow.com/questions/13965643/auto-reloading-war-in-jetty-standalone

This is for deployment and management of tomcat, but the ideas (and the manifests) are similar: http://www.tomcatexpert.com/blog/2010/04/29/deploying-tomcat-applications-puppet

This is for a, IMO, better alternative to directly copying the WAR files - using native packages (eg. RPM) for deploying your application: http://www.slideshare.net/actionjackx/automated-java-deployments-with-rpm

Well I think the main problem is timing: what if all jetty servers are restarting at the same time? — Karussell, Mar 27 '13 at 10:30
Since you're the one setting the version of the application to be installed on each node (inside the site.pp or the ENC), you (your script) should make sure that the new node is up and running. I will fix the syntax of the manifest snippets. — zorlem, Mar 27 '13 at 13:04
Redeployment without restart won't work over a long timespan (see permgenError problems on stackoverflow). Especially if you deploy lots of wars/jars into it. — Karussell, Mar 27 '13 at 14:15

Rolling update with puppet, ansible or fabric

5 Answers5