6

Say I have application servers, database servers, and a few dns-round-robin load balancers. All this powered by images created with Packer with deployment managed with Terraform. How do I change the image of the database servers without nuking their data when the instances get destroyed and recreated?

The simplest thing I can think of would be to turn off writes, snapshot the database, and then restore the snapshot to the new servers. But it feels really wrong to rely on manual fiddling like that, and it also feels wrong to take the service down for a simple upgrade. There is a cleaner and better way, right?

womble
  • 95,029
  • 29
  • 173
  • 228
jpadvo
  • 183
  • 1
  • 5

2 Answers2

5

There's no simple answer to this question.

Using an architecture designed around images (commonly referred to as "immutable infrastructure") works fantastically for stateless services, like your application servers.

It's most definitely possible to expand that to your stateful services with the right tools, failover systems and upgrade paths, but those are usually overkill for simple systems (as the one you describe).

One thing to keep in mind when using these tools is that you don't have to go "all in". Packer and Terraform are very much designed to work only where you want them. They don't enforce a pattern across all of your systems on purpose.

Practically speaking, the best way to handle this problem is maintain your database servers differently, outside of Packer (building the initial image, yes! But not necessarily upgrading them in the same way as the stateless web servers) or outsource managing the state to someone else. Notable options include Heroku Postgres or AWS RDS.

To round it out – yes, it's possible, but with our current tooling it's probably more trouble then it's worth at smaller scale or with simple architectures.

Packer and Terraform can still be a huge boon in other aspects of the very same infrastructure – Terraform, for example, could provision a Heroku database for use in your DigitalOcean Application servers in a very straight forward way. Packer can handle upgrading and releasing your application server images, and likewise for development.

Jack P.
  • 174
  • 2
  • 1
    Ah, the simplest way to win is by refusing to play the game. I bet Hashicorp will come out with software for managing databases someday. Anyway, I can get on with my new infrastructure now, many thanks! – jpadvo Jul 31 '14 at 19:55
  • 1
    Alternately, you could store the database files on a disk which is separate from the base install image, and use a Terraform provisioner to mount the filesystem on that disk, run a tool to upgrade the database files (if necessary), and point the database program on the new base image to the database files. (This post is pretty old so I'm not sure if that would have been possible back in the day, but it would work ok now.) – Dan Oct 29 '16 at 03:05
2

I think Terraform now has the features required here. The basic pattern is to define your data volumes separately and attach these to the instances, so that when the instance is destroyed and a new one created (e.g. from a new AMI built by Packer) the existing volume can be attached to the new instance.

So the detailed steps with Terraform would be:

  • define aws_ebs_volume resources
  • attach these to your instances with aws_volume_attachment (here I've used device_name = "/dev/xvdh")
  • ensure your aws_ebs_volume resource include the lifecycle rule prevent_destroy = true (so they will never be deleted by terraform)
  • ensure your aws_volume_attachment resource includes skip_destroy = true (on upgrade terraform would fail to destroy these whilst the volume is mounted, and stopping the instances will destroy the attachment anyway so no need for terraform to attempt it)

The final step is to ensure that the instance mounts the volume on startup. Which can be achieved with the following in the user_data of your aws_instance resource:

#!/bin/bash
mkdir /data #create mount point
mount /dev/xvdh /data #mount it

For the above to work you'll need to prepare the volume by creating a filesystem, but this is only required once:

mkfs -t ext4 /dev/xvdh

More details are available in Terraform issue #2740.

joelittlejohn
  • 126
  • 5
  • 1
    good points about lifecycle rules. i recommend snapshotting the ebs too which you can do with an `ebs_snapshot` in a `destroy-time provisioner` or `creation-time provisioner` in a `null_resource`. also, add your mount to `/etc/fstab` as shown in the referenced issue. – Mike D Apr 08 '17 at 00:13