20

It's possible to force the re-creation of a EC2 or RDS instance using cloudformation stacks?

My stack goes stuck in a point where simply destroying and creating the resource will fix it, instead of that I had to delete entire stack to continue work.

edit:

This issue hit me twice. First I created an AWS::RDS::Instance with some defaults and then tried to downgrade it to "EngineVersion" : "5.5". Changing this its suposed to happen with some interruption, but mysql instances cannot be downgraded from 5.6 to 5.5 so the stack was left in UPDATE_FAILED state and I cannot be able to recreate RDS without a nasty trick.

The other occurrence was that i have several "AWS::EC2::Instance" which downloads and executes an script from it's "UserData" obviously if Y change the downloaded script I must recrete the instance, and theres no way to do so. Once again I use the same nasty trick to get the machine recreated.

The nasty trick:

Instead of using an autoscaling group of one machine, I solved both problems changing the availability zone in the properties... but left me with a bad taste

theist
  • 1,199
  • 2
  • 9
  • 24
  • Need more info to answer. Do your instances freeze on startup? Does a service become unresponsive? If you're looking to manually recreate an EC2 instance, you can create an auto-scaling group with one instance. When you terminate the instance, another will be created. – Edwin Sep 19 '13 at 04:54
  • edited for clarify. I also asked here: https://forums.aws.amazon.com/thread.jspa?threadID=135295&tstart=0 – theist Sep 19 '13 at 15:59
  • This doesn't directly answer your question, but for re-running UserData scripts when changed, you could look into `cfn-hup`: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-hup.html – Reed Kraft-Murphy Nov 09 '13 at 09:20

3 Answers3

14

For instance store-backed EC2 instances, one trick is to add a comment to the user data script containing a version number, date, or similar, then change that whenever you want the instance recreated:

{
    "Resources" : {
        "MyEC2Instance" : {
            "Type" : "AWS::EC2::Instance",
            "Properties" : {
                // ... other properties ...
                "UserData": { 
                    "Fn::Base64" : {
                        "Fn::Join" : [ ":", [
                        "#!/bin/bash\n",
                        "# Version: 1.0\n",
                        // ... rest of user data ...
                    ]]}
            }
        }
    }
}

Any change to UserData will cause the instance to be replaced (i.e., regenerated). The behavior of the user data script should be the same, though, since the only modification is a comment. Note that this doesn't work for EBS-backed instances.

For RDS, you could take a DB snapshot of the current RDS instance, then modify your template to use that snapshot with DBSnapshotIdentifier:

{
    "Resources" : {
        "MyDB" : {
        "Type" : "AWS::RDS::DBInstance",
        "Properties" : {
            // ... other properties ...
            "DBSnapshotIdentifier": "<db snapshot ID>"
        }
    }    
}

Whenever DBSnapshotIdentifier is changed, the database instance will be replaced. Using snapshots will also let you keep the data from when the snapshot was made. (If you want to wipe the data, you could create an empty snapshot and pass that as input. Or delete and recreate the entire CloudFormation stack.)

A more generic approach is to change the logical name of the resource. From Modifying a Stack Template in the CloudFormation docs:

For most resources, changing the logical name of a resource is equivalent to deleting that resource and replacing it with a new one. Any other resources that depend on the renamed resource also need to be updated and might cause them to be replaced. Other resources require you to update a property (not just the logical name) in order to trigger an update.

markusk
  • 485
  • 6
  • 9
  • Seems that the only solution is to do "dirty tricks" I reached a similar solution (forcing availability zones changes) some time after asking :) – theist Jul 27 '15 at 09:57
  • 5
    Just want to point out that the instance is replaced and thus the UserData executed when the EC2 instance is instance store -backed. If it is EBS backed, the change of UserData will only make the instance restart and UserData is not executed again. You can use cfn-hup to get the UserData run again even in this case, but the instance stays the same. – Kaitsu Sep 30 '15 at 11:37
  • @Kaitsu: Thanks, that's a very valuable clarification. I updated my answer accordingly. – markusk Oct 02 '15 at 05:35
  • @Kaitsu but if you manually re-run the script (located at /var/lib/cloud/instance/scripts/part-001) you have to make sure the script is defensive to running the same commands multiple times :( – c24w Jun 01 '17 at 10:51
1

If you put it into an AutoScalingGroup, you can edit the AutoScalingGroup min/max/default to 0, then as soon as it starts to destroy the old instance, you can then put the min/max/default to 1/1/1 and presto: new instance.

Tim Bassett
  • 111
  • 2
0

If your EC2 are into an AutoScalingGroup you can set the AutoScalingGroupName property with a version number in it.

Every time you change that version number CFN will: 1. create a new auto-scaling group and spin up the desired instances 2. kill instances in the old auto-scaling group and delete it

Here is a piece of code from my stack where I use this technique to force a large number of EC2 machines to re-create and automatically pull new software from S3.

AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
        AutoScalingGroupName: !Sub "${StackName}-${ServiceName}-${ServiceVersion}"
marcopeg
  • 101
  • 1