24

I'm looking for a way to do Blue/Green deployments with CloudFront.

Does anyone have a good solution for moving from one CloudFront distribution to another or does everyone really just create their distribution and then never ever touch it again?

My CloudFront distribution consists of one S3 origin for static content (javascript, etc) and a custom origin pointing to an AWS ELB.

No Changes To CloudFront

Under normal circumstances we don't make any changes to our CloudFront distribution at all. We version our static content in the S3 origin by changing the name of the static content files in S3 and do rolling deployments to EC2 instances under the Elastic Load Balancer (ELB). However, there are times when we need to test and make changes to the CloudFront distribution itself or have significant enough changes to our environment that we need to point to a new ELB in a new environment.

Two CloudFront Distributions

The first option I attempted was to have two separate CloudFront Web Distributions, one for my current, or A, environment and one for my new, or B, environment. I attempted to use a Route53 weighted routing policy where I added two records for my www.domain.com Route53 record, one pointing to CloudFront Distribution A with a weight of 1 and the other pointing to CloudFront Distribution B with a weight of 0. The plan would be to change the weights when I want to move from distribution A to distribution B. However, only one CloudFront distribution at a time can have the www.domain.com Alternate Domain Names (CNAMEs) registered or you get the following error:

com.amazonaws.services.cloudfront.model.CNAMEAlreadyExistsException: One or more of the CNAMEs you provided are already associated with a different resource. (Service: AmazonCloudFront; Status Code: 409; Error Code: CNAMEAlreadyExists; Request ID: ef84a5f0-44e7-11e5-9315-0ba167bb108a)

One CloudFront Distribution

The second option is to keep one CloudFront web distribution. I have S3 and custom origins pointing to both my A and B environments and then I update the CloudFront Cache Behavior to point to the other origin when I want to move from one environment to another. This is extremely messy because these updates take 15-60 minutes, there is no visibility into the progress of the update and depending on the nature of your change you may need to follow that up with a CloudFront Invalidation so you aren't serving cached content from the old environment along with new content.

Thanks for your advice!

Peter M
  • 953
  • 2
  • 14
  • 27
  • Did you found any solution? We have the same problem in our project and way we do it now - we change cloudfront endpoint manually in our project. – Dmytriy Voloshyn Dec 10 '15 at 16:22
  • 1
    unfortunately no -- I don't think there is a good one. The best practice is to use one cloudfront distribution, version any content in s3 buckets and use route53 weighted dns records for origins pointing to dynamic content. then you just update route53 to change dynamic content and you don't need to touch cloudfront. We do maintain separate cloudfront distribution for dev and qa. EX: http://stackoverflow.com/questions/9130555/django-static-files-versioning – Peter M Dec 10 '15 at 18:23
  • This is a great question, and the problem is still difficult to solve in 2022 – rpivovar Mar 29 '22 at 03:20

5 Answers5

13

Two Cloudfront Distributions

Since AWS allows for overlap between wildcard alternate CNAMEs in the same AWS account, you can switch between two cloudfront distributions in the following manner:

  • Use www.domain.com as Alternate CNAME for Prod distribution 1
  • Use *.domain.com as Alternate CNAME for Prod distribution 2
  • Point your CNAME DNS www.domain.com to distribution 1 or distribution 2. (*.cloudfront.net).
  • Remove the alternate CNAME from the distribution which you don't want to serve the content from anymore.

However, the two different distribution DNSs (*.cloudfront.net) may point to the same edge nodes, which means that the way your content is served is by matching the cloudfront.net CNAME to the Edge nodes serving it and then to match by hostname. In this case, if both your distributions are using the same edge nodes (it can be checked for example with dig) the cut will not be clean.

e.g. If both distributions share one or more edge nodes, distribution 1 with Alt CNAME www.domain.com will take precedence over distribution 2 with the more generic *.domain.com until the CNAME gets removed from the distribution 1 config in all edge nodes. So the version retrieved from one request may be different from the other during the transition period.

mpaf
  • 246
  • 3
  • 6
  • Due to the extended change proliferation time in CloudFront, this really is your only option. – CloudWalker Jan 27 '16 at 22:29
  • Thanks -- this is an extremely interesting answer. I never thought about doing it this way. I'm going to mark it correct because it does cut from one distribution to another, however, I need to avoid the extended proliferation time and the risk of serving the wrong content during the transition, so I can't use it in my case. I agree with @CloudWalker that there isn't any other option. – Peter M Jan 28 '16 at 14:49
10

A bit late in the game here, but for anyone else looking for this. I believe this can be done using lambda@edge. Similar to A/B tests. https://docs.aws.amazon.com/lambda/latest/dg/lambda-edge.html. You can implement a lambda function triggered when a user requests a url. Choose to serve the blue/green content from different origins or url prefix. A cookie value will determine which deployment will be served. When the first request arrives (no cookie) set the the cookie randomly say 95% blue 5% green.

Gary H
  • 101
  • 1
  • 3
3

Shooting from the hip, how long does it take to switch the origin within a cloud front distribution? 1) deploy new code behind ELB, warm it up 2) add this ELB as a new origin to your cloud front distribution while removing the old origin 3) once cutover, tear down old code behind old ELB.

To get away from the delays associated with CloudFront updates or DNS updates, you could swap the autoscale group behind your ELB. 1) deploy new code in new ASG, warm it up 2) register your existing ELB with this new ASG 3) de-register the old ASG from your ELB 4) once cutover, tear down old ASG.

Ken Krueger
  • 131
  • 1
2

I've also been doing research into this topic and have a work-around (see below).

Background:

CloudFront requires the CNAME in the distribution config to be unique across your entire account. So controlling blue/green via DNS to different distributions will not work. There is a hack rolling around that would use wild cards but that makes no guarantee that the correct files are served. Controlling blue/green via DNS and CloudFront is not feasible.

Further, changing any config in CloudFront (including CNAME) results in 15-60 min of waiting while the changes are propagated to the edge servers. Also, not an ideal setup.

Work around:

For single page app, this configuration that may do the trick:

  • App URL: app.mydomain.com/app
  • S3 Structure:
    • app/ (your live site)
    • app2/ (your not so live site)

Now configure CloudFront to use your bucket to serve the files. At this point, it all comes down to you cache settings. Since CloudFront takes forever, set the CacheControl header on our S3 objects. For index.html, we use 5 minutes, everything else, 1 day. When it comes time to switch, swap the S3 directory names. Within 5 minutes, the app will be live for all intents and purposes.

For apps that are already running, we have the build version built into the code and a build config json file on the root of the app. Then the app will occasionally request the json file, check the version, if it's out of date, prompt for a refresh.

Limitations

You cannot performs soak tests very well. I suppose it's possible to increase the TTL of index.html to a few hours or days (depending on your need), that would help ensure the clients will get new versions as their local cache expires.

vangorra
  • 121
  • 2
2

In this blog post the author implements a-b testing via Lambda@Edge working off of the code in the AWS documentation (you can see their examples here: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html).

Basically what you do is you create a single Cloudfront distribution pointing to two different origins. Then you can use the Lambda@Edge to direct the traffic to either origin (via Cookies), and of course you can implement other things via the Lambda such as weighting the traffic or flipping it etc. It's also easy to add further origins and logic.

user3399551
  • 23
  • 2