I've been trying to think through a problem in utilizing Terraform (though any infra language would do, like CloudFormation, ARM or whatever Google uses as this is not a cloud applicable problem to my knowledge). I've often done canaries in software engineering, and its pretty much a staple besides red/black (or blue/green, pick your word poison). To build a scenario, let's say that we already have our networking layer in in place, and it will not be changed. I'll use AWS to draw this out into manageable chunks.
Let's say our VPC has the following attributes
- Is a multi-tier VPC in a random AWS region
- Contains 2 Private and 1 Public Subnet in the specific VPC CIDR
- Has Routing Tables that are attached to the appropriate subnets
- A NAT instance with ASG (Auto Scaling Group) to serve as default gateway for the private subnets
- Security Groups to use with EC2 instances that might be created
- SNS notifications for said Auto Scaling events
Pretty standard. Okay we have our baseline, let's say our application will be something leveraging AWS specific services.
- Some EC2's with LCs and ASG defined
- Load balancing with oldschool ELB
- Redis for in mem caching
- A Postgre RDS for whatever
How would you canary this? Let's say we want to test out ALB instead of ELB, or remove Redis entirely and replace it with Elasticache. We evidently don't want to roll this straight to production, it'd be suicide.
Would I consider something like a second Route53 coupled with Traffic Flow (AWS' version of traffic shaping to my knowledge) and a set of the desired infrastructure to gradually give load to see if when or how it breaks? I believe the R53 portions in particular are traffic weighting and failover policies? Please let me know your thoughts, has been something tickling the back of my brain for months.
