I have been trying to setup kubernetes for a week now as part of an evaluation, but I can only find
- extremely complex manual solutions that are already out of date or outright broken
- insanely simple solutions that break a mind numbing amount of best practices
- "get to know kubernetes cluster all on one box"
My base requirements going in are:
- I have AWS
- I have VPCs, Subnets + peering already created
- I need to be able to build one cluster per environment in the appropriate VPC
- Ubuntu 16.04
- I should have control over the SGs and instances (terraform + ansible)
- Absolutely no single points of failure.
- I don't want a solution that creates unnecessary layers by adding tools or controllers above the masters. (WHY?!!)
- No blatant security holes
I've tried a couple solutions, but they have failed spectacularly, and require #7
kubeadm (breaks #6, #7)
- I gave this a try because it was supposed to be a reasonable solution for using existing infrastructure.
- I was able to get a cluster operational with 1 master and 2 nodes, but it just didn't seem to be working fully (kubernetes-dashboard was inaccessible)
- In the end, even the tool itself didn't recommend production due to the fact that the docs state the master node cannot be made HA.
juju / conjure-up with "canonical kubernetes" (breaks #2, #3, #5, #6, #7, #8)
- did not allow specification of VPC. Nevermind managing SGs, instances, working via bastion host.
- It actually created instances in ec2-classic...?
- instance naming in AWS is awful (can't tell what's what - masters vs etcd vs ??)
- 15 security groups for a 3 worker cluster, 9 of which are completely empty?
- conjure-down fails with 'NoneType' object has no attribute 'controller'
- It created a single point of failure in the "controller machine"
- INGRESS TCP 22 0.0.0.0/0 <- I am going to cry (this is why ec2-classic went away years ago)