0

We're trying to figure out a way to periodically have system security patches for our K8s cluster, to keep our system safe and meet the security requirements.

our K8s clusters are running in different clouds, AWS, Azure, Bare metal, etc.

for clouds, we can change our IAM image to update to the latest, replace the old image, launch new nodes, and drain the old nodes. For bare metal one, we need to drain the old nodes, and then patch, and add them back.

Not sure if there is any other way to do that automatically. we don't want to do this work each month in each clouds. maybe there is a better solution?

Ethan Xu
  • 343
  • 2
  • 10
  • You can automate this using your existing configuration management system e.g. ansible, etc. – Michael Hampton Aug 26 '20 at 00:35
  • 1
    I think you meant `AMI`, not `IAM`; also, if you haven't yet considered it, [Flatcar Linux](https://www.flatcar-linux.org/) auto-updates itself, and there's an [operator that will drain the Node before reboot](https://github.com/kinvolk/flatcar-linux-update-operator#readme) – mdaniel Aug 26 '20 at 04:16
  • @mdaniel yeah it's AMI, I think our company will still choose a major distribution rather than a very small one. but thanks. – Ethan Xu Aug 27 '20 at 04:47
  • @MichaelHampton yes we use Ansible, I'm thinking about is there any tool running on k8s which can control or update the cluster – Ethan Xu Aug 27 '20 at 04:50
  • 1
    Orchestration like this tends to be very specific to your organization's requirements. I would be very surprised to find anything off the shelf that you could use without significant customization. Perhaps roles that handle small parts of it, like creating an AMI, would be available. But if you write your own playbooks, then they will do exactly what you need them to do, every time. – Michael Hampton Aug 27 '20 at 05:11

1 Answers1

1

Packer by HashiCorp is a free tool for automating the creation of machine images.

Packer can build machine images for different clouds including AWS, Azure, and self hosted virtualisation platforms like VMware.

Using Packer, you can consume published reference machines (Amazon published AMI's for example), automate the install of patches, updates, and any custom configuration, before publishing the patched image back to your cloud platform, ready to be consumed.

Packer can also be used to seed standard configuration management tools into cloud images. For example perform ansible or puppet runs, and/or bake in the necessary configuration for these to run when a instance starts up.

Terraform (available for free), also by HashiCorp, lets you automate the configuration of compute platforms, such as AWS/Azure/VMware, allowing you to update the configuration automatically.

If you wished to deploy or update the AMI used by nodes in a running Kubernetes cluster, you could likely do this by creating a new launch configuration pointing at the new Packer built image, using Terraform.

One way you could potentially reduce the amount of work, is to modify the boot script (cloud init/user data) of the instance group, to run something like a 'yum update' on boot. This way you could use the pre-made images, and simply need to update the AMI ID to the latest version, each time these are released. This could likely be done using Terraform.

KHobbits
  • 1,090
  • 7
  • 13
  • thanks! that's awesome. I can use packer to build image for all the clouds, so the only thing I need to do it replace image id for different cloud. – Ethan Xu Sep 04 '20 at 05:04
  • I've got a packer config set up, which builds an image for AWS, Azure and GCP in parallel. I specify the base image for each cloud, and a bash script to perform on each with my custom configurations (including configuring puppet), which are the same for each cloud. When packer runs, it creates me an updated image in all 3 providers, into the regions I'm using. Right now I then update my groups in the relevant cloud to point to the updated image manually, spin one up to test it, and call it a day. The output of packer, tells me which image id it creates in each provider which makes it easy. – KHobbits Sep 04 '20 at 09:43
  • For the non-virtualized hosts... I have a pxe boot image, with more or less the same config as I have in packer. I guess it could also be possible to use something like vmware esxi free, on each host, if you wanted a bit more uniformity. – KHobbits Sep 04 '20 at 09:46