0

We use RKE for setting up the cluster with 4 master node (VM, 8 core CPUcurrentlysourcesstagingahorriblea workers and 16G ram) and 5 worker node(bare-metal, HPE G10, 88 core CPU, 256G ram)

-Kubernetes version:1.15

-Host OS: coreos

and increase max pod per node to 250 and currently we have about 500-600 total pods. Most of our stack is php-fpm.

Yesterday we ran a load test with autocannon from multiple sources on our production (staging namespace :| ) and yes I know it's horrible but we think cluster is ok and tolerate it due to our monitoring background history.

I want to know do you recommend that we use many workers (VM) instead of few workers (bare-metal)?

Dave M
  • 4,494
  • 21
  • 30
  • 30
alireza71
  • 101
  • 3

1 Answers1

1

As always: it depends on your needs.

Let's dig into that: first step is to compare the two different deployment mode (few larger workers and many smaller workers) with the very same typology: both VM or both Bare Metal for simplyfing your comparison. Giving that we have:

Benefits of few larger nodes

  • less node management: you have to install, patch and upgrade a smaller number of Coreos
  • less overhead on master node

Benefits of many smaller nodes

  • Redundancy and high availability: if one node fails its resources can be easily rescheduled on survived nodes
  • Less kubelet overhead: with less pods per worker, kubelet has to do less operation like liveness and readiness per node
  • smaller nodes means you're closer to your actual resource needs: if you need 60 CPU and you have 2 nodes with 50 CPU each, you're paying for 40 unused CPU. If you have 3 nodes with 20 CPU instead, you're optimizing your costs.

I don't know your needs but usually I prefer a more High Available solution.

Now step back to VM vs bare-metal. You have to add the virtualization layer overhead to the disadvantages of many smaller nodes. Is smaller nodes solution still worth it? Again, it depends on your needs

PS: why an even number of master nodes? Kubernetes needs an odd number of etcd nodes

oldgiova
  • 406
  • 2
  • 5