0

Some of the nodes in my cluster disappeared just now according to Kubernetes Dashboard and kubectl. In the Google Console, it shows the old amount. This results in many pods not being schedulable due to missing resources.

I tried to increase the node pools now, which is taking much longer than normal, and some new nodes were added, but the amount is still wrong.

Also, in the preemptible pool, no new nodes were added.

What can I do now?

Node Pools according to Google Cloud Console

Only one node in Kubernetes Dashboard

Only one node in kubectl

Sebastian Rösch
  • 101
  • We are currently experiencing the same issue in our production cluster. It seems as if all preemtible nodes are gone. – Sebastian Rösch Oct 25 '18 at 08:19
  • In the preemtible VMs documentation, it is stated that "Preemptible instances are finite Compute Engine resources, so they might not always be available." I get that they can randomly restart, which our cluster can tolerate without a problem. But so far, new nodes were available within minutes. Now, the preemptible VMs have been gone for more than a day and it looks like we're still paying for them. Also, shouldn't there be a warning when no preemptible VMs are available? – Sebastian Rösch Oct 25 '18 at 09:00

2 Answers2

0

It sounds like you are running into a quota or limit issue. I would start checking that:

https://cloud.google.com/compute/quotas https://cloud.google.com/kubernetes-engine/quotas

Marc
  • 33
  • 11
  • Thanks, Marc, I checked but am well below the limit on each resource quota. And the nodes were there before, so they shouldn't just disappear when I hit the quota, I hope. – Sebastian Rösch Oct 24 '18 at 17:21
0

To assist you further and help investigate your cluster issue to answer all your questions. I would recommend that you open an Issue Trackers as this platform is meant for defect reports.

ALex_hha
  • 7,025
  • 1
  • 23
  • 39
M.E Taame
  • 1
  • 2