0

Is it possible to make Kubernetes run a Job on a node after it has started, and do not run any Pods until that Job is finished?

My first idea was to use taints (start node with a taint that only Job can tolerate, then remove taint when the Job has completed). However, I'm using AWS EKS, and I want to use "managed nodes" feature if possible, which does not allow to specify node taints. It also does not allow to use custom instance bootstrap script, which was my other option.

Thunderbeef
  • 161
  • 5

1 Answers1

1

The thing I used for that problem is nidhogg where I would keep "workload" Pods off of a Node until the GPU driver and CNI bootstrapping were finished

The difference from the "without using taints" that you specified is that nidhogg dynamically applies and remove taints, and doesn't require setting them at Node provision time

It may require adjusting your workload to be aware of the taints that nidhogg will apply

An alternative is to have the Job that you're describing apply a label to the Node when it's finished (which it can do via the serviceAccount Token injected into the Pod), and then use a nodeSelector: in the workloads to only target Nodes which have that label

mdaniel
  • 2,338
  • 1
  • 8
  • 13
  • Interesting. Does nighogg guarantees that no Pod will be started on a new node before it has a chance to taint it? – Thunderbeef Jun 17 '20 at 20:55