0

We are in the development phase. We are using Kubernetes cluster consisting of master and slave EC2 instances. We are submitting tasks to Kubernetes cluster using Kubernetes_Pod_Operator of Airflow. We are looking to scale this process. So we are going to use Celeryexecutor of Airflow which is used to concurrently submit and schedule tasks on Airflow.

So the question is, should we care about number of tasks submitted to Kubernetes or irrespective of number of tasks submitted to Kubernetes, Kubernetes will service all the tasks without failure by any internal queuing ?

Adrián
  • 188
  • 2
Free Coder
  • 41
  • 1
  • 4

1 Answers1

1

First of all, note that Kubernetes is not meant to do this kind of operations, precisely because there is no queuing idea in Kubernetes. So, you do care about the number of submitted tasks.

From the source code of "kubernetes_pod_operator.py" it seems that it just creates a pod in the right namespace, with the right image, etc.

In the end it is a pod, so it will do a job and finish (status: completed).

Taking this into consideration, it will depend on the jobs it will have to run, and your machine type. An example:

Say you are running simple pipelines that consume about 0.1CPU and few MB of memory. If your nodes are 4-CPU machines (lets assume you have enough memory), then you can run ~40 concurrent jobs per node. If you run more, you will get an error (saying your pods can't be scheduled).

So,

  • (recommended) If you can actually determine a standard resource consumption per task (per pod), I would recommend you to set resource requests and limits per pods (As by default a pod can consume the 100% of the nodes resources), and always try to run the maximum amount of pods. You will need to keep the track of the number of pods.
  • (not recommended) If you can't determine the pod consumption, you could either monitor the nodes and add tasks as long as they have enough room, or to try creating the pods with exponential back-off, if you get an error on pod creation because it can't be scheduled.

Hope it helps. Again, this is not something that I am used to see on kubernetes.

suren
  • 161
  • 4