- When a work item is "ready", a "Scheduler" should dynamically start a "Worker" Docker image.
Many Kubernetes functions are now built in using custom resources which makes Kubernetes more modular. Custom resources let you store and retrieve data but when you combine it with custom controller
, custom resources provide a true declarative API
.
For you task you want to check an Operator pattern which is a combination of custom resources and custom controllers.
You Operator will have to watch for your work item
state and once it is ready it triggers the Kubernetes Job that runs yours docker Worker
image to completion.
Worth to mention that instead of operator you may use go-client or python-client to access Kubernetes RESTful API interface. client-go
is very powerful lib for developing K8s custom controllers.
For more officially supported client lists please visit official documentation.
ClusterAutoscaler
feature can used to scale up Kubernetes cluster when you need to run a Job and then scale it down when the job is finished.
If you wishes to read more about operator
Alternatively you may want to look into some kind CI/CD scheduler like Jenkins.
Here are some good examples of kubernetes with jenkins:
But in case of you are looking for something like message queue for managing task instead Jenkins it might be more convenient to have it all in the same Kubernetes cluster as Deployments and StatefulSets.
Here you have more information about message queue managemet:
Lastly a small comparison between Kubernetes and Docker that can give you some more insight which one is more suitable for you.