I've an application. It can generate CSV files. It consists of:
- A Mysql db.
- Message Queue.
- Some coordinator services.
- A set of worker services.
I do not have access to the source for the application, and cannot change it.
The application was not designed with Kubernetes in mind, but nonetheless it seems like a Kubernetes StatefulSet would happily accommodate it. (This is an assumption I'm making, and I'm open to other options.)
Anyways, what is vexing me is how to setup the following use case:
- I want to be able to create a job that will start a StatefulSet, submit some work to it, collect the CSV output, and shut down the StatefulSet afterwords.
- And I want some way to limit the number of concurrent StatefulSets to a configurable number that can run on the cluster (for example, maxsets=2 or something).
- I'd like jobs submitted in excess of the resource limitations in (2) to queue.
The jobs are completely independent of one another, can be kicked off at any time, and the number of jobs is not known in advance.
I believe that (1), on its own could probably be handled with a Kubernetes 'job' construct.
My Question therefore: What are some ways to deal with (2) and (3)?
NB: The resource usage profile of the application is a bit chaotic, so using basic resource constraints on CPU/MEM or whatever probably won't be sufficient.