I am planning to run some data processing workloads on AWS Batch in a highly parallel manner (>50 concurrent jobs). These workloads have the unfortunate characteristic that they become more memory intensive over time. It's not a memory leak but it is inefficient memory management that I can't control. So at first, the memory footprint is minuscule but by the end of these jobs (~20mins), each task requires about 8GB of memory.
My question is how does AWS Batch decide where to send a job when it starts (which EC2 instance or Fargate instance) and when to start scaling out to multiple instances? If it sends all 50 jobs to a single EC2, that instance will be forced to scale up to 400GB of memory. But Batch can't possibly know that the memory footprint will increase over time.
I'm clearly missing something about how this works but I can't find any documentation on how Batch scales.
