Will Ubuntu automatically queue hundreds of processes?

Question

I'm creating a service that will be calling potentially hundreds or even 1000s of ffmpeg commands at once. Will Ubuntu 16.04 automatically queue these jobs?

Is there some sort of app that does a better job than its native methods?

Answer 1

Any Linux distribution will have no problem managing multiple, maybe hundreds, process instance.

However, for best performance you should only use as much processes as the hardware threads your machine can manage.

Adding many more processes (eg: 16 processes instances on a 8 threads machine) will lower performance due to heavy cache trashing and resource overcommit.

Answer 2

No, if you launch thousands of processes, the kernel is going to do its level best to run those thousands of processes. Which, for obvious reasons isn't going to end well.

To run jobs in serial like this, put together a proper job queue system using something like gearman or RabbitMQ. This way, jobs get queued up in a message queue, and your worker process consumes those messages as available and only runs a small number of processes at a time.