Batch Job Processing vs Linux Kernel

Question

I have several large hugin scripted jobs which I would like to have processed in a timely manner. Each script will sometimes use all of the machine's cores at times, and other times be single threaded for some time.

I'm wondering about the fastest way to get all of my stitching jobs done. Would it be best to let the Linux kernel sort it out? For example, let all of the scripts slam the cpu at once? Or is it better to space the jobs apart? - Something like sun grid engine / Maui (there is only one server)? Or alternatively, rewrite all of my scripts into a single script? This last option wouldn't be very portable.

Answer 1

As long as your resource constraint is CPU time, you can let the kernel scheduler do its job and you will get a somewhat optimal result.

However, if you have other constraints like RAM usage or disk IO, it might be actually beneficial to space out the jobs in some way, but you need to observe your system yourself to find out what the actual constraints are to come up with an optimal strategy.

Using a classic job scheduler would most likely help not a lot or at all as long as the phases of single- vs multithread processing happen in the same process/job.

Answer 2

How fast do you need to go?

One approach is to keep adding jobs until your resources are at some limit, like a CPU utilization threshold. That keeps process scheduling overhead and memory use manageable. For example, GNU make has --load-average which will pause new jobs until load average is below the threshold.

It is worth understanding what part of the process is single threaded. Look at source code where available. Sometimes not much can be done about this in the short term except getting CPUs with good single thread performance.

Some cloud-style applications can be easily scaled out to multiple hosts, often automatically. If the application can be structured that way, one host can burst to maybe hundreds on demand.

There are many tools to use when working on a performance methodology. One reference I recommend: Red Hat Enterprise Linux Performance Tuning Guide.