3

Is it possible in Sun Grid Engine to have array jobs where each subtask has a unique requirement? For example I may have an array job for which each task has a small unique requirement but I do not want to have to launch each job seperately.

Kamil Kisiel
  • 11,946
  • 7
  • 46
  • 68
Amit
  • 31
  • 1

2 Answers2

1

I assume you mean resource requirements. If so (and probably even if not), the answer is no. The definition of an array job is that it's a set of tasks that all share a common job definition. The resource requirements are set at the job level.

For reference, see the definition of a job (JB_Type: source/libs/sgeobj/sge_job_JB_L.h) versus the definition of a task (JAT_Type: source/browse/gridengine/source/libs/sgeobj/sge_ja_task_JAT_L.h). The JB_Type is about defining the job's parameters, while the JAT_Type is about tracking a task's execution.

If you're looking for a convenient way to automate submitting and tracking lots of jobs, have a look at DRMAA. API bindings for SGE are available in C, Java, Python, Ruby, and Perl.

Daniel

Daniel Templeton
  • 11
  • 1
0

Each sub task knows it's job id. This can be pulled from the environment (SGE_TASK_ID).

When working with array jobs, I have wrapper code that handles unique arguments for each job.

I start by generating a flat file of the arguments for each job. This can be CSV, JSON, text, whatever you like, as long as you can do a lookup based on the SGE_TASK_ID. This file gets saved to some common location.

The job is submitted with the path to this file. When each job runs, it knows to look in this file and find it's bundle of arguments for execution.

Gary Richardson
  • 1,767
  • 3
  • 19
  • 21