3

Has anyone had luck randomizing Slurm node allocations? We have a small cluster of 12 nodes that could be used by anywhere from 1-8 people at a time with jobs of various size/length. When testing our new Slurm setup, jobs always go to the first node in the partition if there are no other users for both interactive and batch jobs. Is there a way to randomize this scheduling?

It seems like, depending on a user's timeline, they could consistently get the same nodes and this could disguise issues in hardware/configuration that might otherwise be visible. Our nodes are always exclusive, so we're only looking at randomizing the node-level scheduling...

tnallen
  • 31
  • 1

3 Answers3

2

Look at the Weight setting in slurm.conf

The priority of the node for scheduling purposes. All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements. For example, a heterogeneous collection of nodes might be placed into a single partition for greater system utilization, responsiveness and capability. It would be preferable to allocate smaller memory nodes rather than larger memory nodes if either will satisfy a job's requirements. The units of weight are arbitrary, but larger weights should be assigned to nodes with more processors, memory, disk space, higher processor speed, etc. Note that if a job allocation request can not be satisfied using the nodes with the lowest weight, the set of nodes with the next lowest weight is added to the set of nodes under consideration for use (repeat as needed for higher weight values). If you absolutely want to minimize the number of higher weight nodes allocated to a job (at a cost of higher scheduling overhead), give each node a distinct Weight value and they will be added to the pool of nodes being considered for scheduling individually. The default value is 1.

Tux_DEV_NULL
  • 1,083
  • 7
  • 11
1

You can add "LLN=YES" to the partition.

LLN Schedule resources to jobs on the least loaded nodes (based upon the number of idle CPUs).

Zhibin Lu
  • 11
  • 1
0

I don't believe that it is possible to randomize node allocation without altering the code or providing your own plugin. There are many way to affect which nodes will be chosen by a given job, but none of them are random. As @Tux_DEV_NULL noted you can use weight to prefer a subset of nodes, but unless you're randomly changing weights you'll still end up on the same node again and again if your entire cluster is idle. I believe that a nodes uptime is used as the ultimate tie breaker.

If you're worried about missing misconfigured or broken nodes then you should use a node health check script using the HealthCheckProgram parameter in slurm.conf. A good one to use can be found here: https://github.com/mej/nhc

Michael Robbert
  • 1
  • 2