This is an old revision of the document!
This page presents some internals of OAR's management of resources, in order to give administrators some useful knowledge and help creating relevant definitions of the resources with regards to CPUs-cores topologies as well as CPUs-GPUs.
The OAR database resources
tables provides several kinds of information:
They are called OAR resources properties.
In database these 4 kinds of resource properties are all stored as columns of the resources
table. A rows then gives the set of properties for one resource.
Given a hierarchy (chosen by the administrator for its cluster setup, for instance cluster/switch/host/cpu/core, but thread could be added, or other customizations), one row in the table gives information for the lowest level of resources of the hierarchy (e.g. core). Then groups of lines define higher levels, like Russian dolls (e.g. #C
rows for CPUs, #H*#C
rows for hosts, …).
One rule must be kept in mind: any unique object in the resources hierarchy must have a unique id among its set of object. For example:
Then, when it comes to the hardware identifiers (cpusets, or see below for GPU devices id), the administrator must take a special attention so that a correct mapping is done between the logical hierarchy (e.g. id of the host, CPUs, cores, hyperthreads) and the hardware processing unit ids (cpuset value). Using a tool such as hwloc
may help at this point.
The basic commands to work with resources are:
oarnodes
command gives an extract of the resources table. oarnodesetting
command can be used to modify or create rows in the resources table. oarproperty
command can be used to modify resource properties (columns of the table).
But two meta-command are provided to build the resource table (using underneath the oarpropery
and oarnodesetting
commands):
oar_resource_init
first looks at the machines hardware topology (connecting to them using SSH) and then provides the relevant oarnodesetting
commands ;oar_resource_add
builds oarnodesetting
commands from a model defined in the command line.See the next section for an example of resource table.
Support of Nvidia GPU devices was added to OAR and will ship with OAR 2.5.8 (already ship with RC versions, starting with 2.5.8 RC6).
Meanwhile, for those who cannot wait and since this only involves some configuration of the resources and using the latest version of the job resource manager script taken from git (the job resource manager is part of the configuration files of OAR, which the administrator can modify), one can backport the feature to released versions (e.g. 2.5.7).
See:
The next release of OAR will provide tools to help setup GPU resources (see the oar_resources_add
command included in OAR starting from version 2.5.8 RC6), but for now, here are some explanations for the setup (for advanced OAR admins only).
In order to enable the mechanism, you have to:
$ENABLE_DEVICESCG = “YES”;
)oarproperty -a gpudevice
)
For step 4, several scenarios are possible. Let's consider here a first scenario where you have 2 GPUs on your nodes, one attached to the first CPU's PCI-e bus, and the second to the 2nd CPU's PCI-e bus (see lstopo
's output for instance to know that). You can then use oarnodesetting
to set the gpudevice
property to 0 (matching the /dev/nvidia0
Linux device) for the resources associated with the first CPU, and 1 (matching the /dev/nvidia1
Linux device) for those associated to the 2nd CPU, for every hosts.
Users would then only have access to 1 GPU of N hosts if requesting a single CPU per host, e.g.:
$ oarsub -l /host=N/cpu=1 ...
or could even request GPUs directely, e.g.:
$ oarsub -l /host=N/gpudevice=1
Result would be equivalent (1 GPU = 1 CPU).
(Warning giving host=N is mandatory in the second variant of the oarsub
command, or one would reserve all hosts of the cluster with a same GPU id.)
Running the nvidia-smi
command in the job should show what GPU is available.
Also, if some nodes do not have any GPU, you could set the value of the property for the corresponding resources to gpudevice=-1
, and let the users add to the oarsub
command a -p “gpudevice >=0”
in order to get resources with GPUs.
Lets assume now that you have a cluster of 3 nodes with 32 GB of RAM and per node:
lstopo
)Obviously, cores and GPUs appear at the same level in the hierarchy of resources we can define, giving two possible hierarchy definitions:
In order to provide a single hierarchy, we have to make a arbitrary choice (not related to a real hardware hierarchy), and choose for instance to associate 3 cores to a GPU. The hierarchy then becomes:
Lets translate that to technical words:
See the oarproperty
command manual, in order to create cluster, host, cpu, core, gpu, gpudevice, cpumode, gpumodel.
Finally, lets now assume that your OAR server also manages a second cluster with no GPU, e.g. 2 nodes with 24GB of RAM and 2 CPU each of 4 cores.
We can now define the resources in the resource table of OAR as follows (1 lines par core, 3 lines par GPU, 6 lines per CPU, 12 lines per host…):
cluster | host | cpu | core | cpuset | gpu | gpudevice | mem | cpumodel | gpumodel |
---|---|---|---|---|---|---|---|---|---|
cluster1 | host1 | 1 | 1 | 0 | 1 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 1 | 2 | 1 | 1 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 1 | 3 | 2 | 1 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 1 | 4 | 3 | 2 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 1 | 5 | 4 | 2 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 1 | 6 | 5 | 2 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 2 | 7 | 6 | 3 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 2 | 8 | 7 | 3 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 2 | 9 | 8 | 3 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 2 | 10 | 9 | 4 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 2 | 11 | 10 | 4 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host1 | 2 | 12 | 11 | 4 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 3 | 13 | 0 | 5 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 3 | 14 | 1 | 5 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 3 | 15 | 2 | 5 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 3 | 16 | 3 | 6 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 3 | 17 | 4 | 6 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 3 | 18 | 5 | 6 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 4 | 19 | 6 | 7 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 4 | 20 | 7 | 7 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 4 | 21 | 8 | 7 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 4 | 22 | 9 | 8 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 4 | 23 | 10 | 8 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host2 | 4 | 24 | 11 | 8 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 5 | 25 | 0 | 9 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 5 | 26 | 1 | 9 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 5 | 27 | 2 | 9 | 0 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 5 | 28 | 3 | 10 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 5 | 29 | 4 | 10 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 5 | 30 | 5 | 10 | 1 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 6 | 31 | 6 | 11 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 6 | 32 | 7 | 11 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 6 | 33 | 8 | 11 | 2 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 6 | 34 | 9 | 12 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 6 | 35 | 10 | 12 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster1 | host3 | 6 | 36 | 11 | 12 | 3 | 32 | Xeon Gold 6128 | Geforce 1080ti |
cluster2 | host4 | 7 | 37 | 0 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 7 | 38 | 1 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 7 | 39 | 2 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 7 | 40 | 3 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 8 | 41 | 4 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 8 | 42 | 5 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 8 | 43 | 6 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host4 | 8 | 44 | 7 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 9 | 45 | 0 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 9 | 46 | 1 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 9 | 47 | 2 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 9 | 48 | 3 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 10 | 49 | 4 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 10 | 50 | 5 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 10 | 51 | 6 | 0 | 0 | 24 | Xeon Gold 5122 | |
cluster2 | host5 | 10 | 52 | 7 | 0 | 0 | 24 | Xeon Gold 5122 |
(Remember that OAR properties (columns) that define the resource hiearachy must be unique values for each unique entity, but system properties (cpuset, gpudevice) loops on each host)
See the oarnodesetting
command manual, in order to create the table rows and set the values. Help yourselfs with scripting the command calls (e.g. use shell for
loops).
Finally, you can look at the created rows using the oarnodes
command, or even look at the resources table in the database.
Once this is done, users can request one of the 12 GPUs as follows:
$ oarsub -p "gpu > 0" -l gpu=1
Or 2 GPUs (possibly not on the same host) as follows:
$ oarsub -p "gpu > 0" -l gpu=2
Or 2 GPUs on a same host as follows:
$ oarsub -p "gpu > 0" -l host=1/gpu=2
When reserving 1 GPU, the user obviously gets the 3 cores associated to the GPUs.
GPU job can be tied to GPU resources (where gpu > 0
) with the following admission rule (see oaradmissionrules
), so that users don't have to set -p “gpu > 0”
in their command lines:
foreach my $mold (@{$ref_resource_list}){ foreach my $r (@{$mold->[0]}){ my $gpu_request = 0; foreach my $resource (@{$r->{resources}}) { if ($resource->{resource} eq "gpu") { $gpu_request = 1; } } if ($gpu_request) { if ($r->{property} ne ""){ $r->{property} = "($r->{property}) AND gpu > 0"; }else{ $r->{property} = "gpu > 0"; } print("[ADMISSION RULE] Tie job resource request for GPU to resources with GPU\n"); } } }
Warning: make sure to look at lstopo output in order to correctly associate cpuset and gpudevices, e.g. not associating cores and GPUs not attached to a same CPU.
In OAR 2.5.8 (starting from 2.5.8 RC6), the oar_resource_add command provides some support to create GPU resources in OAR as well as CPU-Core resources with relevant topologies.