Support of Nvidia GPU devices was added to OAR and will ship with OAR 2.5.8 (already ship with the last RC version, e.g. 2.5.8 RC6).

Meanwhile, for those who cannot wait and since this only involves some configuration of the resources and using the last version of the job resource manager script taken from the sources (the job resource manager is part of the configuration files of OAR, which the administrator can modify), one can backport the feature to released versions (e.g. 2.5.7).

See:

Next releases of OAR may provide tools to help setup GPU resources, but for now, here are some explanations for the setup (for advanced OAR admins only).

In order to enable the mechanism, you have to:

  1. use the last job resource manager (see above)
  2. enable the device cgroup mechanism in it ($ENABLE_DEVICESCG = “YES”;)
  3. add a resource property for the gpu devices (oarproperty -a gpudevice)
  4. set the values for this new resource property for all resources.

First scenario, simple

For step 4, several scenarios are possible. Let's consider here a first scenario where you have 2 GPUs on your nodes, one attached to the first CPU's PCI-e bus, and the second to the 2nd CPU's PCI-e bus (see lstopo's output for instance to know that). You can then use oarnodesetting to set the gpudevice property to 0 (matching the /dev/nvidia0 Linux device) for the resources associated with the first CPU, and 1 (matching the /dev/nvidia1 Linux device) for those associated to the 2nd CPU, for every hosts.

Users would then only have access to 1 GPU of N hosts if requesting a single CPU per host, e.g.:

$ oarsub -l /host=N/cpu=1 ...

or could even request GPUs directely, e.g.:

$ oarsub -l /host=N/gpudevice=1 

Result would be equivalent (1 GPU = 1 CPU). (Warning giving host=N is mandatory in the second variant of the oarsub command, or one would reserve all hosts of the cluster with a same GPU id.)

Running the nvidia-smi command in the job should show what GPU is available.

Also, if some nodes do not have any GPU, you could set the value of the property for the corresponding resources to gpudevice=-1, and let the users add to the oarsub command a -p “gpudevice >=0” in order to get resources with GPUs.

Second scenario, more complexe

Lets assume now that you have a cluster of 3 nodes with 32 GB of RAM and per node:

  • 2 CPUs of 6 cores each
  • 4 GPUs, 2 attached to the PCIe bus of the first CPU and 2 attached to the PCIe bus of the second CPU (e.g. as reported by lstopo)

Obviously, cores and GPUs appear at the same level in the hierarchy of resources we can define, giving two possible hierarchy definitions:

  • /cluster/host/cpu/core
  • /cluster/host/cpu/gpu

In order to provide a single hierarchy, we have to make a arbitrary choice (not related to a real hardware hierarchy), and choose for instance to associate 3 cores to a GPU. The hierarchy then becomes:

  • /cluster/host/cpu/gpu/core

Lets translate that to technical words:

  • first we have to define the resources hierarchy levels: cluster, host, cpu, core, gpu
  • then we have to define the gpu resource for the system mapping: gpudevice (i.e. the GPU equivalent of cpuset for the cores)
  • finally any additional resource property can be added, like mem for host memory, cpumodel for the CPU model, gpumodel, etc.

See the oarproperty command manual, in order to create cluster, host, cpu, core, gpu, gpudevice, cpumode, gpumodel.

Finally, lets now assume that your OAR server also manages a second cluster with no GPU, e.g. 2 nodes with 24GB of RAM and 2 CPU each of 4 cores.

We can now define the resources in the resource table of OAR as follows (1 lines par core, 3 lines par GPU, 6 lines per CPU, 12 lines per host…):

cluster host cpu core cpuset gpu gpudevice mem cpumodel gpumodel
cluster1 host1 1 1 0 1 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 1 2 1 1 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 1 3 2 1 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 1 4 3 2 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 1 5 4 2 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 1 6 5 2 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 2 7 6 3 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 2 8 7 3 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 2 9 8 3 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 2 10 9 4 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 2 11 10 4 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host1 2 12 11 4 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 3 13 0 5 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 3 14 1 5 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 3 15 2 5 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 3 16 3 6 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 3 17 4 6 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 3 18 5 6 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 4 19 6 7 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 4 20 7 7 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 4 21 8 7 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 4 22 9 8 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 4 23 10 8 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host2 4 24 11 8 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 5 25 0 9 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 5 26 1 9 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 5 27 2 9 0 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 5 28 3 10 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 5 29 4 10 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 5 30 5 10 1 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 6 31 6 11 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 6 32 7 11 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 6 33 8 11 2 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 6 34 9 12 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 6 35 10 12 3 32 Xeon Gold 6128 Geforce 1080ti
cluster1 host3 6 36 11 12 3 32 Xeon Gold 6128 Geforce 1080ti
cluster2 host4 7 37 0 0 0 24 Xeon Gold 5122
cluster2 host4 7 38 1 0 0 24 Xeon Gold 5122
cluster2 host4 7 39 2 0 0 24 Xeon Gold 5122
cluster2 host4 7 40 3 0 0 24 Xeon Gold 5122
cluster2 host4 8 41 4 0 0 24 Xeon Gold 5122
cluster2 host4 8 42 5 0 0 24 Xeon Gold 5122
cluster2 host4 8 43 6 0 0 24 Xeon Gold 5122
cluster2 host4 8 44 7 0 0 24 Xeon Gold 5122
cluster2 host5 9 45 0 0 0 24 Xeon Gold 5122
cluster2 host5 9 46 1 0 0 24 Xeon Gold 5122
cluster2 host5 9 47 2 0 0 24 Xeon Gold 5122
cluster2 host5 9 48 3 0 0 24 Xeon Gold 5122
cluster2 host5 10 49 4 0 0 24 Xeon Gold 5122
cluster2 host5 10 50 5 0 0 24 Xeon Gold 5122
cluster2 host5 10 51 6 0 0 24 Xeon Gold 5122
cluster2 host5 10 52 7 0 0 24 Xeon Gold 5122

(Remember that OAR properties (columns) that define the resource hiearachy must be unique values for each unique entity, but system properties (cpuset, gpudevice) loops on each host)

See the oarnodesetting command manual, in order to create the table rows and set the values. Help yourselfs with scripting the command calls (e.g. use shell for loops).

Finally, you can look at the created rows using the oarnodes command, or even look at the resources table in the database.

Once this is done, users can request one of the 12 GPUs as follows:

$ oarsub -p "gpu > 0" -l gpu=1

Or 2 GPUs (possibly not on the same host) as follows:

$ oarsub -p "gpu > 0" -l gpu=2

Or 2 GPUs on a same host as follows:

$ oarsub -p "gpu > 0" -l host=1/gpu=2

When reserving 1 GPU, the user obviously gets the 3 cores associated to the GPUs.

Warning: make sure to look at lstopo output in order to correctly associate cpuset and gpudevices, e.g. not associating cores and GPUs not attached to a same CPU.

The oar_resource_add command

In OAR 2.5.8 (starting from 2.5.8 RC6), the oar_resource_add command provides some support to create GPU resources in OAR.

wiki/managing_gpu_resources.txt · Last modified: 2018/07/05 18:00 by neyron
Recent changes RSS feed GNU Free Documentation License 1.3 Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki