Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
wiki:managing_resources_cpu_gpu [2019/06/25 14:25] – [Second scenario, more complexe] neyronwiki:managing_resources_cpu_gpu [2020/03/03 14:10] (current) – [The oar_resource_add command] neyron
Line 11: Line 11:
 In database these 4 kinds of resource properties are all stored as **columns** of the ''resources'' table. A rows then gives the set of properties for one resource. In database these 4 kinds of resource properties are all stored as **columns** of the ''resources'' table. A rows then gives the set of properties for one resource.
  
-**Given a hierarchy** (chosen by the administrator for its cluster setup, for instance cluster/switch/host/cpu/core, but thread could be added, or other customizations), one row in the table gives information for the **lowest level of resources of the hierarchy** (e.g. core). Then **groups of lines** define higher levels, like Russian dolls (e.g. ''#C'' rows for CPUs, ''#H*#C'' rows for hosts, ...).+**Given a hierarchy** (chosen by the administrator for his cluster setup, for instance cluster/switch/host/cpu/core, but thread could be added, or other customizations), one row in the table gives information for the **lowest level of resources of the hierarchy** (e.g. core). Then **groups of lines** define higher levels, like Russian dolls (e.g. ''#C'' rows for CPUs, ''#H*#C'' rows for hosts, ...).
      
 One rule must be kept in mind: **any unique object in the resources hierarchy must have a unique id among its set of object**. For example: One rule must be kept in mind: **any unique object in the resources hierarchy must have a unique id among its set of object**. For example:
Line 33: Line 33:
 ====== Managing GPUs ====== ====== Managing GPUs ======
  
-Support of Nvidia GPU devices was added to OAR and will ship with OAR 2.5.8 (already ship with RC versions, starting with 2.5.8 RC6). +Support of Nvidia GPU devices was added to OAR and ships with OAR 2.5.8.
- +
-Meanwhile, for those who cannot wait and since this only involves some configuration of the resources and using the latest version of the job resource manager script taken from git (the job resource manager is part of the configuration files of OAR, which the administrator can modify), one can //backport// the feature to released versions (e.g. 2.5.7). +
- +
-See: +
-  * the latest job resource manager script: https://raw.githubusercontent.com/oar-team/oar/2.5/sources/core/tools/job_resource_manager_cgroups.pl +
-  * some explanations about it: https://github.com/oar-team/oar/issues/107 +
- +
-The next release of OAR will provide tools to help setup GPU resources (see the ''oar_resources_add'' command included in OAR starting from version 2.5.8 RC6), but for now, here are some explanations for the setup (for //advanced OAR admins// only)+
  
 +OAR provides tools to help setup GPU resources (see the ''oar_resources_add'' command), but here are some explanations for the setup. 
  
 In order to enable the mechanism, you have to: In order to enable the mechanism, you have to:
-  - use the last job resource manager (see above)+  - use the latest version of the job resource manager (job_resource_manager_cgroup.pl shipped with the latest version of OAR, at least version 2.5.8)
   - enable the device cgroup mechanism in it (''$ENABLE_DEVICESCG = "YES";'')   - enable the device cgroup mechanism in it (''$ENABLE_DEVICESCG = "YES";'')
   - add a resource property for the gpu devices (''oarproperty -a gpudevice'')   - add a resource property for the gpu devices (''oarproperty -a gpudevice'')
Line 69: Line 62:
 Also, if some nodes do not have any GPU, you could set the value of the property for the corresponding resources to ''gpudevice=-1'', and let the users add to the ''oarsub'' command a ''-p "gpudevice >=0"'' in order to get resources with GPUs. Also, if some nodes do not have any GPU, you could set the value of the property for the corresponding resources to ''gpudevice=-1'', and let the users add to the ''oarsub'' command a ''-p "gpudevice >=0"'' in order to get resources with GPUs.
  
-===== Second scenario, more complex =====+ 
 +But be **warned**, that the following commands will mostly not provide what a user would expect: 
 +<code bash> 
 +$ oarsub -l gpudevice=1  
 +</code> 
 +will gives all resources matching one identifier of gpudevices, which means all nodes limited to their first gpus (''gpudevice=0''), or second gpus (''gpudevice=1''), or all nodes which have no gpus (''gpudevice=-1''). 
 + 
 +<code bash> 
 +$ oarsub -l gpudevice=N 
 +</code> 
 +with N > 1 makes even less sense. See the setup proposed in the section below if you want to let your users request N gpus like that (using ''oarsub -l gpu=N''). 
 + 
 +**We strongly suggest to setup the second scenario below, which defines the ''gpu'' hierarchy property, along with the ''gpudevice'' hardware resource identifier.** 
 + 
 + 
 + 
 +===== Second scenario, full featured =====
 Lets assume now that you have a cluster of 3 nodes with 32 GB of RAM and per node: Lets assume now that you have a cluster of 3 nodes with 32 GB of RAM and per node:
   * 2 CPUs of 6 cores each   * 2 CPUs of 6 cores each
Line 81: Line 90:
  
 Lets translate that to technical words: Lets translate that to technical words:
-  * first we have to define the resources hierarchy levels: cluster, host, cpu, core, gpu +  * first we have to define the resources hierarchy levels: cluster, host, cpu, gpu, core 
-  * then we have to define the gpu resource for the system mapping: gpudevice (i.e. the GPU equivalent of cpuset for the cores)+  * then we have to define the GPU resource for the system mapping: gpudevice (i.e. the GPU'equivalent of cpuset used to identify the hardware core ids)
   * finally any additional resource property can be added, like mem for host memory, cpumodel for the CPU model, gpumodel, etc.   * finally any additional resource property can be added, like mem for host memory, cpumodel for the CPU model, gpumodel, etc.
  
Line 167: Line 176:
  
 When reserving 1 GPU, the user obviously gets the 3 cores associated to the GPUs. When reserving 1 GPU, the user obviously gets the 3 cores associated to the GPUs.
 +
 +Finally, GPU jobs can be tied to GPU resources (where ''gpu > 0'') with the following admission rule (see ''oaradmissionrules''), so that users don't have to set ''-p "gpu > 0"'' in their command lines:
 +<code perl>
 +foreach my $mold (@{$ref_resource_list}){
 +  foreach my $r (@{$mold->[0]}){
 +    my $gpu_request = 0;
 +    foreach my $resource (@{$r->{resources}}) {
 +      if ($resource->{resource} eq "gpu") {
 +        $gpu_request = 1;
 +      }
 +    }
 +    if ($gpu_request) {
 +      if ($r->{property} ne ""){
 +        $r->{property} = "($r->{property}) AND gpu > 0";
 +      }else{
 +        $r->{property} = "gpu > 0";
 +      }
 +      print("[ADMISSION RULE] Tie job resource request for GPU to resources with GPU\n");
 +    }
 +  }
 +}
 +</code>
  
 Warning: make sure to look at lstopo output in order to correctly associate cpuset and gpudevices, e.g. not associating cores and GPUs not attached to a same CPU. Warning: make sure to look at lstopo output in order to correctly associate cpuset and gpudevices, e.g. not associating cores and GPUs not attached to a same CPU.
  
-====== The oar_resource_add command ======+Warning: mind the fact that with the defined hierarchy ''host/cpu/gpu/core'', it makes no sense to use such a oarsub command as follows: 
 +<code bash> 
 +$ oarsub -l host=1/core=8/gpu=
 +</code> 
 +In that case, you select 1 host, with 8 of its cores, and 2 GPUs of each cores. But since for each core, there is at most 1 gpu value, that makes no sense.
  
-In OAR 2.5.(starting from 2.5.8 RC6), the oar_resource_add command provides some support to create GPU resources in OAR as well as CPU-Core resources with relevant topologies.+Also, that: 
 +<code bash> 
 +$ oarsub -l host=1/core=8/gpu=1 
 +</code> 
 +is equivalent to: 
 +<code bash> 
 +$ oarsub -l host=1/core=8 
 +</code> 
 +The user will get 1 host with 8 cores of itNothing is said about what or how many GPUs will be available in the job. 
 +====== The oar_resource_add command ====== 
 +The oar_resource_add command provides some support to create GPU resources in OAR as well as CPU-Core resources with relevant topologies.
  
wiki/managing_resources_cpu_gpu.txt · Last modified: 2020/03/03 14:10 by neyron
Recent changes RSS feed GNU Free Documentation License 1.3 Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki