Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
Next revisionBoth sides next revision
wiki:managing_resources_cpu_gpu [2018/10/16 09:10] – [Managing GPUs] neyronwiki:managing_resources_cpu_gpu [2019/11/19 17:56] – [Second scenario, more complex] neyron
Line 2: Line 2:
  
 ====== Managing processing unit topologies ====== ====== Managing processing unit topologies ======
-OAR'resources tables provides several kinds of information: +The OAR database ''resources'' tables provides several kinds of information: 
-  1. resources states (Alive/Absent/Dead/..., maintenance mode, availability...) ; +  resources **states** (Alive/Absent/Dead/..., maintenance mode, availability...) ; 
-  2. resources characteristics (CPU speed, CPU model, memory size...) ; +  resources **characteristics** (CPU speed, CPU model, memory size...) ; 
-  3. and also it defines the computing resources hierarchy (topology) ; +  and also it defines the computing resources **hierarchy** (topology) ; 
-  4. and finally some hardware related identifiers (e.g. cpusets). +  and finally some **hardware** related **identifiers** (e.g. cpusets). 
-They are called OAR resources ''properties''+They are called OAR resources **properties**
  
-In database these kinds of properties are all stored as columns of the table. A rows then gives the set of properties for one resource.+In database these kinds of resource properties are all stored as **columns** of the ''resources'' table. A rows then gives the set of properties for one resource.
  
-Given a hierarchy (chosen by the administrator for its cluster setup, for instance cluster/switch/host/cpu/core, but thread could be added, or other customizations), one row in the table gives information for the lowest level of resources of the hierarchy (e.g. core). Then groups of lines define higher levels, like russian dolls (e.g. ''#C'' rows for CPUs, ''#H*#C'' rows for hosts, ...).+**Given a hierarchy** (chosen by the administrator for its cluster setup, for instance cluster/switch/host/cpu/core, but thread could be added, or other customizations), one row in the table gives information for the **lowest level of resources of the hierarchy** (e.g. core). Then **groups of lines** define higher levels, like Russian dolls (e.g. ''#C'' rows for CPUs, ''#H*#C'' rows for hosts, ...).
      
-One rule must be kept in mind: any unique object in the resources hierarchy must have a unique id among its set of object. For example:+One rule must be kept in mind: **any unique object in the resources hierarchy must have a unique id among its set of object**. For example:
   * any of the cores (of any of the CPUs, of any of the hosts...) must have a unique id among the whole set of cores ;   * any of the cores (of any of the CPUs, of any of the hosts...) must have a unique id among the whole set of cores ;
   * any of the CPUs (of any of the nodes, of any of the clusters...)  must have a unique id among the whole set of CPUs ;   * any of the CPUs (of any of the nodes, of any of the clusters...)  must have a unique id among the whole set of CPUs ;
   * and so one of any resource.   * and so one of any resource.
  
-Then, when it comes to the hardware identifiers (cpusets, or see below for GPU devices id), the administrator must take a special attention so that a correct mapping is done between the logical hierarchy (e.g. id of the host, CPUs, cores, hyperthreads) and the hardware processing unit ids (cpuset value). Using a tool such as ''hwloc'' may help at this point.+Then, when it comes to the hardware identifiers (cpusets, or see below for GPU devices id), the administrator must take a special attention so that a **correct mapping** is done between the **logical hierarchy** (e.g. id of the host, CPUs, cores, hyperthreads) and the **hardware** processing unit **ids** (cpuset value). Using a tool such as ''hwloc'' may help at this point.
  
-The ''oarnodes'' command gives an extract of the resources table. The ''oarnodesetting'' command can be used to modify or create rows in the resources table. The ''oarproperty'' command can be used to modify resource properties (columns of the table).+The basic commands to work with resources are: 
 +  * The ''oarnodes'' command gives an extract of the resources table.  
 +  * The ''oarnodesetting'' command can be used to modify or create rows in the resources table.  
 +  * The ''oarproperty'' command can be used to modify resource properties (columns of the table).
  
-Two meta-command are provided to build the resource table (using underneath the ''oarpropery'' and ''oarnodesetting'' commands):+But two meta-command are provided to build the resource table (using underneath the ''oarpropery'' and ''oarnodesetting'' commands):
   * ''oar_resource_init'' first looks at the machines hardware topology (connecting to them using SSH) and then provides the relevant ''oarnodesetting'' commands ;   * ''oar_resource_init'' first looks at the machines hardware topology (connecting to them using SSH) and then provides the relevant ''oarnodesetting'' commands ;
   * ''oar_resource_add'' builds ''oarnodesetting'' commands from a model defined in the command line.   * ''oar_resource_add'' builds ''oarnodesetting'' commands from a model defined in the command line.
Line 66: Line 69:
 Also, if some nodes do not have any GPU, you could set the value of the property for the corresponding resources to ''gpudevice=-1'', and let the users add to the ''oarsub'' command a ''-p "gpudevice >=0"'' in order to get resources with GPUs. Also, if some nodes do not have any GPU, you could set the value of the property for the corresponding resources to ''gpudevice=-1'', and let the users add to the ''oarsub'' command a ''-p "gpudevice >=0"'' in order to get resources with GPUs.
  
-===== Second scenario, more complexe =====+===== Second scenario, more complex =====
 Lets assume now that you have a cluster of 3 nodes with 32 GB of RAM and per node: Lets assume now that you have a cluster of 3 nodes with 32 GB of RAM and per node:
   * 2 CPUs of 6 cores each   * 2 CPUs of 6 cores each
Line 164: Line 167:
  
 When reserving 1 GPU, the user obviously gets the 3 cores associated to the GPUs. When reserving 1 GPU, the user obviously gets the 3 cores associated to the GPUs.
 +
 +GPU job can be tied to GPU resources (where ''gpu > 0'') with the following admission rule (see ''oaradmissionrules''), so that users don't have to set ''-p "gpu > 0"'' in their command lines:
 +<code perl>
 +foreach my $mold (@{$ref_resource_list}){
 +  foreach my $r (@{$mold->[0]}){
 +    my $gpu_request = 0;
 +    foreach my $resource (@{$r->{resources}}) {
 +      if ($resource->{resource} eq "gpu") {
 +        $gpu_request = 1;
 +      }
 +    }
 +    if ($gpu_request) {
 +      if ($r->{property} ne ""){
 +        $r->{property} = "($r->{property}) AND gpu > 0";
 +      }else{
 +        $r->{property} = "gpu > 0";
 +      }
 +      print("[ADMISSION RULE] Tie job resource request for GPU to resources with GPU\n");
 +    }
 +  }
 +}
 +</code>
  
 Warning: make sure to look at lstopo output in order to correctly associate cpuset and gpudevices, e.g. not associating cores and GPUs not attached to a same CPU. Warning: make sure to look at lstopo output in order to correctly associate cpuset and gpudevices, e.g. not associating cores and GPUs not attached to a same CPU.
wiki/managing_resources_cpu_gpu.txt · Last modified: 2020/03/03 14:10 by neyron
Recent changes RSS feed GNU Free Documentation License 1.3 Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki