While some batch schedulers (e.g. Slurm) do some tricks to transparently pass some environment variables from the head node shell to all execution nodes given to mpirun, OAR does not (OAR provides no more than what OpenSSH does, be it used directly as ssh or through the oarsh wrapper). Therefore, in order to have more than the default environment variables (OMPI_* variables) passed/set on execution nodes, one can use the –mca mca_base_env_list “ENV1;ENV2;ENV3” option of mpirun (or -x ENV1 -x ENV2 -x ENV3 depending on the Openmpi version) .

Example: assuming the MY_ENV1 and MY_ENV2 environment variables are defined in the shell calling mpirun, the value of MY_ENV1 and MY_ENV2 will be available on all execution nodes, along with MY_ENV3 set to value3 when calling mpirun as follows:

mpirun -machinefile $OAR_NODE_FILE --mca orte_rsh_agent "oarsh" --mca mca_base_env_list "MY_ENV1;MY_ENV2;MY_ENV3=value3" ~/bin/test.sh

Settings:

orte_rsh_agent="oarsh"
mca_base_env_list="MY_ENV1;MY_ENV2;MY_ENV3=value3"

can also be set in the ~/.openmpi/mca-params.conf file, so that the command becomes:

mpirun -machinefile $OAR_NODE_FILE ~/bin/test.sh

This could especially be useful to pass OpenMP variables, such as OMP_NUM_THREADS.

More info [https://www.open-mpi.org/doc/v4.0/man1/mpirun.1.php#toc22 in OpenMPI manual pages].