intel mpi tuning on dodrio

[stdweird]

on dodrio intel mpi 2020 and later, we are advised to set following

export FI_PROVIDER=mlx
export I_MPI_FABRICS=shm:ofi
export I_MPI_SHM=clx_avx2

and sometimes

export I_MPI_OFI_EXPERIMENTAL=1

not sure what they mean, and how they are different from defaults

[lexming]

I took a look at these settings:

• FI_PROVIDER=mlx sets the transport driver to Mellanox. This is usually enabled at runtime automatically, so the setting is not really needed but it doesn't hurt either
• I_MPI_FABRICS=shm:ofi is default
• I_MPI_SHM=clx_avx2 fixes the shared memory transport to clx_avx2. Depending on your CPU there might be better options, such as clx_avx512, and the list of options is long: https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-reference-linux/top/environment-variable-reference/environment-variables-for-fabrics-control/shared-memory-control.html. I do not see any reason to fix this setting over auto, which is the default and should automatically pick the best one.