MPI: Difference between revisions

From Research Computing Center Wiki
Jump to navigation Jump to search
 
(10 intermediate revisions by 2 users not shown)
Line 9: Line 9:
   
   
The following MPI libraries are available:
The following MPI libraries are available:
<!--
<!--
====MVAPICH2====
====MVAPICH2====
Line 31: Line 30:


     Versions:
     Versions:
        MVAPICH2/2.3.2-GCC-7.3.0-2.30-EDR
        MVAPICH2/2.3.2-GCC-8.3.0-EDR
        MVAPICH2/2.3.2-iccifort-2013_sp1.0.080-EDR
        MVAPICH2/2.3.2-iccifort-2015.2.164-GCC-4.8.5-EDR
        MVAPICH2/2.3.2-iccifort-2018.1.163-GCC-6.4.0-2.28-EDR
        MVAPICH2/2.3.2-iccifort-2019.5.281-EDR
        MVAPICH2/2.3.4-GCC-7.3.0-2.30
        MVAPICH2/2.3.4-GCC-8.3.0
        MVAPICH2/2.3.4-iccifort-2019.5.281
         MVAPICH2/2.3.6-GCC-9.3.0
         MVAPICH2/2.3.6-GCC-9.3.0
        MVAPICH2/2.3.7-1-GCC-9.3.0
     Other possible modules matches:
     Other possible modules matches:
         gmvapich2
         gmvapich2
Line 54: Line 45:
   For example:
   For example:


     $ module spider MVAPICH2/2.3.6-GCC-9.3.0
     $ module spider MVAPICH2/2.3.7-1-GCC-9.3.0
-----------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------
</pre>
</pre>
Line 60: Line 51:
Once the appropriate module is loaded, you can compile code with '''mpicc, mpicxx,''' '''mpic++,''' '''mpifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
Once the appropriate module is loaded, you can compile code with '''mpicc, mpicxx,''' '''mpic++,''' '''mpifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
-->
-->
====OpenMPI====
====OpenMPI====


Line 68: Line 58:
</pre>
</pre>


Module names that have an -QDR or -EDR extension only work for the corresponding IB network. Module names without such an extension can be used on either IB network. The module names have the format OpenMPI/''Version''-''CompilerToolchain''-''ToolchainVersion''.
The module names have the format OpenMPI/''Version''-''CompilerToolchain''-''ToolchainVersion''.


For example, these are some of the modules available:
For example, these are some of the modules available:
Line 82: Line 72:


     Versions:
     Versions:
        OpenMPI/1.6.5
        OpenMPI/1.8.4-iccifort-2013_sp1.0.080
        OpenMPI/1.8.4-iccifort-2015.2.164-GCC-4.8.5
        OpenMPI/2.1.2-iccifort-2018.1.163-GCC-6.4.0-2.28-test
        OpenMPI/2.1.2-iccifort-2018.1.163-GCC-6.4.0-2.28
        OpenMPI/3.1.1-gcccuda-2018b
        OpenMPI/3.1.1-iccifort-2018.1.163-GCC-6.4.0-2.28
        OpenMPI/3.1.4-GCC-8.3.0-nosrun
        OpenMPI/3.1.4-GCC-8.3.0-test
         OpenMPI/3.1.4-GCC-8.3.0
         OpenMPI/3.1.4-GCC-8.3.0
         OpenMPI/3.1.4-gcccuda-2019b
         OpenMPI/3.1.4-iccifort-2019.5.281
        OpenMPI/4.0.3-GCC-9.3.0
         OpenMPI/4.0.5-GCC-10.2.0
         OpenMPI/4.0.5-GCC-10.2.0
        OpenMPI/4.0.5-gcccuda-2020b
        OpenMPI/4.1.1-GCC-10.3.0-EDR
        OpenMPI/4.1.1-GCC-10.3.0-QDR
         OpenMPI/4.1.1-GCC-10.3.0
         OpenMPI/4.1.1-GCC-10.3.0
        OpenMPI/4.1.1-GCC-11.2.0
        OpenMPI/4.1.4-GCC-11.3.0
        OpenMPI/4.1.4-GCC-12.2.0
        OpenMPI/4.1.4-intel-compilers-2022.1.0
        OpenMPI/4.1.5-GCC-12.3.0


-----------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------
   For detailed information about a specific "OpenMPI" package (including how to load the modules) use the module's full name.
   For detailed information about a specific "OpenMPI" package (including how to load the modules) use the module's full name.
   Note that names that have a trailing (E) are extensions provided by other modules.
   Note that names that have a trailing (E) are extensions provided by other modules.
   For example:
   For example:


     $ module spider OpenMPI/4.1.1-GCC-10.3.0
     $ module spider OpenMPI/4.1.5-GCC-12.3.0
-----------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------
</pre>
</pre>
Line 114: Line 96:
You can find all Intel MPI modules available on Sapelo2 by running the following command on a Sapelo2 node:<pre class="gcommand">
You can find all Intel MPI modules available on Sapelo2 by running the following command on a Sapelo2 node:<pre class="gcommand">
module spider impi
module spider impi
</pre>The module names have the format impi/''Version''-''CompilerToolchain''-''ToolchainVersion''.
</pre>The module names have the format impi/''Version''-''CompilerToolchain''-''ToolchainVersion.''


For example, these are some of the modules available:<pre class="gcomment">
For example, these are some of the modules available:<pre class="gcomment">
Line 126: Line 108:


     Versions:
     Versions:
         impi/2018.5.288-iccifort-2019.5.281
         impi/2021.4.0-intel-compilers-2021.4.0
         impi/2019.9.304-iccifort-2020.4.304
        impi/2021.6.0-intel-compilers-2022.1.0
         impi/2021.9.0-intel-compilers-2023.1.0
     Other possible modules matches:
     Other possible modules matches:
         iimpi
         iimpi
Line 141: Line 124:
   For example:
   For example:


     $ module spider impi/2019.9.304-iccifort-2020.4.304
     $ module spider impi/2021.9.0-intel-compilers-2023.1.0
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
</pre>Once the appropriate module is loaded, you can compile code with '''mpicc, mpiicc, mpicxx, mpiicpc,''' '''mpiifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
</pre>Once the appropriate module is loaded, you can compile code with '''mpicc, mpiicc, mpicxx, mpiicpc,''' '''mpiifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
===MPI commands and how to launch MPI programs===
{|  width="100%" border="1"  cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | MPI library
! scope="col" | Version
! scope="col" | Base toolchain
! scope="col" | Toolchain
! scope="col" | Fortran
! scope="col" | C
! scope="col" | C++
! scope="col" | How to launch with Slurm
|-
| OpenMPI || 3.1.4-GCC-8.3.0 || GCCcore-8.3.0 || foss/2019b || mpif90 || mpicc || mpicxx || srun --mpi=pmi2
|-
| OpenMPI || 3.1.4-iccifort-2019.5.281 || GCCcore-8.3.0 || iomkl/2019b || mpif90 || mpicc || mpicxx || srun --mpi=pmi2
|-
| OpenMPI || 4.0.5-GCC-10.2.0 || GCCcore-10.2.0 || foss/2020b || mpif90 || mpicc || mpicxx || srun --mpi=pmi2
|-
| OpenMPI || 4.1.1-GCC-11.2.0 || GCCcore-11.2.0 || foss/2021b || mpif90 || mpicc || mpicxx || srun
|-
| OpenMPI || 4.1.4-GCC-11.3.0 || GCCcore-11.3.0 || foss/2022a || mpif90 || mpicc || mpicxx || srun
|-
| OpenMPI || 4.1.4-intel-compilers-2022.1.0 || GCCcore-11.3.0 || iomkl/2022a || mpif90 || mpicc || mpicxx || srun
|-
| OpenMPI || 4.1.4-GCC-12.2.0 || GCCcore-12.2.0 || foss/2022b || mpif90 || mpicc || mpicxx || srun
|-
| OpenMPI || 4.1.5-GCC-12.3.0 || GCCcore-12.3.0 || foss/2023a || mpif90 || mpicc || mpicxx || srun
|-
| Intel MPI || 2021.4.0-intel-compilers-2021.4.0 || GCCcore-11.2.0 || intel/2021b || mpiifort || mpiicc || mpiicpc || srun
|-
| Intel MPI || 2021.6.0-intel-compilers-2022.1.0 || GCCcore-11.3.0 || intel/2022a || mpiifort || mpiicc || mpiicpc || srun
|-
| Intel MPI || 2021.9.0-intel-compilers-2023.1.0 || GCCcore-12.3.0 || intel/2023a || mpiifort || mpiicc || mpiicpc || srun
|-
|}
====Note====
If your MPI job receives any of the following or similar errors:
*PMIX ERROR: OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750
*PMIX ERROR: UNPACK-PAST-END in file base/bfrop_base_unpack.c at line 750
*PMIX ERROR: UNPACK-INADEQUATE-SPACE in file base/gds_base_fns.c at line 138
*UNPACK-PMIX-VALUE: UNSUPPORTED TYPE 126
then please use <code>srun --mpi=pmi2</code> to start the MPI application.


----
----
Line 150: Line 180:
==MPI Libraries for parallel jobs on the teaching cluster==
==MPI Libraries for parallel jobs on the teaching cluster==


All compute nodes on the teaching cluster are connected via a '''gigabit Ethernet''' network. MPI libraries currently installed on the teaching cluster are listed below.
All compute nodes on Sapelo2 have Infiniband (IB) interconnect via '''EDR''' Infiniband network (100Gbps). Various IB-enabled MPI libraries are available and users can set the environment variables for the MPI library of choice by loading the corresponding module file.
 
For more information on Environment Modules, please see the [[Lmod]] page.


The following MPI libraries are available:
<!--
====MVAPICH2====
====MVAPICH2====


Line 197: Line 231:


Once the appropriate module is loaded, you can compile code with '''mpicc, mpicxx,''' '''mpic++,''' '''mpifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
Once the appropriate module is loaded, you can compile code with '''mpicc, mpicxx,''' '''mpic++,''' '''mpifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
-->
====OpenMPI====
====OpenMPI====


Line 218: Line 253:


     Versions:
     Versions:
        OpenMPI/1.8.4-iccifort-2013_sp1.0.080
        OpenMPI/1.8.4-iccifort-2015.2.164-GCC-4.8.5
        OpenMPI/2.1.2-iccifort-2018.1.163-GCC-6.4.0-2.28
        OpenMPI/3.1.1-gcccuda-2018b
         OpenMPI/3.1.4-GCC-8.3.0
         OpenMPI/3.1.4-GCC-8.3.0
         OpenMPI/3.1.4-gcccuda-2019b
         OpenMPI/4.1.1-GCC-11.2.0
        OpenMPI/4.1.2-GCC-11.2.0
        OpenMPI/4.1.4-GCC-11.3.0
        OpenMPI/4.1.4-GCC-12.2.0


------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------
   For detailed information about a specific "OpenMPI" package (including how to load the modules) use the module's full name.
   For detailed information about a specific "OpenMPI" package (including how to load the modules) use the module's full name.
   Note that names that have a trailing (E) are extensions provided by other modules.
   Note that names that have a trailing (E) are extensions provided by other modules.
   For example:
   For example:


     $ module spider OpenMPI/3.1.4-gcccuda-2019b
     $ module spider OpenMPI/4.1.4-GCC-12.2.0
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
</pre>
</pre>
Line 250: Line 284:
       Intel MPI Library, compatible with MPICH ABI
       Intel MPI Library, compatible with MPICH ABI


 
    Versions:
        impi/2021.4.0-intel-compilers-2021.4.0
        impi/2021.6.0-intel-compilers-2022.1.0
        impi/2021.9.0-intel-compilers-2023.1.0
     Other possible modules matches:
     Other possible modules matches:
         iimpi
         iimpi


    This module can be loaded directly: module load impi/2018.5.288-iccifort-2019.5.281
    Help:
      Description
      ===========
      Intel MPI Library, compatible with MPICH ABI
     
     
      More information
      ================
      - Homepage: https://software.intel.com/en-us/intel-mpi-library/
     
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
   To find other possible module matches execute:
   To find other possible module matches execute:


       $ module -r spider '.*impi.*'
       $ module -r spider '.*impi.*'
------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "impi" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:
    $ module spider impi/2021.9.0-intel-compilers-2023.1.0
------------------------------------------------------------------------------------------------------------------------------------
</pre>Once the appropriate module is loaded, you can compile code with '''mpicc, mpiicc, mpicxx, mpiicpc,''' '''mpiifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.
</pre>Once the appropriate module is loaded, you can compile code with '''mpicc, mpiicc, mpicxx, mpiicpc,''' '''mpiifort,''' '''mpif90,''' '''mpif77,''' etc. and you can run applications that were linked to the '''MPI libraries''' loaded by the module.


[[#top|Back to Top]]
[[#top|Back to Top]]

Latest revision as of 14:37, 22 May 2024


MPI Libraries for parallel jobs on Sapelo2

All compute nodes on Sapelo2 have Infiniband (IB) interconnect via EDR Infiniband network (100Gbps). Various IB-enabled MPI libraries are available and users can set the environment variables for the MPI library of choice by loading the corresponding module file.

For more information on Environment Modules, please see the Lmod page.

The following MPI libraries are available:

OpenMPI

You can find all OpenMPI modules available on Sapelo2 by running the following command on a Sapelo2 node:

module spider OpenMPI

The module names have the format OpenMPI/Version-CompilerToolchain-ToolchainVersion.

For example, these are some of the modules available:

[shtsai@ss-sub1 ~]$ module spider OpenMPI

-----------------------------------------------------------------------------------------------------------------------------------------------
  OpenMPI:
-----------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      The Open MPI Project is an open source MPI-3 implementation.

     Versions:
        OpenMPI/3.1.4-GCC-8.3.0
        OpenMPI/3.1.4-iccifort-2019.5.281
        OpenMPI/4.0.5-GCC-10.2.0
        OpenMPI/4.1.1-GCC-10.3.0
        OpenMPI/4.1.1-GCC-11.2.0
        OpenMPI/4.1.4-GCC-11.3.0
        OpenMPI/4.1.4-GCC-12.2.0
        OpenMPI/4.1.4-intel-compilers-2022.1.0
        OpenMPI/4.1.5-GCC-12.3.0

------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "OpenMPI" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider OpenMPI/4.1.5-GCC-12.3.0
-----------------------------------------------------------------------------------------------------------------------------------------------

Once the appropriate module is loaded, you can compile code with mpicc, mpiCC, mpicxx, mpic++, mpifort, mpif90, mpif77, etc. and you can run applications that were linked to the MPI libraries loaded by the module.

Intel MPI

You can find all Intel MPI modules available on Sapelo2 by running the following command on a Sapelo2 node:

module spider impi

The module names have the format impi/Version-CompilerToolchain-ToolchainVersion. For example, these are some of the modules available:

[shtsai@ss-sub1 ~]$ module spider impi

------------------------------------------------------------------------------------------------------------------------------------
  impi:
------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Intel MPI Library, compatible with MPICH ABI

     Versions:
        impi/2021.4.0-intel-compilers-2021.4.0
        impi/2021.6.0-intel-compilers-2022.1.0
        impi/2021.9.0-intel-compilers-2023.1.0
     Other possible modules matches:
        iimpi

------------------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*impi.*'

------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "impi" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider impi/2021.9.0-intel-compilers-2023.1.0
------------------------------------------------------------------------------------------------------------------------------------

Once the appropriate module is loaded, you can compile code with mpicc, mpiicc, mpicxx, mpiicpc, mpiifort, mpif90, mpif77, etc. and you can run applications that were linked to the MPI libraries loaded by the module.

MPI commands and how to launch MPI programs

MPI library Version Base toolchain Toolchain Fortran C C++ How to launch with Slurm
OpenMPI 3.1.4-GCC-8.3.0 GCCcore-8.3.0 foss/2019b mpif90 mpicc mpicxx srun --mpi=pmi2
OpenMPI 3.1.4-iccifort-2019.5.281 GCCcore-8.3.0 iomkl/2019b mpif90 mpicc mpicxx srun --mpi=pmi2
OpenMPI 4.0.5-GCC-10.2.0 GCCcore-10.2.0 foss/2020b mpif90 mpicc mpicxx srun --mpi=pmi2
OpenMPI 4.1.1-GCC-11.2.0 GCCcore-11.2.0 foss/2021b mpif90 mpicc mpicxx srun
OpenMPI 4.1.4-GCC-11.3.0 GCCcore-11.3.0 foss/2022a mpif90 mpicc mpicxx srun
OpenMPI 4.1.4-intel-compilers-2022.1.0 GCCcore-11.3.0 iomkl/2022a mpif90 mpicc mpicxx srun
OpenMPI 4.1.4-GCC-12.2.0 GCCcore-12.2.0 foss/2022b mpif90 mpicc mpicxx srun
OpenMPI 4.1.5-GCC-12.3.0 GCCcore-12.3.0 foss/2023a mpif90 mpicc mpicxx srun
Intel MPI 2021.4.0-intel-compilers-2021.4.0 GCCcore-11.2.0 intel/2021b mpiifort mpiicc mpiicpc srun
Intel MPI 2021.6.0-intel-compilers-2022.1.0 GCCcore-11.3.0 intel/2022a mpiifort mpiicc mpiicpc srun
Intel MPI 2021.9.0-intel-compilers-2023.1.0 GCCcore-12.3.0 intel/2023a mpiifort mpiicc mpiicpc srun

Note

If your MPI job receives any of the following or similar errors:

  • PMIX ERROR: OUT-OF-RESOURCE in file base/bfrop_base_unpack.c at line 750
  • PMIX ERROR: UNPACK-PAST-END in file base/bfrop_base_unpack.c at line 750
  • PMIX ERROR: UNPACK-INADEQUATE-SPACE in file base/gds_base_fns.c at line 138
  • UNPACK-PMIX-VALUE: UNSUPPORTED TYPE 126

then please use srun --mpi=pmi2 to start the MPI application.


Back to Top

MPI Libraries for parallel jobs on the teaching cluster

All compute nodes on Sapelo2 have Infiniband (IB) interconnect via EDR Infiniband network (100Gbps). Various IB-enabled MPI libraries are available and users can set the environment variables for the MPI library of choice by loading the corresponding module file.

For more information on Environment Modules, please see the Lmod page.

The following MPI libraries are available:

OpenMPI

You can find all OpenMPI modules available on the teaching cluster by running the following command:

module spider OpenMPI

The module names have the format OpenMPI/Version-CompilerToolchain-ToolchainVersion.

For example, these are some of the modules available:

zhuofei@teach-sub1 ~$ ml spider OpenMPI

------------------------------------------------------------------------------------------------------------------------------------
  OpenMPI:
------------------------------------------------------------------------------------------------------------------------------------
    Description:
      The Open MPI Project is an open source MPI-3 implementation.

     Versions:
        OpenMPI/3.1.4-GCC-8.3.0
        OpenMPI/4.1.1-GCC-11.2.0
        OpenMPI/4.1.2-GCC-11.2.0
        OpenMPI/4.1.4-GCC-11.3.0
        OpenMPI/4.1.4-GCC-12.2.0

--------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "OpenMPI" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider OpenMPI/4.1.4-GCC-12.2.0
------------------------------------------------------------------------------------------------------------------------------------

Once the appropriate module is loaded, you can compile code with mpicc, mpiCC, mpicxx, mpic++, mpifort, mpif90, mpif77, etc. and you can run applications that were linked to the MPI libraries loaded by the module.

Intel MPI

You can find all Intel MPI modules available on the teaching cluster by running the following command:

module spider impi

The module names have the format impi/Version-CompilerToolchain-ToolchainVersion. For example, these are some of the modules available:

zhuofei@teach-sub1 ~$ ml spider impi

------------------------------------------------------------------------------------------------------------------------------------
  impi: impi/2018.5.288-iccifort-2019.5.281
------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Intel MPI Library, compatible with MPICH ABI

     Versions:
        impi/2021.4.0-intel-compilers-2021.4.0
        impi/2021.6.0-intel-compilers-2022.1.0
        impi/2021.9.0-intel-compilers-2023.1.0
     Other possible modules matches:
        iimpi

------------------------------------------------------------------------------------------------------------------------------------
  To find other possible module matches execute:

      $ module -r spider '.*impi.*'

------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "impi" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider impi/2021.9.0-intel-compilers-2023.1.0
------------------------------------------------------------------------------------------------------------------------------------

Once the appropriate module is loaded, you can compile code with mpicc, mpiicc, mpicxx, mpiicpc, mpiifort, mpif90, mpif77, etc. and you can run applications that were linked to the MPI libraries loaded by the module.

Back to Top