Code Compilation on Sapelo2

Where should I compile my code?

IMPORTANT NOTE: Please do NOT compile code on the login node. Instead, please first start an interactive session with interact command and compile the code on the interactive node.

Code compilation can be done in an interactive session. To start an interactive session, first login into Sapelo2 and from there issue the interact command

interact

If you plan to run the code on an AMD node, you can start an interactive session on an AMD node to compile the code. To start an interactive on an AMD node, use the command

interact --constraint AMD

If you plan to run the code on an Intel node, you can start an interactive session on an Intel node to compile the code. To start an interactive on an Intel node, use the command

interact --constraint Intel

For detailed information on how to access the compute node interactively for code compilation, please see Running Jobs on Sapelo2.

---

Compilers

A number of Fortran and C/C++ compilers, as well as Java and scripting languages such as Perl and Python, are available on Sapelo2.

Summary of main Fortran and C/C++ compilers installed

	Portland Group (PGI)	Intel	GNU	OpenMPI	File extension
Fortran77	pgfortran	ifort	gfortran	mpif77	.f
Fortran90	pgfortran	ifort	gfortran	mpif90	.f90
Fortran95	pgfortran	ifort	gfortran	mpifort	.f95
Fortran2003	pgfortran	ifort	gfortran	mpifort	.f
C	pgcc	icc	gcc	mpicc	.c
C++	pgc++	icpc	g++	mpicxx	.C, .cpp, .cc

The various compiler suites are provided by their environment modules.

GNU compiler suites

The following command will show all the modules that provide GCC compiler suites:

module spider GCC

Sample partial output of this command:

[shtsai@b1-1 ~]$ module spider GCC
-----------------------------------------------------------------------------------------------------------------
  GCC:
-----------------------------------------------------------------------------------------------------------------
    Description:
      The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as
      libraries for these languages (libstdc++, libgcj,...).

     Versions:
        GCC/6.4.0-2.28
        GCC/7.3.0-2.30
        GCC/8.3.0
        GCC/9.2.0
        GCC/9.3.0
        GCC/10.2.0
        GCC/10.3.0
     Other possible modules matches:
        GCCcore  gcccuda
-----------------------------------------------------------------------------------------------------------------

This output indicates that the following versions of GCC compilers are available:

• Version 6.4.0, with binutils 2.28, provided by the GCC/6.4.0-2.28 module, includes C, C++, and Fortran compilers.
• Version 7.3.0, with binutils 2.30, provided by the GCC/7.2.0-2.30 module, includes C, C++, and Fortran compilers.
• Version 8.3.0, with binutils 2.32, provided by the GCC/8.3.0 module, includes C, C++, and Fortran compilers.
• Version 9.2.0, with binutils 2.32, provided by the GCC/9.2.0 module, includes C, C++, and Fortran compilers.
• Version 9.3.0, with binutils 2.34, provided by the GCC/9.3.0 module, includes C, C++, and Fortran compilers.
• Version 10.2.0, with binutils 2.35, provided by the GCC/10.2.0 module, includes C, C++, and Fortran compilers.
• Version 10.3.0, with binutils 2.36.1, provided by the GCC/10.3.0 module, includes C, C++, and Fortran compilers.

We suggest that you run the module spider GCC command to check an updated list of GCC compilers available on the cluster.

PGI compiler suites

The following command will show all the modules that provide PGI compiler suites:

module spider PGI

Sample output of this command

[shtsai@b1-1 ~]$  module spider PGI
-----------------------------------------------------------------------------------------------------------------
  PGI:
-----------------------------------------------------------------------------------------------------------------
    Description:
      C, C++ and Fortran compilers from The Portland Group - PGI

     Versions:
        PGI/17.9
        PGI/19.10-GCC-8.3.0-2.32
-----------------------------------------------------------------------------------------------------------------

This output indicates that the PGI v.17.9 and PGI v. 19.10 compiler suites are available currently.

We suggest that you run the module spider PGI command to check an updated list of PGI compilers available on the cluster.

Intel compiler suites

The following command will show all the modules that provide Intel compiler suites:

module spider iccifort

Sample output of this command

[shtsai@b1-1 ~]$ ml spider iccifort
-----------------------------------------------------------------------------------------------------------------------------------
  iccifort:
-----------------------------------------------------------------------------------------------------------------------------------
    Description:
      Intel C, C++ & Fortran compilers

     Versions:
        iccifort/2013_sp1.0.080
        iccifort/2015.2.164-GCC-4.8.5
        iccifort/2018.1.163-GCC-6.4.0-2.28
        iccifort/2019.5.281
        iccifort/2020.4.304
-----------------------------------------------------------------------------------------------------------------------------------

This output indicates that the following versions of the Intel compiler suites are available:

• Version 13 SP1, provided by the iccifort/2013_sp1.0.080 module.
• Version 15.2, provided by the iccifort/2015.2.164-GCC-4.8.5 module.
• Version 2018.0.1.163, provided by the iccifort/2018.1.163-GCC-6.4.0-2.28 module.
• Version 2019.5.281, provided by the iccifort/2019.5.281
• Version 2020.4.304, provided by the iccifort/2020.4.304

We suggest that you run the module spider iccifort command to check an updated list of Intel compilers available on the cluster.

LLVM compiler suites

The following command will show all the modules that provide LLVM compilers:

module spider LLVM

Sample output of this command

[shtsai@b1-1 ~]$ module spider LLVM
------------------------------------------------------------------------------------------------------------------------------------
  LLVM:
------------------------------------------------------------------------------------------------------------------------------------
    Description:
      The LLVM Core libraries provide a modern source- and target-independent optimizer, along with code generation support for many popular CPUs (as well as some less common ones!) These libraries are
      built around a well specified code representation known as the LLVM intermediate representation ("LLVM IR"). The LLVM Core libraries are well documented, and it is particularly easy to invent your own
      language (or port an existing compiler) to use LLVM as an optimizer and code generator.

     Versions:
        LLVM/8.0.1-GCCcore-8.3.0
        LLVM/9.0.0-GCCcore-8.3.0
        LLVM/9.0.1-GCCcore-8.3.0
        LLVM/10.0.1-GCCcore-10.2.0
        LLVM/11.0.0-GCCcore-10.2.0
-----------------------------------------------------------------------------------------------------------------------------------

How to load a compiler module

To use any of the compiler suite, please first load the corresponding module. For example, to use the GNU 8.3.0 compiler suite, load the module with

module load GCC/8.3.0

Once this module is loaded the gcc, g++, and gfortran for GCC v. 8.3.0 will be available in your path.

Please note that you can only have one compiler module loaded at a time.

Some commonly used compiler options

PGI compiler suite

Option	Description
-O0	Specifies no optimization, recommended for code debugging
-O1	Specifies local optimization
-O2	Specifies global optimization (this is the default, same as using -O)
-O3	Includes -O1, -O2 and more aggressive optimization. Use with care.
-fast	Chooses generally good optimization options for the platform. Type pgcc -fast -help to see the equivalent options.
-Mbounds	Performs runtime array bound check, recommended for code debugging
-g	Produces symbolic debug information in the object files.
-r8	Interpret REAL variables as DOUBLE PRECISION.
-B	Allow C++ style comments in C source code; these begin with ‘//’ and continue until the end of the current line. pgcc only.
-Kieee	Perform floating-point operations in strict conformance with the IEEE 754 standard. The default compilation is -Knoieee, which uses faster but very slightly less accurate methods.
-mp	Interpret OpenMP directives to explicitly parallelize regions of code for execution by multiple threads
-acc	Enable OpenACC pragmas and directives to explicitly parallelize regions of code for execution by accelerator devices. Use with the -ta option

NOTE When using optimization options, please check if your code becomes more efficient (in some cases optimization options will slow the code down) and if it still generates correct results. Many other compiler options are available. For more information on the PGI compilers, you can view the manual pages with the commands man pgf90, man pgcc, etc, after loading the PGI module.

Intel compiler suite

Option	Description
-O0	Specifies no optimization, recommended for code debugging
-O2	Enables optimizations for speed. This is the generally recommended optimization level.
-O3	Performs -O2 optimizations and more aggressive loop transformations. Use with care.
-fast	Chooses generally good optimization options for the platform. Type pgcc -fast -help to see the equivalent options.
-Mbounds	Performs runtime array bound check, recommended for code debugging

---

Back to Top

Compiler Toolchains

On Sapelo2 we use the EasyBuild framework to install software applications. The EasyBuild toolchains are also available for users to compile their own code. Each toolchain provides a compiler suite and some basic libraries, such as MPI, BLAS, LAPACK, FFTW, etc.

More information about compiler toolchains, please Available Toolchains and Toolchain Compatibility.

foss toolchains

Most software applications are installed with the foss toolchain, where foss is short for “Free and Open Source Software”.

The foss toolchain consists of:

• binutils (https://www.gnu.org/software/binutils/)
• the GNU Compiler Collection (GCC, https://gcc.gnu.org/), i.e. gcc (C), g++ (C++) and gfortran (Fortran)
• the Open MPI library (https://www.open-mpi.org/)
• the OpenBLAS (http://www.openblas.net/) + LAPACK (http://netlib.org/lapack) libraries
• the ScaLAPACK (http://netlib.org/scalapack) library is also included
• the FFTW library (http://fftw.org/)

You can check the foss toolchain modules that are installed on the cluster with the command

module spider foss

When you load a foss toolchain, all it components will be loaded. For example:

[shtsai@b1-1 ~]$ module list
No modules loaded
[shtsai@b1-1 ~]$ module load foss/2019b
[shtsai@b1-1 ~]$ module list

Currently Loaded Modules:
  1) GCCcore/8.3.0                  6) XZ/5.2.4-GCCcore-8.3.0           11) OpenBLAS/0.3.7-GCC-8.3.0
  2) zlib/1.2.11-GCCcore-8.3.0      7) libxml2/2.9.9-GCCcore-8.3.0      12) gompi/2019b
  3) binutils/2.32-GCCcore-8.3.0    8) libpciaccess/0.14-GCCcore-8.3.0  13) FFTW/3.3.8-gompi-2019b
  4) GCC/8.3.0                      9) hwloc/1.11.12-GCCcore-8.3.0      14) ScaLAPACK/2.0.2-gompi-2019b
  5) numactl/2.0.12-GCCcore-8.3.0  10) OpenMPI/3.1.4-GCC-8.3.0          15) foss/2019b

intel toolchains

The intel toolchain consists of

• the Intel compiler suite
• the Intel MPI libraries
• the Intel Math Kernel Libraries (MKL)

You can check the intel toolchain modules that are installed on the cluster with the command

module spider intel

When you load an intel toolchain, all it components will be loaded. For example:

[shtsai@b1-1 ~]$ module list
No modules loaded
[shtsai@b1-1 ~]$ module load intel/2019b
[shtsai@b1-1 ~]$ module list

Currently Loaded Modules:
  1) GCCcore/8.3.0                 4) iccifort/2019.5.281                   7) imkl/2019.5.281-iimpi-2019b
  2) zlib/1.2.11-GCCcore-8.3.0     5) impi/2018.5.288-iccifort-2019.5.281   8) intel/2019b
  3) binutils/2.32-GCCcore-8.3.0   6) iimpi/2019b

iomkl toolchains

The iomkl toolchain consists of

• the Intel compiler suite
• the OpenMPI libraries
• the Intel Math Kernel Libraries (MKL)

You can check the iomkl toolchain modules that are installed on the cluster with the command

module spider iomkl

Th iomkl toolchains available on the cluster include:

• iomkl/2013_sp1.0.080, includes the Intel 2013.SP1 compiler suite, OpenMPI 1.8.4, MKL 11.1.1.106
• iomkl/2015.02, includes the Intel 2015.2.164 compiler suite, OpenMPI 1.8.4, MKL 11.2.2.164
• iomkl/2018a, includes the Intel 2018.1.163 compiler suite, OpenMPI 2.1.2, MKL 2018.1.163

gmvolf toolchains

The gmvolf toolchain consists of:

• binutils (https://www.gnu.org/software/binutils/)
• the GNU Compiler Collection (GCC, https://gcc.gnu.org/), i.e. gcc (C), g++ (C++) and gfortran (Fortran)
• the MVAPICH2 library (http://mvapich.cse.ohio-state.edu/)
• the OpenBLAS (http://www.openblas.net/) + LAPACK (http://netlib.org/lapack) libraries
• the ScaLAPACK (http://netlib.org/scalapack) library is also included
• the FFTW library (http://fftw.org/)

You can check the gmvolf toolchain modules that are installed on the cluster with the command

module spider gmvolf

---

Back to Top

Linking with libraries

Some library packages are installed along with some compiler toolchains. Examples of these are OpenBLAS, MKL, FFTW, etc. Other libraries are installed as a separate module, for example, Boost and GSL.

If you want to compile a code that uses a library that is not included with compiler toolchain, you will have to load a library module that uses a compatible toolchain. For example, if you want to compile your code with GCC 8.3.0 (or with the foss/2019b toolchain), and you need to use GSL, you can load the GSL/2.6-GCC-8.3.0 module.

Also note that when you load a module for a library or an application, the full path to its installation directory will be stored in an environment variable called EBROOTNAME, where NAME is the name of the application or library. For example, when you load a GSL module, the directory where the GSL libraries are installed will be in an environment variable called EBROOTGSL.

For example:

[shtsai@b1-1 ~]$ module list
No modules loaded
[shtsai@b1-1 ~]$ module load GCC/8.3.0
[shtsai@b1-1 ~]$ echo $EBROOTGCC
/apps/eb/GCCcore/8.3.0
[shtsai@b1-1 ~]$ echo $EBROOTGSL

[shtsai@b1-1 ~]$ module load GSL/2.6-GCC-8.3.0
[shtsai@b1-1 ~]$ echo $EBROOTGSL
/apps/eb/GSL/2.6-GCC-8.3.0
[shtsai@b1-1 ~]$ 

As shown in the example above, when you load an GSL module, an environment variable called EBROOTGSL is defined, and it points to the installation path for GSL.

When you compile your code, you can add the compiler option:

-I${EBROOTGSL}/include

and the linker option

-L${EBROOTGSL}/lib -lgsl -lgslcblas

Example of program compilation that uses GCC 8.3.0 and GSL v. 2.6:

module load GSL/2.6-GCC-8.3.0

gcc -O program.c I${EBROOTGSL}/include -L${EBROOTGSL}/lib -lgsl -lgslcblas -Wl,-rpath=${EBROOTGSL}/lib

Users can include the compilation option e.g. -Wl,-rpath=${EBROOTGSL}/lib to include the library directory in the runtime path. If this option is not included, then at runtime the user has to load the GSL module again, in order to define the environment variable LD_LIBRARY_PATH.

---

Back to Top