Code Compilation on pcluster

Compilers

Compilers Installed on IBM p655 (pcluster)

	IBM XL set	GNU	MPI
Fortran77	xlf, xlf_r		mpxlf, mpxlf_r
Fortran90	xlf90, xlf90_r		mpxlf90, mpxlf90_r
C	cc, cc_r, xlc, xlc_r	gcc	mpcc, mpcc_r
C++	xlC,xlC_r	g++	mpCC, mpCC_r

Note

The extension _r indicates thread safe versions of the corresponding compilers.

Default Compiler Versions

• IBM XL Fortran: version 9.1 (updated on 01/08/2007)
• IBM XL C/C++: version 7 (updated on 01/08/2007)
• GNU gcc/g++: version 3.3.2

Note

Older versions of the IBM compilers, namely XL Fortran version 8.1 and Visual Age C/C++ version 6, which were the default compilers from pcluster go-live to 01/03/2007, are still installed, but in a different path. These older versions can be used to compile serial programs (they cannot be used to compile MPI jobs). Examples of how to set the path to use the older versions of the compilers

To use the IBM C/C++ compilers, version 6 :

For csh/tcsh

pcluster>   source /usr/local/bin/use_ibm_c_v6.csh

For bash/ksh

pcluster>   source /usr/local/bin/use_ibm_c_v6.sh 

To use the IBM XL Fortran compilers, version 8.1 :

For csh/tcsh

pcluster>   source /usr/local/bin/use_xlf_v8.csh

For bash/ksh

pcluster>   source /usr/local/bin/use_xlf_v8.sh

The default versions of the IBM compilers can be reset to the newer ones with the following commands:

Reset the default IBM C/C++ compilers to version 7 :

For csh/tcsh

pcluster>   source /usr/local/bin/use_ibm_c_v7.csh

For bash/ksh

pcluster>   source /usr/local/bin/use_ibm_c_v7.sh

Reset the default IBM XL Fortran compilers to version 9.1 :

For csh/tcsh

pcluster>   source /usr/local/bin/use_xlf_v9.csh

For bash/ksh

pcluster>   source /usr/local/bin/use_xlf_v9.sh 

Commonly used compilation flags for the IBM compilers (IBM XL set and Visual Age):

• -O3 more aggressive optimization (default is -O2), should be used with the -qstrict flag
• -qstrict used with -O3 to ensure that the semantics of the program is not changed
• -qarch=pwr4 generates binaries tuned for the Power4 processor. Recommended.
• -qtune=pwr4 generates binaries tuned for the Power4 processor. Recommended.
• -C performs runtime array bound check (will slow execution - use only for debugging) Fortran only.
• -qcheck=bounds performs runtime array bound check (will slow execution - use only for debugging) C/C++ only.
• -q64 -qwarn64 uses 64-bit memory addressing
• -qsmp=omp for OpenMP and hybrid MPI/OpenMP codes. You must also use the thread-safe compilers (ending with _r)
• -qcpluscmt permits "//" to introduce a comment line that lasts until the end of the source line, as in C++. By default the Visual Age C compiler only accepts comments delimited by "/* */". C only.

Examples

1. To compile a Fortran90 serial code:

pcluster>  xlf90 -O3 -qstrict -qarch=pwr4 -qtune=pwr4 -qsuffix=f=f90 program.f90

An executable a.out will be created. Use the -o flag to specify a different executable name. In the following example the executable is called "program".

pcluster>  xlf90 program.f -o program

Note By default Fortran90 programs on the pcluster have suffix .f instead of .f90 (If your Fortran90 program has an .f90 suffix, you can compile it with the flag -qsuffix=f=f90, as in the example above.).

2. To compile a Fortran77 MPI code:

pcluster>  mpxlf_r -O3 -qstrict -qarch=pwr4 -qtune=pwr4 program.f

Note The IBM compilers mpxlf, mpxlf_r, mpcc, mpcc_r, etc automatically link the IBM MPI libraries. Therefore, when using these compilers it is not necessary to use the -lmpi flag.

3. To compile a C++ MPI code:

pcluster>  mpCC_r -cpp program.C

4. To compile an OpenMP/C code:

pcluster>  mpcc_r -qsmp=omp program.c

Memory limits:

By default, programs are compiled with 32-bit memory addressing and can access up to 256MB of memory. With Fortran (xlf, xlf90, etc) if you use the compiler/linker flag -bmaxdata:0x80000000 the program can use up to 2GB of memory. Note that the number following -bmaxdata: corresponds to the amount of memory (hexadecimal notation) and 0x80000000 is the maximum acceptable value. To use more than 2GB with Fortran and more than 256MB with C/C++, you have to use 64-bit memory addressing. This can be done by adding the compiler flags: -q64 -qwarn64

Notes

By default, the IBM compilers use 80-bit arithmetic libraries to perform addition and multiplication. To turn this feature off, use the flag -qfloat=nomaf to compile the code. Type 'xlf', 'xlc', 'xlC', 'mpxlf', 'mpcc', or 'mpCC' (without the quotes) to get more information about these compilers and the compilation options available.

In general the IBM XL Fortran and Visual Age C/C++ compilers perform significantly better on the Power4 system than the corresponding GNU compilers, therefore it is our recommendation that you avoid using the latter whenever possible.

The MPI library MPICH is installed in /usr/local/mpich but for better performance we recommend that you use the IBM MPI libraries whenever possible.