OpenMP

Compiling OpenMP code on Sapelo2

The compilers installed on Sapelo2 support shared memory applications using OpenMP. Here are the compiler options to include when using OpenMP:

Compiler	Commands	Option
PGI	pgcc, pgCC, pgf90, etc.	-mp
Intel	icc, ifort	-openmp
GNU	gcc, gfortran, g++, gcc44, etc.	-fopenmp

For more information about the compilers on Sapelo2, please see Code Compilation on Sapelo2.

Defining number of OpenMP threads

The number of OpenMP threads can be specified using the environment variable OMP_NUM_THREADS. For example, to define 4 threads:

For bash/sh:

export OMP_NUM_THREADS=4

For csh/tcsh:

setenv OMP_NUM_THREADS 4

Setting thread-core binding

In general, an OpenMP code will run a lot more efficiently when threads are pinned to a given core on a compute node. Please set the environment variable OMP_PROC_BIND=true to specify threads' affinity policy (thread-core bind).

For bash/sh:

export OMP_PROC_BIND=true

For csh/tcsh:

setenv OMP_PROC_BIND  true

Sample script to run a shared memory job using OpenMP

Sample job submission script (sub.sh) to run a program that uses 4 OpenMP threads:

#PBS -S /bin/bash
#PBS -q batch
#PBS -N testjob
#PBS -l nodes=1:ppn=4
#PBS -l walltime=48:00:00
#PBS -l mem=30gb
#PBS -M username@uga.edu 
#PBS -m abe

cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=4
export OMP_PROC_BIND=true

echo
echo "Job ID: $PBS_JOBID"
echo "Queue:  $PBS_QUEUE"
echo "Cores:  $PBS_NP"
echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
echo

time ./a.out > outputfile

Please note that the number of cores requested by this job (ppn=4) is the same as the number of OpenMP threads that the application will use (export OMP_NUM_THREADS=4).

Sample job submission command on Sapelo2

To submit a shared memory job that uses 4 OpenMP threads:

qsub sub.sh

For more information about running jobs on Sapelo2, please see Running Jobs on Sapelo2.