OpenMP

Compiling OpenMP code on Sapelo2

The compilers installed on Sapelo2 support shared memory applications using OpenMP. Here are the compiler options to include when using OpenMP:

Compiler	Commands	Compile Option
PGI	pgcc, pgCC, pgfortran, etc.	-mp
Intel	icc, ifort	-openmp
GNU	gcc, g++, gfortran, etc.	-fopenmp

For more information about the compilers on Sapelo2, please see Code Compilation on Sapelo2.

Defining number of OpenMP threads

The number of OpenMP threads can be specified using the environment variable OMP_NUM_THREADS. For example, to define 4 threads:

For bash/sh:

export OMP_NUM_THREADS=4

For csh/tcsh:

setenv OMP_NUM_THREADS 4

Setting thread-core binding

In general, an OpenMP code will run a lot more efficiently when threads are pinned to a given core on a compute node. Please set the environment variable OMP_PROC_BIND=true to specify threads' affinity policy (thread-core bind).

For bash/sh:

export OMP_PROC_BIND=true

For csh/tcsh:

setenv OMP_PROC_BIND  true

Sample script to run a shared memory job using OpenMP

Sample job submission script (sub.sh) to run a program that uses 4 OpenMP threads:

#!/bin/bash
#SBATCH --job-name=mctest             # Job name
#SBATCH --partition=batch             # Partition (queue) name
#SBATCH --ntasks=1                    # Run a single task	
#SBATCH --cpus-per-task=4             # Number of CPU cores per task
#SBATCH --mem=4gb                     # Job memory request
#SBATCH --time=02:00:00               # Time limit hrs:min:sec
#SBATCH --output=mctest.%j.out        # Standard output log
#SBATCH --error=mctest.%j.err         # Standard error log

#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu  # Where to send mail	

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=4
export OMP_PROC_BIND=true

module load foss/2019b                # load the appropriate module file, e.g. foss/2019b

time ./a.out

Please note that the number of cores requested by this job (--cpus-per-task=4) is the same as the number of OpenMP threads that the application will use (export OMP_NUM_THREADS=4).

Sample job submission command on Sapelo2

To submit a shared memory job that uses 4 OpenMP threads:

sbatch sub.sh

For more information about running jobs on Sapelo2, please see Running Jobs on Sapelo2.