Sample Scripts

Sample batch job submission scripts on Sapelo2

OpenMPI

Sample job submission script (sub.sh) to run an OpenMPI application:

#PBS -S /bin/bash
#PBS -q batch
#PBS -N testjob
#PBS -l nodes=2:ppn=48:AMD
#PBS -l walltime=48:00:00
#PBS -l mem=2gb
#PBS -M username@uga.edu 
#PBS -m abe

cd $PBS_O_WORKDIR

ml OpenMPI/2.1.1-GCC-6.4.0-2.28

echo
echo "Job ID: $PBS_JOBID"
echo "Queue:  $PBS_QUEUE"
echo "Cores:  $PBS_NP"
echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
echo "mpirun: $(which mpirun)"
echo

mpirun ./a.out > outputfile

OpenMP

Sample job submission script (sub.sh) to run an OpenMP (threaded) large memory application:

#PBS -S /bin/bash
#PBS -q batch
#PBS -N testjob
#PBS -l nodes=1:ppn=10:HIGHMEM
#PBS -l walltime=48:00:00
#PBS -l mem=300gb
#PBS -M username@uga.edu 
#PBS -m abe

cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=10

echo
echo "Job ID: $PBS_JOBID"
echo "Queue:  $PBS_QUEUE"
echo "Cores:  $PBS_NP"
echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
echo

time ./a.out > outputfile

A high memory job

Sample job submission script (sub.sh) to run an application that needs to use an Intel HIGHMEM node:

#PBS -S /bin/bash
#PBS -q highmem_q
#PBS -N testjob
#PBS -l nodes=1:ppn=12:Intel
#PBS -l walltime=48:00:00
#PBS -l mem=400gb
#PBS -M username@uga.edu 
#PBS -m abe

cd $PBS_O_WORKDIR

ml Velvet 

velvetg [options] > outputfile

GPU/CUDA

Sample job submission script (sub.sh) to run a GPU-enabled (e.g. CUDA) application:

#PBS -S /bin/bash
#PBS -q gpu_q
#PBS -N testjob
#PBS -l nodes=1:ppn=4:gpus=1:GPU
#PBS -l walltime=48:00:00
#PBS -l mem=2gb
#PBS -M username@uga.edu 
#PBS -m abe

cd $PBS_O_WORKDIR

ml CUDA/9.0.176

echo
echo "Job ID: $PBS_JOBID"
echo "Queue:  $PBS_QUEUE"
echo "Cores:  $PBS_NP"
echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
echo

time ./a.out > outputfile

Note: Please note the additional gpus=1 option in the header line. This option should be used to request the number of GPU cards to be used (e.g. to request 2 GPU cards, use gpus=2).

The GPU devices allocated to a job are listed in a file whose name is stored in the queueing system environment variable PBS_GPUFILE. You can print what this file name is with the command (add it to your job submission file):

echo $PBS_GPUFILE

To get a list of the numbers of the GPU devices allocated to your job, separated by a blank space, use the command:

CUDADEV=$(cat $PBS_GPUFILE | rev | cut -d"u" -f1)

echo "List of devices allocated to this job:"

echo $CUDADEV

To remove the blank space between two device numbers in the CUDADEV variable above, use the command:

CUDADEV=$(cat $PBS_GPUFILE | rev | cut -d"u" -f1)

GPULIST=$(echo $CUDADEV | sed 's/ //')

echo "List of devices allocated to this job (no blank spaces between devices):"

echo $GPULIST

Some GPU/CUDA applications require that a list of the GPU devices be given as an argument to the application. If the application needs a blank space separated device number list, use the $CUDADEV variable as an argument. If no blank space is allowed in the list, you can use the $GPULIST variable as an argument to the application.

Hybrid MPI/shared-memory using OpenMPI

Sample job submission script (sub.sh) to run a parallel job that uses 3 MPI processes with OpenMPI and each MPI process runs with 12 threads:

#PBS -S /bin/bash
#PBS -j oe
#PBS -q batch
#PBS -N testhybrid
#PBS -l nodes=3:ppn=12:AMD
#PBS -l mem=60g
#PBS -l walltime=4:00:00
#PBS -M username@uga.edu 
#PBS -m abe

ml OpenMPI/2.1.1-GCC-6.4.0-2.28

echo
echo "Job ID: $PBS_JOBID"
echo "Queue:  $PBS_QUEUE"
echo "Cores:  $PBS_NP"
echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
echo "mpirun: $(which mpirun)"
echo

cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=12

perl /usr/local/bin/makehostlist.pl $PBS_NODEFILE $PBS_NUM_PPN $PBS_JOBID

mpirun -machinefile  host.$PBS_JOBID.list ./a.out

Running an array job

Sample job submission script (sub.sh) to submit an array job with 10 elements. In this example, each array job element will run the a.out binary using an input file called input_0, input_1, ..., input_9.

#PBS -S /bin/bash
#PBS -j oe
#PBS -q batch
#PBS -N myarrayjob
#PBS -l nodes=1:ppn=1:AMD
#PBS -l walltime=4:00:00
#PBS -t 0-9

cd $PBS_O_WORKDIR

time ./a.out < input_$PBS_ARRAYID

Running a singularity job

For example, to run Trinity using the singularity image:

#PBS -S /bin/bash
#PBS -N j_s_trinity
#PBS -q highmem_q
#PBS -l nodes=1:ppn=1
#PBS -l walltime=480:00:00
#PBS -l mem=100gb
 
cd $PBS_O_WORKDIR

singularity exec /usr/local/singularity-images/trinity-2.5.1--0.simg COMMAND OPTION

where COMMAND should be replaced by the specific command and options, such as:

#PBS -S /bin/bash
#PBS -N j_s_trinity
#PBS -q highmem_q
#PBS -l nodes=1:ppn=16
#PBS -l walltime=480:00:00
#PBS -l mem=100gb
 
cd $PBS_O_WORKDIR

singularity exec /usr/local/singularity-images/trinity-2.5.1--0.simg Trinity --seqType <string> --max_memory 100G --CPU 8 --no_version_check 1>job.out 2>job.err