@@ Line 1: / Line 1: @@
-==Sample batch job submission scripts on Sapelo2==
+[[Category:sapelo2]]
-====OpenMPI====
+[[Sample batch job submission scripts on Sapelo2]]
-Sample job submission script (sub.sh) to run an OpenMPI application:
+[[Sample batch job submission scripts on the teaching cluster]]
-<pre class="gscript">
-#PBS -S /bin/bash
-#PBS -q batch
-#PBS -N testjob
-#PBS -l nodes=2:ppn=48:AMD
-#PBS -l walltime=48:00:00
-#PBS -l mem=2gb
-#PBS -M username@uga.edu
-#PBS -m abe
-cd $PBS_O_WORKDIR
-ml OpenMPI/2.1.1-GCC-6.4.0-2.28
-echo
-echo "Job ID: $PBS_JOBID"
-echo "Queue:  $PBS_QUEUE"
-echo "Cores:  $PBS_NP"
-echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
-echo "mpirun: $(which mpirun)"
-echo
-mpirun ./a.out > outputfile
-</pre>
-====OpenMP====
-Sample job submission script (sub.sh) to run an OpenMP (threaded) large memory application:
-<pre class="gscript">
-#PBS -S /bin/bash
-#PBS -q batch
-#PBS -N testjob
-#PBS -l nodes=1:ppn=10:HIGHMEM
-#PBS -l walltime=48:00:00
-#PBS -l mem=300gb
-#PBS -M username@uga.edu
-#PBS -m abe
-cd $PBS_O_WORKDIR
-export OMP_NUM_THREADS=10
-echo
-echo "Job ID: $PBS_JOBID"
-echo "Queue:  $PBS_QUEUE"
-echo "Cores:  $PBS_NP"
-echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
-echo
-time ./a.out > outputfile
-</pre>
-====A high memory job====
-Sample job submission script (sub.sh) to run an application that needs to use an Intel HIGHMEM node:
-<pre class="gscript">
-#PBS -S /bin/bash
-#PBS -q highmem_q
-#PBS -N testjob
-#PBS -l nodes=1:ppn=12:Intel
-#PBS -l walltime=48:00:00
-#PBS -l mem=400gb
-#PBS -M username@uga.edu
-#PBS -m abe
-cd $PBS_O_WORKDIR
-ml Velvet
-velvetg [options] > outputfile
-</pre>
-====GPU/CUDA====
-Sample job submission script (sub.sh) to run a GPU-enabled (e.g. CUDA) application:
-<pre class="gscript">
-#PBS -S /bin/bash
-#PBS -q gpu_q
-#PBS -N testjob
-#PBS -l nodes=1:ppn=4:gpus=1:GPU
-#PBS -l walltime=48:00:00
-#PBS -l mem=2gb
-#PBS -M username@uga.edu
-#PBS -m abe
-cd $PBS_O_WORKDIR
-ml CUDA/9.0.176
-echo
-echo "Job ID: $PBS_JOBID"
-echo "Queue:  $PBS_QUEUE"
-echo "Cores:  $PBS_NP"
-echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
-echo
-time ./a.out > outputfile
-</pre>
-'''Note:''' Please note the additional '''gpus=1''' option in the header line. This option should be used to request the number of GPU cards to be used (e.g. to request 2 GPU cards, use gpus=2).
-The GPU devices allocated to a job are listed in a file whose name is stored in the queueing system environment variable PBS_GPUFILE. You can print what this file name is with the command (add it to your job submission file):
-<pre class="gscript">
-echo $PBS_GPUFILE
-</pre>
-To get a list of the numbers of the GPU devices allocated to your job, separated by a blank space, use the command:
-<pre class="gscript">
-CUDADEV=$(cat $PBS_GPUFILE | rev | cut -d"u" -f1)
-echo "List of devices allocated to this job:"
-echo $CUDADEV
-</pre>
-To remove the blank space between two device numbers in the CUDADEV variable above, use the command:
-<pre class="gscript">
-CUDADEV=$(cat $PBS_GPUFILE | rev | cut -d"u" -f1)
-GPULIST=$(echo $CUDADEV | sed 's/ //')
-echo "List of devices allocated to this job (no blank spaces between devices):"
-echo $GPULIST
-</pre>
-Some GPU/CUDA applications require that a list of the GPU devices be given as an argument to the application. If the application needs a blank space separated device number list, use the $CUDADEV variable as an argument. If no blank space is allowed in the list, you can use the $GPULIST variable as an argument to the application.
-====Hybrid MPI/shared-memory using OpenMPI====
-Sample job submission script (sub.sh) to run a parallel job that uses 3 MPI processes with OpenMPI and each MPI process runs with 12 threads:
-<pre class="gscript">
-#PBS -S /bin/bash
-#PBS -j oe
-#PBS -q batch
-#PBS -N testhybrid
-#PBS -l nodes=3:ppn=12:AMD
-#PBS -l mem=60g
-#PBS -l walltime=4:00:00
-#PBS -M username@uga.edu
-#PBS -m abe
-ml OpenMPI/2.1.1-GCC-6.4.0-2.28
-echo
-echo "Job ID: $PBS_JOBID"
-echo "Queue:  $PBS_QUEUE"
-echo "Cores:  $PBS_NP"
-echo "Nodes:  $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
-echo "mpirun: $(which mpirun)"
-echo
-cd $PBS_O_WORKDIR
-export OMP_NUM_THREADS=12
-perl /usr/local/bin/makehostlist.pl $PBS_NODEFILE $PBS_NUM_PPN $PBS_JOBID
-mpirun -machinefile  host.$PBS_JOBID.list ./a.out
-</pre>
-====Running an array job====
-Sample job submission script (sub.sh) to submit an array job with 10 elements. In this example, each array job element will run the a.out binary using an input file called input_0, input_1, ..., input_9.
-<pre class="gscript">
-#PBS -S /bin/bash
-#PBS -j oe
-#PBS -q batch
-#PBS -N myarrayjob
-#PBS -l nodes=1:ppn=1:AMD
-#PBS -l walltime=4:00:00
-#PBS -t 0-9
-cd $PBS_O_WORKDIR
-time ./a.out < input_$PBS_ARRAYID
-</pre>
-==Sample batch job submission scripts on zcluster==
-In all these sample scripts, the ''working_directory'' refers to your working directory. For example, your working directory could be a subdirectory called projectA in your home directory.
-'''Notes about the job submission script'''
-* You can use the pound sign # to start comment lines in your script, but # cannot be immediately followed by a dollar sign $. The combination #$ is reserved to indicate a qsub option defined in the script.
-* The job submission script does not need to have execution permission.
-====To run a serial job====
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-time ./myprog < myin > myout
-</pre>
-Note that the 'time' command included in the sample script above is optional (this command measures the amount of time it takes to run an executable). Entering a standard input file (myin) and standard output file (myout) is only necessary if your executable requires input from standard input and if it outputs data to standard output.
-You can specify the ''working_directory'' as in the example below:
-<pre class="gscript">
-#!/bin/bash
-cd ${HOME}/projectA
-time ./myprog < myin > myout
-</pre>
-If the job is submitted from within the working directory, you can use the following sample script:
-<pre class="gscript">
-#!/bin/bash
-cd `pwd`
-time ./myprog < myin > myout
-</pre>
-====To run a Parallel job====
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-time ./myprog -number-of-thread 4
-</pre>
-Note that the '-number-of-thread' in the sample script above is an example of thread number parameter. Each program has own name of such parameter.
-To submit a parallel job using 4 threads with script sub.sh to the rcc-30d batch queue and have the standard output and error go to the files ''sub.sh'''''.o'''''jobid'' and ''sub.sh'''''.e'''''jobid'', respectively, in the directory from which the job was submitted, use
-<pre class="gcommand">
-qsub -q rcc-30d -pe thread 4 sub.sh
-</pre>
-====To run an R program====
-Sample script to run an R program called program.R:
-<pre class="gscript">
-#!/bin/bash
-cd `pwd`
-time /usr/local/R/2.15.2/bin/R CMD BATCH program.R
-</pre>
-====To run a Matlab program====
-Sample script to run a Matlab program called plotsin.m:
-<pre class="gscript">
-#!/bin/bash
-cd `pwd`
-matlab -nodisplay < plotsin.m > matlab.out
-exit
-</pre>
-===Parallel Jobs using MPI libraries===
-<u>To run a parallel MPI job using MPICH/PGI</u>
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-echo "Got $NSLOTS processors."
-echo "Machines:"
-cat $TMPDIR/machines
-/usr/local/pgi/linux86-64/2012/mpi/mpich/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines ./myprog
-</pre>
-Note that lines 3, 4, and 5 in this script are optional.
-<u>To run a parallel MPI job using the default MPICH2 (PGI) compilers.</u> For example, if you compiled the code with mpicc or mpif90, etc, without full path.
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-export LD_LIBRARY_PATH=/usr/local/mpich2/1.4.1p1/pgi123/lib:${LD_LIBRARY_PATH}
-mpirun -np $NSLOTS ./myprog
-</pre>
-<u>To run a parallel MPI job using MPICH2 and e.g. GNU 4.4.4 compilers</u>
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-export LD_LIBRARY_PATH=/usr/local/mpich2/1.4.1p1/gcc_4.4.4/lib:${LD_LIBRARY_PATH}
-/usr/local/mpich2/1.4.1p1/gcc_4.4.4/bin/mpirun -np $NSLOTS ./myprog
-</pre>
-Note that with MPICH2 it is not necessary to include the -machinefile option when submitting the job to a batch queue. When using other MPICH2 compilations, such as for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script.
-<u>To run a parallel MPI job using OpenMPI 1.4.4 and e.g. GNU 4.1.2 compilers</u>
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-export LD_LIBRARY_PATH=/usr/local/openmpi/1.4.4/gcc412/lib:${LD_LIBRARY_PATH}
-/usr/local/openmpi/1.4.4/gcc412/bin/mpirun ––mca btl "tcp,self" -np $NSLOTS ./myprog
-</pre>
-<u>To run a parallel MPI job using OpenMPI 1.6.2 and e.g. GNU 4.7.1 compilers</u>
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-export LD_LIBRARY_PATH=/usr/local/gcc/4.7.1/lib64:/usr/local/openmpi/1.6.2/gcc471/lib:${LD_LIBRARY_PATH}
-/usr/local/openmpi/1.6.2/gcc471/bin/mpirun ––mca btl "tcp,self" -np $NSLOTS ./myprog
-</pre>
-Note that with OpenMPI you can use the mpirun command and there is no need to include the -machinefile option. When using other OpenMPI compilations, such as the one for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script. To use OpenMPI over Infiniband, remove the mpirun option
-––mca btl "tcp,self"
-from the script.
-<u>To run a parallel MPI job using MVAPICH2/GNU over Infiniband</u>
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-export LD_LIBRARY_PATH=/usr/local/mvapich2/1.8/gcc444/lib:${LD_LIBRARY_PATH}
-/usr/local/mvapich2/1.8/gcc444/bin/mpirun -np $NSLOTS ./myprog
-</pre>
-===To run a shared memory job using OpenMP===
-Sample script to run a program that uses 4 OpenMP threads:
-<pre class="gscript">
-#!/bin/bash
-cd working_directory
-export OMP_NUM_THREADS=4
-time ./myprog
-</pre>

Difference between revisions of "Sample Scripts"

Latest revision as of 14:14, 7 January 2021

Navigation menu

Search