Running Jobs on the teaching cluster

Using the Queueing System

The login node for the teaching cluster should be used for text editing, and job submissions. No jobs should be run directly on the login node. Processes that use too much CPU or RAM on the login node may be terminated by GACRC staff, or automatically, in order to keep the cluster running properly. Jobs should be run using the Slurm queueing system. The queueing system should be used to run both interactive and batch jobs.

---

Back to Top

Queues defined on the teaching cluster

There are different queues defined on the teaching cluster. The SLURM queueing system refers to queues as partition. Users are required to specify, in the job submission script or as job submission command line arguments, the queue and the resources needed by the job in order for it to be assigned to compute node(s) that have enough available resources (such as number of cores, amount of memory, GPU cards, etc).

The table below summarizes the partitions (queues) defined and the compute nodes that they target:

Queue Name	Node Type	Node Number	Description	Notes
batch	Intel	37	12-core, 48GB RAM, Intel Xeon	Regular nodes.
batch	Intel	2	8-core, 48GB RAM, Intel Xeon	Regular nodes.
highmem	AMD	2	48-core, 128GB RAM, AMD Opteron	For high memory jobs.
highmem	Intel	3	8-core, 192GB RAM, Intel Xeon	For high memory jobs.
gpu	GPU, M2070	1	12-core, 48GB RAM, Intel Xeon, 8 NVIDIA M2070 GPUs	For GPU-enabled jobs.
interq	AMD	3	32-core, 64GB RAM, AMD Opteron	For interactive jobs.

You can check all partitions (queues) defined in the cluster with the command

sinfo

---

Back to Top

Job submission Scripts

Users are required to specify the number of cores, the amount of memory, the queue name, and the maximum wallclock time needed by the job.

Header lines

Basic job submission script

At a minimum, the job submission script needs to have the following header lines:

#!/bin/bash
#SBATCH --partition=batch
#SBATCH --job-name=test
#SBATCH --ntasks=2
#SBATCH --time=2:00:00
#SBATCH --mem=2gb

Commands to run your application should be added after these header lines.

Header lines explained

• #!/bin/bash : used to specify using /bin/bash shell
• #SBATCH --partition=batch : used to specify the partition (queue) name, e.g. batch
• #SBATCH --job-name=test : used to specify the name of the job, e.g. test
• #SBATCH --ntasks=2 : used to specify the number of tasks (e.g. 2).
• #SBATCH --time=2:00:00 : used to specify the maximum allowed wall clock time in dd:hh:mm:ss format for the job (e.g 2h).
• #SBATCH --mem=2gb : used to specify the maximum memory allowed for the job (e.g. 2GB)

Below are some of the most commonly used queueing system options to configure the job.

Options to request resources for the job

• -t, --time=time

   Set a limit on the total run time. Acceptable formats include  "minutes", "minutes:seconds",  "hours:minutes:seconds",  "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds"

• --mem=MB

   Maximum memory per node the job will need in MegaBytes

• --mem-per-cpu=MB

   Memory required per allocated CPU in MegaBytes

• -N, --nodes=num

   Number of nodes are required. Default is 1 node

• -n, --ntasks=num

   Maximum number tasks will be launched. Default is one task per node

• --ntasks-per-node=ntasks

   Request that ntasks be invoked on each node

• -c, --cpus-per-task=ncpus

   Require ncpus number of CPU cores per task. Without this option, allocate one core per task

Please try to request resources for your job as accurately as possible, because this allows your job to be dispatched to run at the earliest opportunity and it helps the system allocate resources efficiently to start as many jobs as possible, benefiting all users.

Options to manage job notification and output

• -J, --job-name jobname

   Give the job a name. The default is the filename of the job script. Within the job, $SBATCH_JOB_NAME expands to the job name

• -o, --output=path/for/stdout

   Send stdout to path/for/stdout. The default filename is slurm-${SLURM_JOB_ID}.out, e.g. slurm-12345.out, in the directory from which the job was submitted 

• -e, --error=path/for/stderr

   Send stderr to path/for/stderr.

• --mail-user=username@uga.edu

   Send email notification to the address you specified when certain events occur.

• --mail-type=type

   The value of type can be set to NONE, BEGIN, END, FAIL, ALL.

Options to set Array Jobs

If you wish to run an application binary or script using e.g. different input files, then you might find it convenient to use an array job. To create an array job with e.g. 10 elements, use

#SBATCH -t 0-9

or

#SBATCH --array=0-9

The ID of each element in an array job is stored in the variable SLURM_ARRAY_TASK_ID. The variable SLURM_ARRAY_JOB_ID will be expanded into the jobid of the array job. Each array job element runs as an independent job, so multiple array elements can run concurrently, if resources are available.

Option to set job dependency

You can set job dependency with the option -d or --dependency=dependency-list. For example, if you want to specify that one job only starts after job with jobid 1234 finishes, you can add the following header line in the job submission script of the job:

#SBATCH --dependency=afterok:1234

Having this header line in the job submission script will ensure that the job is only dispatched to run after job 1234 has completed successfully.

Other content of the script

Following the header lines, users can include commands to change to the working directory, to load the modules needed to run the application, and to invoke the application. For example, to use the directory from which the job is submitted as the working directory (where to find input files or binaries), add the line

cd $SLURM_SUBMIT_DIR

You can then load the needed modules. For example, if you are running an R program, then include the line

module load R/3.4.4-foss-2016b-X11-20160819-GACRC

Then invoke your application. For example, if you are running an R program called add.R which is in your job submission directory, use

R CMD BATCH add.R

Environment Variables exported by batch jobs

When a batch job is started, a number of variables are introduced into the job's environment that can be used by the batch script in making decisions, creating output files, and so forth. Some of these variables are listed in the following table:

Variable	Description
SLURM_ARRAY_JOB_ID	Job id of an array job
SLURM_ARRAY_TASK_ID	Value of job array index for this job
SLURM_CPUS_ON_NODE	Number of CPUS on the allocated node.
SLURM_CPUS_PER_TASK	Number of cpus requested per task. Only set if the --cpus-per-task option is specified.
SLURM_JOB_ID	Unique pbs job id
SLURM_JOB_NAME	User specified jobname
SLURM_JOB_CPUS_PER_NODE	Count of processors available to the job on this node.
SLURM_JOB_NAME	Name of the job.
SLURM_JOB_NODELIST	List of nodes allocated to the job.
SLURM_JOB_NUM_NODES	Total number of nodes in the job's resource allocation.
SLURM_JOB_PARTITION	Name of the partition (i.e. queue) in which the job is running.
SLURM_NTASKS	Same as -n, --ntasks
SLURM_NTASKS_PER_NODE	Number of tasks requested per node. Only set if the --ntasks-per-node option is specified.
SLURM_SUBMIT_DIR	The directory from which sbatch was invoked.
SLURM_TASKS_PER_NODE	Number of tasks to be initiated on each node.

---

Back to Top

Sample job submission scripts

Serial (single-processor) Job

Sample job submission script (sub.sh) to run an R program called add.R using a single core:

#!/bin/bash
#SBATCH --job-name=testserial         # Job name
#SBATCH --partition=batch             # Partition (queue) name
#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu  # Where to send mail	
#SBATCH --ntasks=1                    # Run on a single CPU
#SBATCH --mem=1gb                     # Job memory request
#SBATCH --time=02:00:00               # Time limit hrs:min:sec
#SBATCH --output=testserial.%j.out    # Standard output and error log

cd $SLURM_SUBMIT_DIR

module load R/3.4.4-foss-2016b-X11-20160819-GACRC

R CMD BATCH add.R

In this sample script, the standard output and error of the job will be saved into a file called testserial.o%j, where %j will be automatically replaced by the job id of the job.

MPI Job

Sample job submission script (sub.sh) to run an OpenMPI application. In this example the job requests 16 cores and further specifies that these 16 cores need to be divided equally on 2 nodes (8 cores per node) and the binary is called mympi.exe:

#!/bin/bash
#SBATCH --job-name=mpitest      # Job name
#SBATCH --partition=batch             # Partition (queue) name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu    # Where to send mail	
#SBATCH --ntasks=16                  # Number of MPI ranks
#SBATCH --cpus-per-task=1            # Number of cores per MPI rank 
#SBATCH --nodes=2                    # Number of nodes
#SBATCH --ntasks-per-node=8          # How many tasks on each node
#SBATCH --mem-per-cpu=600mb          # Memory per processor
#SBATCH --time=02:00:00              # Time limit hrs:min:sec
#SBATCH --output=mpitest.%j.out         # Standard output and error log

cd $SLURM_SUBMIT_DIR

module load OpenMPI/1.10.3-GCC-5.4.0-2.26

mpirun ./mympi.exe

OpenMP (Multi-Thread) Job

Sample job submission script (sub.sh) to run a program that uses OpenMP with 6 threads. Please set --ntasks=1 and set --cpus-per-task to the number of threads you wish to use. The name of the binary in this example is a.out.

#!/bin/bash
#SBATCH --job-name=mctest      # Job name
#SBATCH --partition=batch             # Partition (queue) name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu    # Where to send mail	
#SBATCH --ntasks=1                   # Run a single task	
#SBATCH --cpus-per-task=6            # Number of CPU cores per task
#SBATCH --mem=4gb                    # Job memory request
#SBATCH --time=02:00:00              # Time limit hrs:min:sec
#SBATCH --output=mctest.%j.out          # Standard output and error log

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=6  

module load foss/2016b  # load the appropriate module file, e.g. foss/2016b

time ./a.out

High Memory Job

Sample job submission script (sub.sh) to run a velvet application that needs to use 50GB of memory and 4 threads:

#!/bin/bash
#SBATCH --job-name=highmemtest      # Job name
#SBATCH --partition=highmem            # Partition (queue) name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu    # Where to send mail	
#SBATCH --ntasks=1                   # Run a single task	
#SBATCH --cpus-per-task=4          # Number of CPU cores per task
#SBATCH --mem=50gb                    # Job memory request
#SBATCH --time=02:00:00              # Time limit hrs:min:sec
#SBATCH --output=highmemtest.%j.out     # Standard output and error log

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=4

module load Velvet

velvetg [options]

Hybrid MPI/shared-memory using OpenMPI

Sample job submission script (sub.sh) to run a parallel job that uses 4 MPI processes with OpenMPI and each MPI process runs with 3 threads:

#!/bin/bash
#SBATCH --job-name=hybridtest
#SBATCH --partition=batch             # Partition (queue) name
#SBATCH --mail-type=END,FAIL   # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail	
#SBATCH --nodes=2              # Number of nodes
#SBATCH --ntasks=4             # Number of MPI ranks
#SBATCH --ntasks-per-node=2    # Number of MPI ranks per node
#SBATCH --cpus-per-task=3      # Number of OpenMP threads for each MPI process/rank
#SBATCH --mem-per-cpu=2000mb   # Per processor memory request
#SBATCH --time=2-00:00:00      # Walltime in hh:mm:ss or d-hh:mm:ss (2 days in the example)
#SBATCH --output=hybridtest.%j.out

cd $SLURM_SUBMIT_DIR
 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

mpirun ./myhybridprog.exe

Array job

Sample job submission script (sub.sh) to submit an array job with 10 elements. In this example, each array job element will run the a.out binary using an input file called input_0, input_1, ..., input_9.

#!/bin/bash
#SBATCH --job-name=arrayjobtest   # Job name
#SBATCH --partition=batch             # Partition (queue) name
#SBATCH --ntasks=1                  # Run a single task
#SBATCH --mem=1gb                   # Job Memory
#SBATCH --time=10:00:00             # Time limit hrs:min:sec
#SBATCH --output=array_%A-%a.out    # Standard output and error log
#SBATCH --array=0-9                 # Array range

cd $SLURM_SUBMIT_DIR

module load foss/2016b # load any needed module files, e.g. foss/2016b

time ./a.out < input_$SLURM_ARRAY_TASK_ID

GPU/CUDA

To be added.

---

Back to Top

How to submit a job to the batch queue

With the resource requirements specified in the job submission script (sub.sh), submit your job with

sbatch <scriptname>

For example

sbatch sub.sh

Once the job is submitted, the Job ID of the job (e.g. 12345) will be printed on the screen.

---

Back to Top

Discovering if the queue is busy

To be added.

---

Back to Top

How to open an interactive session

An interactive session on a compute node can be started with the command

qlogin

This command will start an interactive session with one core on one of the interactive nodes. The qlogin command is an alias for

To be added.

so it can be used to start an interactive session on a node with feature inter and with a walltime of 12h.

---

Back to Top

How to run an interactive job with Graphical User Interface capabilities

To be added.

---

Back to Top

How to check on running or pending jobs

To list all running and pending jobs (by all users), use the command

squeue

or

squeue -l

For detailed information on how to monitor your jobs, please see Monitoring Jobs on the teaching cluster.

---

Back to Top

How to delete a running or pending job

To delete one of your running or pending job, use the command

scancel <jobid>

For example, to delete a job with Job ID 12345 use

scancel 12345

---

Back to Top

How to check resource utilization of a finished job

To be added.

---

Back to Top