Research Computing Center Wiki - User contributions [en]

GPU

2024-12-13T02:39:03Z

Jerky: Removed link to pgroup.com which led to HPC SDK. It had the only remaining occurrence of 'pgroup' in the Wiki, so I considered it obsolete.

[[Category:Sapelo2]]

==GPU Computing on Sapelo2==

===Hardware===
For a description of the Graphics Processing Units (GPU) device specifications, please see [[GPU Hardware]].

The following table summarizes the GPU devices available on sapelo2:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Number of nodes
! scope="col" | CPU cores per node
! scope="col" | Host memory per node
! scope="col" | CPU processor
! scope="col" | GPU model
! scope="col" | GPU devices per node
! scope="col" | Device memory
! scope="col" | GPU compute capability
! scope="col" | Minimum CUDA version
! scope="col" | Partition Name
! scope="col" | Notes
|-
|-
| 12 || 64 || 1TB || Intel Sapphire Rapids || H100 || 4 || 80GB || 9.0 || 11.8 || gpu_p, gpu_30d_p || Need to request --gres=gpu:H100, e.g.,
<nowiki>#</nowiki>SBATCH --partition=gpu_p

<nowiki>#</nowiki>SBATCH --gres=gpu:H100:1

<nowiki>#</nowiki>SBATCH --time=7-00:00:00
|-
| 14 || 64 || 1TB || AMD Milan || A100 || 4 || 80GB || 8.0 || 11.0 || gpu_p, gpu_30d_p || Need to request --gres=gpu:A100, e.g.,
<nowiki>#</nowiki>SBATCH --partition=gpu_p

<nowiki>#</nowiki>SBATCH --gres=gpu:A100:1

<nowiki>#</nowiki>SBATCH --time=7-00:00:00
|-
| 12 || 128 || 745GB || AMD Genoa || L4 || 4 || 24GB || 8.9 || 11.8 || gpu_p, gpu_30d_p || Need to request --gres=gpu:L4, e.g.,
<nowiki>#</nowiki>SBATCH --partition=gpu_p

<nowiki>#</nowiki>SBATCH --gres=gpu:L4:1

<nowiki>#</nowiki>SBATCH --time=7-00:00:00
|-
| 2 || 32 || 192GB || Intel Skylake || P100 || 1 || 16GB || 6.0 || 8.0|| gpu_p, gpu_30d_p ||Need to request --gres=gpu:P100, e.g.,
<nowiki>#</nowiki>SBATCH --partition=gpu_p

<nowiki>#</nowiki>SBATCH --gres=gpu:P100:1

<nowiki>#</nowiki>SBATCH --time=7-00:00:00
|-
| 1 || 64 || 1TB || AMD Milan || A100 || 4 || 80GB || 8.0 || 11.0 || buyin partition || rowspan="8" | Available on '''batch''' for all users up to '''4 hours''', e.g.,
<nowiki>#</nowiki>SBATCH --partition=batch

<nowiki>#</nowiki>SBATCH --gres=gpu:A100:1 or

<nowiki>#</nowiki>SBATCH --gres=gpu:L4:1 or

<nowiki>#</nowiki>SBATCH --gres=gpu:V100:1 or

<nowiki>#</nowiki>SBATCH --gres=gpu:V100S:1

<nowiki>#</nowiki>SBATCH --time=4:00:00
|-
| 2 || 64 || 745GB || AMD Genoa || L4 || 4 || 24GB || 8.9 || 11.8 || buyin partition
|-
| 2 || 28 || 192GB || Intel Skylake || V100 || 1 || 16GB || 7.0|| 9.0 || buyin partition
|-
| 2 || 32 || 192GB || Intel Skylake || V100 || 1 || 16GB || 7.0 || 9.0 || buyin partition
|-
| 2 || 32 || 384GB || Intel Skylake || V100 || 1 || 32GB || 7.0 || 9.0 || buyin partition
|-
| 2 || 64 || 128GB || AMD Naples || V100 || 2 || 32GB || 7.0 || 9.0 || buyin partition
|-
| 1 || 64 || 128GB || AMD Naples || V100 || 1 || 32GB || 7.0 || 9.0 || buyin partition
|-
| 4 || 64 || 128GB || AMD Rome || V100S || 1 || 32GB || 7.0 || 9.0 || buyin partition
|-
|}

'''Note:'''

1. The GPU compute capability of a device, also sometimes called its “SM version”, identifies the features supported by the GPU hardware. For more information, please see [https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capability NVIDIA compute capability]

===Software===
Sapelo2 has several tools for GPU programming and many CUDA-enabled applications. For example:

'''1. NVIDIA CUDA toolkit'''

Several versions of the CUDA toolkit are available. Please see our [[CUDA-Sapelo2|CUDA]] page.


'''2. cuDNN'''

The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks.

To see all modules of cuDNN installed on Sapelo2, please use the command

<pre class="gcommand">
ml spider cuDNN
</pre>

'''3. NCCL'''

The NVIDIA Collective Communications Library (NCCL) implements multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs.

To see all modules of cuDNN installed on Sapelo2, please use the command

<pre class="gcommand">
ml spider NCCL
</pre>

'''4. OpenACC'''

Using the NVIDIA HPC SDK compiler suite, provided by the NVHPC module on Sapelo2, programmers can accelerate applications on x64+accelerator platforms by adding OpenACC compiler directives to Fortran and C programs and then recompiling with appropriate compiler options. Please see https://developer.nvidia.com/hpc-sdk .

OpenACC is also supported by GNU compilers, especially the latest versions, e.g. GNU 7.2.0, installed on Sapelo2. For more information on OpenACC support by GNU compilers, please refer to https://gcc.gnu.org/wiki/OpenACC

For information on versions of compilers installed on Sapelo2, please see [[Code Compilation on Sapelo2]].

'''5. CUDA-enabled applications'''

CUDA-enabled applications typically have a version suffix in the module name to indicate the version of CUDA that they were built with.

==== Software modules that are supported on the H100 and L4 nodes ====
New modules that are being installed centrally using CUDA versions 12.1.1 or higher will include support for GPU compute capability up to 9.0. Some examples are:

* PyTorch/2.1.2-foss-2023a-CUDA-12.1.1 (note that this version uses the foss-2023a toolchain)

* GROMACS/2023.3-foss-2023a-CUDA-12.1.1-PLUMED-2.9.0

* GROMACS/2023.4-foss-2023a-CUDA-12.1.1

* magma/2.7.2-foss-2023a-CUDA-12.1.1

* NCCL/2.18.3-GCCcore-12.3.0-CUDA-12.1.1

* torchvision/0.16.2-foss-2023a-CUDA-12.1.1

====Software modules not supported on the H100 and L4 nodes====

Some modules that use CUDA 12.1.1 were installed before the H100 and L4 devices were added to the cluster and these only have support for GPU compute capability up to 8.0. Modules that do not work on the H100 and the L4 nodes include:

* module that use CUDA versions below 11.8.0

* PyTorch/2.1.2-foss-2022a-CUDA-12.1.1.lua (note that this version uses the foss-2022a toolchain)

* transformers/4.41.2-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* transformers/4.37.0-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* controlnet-aux/0.0.7-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* diffusers/0.25.1-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* torchvision/0.16.2-foss-2022a-CUDA-12.1.1.lua

* bitsandbytes/0.42.0-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* accelerate/0.26.1-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* timm/0.9.12-foss-2022a-CUDA-12.1.1.lua

* flash-attn/2.5.9.post1-foss-2022a-PyTorch-2.1.2-CUDA-12.1.1.lua

* magma/2.7.2-foss-2022a-CUDA-12.1.1

* NCCL/2.18.3-GCCcore-11.3.0-CUDA-12.1.1

===Running Jobs===
For information on how to run GPU jobs on Sapelo2, please refer to [[Running Jobs on Sapelo2]].

'''Important notes:'''

1. If a job requests
<pre class="gscript">
#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:1
</pre>

then it can get allocated any GPU device type (i.e. P100, A100, L4, or H100). If you opt for requesting a GPU device without specifying its type, please make sure that the application or code you are running works on all device types.

2. If the application that you are running uses an older version of CUDA, for example CUDA/11.4.1 or CUDA/11.7.0, please request an explicit GPU device that supports the CUDA version. For example, request an A100 device with
<pre class="gscript">
#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:A100:1
</pre>

3. If the application or code that you are running does not need double precision operations, and it does not need over 24GB of GPU device memory, you could get a faster job throughput by running it on an L4 device, which can be requested with
<pre class="gscript">
#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:L4:1
</pre>

4. If the application or code that you are running is supported by the H100 device, you can request an H100 device with
<pre class="gscript">
#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:H100:1
</pre>

5. If the application or code that you are running works and an A100 and an H100 device, and you would like the job to be allocated to a node equipped with either an A100 or H100, you can use these header lines for your job
<pre class="gscript">
#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:1
#SBATCH --constraint="SapphireRapids|Milan"
</pre>
By not specifying a specific GPU model when requesting a generic resource ("--gres"), you allow SLURM to allocate your job to any node on the GPU partition (assuming "--partition gpu_p" is used) with available resources. The GPU partition has nodes with GPUs other than A100s and H100s, so to prevent the job from running on a node with another GPU model (i.e. an L4 or P100), restrict your job to run only on nodes with either a Milan or SapphireRapids processor. In the GPU partition, only nodes with an A100 or H100 GPU have Milan or SapphireRapids processors, respectively.

Training

2024-09-18T21:03:44Z

Jerky:

==GACRC Training==

The GACRC regularly hosts training sessions on a number of subjects relevant to the use of our computational and storage resources. Scheduled trainings will be announced through the GACRC mailing list.

'''NOTE: New users are required to attend a Sapelo2 cluster introductory training session and information about that will be sent once an account is requested.'''

==Regular Training Announcement==
In '''September 2024''', the GACRC is hosting 7 training sessions (3 Linux basics and 3 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (3 sessions)

2. Sapelo2 cluster new user training (3 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session)

In '''October 2024''', the GACRC is hosting 8 training sessions (4 Linux basics and 4 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (4 sessions)

2. Sapelo2 cluster new user training (4 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session)

'''Please Note:''' The training workshops will be offered remotely via Zoom Meeting. Detailed information on how to join the Zoom meeting will be sent to your UGA email account prior to each training session.

==Event Schedule==

===Sapelo2 Cluster New User Training===

Our Sapelo2 training consists of 1 hr 30 mins of instructional videos, followed by a 1 hr 30 min workshop. '''The instructional videos are required to be viewed prior to the training workshop, and they can be found [https://kaltura.uga.edu/playlist/dedicated/176125031/1_4o12v8b4/1_9zi68rgi here]'''

'''Prerequisites:'''

* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.

'''Video Playlist Training Goals:'''

* Understand the layout of Sapelo2

* Understand the Sapelo2 file systems

* Understand the Sapelo2 partitions

* Understand the Sapelo2 software environment

'''Workshop Training Goals:'''

* Understand how to request computing resources and submit a computational batch job following the Sapelo2 cluster general workflow

* Understand how to initiate an interactive job

* Understand how to transfer files to and from the cluster

* Understand how to get support from GACRC support team when you have any issues on cluster

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC
|September 18th, Wednesday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 3rd, Thursday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 11th, Friday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 17th, Thursday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 23rd, Wednesday, 2:00 PM - 4:00 PM
|}

===Linux Training for Linux-inexperienced Cluster New Users===
The Sapelo2 High Performance Computing (HPC) cluster runs a headless Linux distribution as the operating system on each of its constituent nodes. The term headless refers to the fact that these nodes do not have a desktop graphical user interface (GUI) installed by default. Graphical desktop environments consume resources that analyses could otherwise use, so users employ a command-line interface (CLI) instead. To interact with these resources, users connect to a remote terminal via SSH and execute commands.

The Linux Training workshop provides hands-on practice of the fundamental Linux commands necessary to interact with HPC resources.

'''Prerequisite:''' Please watch the introductory videos on Linux, basic Linux terms, and Linux Paths and Directories (total ~17 minutes) '''before attending the training workshop'''.

* [https://kaltura.uga.edu/media/t/1_81u2kfi2/176125031 Linux]
* [https://kaltura.uga.edu/media/t/1_ol51cuyn/176125031 basic Linux terms]
* [https://kaltura.uga.edu/media/t/1_wdyxhgdg/176125031 Linux Paths and Directories]

'''Training Goals:'''

1. Understand fundamental concepts of Linux working environment (filesystem hierarchy, path, PATH, etc.)

2. Know how to use Linux common commands (ls, cd, pwd, cat, more, nano, mkdir, rm, cp, mv, etc.)

3. Understand what is Linux bash shell and know how to make a simple Linux script and run it in Linux environment

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Use Linux on Cluster
|October 1st, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 9th, Wednesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 15th, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 21st, Monday, 1:00 PM - 3:00 PM
|}

===Using Sapelo2 Cluster at the GACRC, Part II===
'''Prerequisites:'''
* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.
* Sapelo2 cluster new user training. Fundamental HPC and Sapelo2 knowledge is required for this advanced Sapelo2 workshop.

'''Training Goals:'''

1. Learn about high-performance computing framework

2. Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

3. How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

4. How can I make my job run more efficiently (through the correct use of software and hardware)?

5. A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC, Part II||September 27th, Friday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC, Part II
|October 25th, Friday, 2:00 PM - 4:00 PM
|}

===Python Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand Python scientific modules and distributions

2. Understand Python general lexical conventions; Python built-in data types, like string, list, tuple, dictionary, etc.

3. Understand Python programming structures and procedural programming using functions

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Python Basics I||Not scheduled
|-
|Python Basics II||Not scheduled
|-
|}

===R Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of R language, e.g. R general lexical conventions, data types, functions, and packages. Part 2 will introduce loops and functions.

2. Be able to manipulate and create data frames using built in functions and the dplyr package.

3. Interact with your file system and submit R code as a batch job to Sapelo 2.

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
| R Basics I||Not scheduled
|-
|R Basics II||Not scheduled
|-
|}

===Conda===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of conda environment

2. Use conda to create and configure your own virtual environments

3. Activate your environments to run python apps from your home directory on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Conda Basics||Not scheduled
|-
|}

==How to Register==

'''Please Note''', the training workshops '''Using Sapelo2 Cluster at the GACRC''' and '''Use Linux on Cluster''' are '''ONLY''' offered to '''new users''' who need computing user accounts on the GACRC Sapelo2 cluster, or any current users who have never attended the GACRC Sapelo2 cluster new user training before. Please ask your group PI/UGA faculty member to send us a request for you, using the GACRC User Account Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25839

If you want to attend '''Python Basics''', '''R''', and '''Conda basics''' training sessions, please send us a request using the GACRC Training Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25852 . In your request, please tell us which session(s) you want to attend.

The GACRC is going to host other training workshops and seminars covering various HPC topics, including HPC fundamental introduction, Linux introductory III (Linux working environment and utilities), Bioinfomatics applications on Sapelo cluster, Perl, R, C/C++/Fortran programming, etc., in the near future. We will announce those events when they are scheduled.

The GACRC Web Training page can be found at https://gacrc.uga.edu/training/ and the GACRC Wiki Training page can be found at https://wiki.gacrc.uga.edu/wiki/Training, from which you can find detailed information about upcoming and past training sessions from GACRC and download training materials.

== Topic Introduction==

Title: '''Sap2test cluster migration training'''

Focus: Slurm queueing system, including Slurm job commands, job environment variables, and job submission headers, etc.

The new software environment on Sap2test

Other important topics related to Sap2test working environment

Title: '''Using Sapelo2 Cluster at the GACRC'''

Focus: Sapelo2 HPC cluster and computational batch job submission workflow

Cluster's storage environment

Computational queues on cluster

Software environment

How to submit computational batch jobs

Other tips and guidelines for users

Title: '''Using Sapelo2 Cluster at the GACRC, Part II'''

Focus: More topics on how to use Sapelo2 cluster

Learn about high-performance computing framework

Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

How can I make my job run more efficiently (through the correct use of software and hardware)?

A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

Title: '''Use Linux on Cluster'''

Focus: Linux OS fundamentals

Linux common commands, filesystem, and shell

Linux shell scripting basics

Common Linux utilities, e.g., grep, sed, find, sort, and awk, etc.

Linux Hands-on practice

Title: '''Python Basics I, II'''

Focus of I: Python language overview, scientific modules and distributions

Python general lexical conventions

Basic built-in data types, like string, list, tuple, dictionary, etc.

Focus of II: Programming structures: control flow and loop

Function: procedural programming with examples, lambda expression, factory function and generator

Title: '''R Basics I, II'''

Focus of I: R language overview,general lexical conventions, data types, functions, and packages.

Basic built-in data types, like string, numeric, list, dataframe etc. Using the dplyr package.

Focus of II: Programming structures: control flow, loops and functions

Title: '''Python on GACRC Sapelo2 Cluster'''

Focus: Install Python packages/modules in a user's home directory on Sapelo2 cluster

Python versions installed on Sapelo2

Python environment details on Sapelo2

How to know a Python package is installed or not on Sapelo2

How to install a Python package in user's home directory on Sapelo2

Title: '''Do It Yourself: Using Conda to create and run python environments to suit your computing needs effortlessly!'''

Focus: Use conda to create and configure your own python virtual environments; Activate your environments to run python apps from your home directory on Sapelo2

What is Conda and its environment

Conda on Sapelo2

Use conda to create and configure your own python virtual environments

Activate your environments to run python apps from your home directory on Sapelo2

Title: '''How to submit and run jobs efficiently and correctly on Sapelo2'''

Focus: Sapelo2 cluster general workflow and correct computing resource requesting

Overview of Sapelo2 cluster with reference tables and operational diagrams

Sapelo2 batch job submission workflow taking global scratch as job working space

How to request computing resources correctly

How to run pipeline tasks and what are advantages/disadvantages of different options

Sapelo2 cluster guideline and practical tips

Title: '''GACRC Storage Environment'''

Focus: Overview of Linux common commands related to file and folder operations

Overview of the storage enviornment of zcluster and Sapelo cluster at GACRC

How to transfer data between local and GACRC storage

New file transfer node xfer2 and how to use it to transfer data between zcluster and the new cluster

GACRC suggestions on good practices on GACRC storage, etc;

Title: '''NCBI Blast application on sapelo'''

Focus: Introduction to BLAST

BLAST job submission to sapelo

Advantages & Disadvantages: NCBI website vs run at sapelo.

Understand BLAST output

Troubleshooting the BLAST results

Title: '''NGS application overview at GACRC'''

Focus: Overview of Bioinformatics software available on HPC clusters at GACRC

It’s a brave new world – NGS and its Applications

Hardware, Software, Databases available at GACRC

NGS project: Logistics and resource considerations

Best practices, common mistakes, troubleshooting and getting help from GACRC

Title: '''Perl Language Basics I, II'''

Focus of I: Overview of Perl language,

Perl general scripting style

Perl fundamental data types

Focus of II: Program structure: control flow and loop

Perl subroutine

Perl I/O

==Download==

====Sapelo2 Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|[[Media:GACRC_Sapelo2_cluster_new_user_training_workshop_v10.8.pdf]]
|-
|}

====Sap2test Migration Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Migrating_to_Slurm_and_new_software_environment.pdf]]
|}

'''Please note:''' To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the '''GACRC's Kaltura channel''' at
https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required).

==== Teaching Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:GACRC-Teaching-cluster-new-user-training-workshop_Fall2024.pdf]]
|-
|}

====Linux Training for New Cluster Users====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Linux_Training_For_New_Users_Of_Cluster_Suchi_04252019.pdf]]
|-
|}

==== Python Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Python_Language_Basics_I_v5.1.pdf]]
|-
|[[Media:Python_Language_Basics_II_v5.1.pdf]]
|-
|[[Media:Python_Basics_v6.1.pdf]]
|-
|}

====R Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:R Language Basics PowerPoint v2.0.1.pdf|Media:R_Language_Basics_PowerPoint_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_Document_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Powerpoint_v1.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Document_v1.0.pdf]]
|-
|}

====Perl Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Perl_Language_Basics_I_Workshop_v1.pdf]]
|-
|}

====Topical Sessions====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|
|-
| [[Media:AI_Resources_on_the_GACRC_Sapelo2_Cluster.pdf]]
|-
| [[Media:Using_Sapelo2_Cluster_at_the_GACRC_Part_II_Rocky8.pdf]]
|-
| [[Media:Using_Conda_on_the_GACRC_Sap2test_cluster_v1.pdf]]
|-
|[[Media:Blast_Workshop_GACRC_02012017.pdf]]
|-
|[[Media:Next-Generation_Sequencing_Applications_at_GACRC_10282016.pdf]]
|-
|}

==== Out-Reach/On-Class Talk====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Dept./Center/Institute
! scope="col" | Type
! scope="col" |Workshop PDF
|-
|CSP seminar - Fall 2024||Out-Reach||[[Media:GACRC_overview_20240820-CSP.pdf]]
|-
| BCMB8330 - Spring2024||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330_Spring2024.pdf]]
|-
|PHYS4601/6601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys4601.pdf]]
|-
|PHYS8601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys8601.pdf]]
|-
|CSP seminar - Fall 2023||Out-Reach||[[Media:GACRC_overview_20230822-CSP.pdf]]
|-
|BCMB8330 - Spring2023||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330.pdf]]
|-
|PHYS4601/6601 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2023_phys4601.pdf]]
|-
|PHYS8602 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] ; [[Media:Gacrc_handout2023_phys8602.pdf]]
|-
|ILS GradFIRST course - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220901-ILS.pdf]]
|-
|FYOS1001 - Fall 2022||Out-Reach||[[Media:High_Performance_Computing_(HPC)_on_GACRC_Sapelo2_Cluster.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220830-CSP.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:Compile_and_Run_HPC_code_on_Sapelo2.pdf]]
|-
|Terry College IT - Spring2022||Out-Reach||[[Media:GACRC_overview_20220506-Terry.pdf]]
|-
|PHYS8601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]]
|-
|PHYS4601/6601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2021_phys4601.pdf]]
|-
|PHYS8602 - Spring2021||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602-2021.pdf]] ; [[Media:Gacrc_handout2021_phys8602.pdf]]
|-
|GENE4220 - Fall2020||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_GENE4220_Fall2020.pdf]]
|-
|College of Veterinary Medicine - Spring2020||Out-Reach (jlslab) ||[[Media:Using_GACRC_Sapelo2_Cluster-Advanced_Topics(1).pdf]]
|-
|Byod Data Center - Fall2019||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Cluster.pdf]]
|-
|Department of Linguistics - Fall2019||On-class (LING6570)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_LING6570_Part2.pdf]]
|-
|The Center for Simulational Physics - Fall2019 ||Out-Reach (Seminar Talk 20190820)||[[Media:Introduction_to_GACRC_Computing_Facility_-_Sapelo2_Cluster_CSP-Fall2019.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS4601/6601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] [[Media:Gacrc_handout2019_phys4601.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS8601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]] [[Media:Gacrc_handout2020_phys8601.pdf]]
|-
|The Center for Simulational Physics ||On-Class (PHYS8602)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] [[Media:Gacrc_handout2019_phys8602.pdf]]
|-
|Food Science - Fall2018 ||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Sapelo2_Cluster_at_GACRC.pdf]]
|-
|The Center for Simulational Physics - Summer2018||Out-Reach (Seminar Talk 20180821)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
| Miller plant science - Summer2018 ||Out-Reach (jlmlab)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
|Biochemistry and Molecular Biology - Spring2018||On-Class (BCMB8330)||[[Media:GACRC_zcluster_Class_Training_BCMB8330_Spring_2018.pdf]]
|-
| The Center for Simulational Physics - Summer2017||Out-Reach (Seminar Talk 20170831) ||[[Media:Introduction_on_HPC_Resources_at_the_GACRC.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS4601/6601) ||[[Media:Phys4601.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS8602)||[[Media:Phys8602.pdf]]
|-
|The Institute of Bioinformatics and the Quantitative Biology Consulting Group|| Out-Reach||[[Media:Introduction_to_HPC_Resources_at_GACRC_BBB_Talk_20151014.pdf]]
|-
|The Center for Simulational Physics|| Out-Reach (Seminar Talk 20160906)||[[Media:Introduction_to_Sapelo_Computing_Resources_at_GACRC_Workshop20160906.pdf]]
|-
|Microbiology||On-Class (MIBO8150) ||[[Media:Introduction_to_HPC_Resources_at_GACRC_MIBO8150_20160926.pdf]]
|-
|Statistics||On-Class (STAT8060)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_Workshop_STAT8060_20150826.pdf]]
|-
|Biochemistry and Molecular Biology||On-Class (BCMB8211) ||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BCMB8211_20160114.pdf]]
|-
|Plant Biology||On-Class (PBIO/BINF8350)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO-BINF8350_20160115.pdf]]
|-
|Plant Biology - Bioinformatics Applications Fall2016||On-Class (PBIO4550)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO_4550_08182016.pdf]]
|-
| Bioinformatics - Essential Computing Skills for Biologists Fall2016||On-Class (BINF4005)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BINF_4005_08312016.pdf]]
|-
|Computers in Experimental Genetics Fall2016||On-Class (GENE4220)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_GENE_4220_10192016.pdf]]
|-
|Statistics - Advanced Applications and Computing in R Fall2016||On-Class (STAT8330)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_STAT8330_11022016.pdf]]
|}

'''NOTE:''' The slides may become outdated and you should always check GACRC Wiki for up to date information.

== Past Sessions ==

[[Pass Sessions in 2021]]

[[Past Sessions in 2020]]

[[Past Sessions in 2019]]

[[Past Sessions in 2018]]

[[Past Sessions in 2017]]

[[Past Sessions in 2016]]

[[Past Sessions in 2015]]

Training

2024-09-18T20:58:37Z

Jerky:

==GACRC Training==

The GACRC regularly hosts training sessions on a number of subjects relevant to the use of our computational and storage resources. Scheduled trainings will be announced through the GACRC mailing list.

'''NOTE: New users are required to attend a Sapelo2 cluster introductory training session and information about that will be sent once an account is requested.'''

==Regular Training Announcement==
In '''September 2024''', the GACRC is hosting 7 training sessions (3 Linux basics and 3 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (3 sessions)

2. Sapelo2 cluster new user training (3 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session) (here's a [[Matlab|link to wiki's Matlab page,]]

from PB)

In '''October 2024''', the GACRC is hosting 8 training sessions (4 Linux basics and 4 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (4 sessions)

2. Sapelo2 cluster new user training (4 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session)

'''Please Note:''' The training workshops will be offered remotely via Zoom Meeting. Detailed information on how to join the Zoom meeting will be sent to your UGA email account prior to each training session.

==Event Schedule==

===Sapelo2 Cluster New User Training===

Our Sapelo2 training consists of 1 hr 30 mins of instructional videos, followed by a 1 hr 30 min workshop. '''The instructional videos are required to be viewed prior to the training workshop, and they can be found [https://kaltura.uga.edu/playlist/dedicated/176125031/1_4o12v8b4/1_9zi68rgi here]'''

'''Prerequisites:'''

* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.

'''Video Playlist Training Goals:'''

* Understand the layout of Sapelo2

* Understand the Sapelo2 file systems

* Understand the Sapelo2 partitions

* Understand the Sapelo2 software environment

'''Workshop Training Goals:'''

* Understand how to request computing resources and submit a computational batch job following the Sapelo2 cluster general workflow

* Understand how to initiate an interactive job

* Understand how to transfer files to and from the cluster

* Understand how to get support from GACRC support team when you have any issues on cluster

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC
|September 18th, Wednesday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 3rd, Thursday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 11th, Friday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 17th, Thursday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 23rd, Wednesday, 2:00 PM - 4:00 PM
|}

===Linux Training for Linux-inexperienced Cluster New Users===
The Sapelo2 High Performance Computing (HPC) cluster runs a headless Linux distribution as the operating system on each of its constituent nodes. The term headless refers to the fact that these nodes do not have a desktop graphical user interface (GUI) installed by default. Graphical desktop environments consume resources that analyses could otherwise use, so users employ a command-line interface (CLI) instead. To interact with these resources, users connect to a remote terminal via SSH and execute commands.

The Linux Training workshop provides hands-on practice of the fundamental Linux commands necessary to interact with HPC resources.

'''Prerequisite:''' Please watch the introductory videos on Linux, basic Linux terms, and Linux Paths and Directories (total ~17 minutes) '''before attending the training workshop'''.

* [https://kaltura.uga.edu/media/t/1_81u2kfi2/176125031 Linux]
* [https://kaltura.uga.edu/media/t/1_ol51cuyn/176125031 basic Linux terms]
* [https://kaltura.uga.edu/media/t/1_wdyxhgdg/176125031 Linux Paths and Directories]

'''Training Goals:'''

1. Understand fundamental concepts of Linux working environment (filesystem hierarchy, path, PATH, etc.)

2. Know how to use Linux common commands (ls, cd, pwd, cat, more, nano, mkdir, rm, cp, mv, etc.)

3. Understand what is Linux bash shell and know how to make a simple Linux script and run it in Linux environment

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Use Linux on Cluster
|October 1st, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 9th, Wednesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 15th, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 21st, Monday, 1:00 PM - 3:00 PM
|}

===Using Sapelo2 Cluster at the GACRC, Part II===
'''Prerequisites:'''
* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.
* Sapelo2 cluster new user training. Fundamental HPC and Sapelo2 knowledge is required for this advanced Sapelo2 workshop.

'''Training Goals:'''

1. Learn about high-performance computing framework

2. Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

3. How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

4. How can I make my job run more efficiently (through the correct use of software and hardware)?

5. A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC, Part II||September 27th, Friday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC, Part II
|October 25th, Friday, 2:00 PM - 4:00 PM
|}

===Python Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand Python scientific modules and distributions

2. Understand Python general lexical conventions; Python built-in data types, like string, list, tuple, dictionary, etc.

3. Understand Python programming structures and procedural programming using functions

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Python Basics I||Not scheduled
|-
|Python Basics II||Not scheduled
|-
|}

===R Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of R language, e.g. R general lexical conventions, data types, functions, and packages. Part 2 will introduce loops and functions.

2. Be able to manipulate and create data frames using built in functions and the dplyr package.

3. Interact with your file system and submit R code as a batch job to Sapelo 2.

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
| R Basics I||Not scheduled
|-
|R Basics II||Not scheduled
|-
|}

===Conda===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of conda environment

2. Use conda to create and configure your own virtual environments

3. Activate your environments to run python apps from your home directory on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Conda Basics||Not scheduled
|-
|}

==How to Register==

'''Please Note''', the training workshops '''Using Sapelo2 Cluster at the GACRC''' and '''Use Linux on Cluster''' are '''ONLY''' offered to '''new users''' who need computing user accounts on the GACRC Sapelo2 cluster, or any current users who have never attended the GACRC Sapelo2 cluster new user training before. Please ask your group PI/UGA faculty member to send us a request for you, using the GACRC User Account Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25839

If you want to attend '''Python Basics''', '''R''', and '''Conda basics''' training sessions, please send us a request using the GACRC Training Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25852 . In your request, please tell us which session(s) you want to attend.

The GACRC is going to host other training workshops and seminars covering various HPC topics, including HPC fundamental introduction, Linux introductory III (Linux working environment and utilities), Bioinfomatics applications on Sapelo cluster, Perl, R, C/C++/Fortran programming, etc., in the near future. We will announce those events when they are scheduled.

The GACRC Web Training page can be found at https://gacrc.uga.edu/training/ and the GACRC Wiki Training page can be found at https://wiki.gacrc.uga.edu/wiki/Training, from which you can find detailed information about upcoming and past training sessions from GACRC and download training materials.

== Topic Introduction==

Title: '''Sap2test cluster migration training'''

Focus: Slurm queueing system, including Slurm job commands, job environment variables, and job submission headers, etc.

The new software environment on Sap2test

Other important topics related to Sap2test working environment

Title: '''Using Sapelo2 Cluster at the GACRC'''

Focus: Sapelo2 HPC cluster and computational batch job submission workflow

Cluster's storage environment

Computational queues on cluster

Software environment

How to submit computational batch jobs

Other tips and guidelines for users

Title: '''Using Sapelo2 Cluster at the GACRC, Part II'''

Focus: More topics on how to use Sapelo2 cluster

Learn about high-performance computing framework

Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

How can I make my job run more efficiently (through the correct use of software and hardware)?

A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

Title: '''Use Linux on Cluster'''

Focus: Linux OS fundamentals

Linux common commands, filesystem, and shell

Linux shell scripting basics

Common Linux utilities, e.g., grep, sed, find, sort, and awk, etc.

Linux Hands-on practice

Title: '''Python Basics I, II'''

Focus of I: Python language overview, scientific modules and distributions

Python general lexical conventions

Basic built-in data types, like string, list, tuple, dictionary, etc.

Focus of II: Programming structures: control flow and loop

Function: procedural programming with examples, lambda expression, factory function and generator

Title: '''R Basics I, II'''

Focus of I: R language overview,general lexical conventions, data types, functions, and packages.

Basic built-in data types, like string, numeric, list, dataframe etc. Using the dplyr package.

Focus of II: Programming structures: control flow, loops and functions

Title: '''Python on GACRC Sapelo2 Cluster'''

Focus: Install Python packages/modules in a user's home directory on Sapelo2 cluster

Python versions installed on Sapelo2

Python environment details on Sapelo2

How to know a Python package is installed or not on Sapelo2

How to install a Python package in user's home directory on Sapelo2

Title: '''Do It Yourself: Using Conda to create and run python environments to suit your computing needs effortlessly!'''

Focus: Use conda to create and configure your own python virtual environments; Activate your environments to run python apps from your home directory on Sapelo2

What is Conda and its environment

Conda on Sapelo2

Use conda to create and configure your own python virtual environments

Activate your environments to run python apps from your home directory on Sapelo2

Title: '''How to submit and run jobs efficiently and correctly on Sapelo2'''

Focus: Sapelo2 cluster general workflow and correct computing resource requesting

Overview of Sapelo2 cluster with reference tables and operational diagrams

Sapelo2 batch job submission workflow taking global scratch as job working space

How to request computing resources correctly

How to run pipeline tasks and what are advantages/disadvantages of different options

Sapelo2 cluster guideline and practical tips

Title: '''GACRC Storage Environment'''

Focus: Overview of Linux common commands related to file and folder operations

Overview of the storage enviornment of zcluster and Sapelo cluster at GACRC

How to transfer data between local and GACRC storage

New file transfer node xfer2 and how to use it to transfer data between zcluster and the new cluster

GACRC suggestions on good practices on GACRC storage, etc;

Title: '''NCBI Blast application on sapelo'''

Focus: Introduction to BLAST

BLAST job submission to sapelo

Advantages & Disadvantages: NCBI website vs run at sapelo.

Understand BLAST output

Troubleshooting the BLAST results

Title: '''NGS application overview at GACRC'''

Focus: Overview of Bioinformatics software available on HPC clusters at GACRC

It’s a brave new world – NGS and its Applications

Hardware, Software, Databases available at GACRC

NGS project: Logistics and resource considerations

Best practices, common mistakes, troubleshooting and getting help from GACRC

Title: '''Perl Language Basics I, II'''

Focus of I: Overview of Perl language,

Perl general scripting style

Perl fundamental data types

Focus of II: Program structure: control flow and loop

Perl subroutine

Perl I/O

==Download==

====Sapelo2 Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|[[Media:GACRC_Sapelo2_cluster_new_user_training_workshop_v10.8.pdf]]
|-
|}

====Sap2test Migration Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Migrating_to_Slurm_and_new_software_environment.pdf]]
|}

'''Please note:''' To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the '''GACRC's Kaltura channel''' at
https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required).

==== Teaching Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:GACRC-Teaching-cluster-new-user-training-workshop_Fall2024.pdf]]
|-
|}

====Linux Training for New Cluster Users====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Linux_Training_For_New_Users_Of_Cluster_Suchi_04252019.pdf]]
|-
|}

==== Python Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Python_Language_Basics_I_v5.1.pdf]]
|-
|[[Media:Python_Language_Basics_II_v5.1.pdf]]
|-
|[[Media:Python_Basics_v6.1.pdf]]
|-
|}

====R Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:R Language Basics PowerPoint v2.0.1.pdf|Media:R_Language_Basics_PowerPoint_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_Document_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Powerpoint_v1.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Document_v1.0.pdf]]
|-
|}

====Perl Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Perl_Language_Basics_I_Workshop_v1.pdf]]
|-
|}

====Topical Sessions====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|
|-
| [[Media:AI_Resources_on_the_GACRC_Sapelo2_Cluster.pdf]]
|-
| [[Media:Using_Sapelo2_Cluster_at_the_GACRC_Part_II_Rocky8.pdf]]
|-
| [[Media:Using_Conda_on_the_GACRC_Sap2test_cluster_v1.pdf]]
|-
|[[Media:Blast_Workshop_GACRC_02012017.pdf]]
|-
|[[Media:Next-Generation_Sequencing_Applications_at_GACRC_10282016.pdf]]
|-
|}

==== Out-Reach/On-Class Talk====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Dept./Center/Institute
! scope="col" | Type
! scope="col" |Workshop PDF
|-
|CSP seminar - Fall 2024||Out-Reach||[[Media:GACRC_overview_20240820-CSP.pdf]]
|-
| BCMB8330 - Spring2024||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330_Spring2024.pdf]]
|-
|PHYS4601/6601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys4601.pdf]]
|-
|PHYS8601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys8601.pdf]]
|-
|CSP seminar - Fall 2023||Out-Reach||[[Media:GACRC_overview_20230822-CSP.pdf]]
|-
|BCMB8330 - Spring2023||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330.pdf]]
|-
|PHYS4601/6601 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2023_phys4601.pdf]]
|-
|PHYS8602 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] ; [[Media:Gacrc_handout2023_phys8602.pdf]]
|-
|ILS GradFIRST course - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220901-ILS.pdf]]
|-
|FYOS1001 - Fall 2022||Out-Reach||[[Media:High_Performance_Computing_(HPC)_on_GACRC_Sapelo2_Cluster.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220830-CSP.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:Compile_and_Run_HPC_code_on_Sapelo2.pdf]]
|-
|Terry College IT - Spring2022||Out-Reach||[[Media:GACRC_overview_20220506-Terry.pdf]]
|-
|PHYS8601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]]
|-
|PHYS4601/6601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2021_phys4601.pdf]]
|-
|PHYS8602 - Spring2021||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602-2021.pdf]] ; [[Media:Gacrc_handout2021_phys8602.pdf]]
|-
|GENE4220 - Fall2020||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_GENE4220_Fall2020.pdf]]
|-
|College of Veterinary Medicine - Spring2020||Out-Reach (jlslab) ||[[Media:Using_GACRC_Sapelo2_Cluster-Advanced_Topics(1).pdf]]
|-
|Byod Data Center - Fall2019||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Cluster.pdf]]
|-
|Department of Linguistics - Fall2019||On-class (LING6570)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_LING6570_Part2.pdf]]
|-
|The Center for Simulational Physics - Fall2019 ||Out-Reach (Seminar Talk 20190820)||[[Media:Introduction_to_GACRC_Computing_Facility_-_Sapelo2_Cluster_CSP-Fall2019.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS4601/6601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] [[Media:Gacrc_handout2019_phys4601.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS8601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]] [[Media:Gacrc_handout2020_phys8601.pdf]]
|-
|The Center for Simulational Physics ||On-Class (PHYS8602)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] [[Media:Gacrc_handout2019_phys8602.pdf]]
|-
|Food Science - Fall2018 ||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Sapelo2_Cluster_at_GACRC.pdf]]
|-
|The Center for Simulational Physics - Summer2018||Out-Reach (Seminar Talk 20180821)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
| Miller plant science - Summer2018 ||Out-Reach (jlmlab)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
|Biochemistry and Molecular Biology - Spring2018||On-Class (BCMB8330)||[[Media:GACRC_zcluster_Class_Training_BCMB8330_Spring_2018.pdf]]
|-
| The Center for Simulational Physics - Summer2017||Out-Reach (Seminar Talk 20170831) ||[[Media:Introduction_on_HPC_Resources_at_the_GACRC.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS4601/6601) ||[[Media:Phys4601.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS8602)||[[Media:Phys8602.pdf]]
|-
|The Institute of Bioinformatics and the Quantitative Biology Consulting Group|| Out-Reach||[[Media:Introduction_to_HPC_Resources_at_GACRC_BBB_Talk_20151014.pdf]]
|-
|The Center for Simulational Physics|| Out-Reach (Seminar Talk 20160906)||[[Media:Introduction_to_Sapelo_Computing_Resources_at_GACRC_Workshop20160906.pdf]]
|-
|Microbiology||On-Class (MIBO8150) ||[[Media:Introduction_to_HPC_Resources_at_GACRC_MIBO8150_20160926.pdf]]
|-
|Statistics||On-Class (STAT8060)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_Workshop_STAT8060_20150826.pdf]]
|-
|Biochemistry and Molecular Biology||On-Class (BCMB8211) ||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BCMB8211_20160114.pdf]]
|-
|Plant Biology||On-Class (PBIO/BINF8350)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO-BINF8350_20160115.pdf]]
|-
|Plant Biology - Bioinformatics Applications Fall2016||On-Class (PBIO4550)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO_4550_08182016.pdf]]
|-
| Bioinformatics - Essential Computing Skills for Biologists Fall2016||On-Class (BINF4005)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BINF_4005_08312016.pdf]]
|-
|Computers in Experimental Genetics Fall2016||On-Class (GENE4220)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_GENE_4220_10192016.pdf]]
|-
|Statistics - Advanced Applications and Computing in R Fall2016||On-Class (STAT8330)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_STAT8330_11022016.pdf]]
|}

'''NOTE:''' The slides may become outdated and you should always check GACRC Wiki for up to date information.

== Past Sessions ==

[[Pass Sessions in 2021]]

[[Past Sessions in 2020]]

[[Past Sessions in 2019]]

[[Past Sessions in 2018]]

[[Past Sessions in 2017]]

[[Past Sessions in 2016]]

[[Past Sessions in 2015]]

Training

2024-09-18T20:25:47Z

Jerky:

==GACRC Training==

The GACRC regularly hosts training sessions on a number of subjects relevant to the use of our computational and storage resources. Scheduled trainings will be announced through the GACRC mailing list.

'''NOTE: New users are required to attend a Sapelo2 cluster introductory training session and information about that will be sent once an account is requested.'''

==Regular Training Announcement==
In '''September 2024''', the GACRC is hosting 7 training sessions (3 Linux basics and 3 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (3 sessions)

2. Sapelo2 cluster new user training (3 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session) (here's a [[Matlab|link to wiki's Matlab page,]]

from PB)

In '''October 2024''', the GACRC is hosting 8 training sessions (4 Linux basics and 4 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (4 sessions)

2. Sapelo2 cluster new user training (4 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session)

'''Please Note:''' The training workshops will be offered remotely via Zoom Meeting. Detailed information on how to join the Zoom meeting will be sent to your UGA email account prior to each training session.

==Event Schedule==

===Sapelo2 Cluster New User Training===

Our Sapelo2 training consists of 1 hr 30 mins of instructional videos, followed by a 1 hr 30 min workshop. '''The instructional videos are required to be viewed prior to the training workshop, and they can be found [https://kaltura.uga.edu/playlist/dedicated/176125031/1_4o12v8b4/1_9zi68rgi here]'''

'''Prerequisites:'''

* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.

'''Video Playlist Training Goals:'''

* Understand the layout of Sapelo2

* Understand the Sapelo2 file systems

* Understand the Sapelo2 partitions

* Understand the Sapelo2 software environment

'''Workshop Training Goals:'''

* Understand how to request computing resources and submit a computational batch job following the Sapelo2 cluster general workflow

* Understand how to initiate an interactive job

* Understand how to transfer files to and from the cluster

* Understand how to get support from GACRC support team when you have any issues on cluster

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC
|September 18th, Wednesday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 3rd, Thursday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 11th, Friday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 17th, Thursday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 23rd, Wednesday, 2:00 PM - 4:00 PM
|}

===Linux Training for Linux-inexperienced Cluster New Users===
The Sapelo2 High Performance Computing (HPC) cluster runs a headless Linux distribution as the operating system on each of its constituent nodes. The term headless refers to the fact that these nodes do not have a desktop graphical user interface (GUI) installed by default. Graphical desktop environments consume resources that analyses could otherwise use, so users employ a command-line interface (CLI) instead. To interact with these resources, users connect to a remote terminal via SSH and execute commands.

The Linux Training workshop provides hands-on practice of the fundamental Linux commands necessary to interact with HPC resources.

'''Prerequisite:''' Please watch the introductory videos on Linux, basic Linux terms, and Linux Paths and Directories (total ~17 minutes) '''before attending the training workshop'''.

'''Training Goals:'''

1. Understand fundamental concepts of Linux working environment (filesystem hierarchy, path, PATH, etc.)

2. Know how to use Linux common commands (ls, cd, pwd, cat, more, nano, mkdir, rm, cp, mv, etc.)

3. Understand what is Linux bash shell and know how to make a simple Linux script and run it in Linux environment

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Use Linux on Cluster
|October 1st, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 9th, Wednesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 15th, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 21st, Monday, 1:00 PM - 3:00 PM
|}

===Using Sapelo2 Cluster at the GACRC, Part II===
'''Prerequisites:'''
* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.
* Sapelo2 cluster new user training. Fundamental HPC and Sapelo2 knowledge is required for this advanced Sapelo2 workshop.

'''Training Goals:'''

1. Learn about high-performance computing framework

2. Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

3. How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

4. How can I make my job run more efficiently (through the correct use of software and hardware)?

5. A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC, Part II||September 27th, Friday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC, Part II
|October 25th, Friday, 2:00 PM - 4:00 PM
|}

===Python Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand Python scientific modules and distributions

2. Understand Python general lexical conventions; Python built-in data types, like string, list, tuple, dictionary, etc.

3. Understand Python programming structures and procedural programming using functions

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Python Basics I||Not scheduled
|-
|Python Basics II||Not scheduled
|-
|}

===R Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of R language, e.g. R general lexical conventions, data types, functions, and packages. Part 2 will introduce loops and functions.

2. Be able to manipulate and create data frames using built in functions and the dplyr package.

3. Interact with your file system and submit R code as a batch job to Sapelo 2.

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
| R Basics I||Not scheduled
|-
|R Basics II||Not scheduled
|-
|}

===Conda===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of conda environment

2. Use conda to create and configure your own virtual environments

3. Activate your environments to run python apps from your home directory on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Conda Basics||Not scheduled
|-
|}

==How to Register==

'''Please Note''', the training workshops '''Using Sapelo2 Cluster at the GACRC''' and '''Use Linux on Cluster''' are '''ONLY''' offered to '''new users''' who need computing user accounts on the GACRC Sapelo2 cluster, or any current users who have never attended the GACRC Sapelo2 cluster new user training before. Please ask your group PI/UGA faculty member to send us a request for you, using the GACRC User Account Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25839

If you want to attend '''Python Basics''', '''R''', and '''Conda basics''' training sessions, please send us a request using the GACRC Training Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25852 . In your request, please tell us which session(s) you want to attend.

The GACRC is going to host other training workshops and seminars covering various HPC topics, including HPC fundamental introduction, Linux introductory III (Linux working environment and utilities), Bioinfomatics applications on Sapelo cluster, Perl, R, C/C++/Fortran programming, etc., in the near future. We will announce those events when they are scheduled.

The GACRC Web Training page can be found at https://gacrc.uga.edu/training/ and the GACRC Wiki Training page can be found at https://wiki.gacrc.uga.edu/wiki/Training, from which you can find detailed information about upcoming and past training sessions from GACRC and download training materials.

== Topic Introduction==

Title: '''Sap2test cluster migration training'''

Focus: Slurm queueing system, including Slurm job commands, job environment variables, and job submission headers, etc.

The new software environment on Sap2test

Other important topics related to Sap2test working environment

Title: '''Using Sapelo2 Cluster at the GACRC'''

Focus: Sapelo2 HPC cluster and computational batch job submission workflow

Cluster's storage environment

Computational queues on cluster

Software environment

How to submit computational batch jobs

Other tips and guidelines for users

Title: '''Using Sapelo2 Cluster at the GACRC, Part II'''

Focus: More topics on how to use Sapelo2 cluster

Learn about high-performance computing framework

Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

How can I make my job run more efficiently (through the correct use of software and hardware)?

A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

Title: '''Use Linux on Cluster'''

Focus: Linux OS fundamentals

Linux common commands, filesystem, and shell

Linux shell scripting basics

Common Linux utilities, e.g., grep, sed, find, sort, and awk, etc.

Linux Hands-on practice

Title: '''Python Basics I, II'''

Focus of I: Python language overview, scientific modules and distributions

Python general lexical conventions

Basic built-in data types, like string, list, tuple, dictionary, etc.

Focus of II: Programming structures: control flow and loop

Function: procedural programming with examples, lambda expression, factory function and generator

Title: '''R Basics I, II'''

Focus of I: R language overview,general lexical conventions, data types, functions, and packages.

Basic built-in data types, like string, numeric, list, dataframe etc. Using the dplyr package.

Focus of II: Programming structures: control flow, loops and functions

Title: '''Python on GACRC Sapelo2 Cluster'''

Focus: Install Python packages/modules in a user's home directory on Sapelo2 cluster

Python versions installed on Sapelo2

Python environment details on Sapelo2

How to know a Python package is installed or not on Sapelo2

How to install a Python package in user's home directory on Sapelo2

Title: '''Do It Yourself: Using Conda to create and run python environments to suit your computing needs effortlessly!'''

Focus: Use conda to create and configure your own python virtual environments; Activate your environments to run python apps from your home directory on Sapelo2

What is Conda and its environment

Conda on Sapelo2

Use conda to create and configure your own python virtual environments

Activate your environments to run python apps from your home directory on Sapelo2

Title: '''How to submit and run jobs efficiently and correctly on Sapelo2'''

Focus: Sapelo2 cluster general workflow and correct computing resource requesting

Overview of Sapelo2 cluster with reference tables and operational diagrams

Sapelo2 batch job submission workflow taking global scratch as job working space

How to request computing resources correctly

How to run pipeline tasks and what are advantages/disadvantages of different options

Sapelo2 cluster guideline and practical tips

Title: '''GACRC Storage Environment'''

Focus: Overview of Linux common commands related to file and folder operations

Overview of the storage enviornment of zcluster and Sapelo cluster at GACRC

How to transfer data between local and GACRC storage

New file transfer node xfer2 and how to use it to transfer data between zcluster and the new cluster

GACRC suggestions on good practices on GACRC storage, etc;

Title: '''NCBI Blast application on sapelo'''

Focus: Introduction to BLAST

BLAST job submission to sapelo

Advantages & Disadvantages: NCBI website vs run at sapelo.

Understand BLAST output

Troubleshooting the BLAST results

Title: '''NGS application overview at GACRC'''

Focus: Overview of Bioinformatics software available on HPC clusters at GACRC

It’s a brave new world – NGS and its Applications

Hardware, Software, Databases available at GACRC

NGS project: Logistics and resource considerations

Best practices, common mistakes, troubleshooting and getting help from GACRC

Title: '''Perl Language Basics I, II'''

Focus of I: Overview of Perl language,

Perl general scripting style

Perl fundamental data types

Focus of II: Program structure: control flow and loop

Perl subroutine

Perl I/O

==Download==

====Sapelo2 Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|[[Media:GACRC_Sapelo2_cluster_new_user_training_workshop_v10.8.pdf]]
|-
|}

====Sap2test Migration Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Migrating_to_Slurm_and_new_software_environment.pdf]]
|}

'''Please note:''' To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the '''GACRC's Kaltura channel''' at
https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required).

==== Teaching Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:GACRC-Teaching-cluster-new-user-training-workshop_Fall2024.pdf]]
|-
|}

====Linux Training for New Cluster Users====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Linux_Training_For_New_Users_Of_Cluster_Suchi_04252019.pdf]]
|-
|}

==== Python Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Python_Language_Basics_I_v5.1.pdf]]
|-
|[[Media:Python_Language_Basics_II_v5.1.pdf]]
|-
|[[Media:Python_Basics_v6.1.pdf]]
|-
|}

====R Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:R Language Basics PowerPoint v2.0.1.pdf|Media:R_Language_Basics_PowerPoint_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_Document_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Powerpoint_v1.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Document_v1.0.pdf]]
|-
|}

====Perl Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Perl_Language_Basics_I_Workshop_v1.pdf]]
|-
|}

====Topical Sessions====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|
|-
| [[Media:AI_Resources_on_the_GACRC_Sapelo2_Cluster.pdf]]
|-
| [[Media:Using_Sapelo2_Cluster_at_the_GACRC_Part_II_Rocky8.pdf]]
|-
| [[Media:Using_Conda_on_the_GACRC_Sap2test_cluster_v1.pdf]]
|-
|[[Media:Blast_Workshop_GACRC_02012017.pdf]]
|-
|[[Media:Next-Generation_Sequencing_Applications_at_GACRC_10282016.pdf]]
|-
|}

==== Out-Reach/On-Class Talk====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Dept./Center/Institute
! scope="col" | Type
! scope="col" |Workshop PDF
|-
|CSP seminar - Fall 2024||Out-Reach||[[Media:GACRC_overview_20240820-CSP.pdf]]
|-
| BCMB8330 - Spring2024||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330_Spring2024.pdf]]
|-
|PHYS4601/6601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys4601.pdf]]
|-
|PHYS8601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys8601.pdf]]
|-
|CSP seminar - Fall 2023||Out-Reach||[[Media:GACRC_overview_20230822-CSP.pdf]]
|-
|BCMB8330 - Spring2023||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330.pdf]]
|-
|PHYS4601/6601 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2023_phys4601.pdf]]
|-
|PHYS8602 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] ; [[Media:Gacrc_handout2023_phys8602.pdf]]
|-
|ILS GradFIRST course - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220901-ILS.pdf]]
|-
|FYOS1001 - Fall 2022||Out-Reach||[[Media:High_Performance_Computing_(HPC)_on_GACRC_Sapelo2_Cluster.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220830-CSP.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:Compile_and_Run_HPC_code_on_Sapelo2.pdf]]
|-
|Terry College IT - Spring2022||Out-Reach||[[Media:GACRC_overview_20220506-Terry.pdf]]
|-
|PHYS8601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]]
|-
|PHYS4601/6601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2021_phys4601.pdf]]
|-
|PHYS8602 - Spring2021||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602-2021.pdf]] ; [[Media:Gacrc_handout2021_phys8602.pdf]]
|-
|GENE4220 - Fall2020||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_GENE4220_Fall2020.pdf]]
|-
|College of Veterinary Medicine - Spring2020||Out-Reach (jlslab) ||[[Media:Using_GACRC_Sapelo2_Cluster-Advanced_Topics(1).pdf]]
|-
|Byod Data Center - Fall2019||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Cluster.pdf]]
|-
|Department of Linguistics - Fall2019||On-class (LING6570)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_LING6570_Part2.pdf]]
|-
|The Center for Simulational Physics - Fall2019 ||Out-Reach (Seminar Talk 20190820)||[[Media:Introduction_to_GACRC_Computing_Facility_-_Sapelo2_Cluster_CSP-Fall2019.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS4601/6601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] [[Media:Gacrc_handout2019_phys4601.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS8601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]] [[Media:Gacrc_handout2020_phys8601.pdf]]
|-
|The Center for Simulational Physics ||On-Class (PHYS8602)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] [[Media:Gacrc_handout2019_phys8602.pdf]]
|-
|Food Science - Fall2018 ||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Sapelo2_Cluster_at_GACRC.pdf]]
|-
|The Center for Simulational Physics - Summer2018||Out-Reach (Seminar Talk 20180821)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
| Miller plant science - Summer2018 ||Out-Reach (jlmlab)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
|Biochemistry and Molecular Biology - Spring2018||On-Class (BCMB8330)||[[Media:GACRC_zcluster_Class_Training_BCMB8330_Spring_2018.pdf]]
|-
| The Center for Simulational Physics - Summer2017||Out-Reach (Seminar Talk 20170831) ||[[Media:Introduction_on_HPC_Resources_at_the_GACRC.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS4601/6601) ||[[Media:Phys4601.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS8602)||[[Media:Phys8602.pdf]]
|-
|The Institute of Bioinformatics and the Quantitative Biology Consulting Group|| Out-Reach||[[Media:Introduction_to_HPC_Resources_at_GACRC_BBB_Talk_20151014.pdf]]
|-
|The Center for Simulational Physics|| Out-Reach (Seminar Talk 20160906)||[[Media:Introduction_to_Sapelo_Computing_Resources_at_GACRC_Workshop20160906.pdf]]
|-
|Microbiology||On-Class (MIBO8150) ||[[Media:Introduction_to_HPC_Resources_at_GACRC_MIBO8150_20160926.pdf]]
|-
|Statistics||On-Class (STAT8060)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_Workshop_STAT8060_20150826.pdf]]
|-
|Biochemistry and Molecular Biology||On-Class (BCMB8211) ||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BCMB8211_20160114.pdf]]
|-
|Plant Biology||On-Class (PBIO/BINF8350)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO-BINF8350_20160115.pdf]]
|-
|Plant Biology - Bioinformatics Applications Fall2016||On-Class (PBIO4550)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO_4550_08182016.pdf]]
|-
| Bioinformatics - Essential Computing Skills for Biologists Fall2016||On-Class (BINF4005)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BINF_4005_08312016.pdf]]
|-
|Computers in Experimental Genetics Fall2016||On-Class (GENE4220)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_GENE_4220_10192016.pdf]]
|-
|Statistics - Advanced Applications and Computing in R Fall2016||On-Class (STAT8330)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_STAT8330_11022016.pdf]]
|}

'''NOTE:''' The slides may become outdated and you should always check GACRC Wiki for up to date information.

== Past Sessions ==

[[Pass Sessions in 2021]]

[[Past Sessions in 2020]]

[[Past Sessions in 2019]]

[[Past Sessions in 2018]]

[[Past Sessions in 2017]]

[[Past Sessions in 2016]]

[[Past Sessions in 2015]]

Training

2024-09-18T20:24:26Z

Jerky: /* Regular Training Announcement */

==GACRC Training==

The GACRC regularly hosts training sessions on a number of subjects relevant to the use of our computational and storage resources. Scheduled trainings will be announced through the GACRC mailing list.

'''NOTE: New users are required to attend a Sapelo2 cluster introductory training session and information about that will be sent once an account is requested.'''

==Regular Training Announcement==
In '''September 2024''', the GACRC is hosting 7 training sessions (3 Linux basics and 3 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (3 sessions)

2. Sapelo2 cluster new user training (3 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session) (here's a [http://cornell.edu/ test hyperlink] from PB)

In '''October 2024''', the GACRC is hosting 8 training sessions (4 Linux basics and 4 Sapelo2 cluster new user trainings and 1 Using Sapelo2 Cluster at the GACRC, Part II training).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (4 sessions)

2. Sapelo2 cluster new user training (4 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session)

'''Please Note:''' The training workshops will be offered remotely via Zoom Meeting. Detailed information on how to join the Zoom meeting will be sent to your UGA email account prior to each training session.

==Event Schedule==

===Sapelo2 Cluster New User Training===

Our Sapelo2 training consists of 1 hr 30 mins of instructional videos, followed by a 1 hr 30 min workshop. '''The instructional videos are required to be viewed prior to the training workshop, and they can be found [https://kaltura.uga.edu/playlist/dedicated/176125031/1_4o12v8b4/1_9zi68rgi here]'''

'''Prerequisites:'''

* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.

'''Video Playlist Training Goals:'''

* Understand the layout of Sapelo2

* Understand the Sapelo2 file systems

* Understand the Sapelo2 partitions

* Understand the Sapelo2 software environment

'''Workshop Training Goals:'''

* Understand how to request computing resources and submit a computational batch job following the Sapelo2 cluster general workflow

* Understand how to initiate an interactive job

* Understand how to transfer files to and from the cluster

* Understand how to get support from GACRC support team when you have any issues on cluster

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC
|September 18th, Wednesday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 3rd, Thursday, 2:00 PM - 4:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 11th, Friday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 17th, Thursday, 10:00 AM - 12:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|October 23rd, Wednesday, 2:00 PM - 4:00 PM
|}

===Linux Training for Linux-inexperienced Cluster New Users===
The Sapelo2 High Performance Computing (HPC) cluster runs a headless Linux distribution as the operating system on each of its constituent nodes. The term headless refers to the fact that these nodes do not have a desktop graphical user interface (GUI) installed by default. Graphical desktop environments consume resources that analyses could otherwise use, so users employ a command-line interface (CLI) instead. To interact with these resources, users connect to a remote terminal via SSH and execute commands.

The Linux Training workshop provides hands-on practice of the fundamental Linux commands necessary to interact with HPC resources.

'''Prerequisite:''' Please watch the introductory videos on Linux, basic Linux terms, and Linux Paths and Directories (total ~17 minutes) '''before attending the training workshop'''.

'''Training Goals:'''

1. Understand fundamental concepts of Linux working environment (filesystem hierarchy, path, PATH, etc.)

2. Know how to use Linux common commands (ls, cd, pwd, cat, more, nano, mkdir, rm, cp, mv, etc.)

3. Understand what is Linux bash shell and know how to make a simple Linux script and run it in Linux environment

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Use Linux on Cluster
|October 1st, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 9th, Wednesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 15th, Tuesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|October 21st, Monday, 1:00 PM - 3:00 PM
|}

===Using Sapelo2 Cluster at the GACRC, Part II===
'''Prerequisites:'''
* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.
* Sapelo2 cluster new user training. Fundamental HPC and Sapelo2 knowledge is required for this advanced Sapelo2 workshop.

'''Training Goals:'''

1. Learn about high-performance computing framework

2. Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

3. How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

4. How can I make my job run more efficiently (through the correct use of software and hardware)?

5. A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC, Part II||September 27th, Friday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC, Part II
|October 25th, Friday, 2:00 PM - 4:00 PM
|}

===Python Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand Python scientific modules and distributions

2. Understand Python general lexical conventions; Python built-in data types, like string, list, tuple, dictionary, etc.

3. Understand Python programming structures and procedural programming using functions

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Python Basics I||Not scheduled
|-
|Python Basics II||Not scheduled
|-
|}

===R Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of R language, e.g. R general lexical conventions, data types, functions, and packages. Part 2 will introduce loops and functions.

2. Be able to manipulate and create data frames using built in functions and the dplyr package.

3. Interact with your file system and submit R code as a batch job to Sapelo 2.

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
| R Basics I||Not scheduled
|-
|R Basics II||Not scheduled
|-
|}

===Conda===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of conda environment

2. Use conda to create and configure your own virtual environments

3. Activate your environments to run python apps from your home directory on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Conda Basics||Not scheduled
|-
|}

==How to Register==

'''Please Note''', the training workshops '''Using Sapelo2 Cluster at the GACRC''' and '''Use Linux on Cluster''' are '''ONLY''' offered to '''new users''' who need computing user accounts on the GACRC Sapelo2 cluster, or any current users who have never attended the GACRC Sapelo2 cluster new user training before. Please ask your group PI/UGA faculty member to send us a request for you, using the GACRC User Account Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25839

If you want to attend '''Python Basics''', '''R''', and '''Conda basics''' training sessions, please send us a request using the GACRC Training Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25852 . In your request, please tell us which session(s) you want to attend.

The GACRC is going to host other training workshops and seminars covering various HPC topics, including HPC fundamental introduction, Linux introductory III (Linux working environment and utilities), Bioinfomatics applications on Sapelo cluster, Perl, R, C/C++/Fortran programming, etc., in the near future. We will announce those events when they are scheduled.

The GACRC Web Training page can be found at https://gacrc.uga.edu/training/ and the GACRC Wiki Training page can be found at https://wiki.gacrc.uga.edu/wiki/Training, from which you can find detailed information about upcoming and past training sessions from GACRC and download training materials.

== Topic Introduction==

Title: '''Sap2test cluster migration training'''

Focus: Slurm queueing system, including Slurm job commands, job environment variables, and job submission headers, etc.

The new software environment on Sap2test

Other important topics related to Sap2test working environment

Title: '''Using Sapelo2 Cluster at the GACRC'''

Focus: Sapelo2 HPC cluster and computational batch job submission workflow

Cluster's storage environment

Computational queues on cluster

Software environment

How to submit computational batch jobs

Other tips and guidelines for users

Title: '''Using Sapelo2 Cluster at the GACRC, Part II'''

Focus: More topics on how to use Sapelo2 cluster

Learn about high-performance computing framework

Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

How can I make my job run more efficiently (through the correct use of software and hardware)?

A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

Title: '''Use Linux on Cluster'''

Focus: Linux OS fundamentals

Linux common commands, filesystem, and shell

Linux shell scripting basics

Common Linux utilities, e.g., grep, sed, find, sort, and awk, etc.

Linux Hands-on practice

Title: '''Python Basics I, II'''

Focus of I: Python language overview, scientific modules and distributions

Python general lexical conventions

Basic built-in data types, like string, list, tuple, dictionary, etc.

Focus of II: Programming structures: control flow and loop

Function: procedural programming with examples, lambda expression, factory function and generator

Title: '''R Basics I, II'''

Focus of I: R language overview,general lexical conventions, data types, functions, and packages.

Basic built-in data types, like string, numeric, list, dataframe etc. Using the dplyr package.

Focus of II: Programming structures: control flow, loops and functions

Title: '''Python on GACRC Sapelo2 Cluster'''

Focus: Install Python packages/modules in a user's home directory on Sapelo2 cluster

Python versions installed on Sapelo2

Python environment details on Sapelo2

How to know a Python package is installed or not on Sapelo2

How to install a Python package in user's home directory on Sapelo2

Title: '''Do It Yourself: Using Conda to create and run python environments to suit your computing needs effortlessly!'''

Focus: Use conda to create and configure your own python virtual environments; Activate your environments to run python apps from your home directory on Sapelo2

What is Conda and its environment

Conda on Sapelo2

Use conda to create and configure your own python virtual environments

Activate your environments to run python apps from your home directory on Sapelo2

Title: '''How to submit and run jobs efficiently and correctly on Sapelo2'''

Focus: Sapelo2 cluster general workflow and correct computing resource requesting

Overview of Sapelo2 cluster with reference tables and operational diagrams

Sapelo2 batch job submission workflow taking global scratch as job working space

How to request computing resources correctly

How to run pipeline tasks and what are advantages/disadvantages of different options

Sapelo2 cluster guideline and practical tips

Title: '''GACRC Storage Environment'''

Focus: Overview of Linux common commands related to file and folder operations

Overview of the storage enviornment of zcluster and Sapelo cluster at GACRC

How to transfer data between local and GACRC storage

New file transfer node xfer2 and how to use it to transfer data between zcluster and the new cluster

GACRC suggestions on good practices on GACRC storage, etc;

Title: '''NCBI Blast application on sapelo'''

Focus: Introduction to BLAST

BLAST job submission to sapelo

Advantages & Disadvantages: NCBI website vs run at sapelo.

Understand BLAST output

Troubleshooting the BLAST results

Title: '''NGS application overview at GACRC'''

Focus: Overview of Bioinformatics software available on HPC clusters at GACRC

It’s a brave new world – NGS and its Applications

Hardware, Software, Databases available at GACRC

NGS project: Logistics and resource considerations

Best practices, common mistakes, troubleshooting and getting help from GACRC

Title: '''Perl Language Basics I, II'''

Focus of I: Overview of Perl language,

Perl general scripting style

Perl fundamental data types

Focus of II: Program structure: control flow and loop

Perl subroutine

Perl I/O

==Download==

====Sapelo2 Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|[[Media:GACRC_Sapelo2_cluster_new_user_training_workshop_v10.8.pdf]]
|-
|}

====Sap2test Migration Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Migrating_to_Slurm_and_new_software_environment.pdf]]
|}

'''Please note:''' To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the '''GACRC's Kaltura channel''' at
https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required).

==== Teaching Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:GACRC-Teaching-cluster-new-user-training-workshop_Fall2024.pdf]]
|-
|}

====Linux Training for New Cluster Users====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Linux_Training_For_New_Users_Of_Cluster_Suchi_04252019.pdf]]
|-
|}

==== Python Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Python_Language_Basics_I_v5.1.pdf]]
|-
|[[Media:Python_Language_Basics_II_v5.1.pdf]]
|-
|[[Media:Python_Basics_v6.1.pdf]]
|-
|}

====R Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:R Language Basics PowerPoint v2.0.1.pdf|Media:R_Language_Basics_PowerPoint_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_Document_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Powerpoint_v1.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Document_v1.0.pdf]]
|-
|}

====Perl Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Perl_Language_Basics_I_Workshop_v1.pdf]]
|-
|}

====Topical Sessions====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|
|-
| [[Media:AI_Resources_on_the_GACRC_Sapelo2_Cluster.pdf]]
|-
| [[Media:Using_Sapelo2_Cluster_at_the_GACRC_Part_II_Rocky8.pdf]]
|-
| [[Media:Using_Conda_on_the_GACRC_Sap2test_cluster_v1.pdf]]
|-
|[[Media:Blast_Workshop_GACRC_02012017.pdf]]
|-
|[[Media:Next-Generation_Sequencing_Applications_at_GACRC_10282016.pdf]]
|-
|}

==== Out-Reach/On-Class Talk====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Dept./Center/Institute
! scope="col" | Type
! scope="col" |Workshop PDF
|-
|CSP seminar - Fall 2024||Out-Reach||[[Media:GACRC_overview_20240820-CSP.pdf]]
|-
| BCMB8330 - Spring2024||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330_Spring2024.pdf]]
|-
|PHYS4601/6601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys4601.pdf]]
|-
|PHYS8601 - Spring2024||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601-Spring2024.pdf]] ; [[Media:Gacrc_handout2024_phys8601.pdf]]
|-
|CSP seminar - Fall 2023||Out-Reach||[[Media:GACRC_overview_20230822-CSP.pdf]]
|-
|BCMB8330 - Spring2023||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330.pdf]]
|-
|PHYS4601/6601 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2023_phys4601.pdf]]
|-
|PHYS8602 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] ; [[Media:Gacrc_handout2023_phys8602.pdf]]
|-
|ILS GradFIRST course - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220901-ILS.pdf]]
|-
|FYOS1001 - Fall 2022||Out-Reach||[[Media:High_Performance_Computing_(HPC)_on_GACRC_Sapelo2_Cluster.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220830-CSP.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:Compile_and_Run_HPC_code_on_Sapelo2.pdf]]
|-
|Terry College IT - Spring2022||Out-Reach||[[Media:GACRC_overview_20220506-Terry.pdf]]
|-
|PHYS8601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]]
|-
|PHYS4601/6601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2021_phys4601.pdf]]
|-
|PHYS8602 - Spring2021||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602-2021.pdf]] ; [[Media:Gacrc_handout2021_phys8602.pdf]]
|-
|GENE4220 - Fall2020||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_GENE4220_Fall2020.pdf]]
|-
|College of Veterinary Medicine - Spring2020||Out-Reach (jlslab) ||[[Media:Using_GACRC_Sapelo2_Cluster-Advanced_Topics(1).pdf]]
|-
|Byod Data Center - Fall2019||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Cluster.pdf]]
|-
|Department of Linguistics - Fall2019||On-class (LING6570)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_LING6570_Part2.pdf]]
|-
|The Center for Simulational Physics - Fall2019 ||Out-Reach (Seminar Talk 20190820)||[[Media:Introduction_to_GACRC_Computing_Facility_-_Sapelo2_Cluster_CSP-Fall2019.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS4601/6601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] [[Media:Gacrc_handout2019_phys4601.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS8601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]] [[Media:Gacrc_handout2020_phys8601.pdf]]
|-
|The Center for Simulational Physics ||On-Class (PHYS8602)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] [[Media:Gacrc_handout2019_phys8602.pdf]]
|-
|Food Science - Fall2018 ||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Sapelo2_Cluster_at_GACRC.pdf]]
|-
|The Center for Simulational Physics - Summer2018||Out-Reach (Seminar Talk 20180821)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
| Miller plant science - Summer2018 ||Out-Reach (jlmlab)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
|Biochemistry and Molecular Biology - Spring2018||On-Class (BCMB8330)||[[Media:GACRC_zcluster_Class_Training_BCMB8330_Spring_2018.pdf]]
|-
| The Center for Simulational Physics - Summer2017||Out-Reach (Seminar Talk 20170831) ||[[Media:Introduction_on_HPC_Resources_at_the_GACRC.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS4601/6601) ||[[Media:Phys4601.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS8602)||[[Media:Phys8602.pdf]]
|-
|The Institute of Bioinformatics and the Quantitative Biology Consulting Group|| Out-Reach||[[Media:Introduction_to_HPC_Resources_at_GACRC_BBB_Talk_20151014.pdf]]
|-
|The Center for Simulational Physics|| Out-Reach (Seminar Talk 20160906)||[[Media:Introduction_to_Sapelo_Computing_Resources_at_GACRC_Workshop20160906.pdf]]
|-
|Microbiology||On-Class (MIBO8150) ||[[Media:Introduction_to_HPC_Resources_at_GACRC_MIBO8150_20160926.pdf]]
|-
|Statistics||On-Class (STAT8060)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_Workshop_STAT8060_20150826.pdf]]
|-
|Biochemistry and Molecular Biology||On-Class (BCMB8211) ||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BCMB8211_20160114.pdf]]
|-
|Plant Biology||On-Class (PBIO/BINF8350)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO-BINF8350_20160115.pdf]]
|-
|Plant Biology - Bioinformatics Applications Fall2016||On-Class (PBIO4550)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO_4550_08182016.pdf]]
|-
| Bioinformatics - Essential Computing Skills for Biologists Fall2016||On-Class (BINF4005)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BINF_4005_08312016.pdf]]
|-
|Computers in Experimental Genetics Fall2016||On-Class (GENE4220)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_GENE_4220_10192016.pdf]]
|-
|Statistics - Advanced Applications and Computing in R Fall2016||On-Class (STAT8330)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_STAT8330_11022016.pdf]]
|}

'''NOTE:''' The slides may become outdated and you should always check GACRC Wiki for up to date information.

== Past Sessions ==

[[Pass Sessions in 2021]]

[[Past Sessions in 2020]]

[[Past Sessions in 2019]]

[[Past Sessions in 2018]]

[[Past Sessions in 2017]]

[[Past Sessions in 2016]]

[[Past Sessions in 2015]]

Running Jobs on Sapelo2

2024-07-05T15:19:40Z

Jerky: Added a row for the 3TB nodes ra4-[3-5] in "hugemem*_p".

[[Category:Sapelo2]]

===Using the Queueing System===

The login node for the Sapelo2 cluster should be used for text editing, and job submissions. '''No jobs should be run directly on the login node.'''
Processes that use too much CPU or RAM on the login node may be terminated by GACRC staff, or automatically, in order to keep the cluster running properly. Jobs should
be run using the Slurm queueing system. The queueing system should be used to run both interactive and batch jobs.

----
[[#top|Back to Top]]

===Batch partitions (queues) defined on the Sapelo2===

There are different partitions defined on Sapelo2. The Slurm queueing system refers to queues as partition. Users are required to specify, in the job submission script or as job submission command line arguments, the partition and the resources needed by the job in order for it to be assigned to compute node(s) that have enough available resources (such as number of cores, amount of memory, GPU cards, etc). Please note, Slurm will not allow a job to be submitted if there are no resources matching your request. Please refer to [[Migrating from Torque to Slurm]] for more info about Slurm queueing system.

The following partitions are defined on the Sapelo2 cluster:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | Time limit
! scope="col" | Max jobs
! scope="col" | Notes
|-
|-
| batch || 7 days || || Regular nodes.
|-
| batch-30d || 30 days || 2 || Regular nodes. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| highmem_p || 7 days || || For high memory jobs
|-
| highmem_30d_p || 30 days || 2 || For high memory jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
|hugemem_p || 7 days ||4 || For jobs needing up to 3TB of memory.
|-
|hugemem_30d_p || 30 days || 4 || For jobs needing up to 3TB of memory.
|-
| gpu_p || 7 days || || For GPU-enabled jobs.
|-
| gpu_30d_p || 30 days || 2 || For GPU-enabled jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| inter_p || 2 days || || Regular nodes, for interactive jobs.
|-
| '''name'''_p || variable || || Partitions that target different groups' buy-in nodes. The '''name''' string is specific to each group.
|-
| scavenge_p || 2 hours || || Partition that targets the buy-in nodes. When there are no available resources in the batch partition, short jobs submitted there might be automatically transferred into scavenge_p, to run on idle buy-in resources. Jobs cannot be submitted directly to this partition.
|-
|}
For more detailed information about the partitions, please see [[Job Submission partitions on Sapelo2]].

The table below summarizes the partitions (queues) defined and the compute nodes that they target:
{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | Node Features
! scope="col" | Node Number
! scope="col" | Description
! scope="col" | Memory for jobs
! scope="col" | Notes
|-
|-
| batch, batch_30d || AMD, Opteron, QDR || || 48-core, 128GB RAM, AMD Opteron, QDR IB interconnect || 122GB || Regular nodes.
|-
| batch, batch_30d || AMD, EPYC, EDR || || 64-core, 128GB RAM, AMD EPYC, IB EDR interconnect || 120GB || Regular nodes
|-
| batch, batch_30d || AMD, EPYC, EDR || || 32-core, 128GB RAM, AMD EPYC, IB EDR interconnect || 120GB || Regular nodes
|-
| batch, batch_30d || AMD, Opteron, QDR || || 48-core, 256GB RAM, AMD Opteron, QDR IB interconnect || 250GB || Regular nodes.
|-
| batch, batch_30d || Intel, Skylake, EDR || || 32-core, 192GB RAM, Intel Xeon Skylake, IB EDR interconnect || 180GB || Regular nodes
|-
| batch, batch_30d || Intel, Broadwell, EDR || || 28-core, 64GB RAM, Intel Xeon Broadwell, IB EDR interconnect || 58GB || Regular nodes
|-
| highmem_p, highmem_30d_p || AMD, EPYC, EDR || || 64-core, 1TB RAM, AMD EPYC, IB EDR interconnect || 950GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || Intel, EDR || || 32-core, 1TB RAM, Intel, IB EDR interconnect || 950GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || AMD, Opteron, EDR || || 48-core, 1TB RAM, AMD Opteron, IB EDR interconnect || 950GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || AMD, Opteron, QDR || || 48-core, 512GB, AMD Opteron, IB QDR interconnect || 500GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || AMD, EPYC, EDR || || 32-core, 512GB RAM, AMD EPYC, IB EDR interconnect || 490GB || For high memory jobs
|-
| hugemem_p, hugemem_30d_p || AMD, EPYC, EDR || || 32-core, 2TB RAM, AMD EPYC, IB EDR interconnect || 2000GB || For high memory jobs
|-
|hugemem_p, hugemem_30d_p
|AMD, EPYC, EDR
|
|48-core, 3TB RAM, AMD EPYC, IB EDR interconnect
|3000GB
|For high memory jobs
|-
| gpu_p, gpu_30d_p || GPU, A100, EDR || || 64-core, 1000GB RAM, AMD EPYC, 4 NVIDIA A100 GPUs, EDR IB interconnect || 1000GB || For GPU-enabled jobs.
|-
| gpu_p, gpu_30d_p || GPU, P100, EDR || || 32-core, 192GB RAM, Intel Xeon Skylake, 1 NVIDIA P100 GPUs, EDR IB interconnect || 180GB || For GPU-enabled jobs.
|-
| gpu_p, gpu_30d_p || GPU, K40, QDR || || 16-core, 128GB RAM, Intel Xeon , 8 NVIDIA K40 GPUs, QDR IB interconnect || 120GB || For GPU-enabled jobs.
|-
| gpu_p, gpu_30d_p || GPU, K20, QDR || || 12-core, 96GB RAM, Intel Xeon , 7 NVIDIA K20Xm GPUs, QDR IB interconnect || 70GB || For GPU-enabled jobs.
|-

|}

You can check all partitions (queues) defined in the cluster with the command
<pre class="gcommand">
sinfo
</pre>

----
[[#top|Back to Top]]

===Job submission Scripts===

Users are required to specify the number of cores, the amount of memory, the partition (queue) name, and the maximum wallclock time needed by the job.

====Header lines====

'''Basic job submission script'''

At a minimum, the job submission script needs to have the following header lines:
<pre class="gscript">
#!/bin/bash
#SBATCH --partition=batch
#SBATCH --job-name=test
#SBATCH --ntasks=1
#SBATCH --time=4:00:00
#SBATCH --mem=10G
</pre>

Commands to run your application should be added after these header lines.

'''Header lines explained:'''

* '''#!/bin/bash''': specify Linux default shell bash
* '''#SBATCH --partition=batch''' : specify the partition (queue) to run on, e.g. ''batch''
* '''#SBATCH --job-name=test''' : specify the job name, e.g. ''test''
* '''#SBATCH --ntasks=1''' : specify the number of tasks (e.g. 1)
* '''#SBATCH --time=4:00:00''' : specify the maximum walltime of the job in the format D-HH:MM:SS (e.g. --time=1- for one day or --time=4:00:00 for 4 hours)
* '''#SBATCH --mem=10G''' : specify the maximum memory per node required by the job (e.g. 10GB)

Below are some of the most commonly used queueing system options to configure the job.

====Options to request resources for the job====

* -t, --time=time
Wall clock time limit of a job running on cluster. Acceptable formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes", and "days-hours:minutes:seconds". '''This is a required option.'''

* --mem=num
Maximum amount of memory in MegaBytes per node required by the job. Different units can be specified using the suffix [K|M|G|T].

* --mem-per-cpu=num
Minimum amount of memory in MegaBytes per allocated CPU. Different units can be specified using the suffix [K|M|G|T].

* -n, --ntasks=num
Number of tasks to run. The default is one task per node. For use with distributed parallelism. See below.

* -N, --nodes=num
Number of nodes allocated to the job. Default is one node.

* --ntasks-per-node=num
Number of tasks invoked on each node. Meant to be used with the --nodes option. For use with distributed parallelism. See below.

* -c, --cpus-per-task=ncpus
Number of CPUs allocated to each task. For use with shared memory parallelism. See below.

* -C, --constraint=<list>
List of node features required by the job. Only nodes having features matching the job constraints will be used to satisfy the request. Multiple constraints may be specified with AND, OR, matching OR, resource counts, etc.

* --gres=<list>
A comma delimited list of generic consumable resources. For example, to request one P100 GPU card: --gres=gpu:P100:1

Please try to request resources for your job as accurately as possible, because this allows your job to be dispatched to run at the earliest opportunity and it helps the system allocate resources efficiently to start as many jobs as possible, benefiting all users.

====Options to manage job notification and output====

* -J, --job-name jobname
Specify a name for the job. The specified name will appear along with the job id number when querying running jobs on the system. The default is the supplied executable program's name. Within the job, $SBATCH_JOB_NAME expands to the job name.

* -o, --output=path/for/stdout
Send stdout to path/for/stdout. The default filename is slurm-${SLURM_JOB_ID}.out, e.g. slurm-12345.out, in the directory from which the job was submitted.

* -e, --error=path/for/stderr
Send stderr to path/for/stderr. If --error is not specified, both stdout and stderr will directed to the file specified by --output.

* --mail-user=username@uga.edu
Send email notification to the address you specified when certain events occur.

* --mail-type=type
Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL, TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 and TIME_LIMIT_50.

By default, email notifications set for an array job will generate one email message for the array job. If you would like to receive an email message for individual array job elements (up to a certain limit), please add ARRAY_TASKS to the --mail-type option.

====Options to set Array Jobs====
If you wish to run an application binary or script using e.g. different input files, then you might find it convenient to use an array job. To create an array job with e.g. 10 elements, use
<pre class="gscript">
#SBATCH -a 0-9
</pre>
or
<pre class="gscript">
#SBATCH --array=0-9
</pre>

Each array job element runs as an independent job, so multiple elements can run concurrently if resources are available. For this reason, the job ID which is stored in SLURM_JOB_ID for each element in an array job will be different and unique. The ID of each element in an array job, i.e., array element index value, is stored in SLURM_ARRAY_TASK_ID. The ID of an array job as whole is stored in SLURM_ARRAY_JOB_ID. For this reason, it will be the same for all elements in an array job. The JodID reported by sq command is a combination of SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID connected by "_".

<pre class="gscript">
sbatch --array=1-3 -N1 sub.sh

will generate a job array containing three jobs. If the sbatch command responds
Submitted batch job 36
then the environment variables will be set as follows:

SLURM_JOB_ID=36
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=37
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=38
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
</pre>

Most Slurm commands recognize the SLURM_ARRAY_JOB_ID plus SLURM_ARRAY_TASK_ID values separated by an underscore as identifying an element of a job array, for example, 36_2 would be equivalent ways to identify the second array element of array job 36.

For more information, please see [[Array Jobs]].

====Option to set job dependency====
You can set job dependency with the option -d or --dependency=''dependency-list''. For example, if you want to specify that one job starts to run after the job 1234 and 1235 have successfully executed (ran to completion with an exit code of zero), you can add the following header line in the job submission script of the job:
<pre class="gscript">
#SBATCH --dependency=afterok:1234:1235
</pre>
Having this header line in the job submission script will ensure that the job is only dispatched to run after job 1234 and 1235 have completed successfully.

You can also use the following header line to specify that one job starts to run after the job 1236 and 1237 start or are cancelled:
<pre class="gscript">
#SBATCH --dependency=after:1236:1237
</pre>

====Options to requeue or not requeue a job when a node crashes====

If a job is running and one or more nodes that it is using crash, the job will stop running and, by default, it will get requeued. When resources become available, the job will start running again, from the beginning, unless the program saves intermediate results and it is able to automatically pick up from where it stopped. The files with the standard error and standard output of the job will get rewritten once the job restarts. Often other output files will get rewritten as well.

If you are running a program that cannot restart, e.g. the program will fail if a certain output file or directory has already been created, or if you would like to preserve the partial results, you can use the following option to prevent the job from being requeued:
<pre class="gscript">
#SBATCH --no-requeue
</pre>
When this option is used, the job will simply stop if a node crashes, it will not be requeued. In this case partial results and the standard error and output of the job will not get overwritten.

Although requeueing jobs is enabled by default now, you can also add the option below if you would like to ensure a job is requeued in the event of a node crash:
<pre class="gscript">
#SBATCH --requeue
</pre>

====Other content of the script====

Following the header lines, users can include commands to change to the working directory, to load the modules needed to run the application, and to invoke the application. For example, to use the directory from which the job is submitted as the working directory (where to find input files or binaries), add the line
<pre class="gscript">
cd $SLURM_SUBMIT_DIR
</pre>
(Note that Slurm jobs start from the submit directory by default, so adding the line above might not be necessary.)

You can then load the needed modules. For example, if you are running an R program, then include the line
<pre class="gscript">
module load R/4.3.1-foss-2022a
</pre>

Then invoke your application. For example, if you are running an R program called add.R which is in your job submission directory, use
<pre class="gscript">
R CMD BATCH add.R
</pre>

====Environment Variables exported by batch jobs====

When a batch job is started, a number of variables are introduced into the job's environment that can be used by the batch script in making decisions, creating output files, and so forth. Some of these variables are listed in the following table:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Variable
! scope="col" | Description
|-
| SLURM_ARRAY_JOB_ID || Job array's master job ID number, i.e., the first Slurm job id of a job array
|-
| SLURM_ARRAY_TASK_COUNT || Total number of tasks (elements) in a job array
|-
| SLURM_ARRAY_TASK_ID || Job array ID (index) number
|-
| SLURM_ARRAY_TASK_MAX || Job array's maximum ID (index) number
|-
| SLURM_ARRAY_TASK_MIN || Job array's minimum ID (index) number
|-
| SLURM_CPUS_ON_NODE || Number of CPUS on the allocated node
|-
| SLURM_CPUS_PER_TASK || Number of cpus requested per task. Only set if the --cpus-per-task option is specified
|-
| SLURM_JOB_ID || Unique Slurm job id
|-
| SLURM_JOB_NAME || Job name
|-
| SLURM_JOB_CPUS_PER_NODE || Count of processors available to the job on this node
|-
| SLURM_JOB_NODELIST || List of nodes allocated to the job
|-
| SLURM_JOB_NUM_NODES ||Total number of nodes in the job's resource allocation
|-
| SLURM_JOB_PARTITION || Name of the partition (i.e. queue) in which the job is running
|-
| SLURM_MEM_PER_NODE || Same as --mem
|-
| SLURM_MEM_PER_CPU || Same as --mem-per-cpu
|-
| SLURM_NTASKS || Same as -n, --ntasks
|-
| SLURM_NTASKS_PER_NODE || Number of tasks requested per node. Only set if the --ntasks-per-node option is specified
|-
| SLURM_SUBMIT_DIR || The directory from which sbatch was invoked
|-
| SLURM_SUBMIT_HOST || The hostname of the computer from which sbatch was invoked
|-
| SLURM_TASK_PID || The process ID of the task being started
|-
| SLURMD_NODENAME || Name of the node running the job script
|-
| CUDA_VISIBLE_DEVICES || GPU devide ID that assigned to the job to use
|-
|}

----
[[#top|Back to Top]]

===Sample job submission scripts===

====Serial (single-processor) Job====

Sample job submission script (sub.sh) to run an R program called add.R using a single core:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=testserial # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=testserial.%j.out # Standard output log
#SBATCH --error=testserial.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load R/4.3.1-foss-2022a

R CMD BATCH add.R
</pre>

In this sample script, the standard output and error of the job will be saved into a file called testserial.o%j, where %j will be automatically replaced by the job id of the job.

====Serial (single-processor) Job on an AMD EPYC Milan processor====

Sample job submission script (sub.sh) to run an R program called add.R using a single core:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=testserial # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --constraint=Milan # node feature
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=testserial.%j.out # Standard output log
#SBATCH --error=testserial.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load R/4.3.1-foss-2022a

R CMD BATCH add.R
</pre>

In this sample script, the standard output and error of the job will be saved into a file called testserial.%j.out and testserial.%j.err, where %j will be automatically replaced by the job id of the job.

====MPI Job====

Sample job submission script (sub.sh) to run an OpenMPI application. In this example the job requests 16 cores and further specifies that these 16 cores need to be divided equally on 2 nodes (8 cores per node) and the binary is called mympi.exe:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=mpitest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks=16 # Number of MPI ranks
#SBATCH --ntasks-per-node=8 # How many tasks on each node
#SBATCH --cpus-per-task=1 # Number of cores per MPI rank
#SBATCH --mem-per-cpu=600mb # Memory per processor
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=mpitest.%j.out # Standard output log
#SBATCH --error=mpitest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load OpenMPI/4.1.4-GCC-11.3.0

srun ./mympi.exe
</pre>

Please note that you need to start the application with '''srun''' and not with '''mpirun''' or '''mpiexec'''.

'''Important note:''' MPI jobs need to be submitted from a Sapelo2 login node, not from an interactive session, in order to get the correct core allocation for the MPI processes.

====MPI Job on nodes connected via the EDR IB fabric====

Sample job submission script (sub.sh) to run an OpenMPI application. In this example the job requests 16 cores and further specifies that these 16 cores need to be divided equally on 2 nodes (8 cores per node) and the binary is called mympi.exe:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=mpitest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --constraint=EDR # node feature
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks=16 # Number of MPI ranks
#SBATCH --ntasks-per-node=8 # How many tasks on each node
#SBATCH --cpus-per-task=1 # Number of cores per MPI rank
#SBATCH --mem-per-cpu=600mb # Memory per processor
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=mpitest.%j.out # Standard output log
#SBATCH --error=mpitest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load OpenMPI/4.1.4-GCC-11.3.0

srun ./mympi.exe
</pre>

Please note that you need to start the application with '''srun''' and not with '''mpirun''' or '''mpiexec'''.

'''Important note:''' MPI jobs need to be submitted from a Sapelo2 login node, not from an interactive session, in order to get the correct core allocation for the MPI processes.

====OpenMP (Multi-Thread) Job====

Sample job submission script (sub.sh) to run a program that uses OpenMP with 6 threads. Please set '''--ntasks=1''' and set '''--cpus-per-task''' to the number of threads you wish to use. The name of the binary in this example is a.out.

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=mctest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=6 # Number of CPU cores per task
#SBATCH --mem=4gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=mctest.%j.out # Standard output log
#SBATCH --error=mctest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=6

module load foss/2022a # load the appropriate module file, e.g. foss/2022a

time ./a.out
</pre>

====High Memory Job====

Sample job submission script (sub.sh) to run a velvet application that needs to use 200GB of memory and 4 threads:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=highmemtest # Job name
#SBATCH --partition=highmem_p # Partition (queue) name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=4 # Number of CPU cores per task
#SBATCH --mem=200gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=highmemtest.%j.out # Standard output log
#SBATCH --error=highmemtest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=4

module load Velvet

velvetg [options]
</pre>

====Hybrid MPI/shared-memory using OpenMPI====

Sample job submission script (sub.sh) to run a parallel job that uses 4 MPI processes with OpenMPI and each MPI process runs with 3 threads:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=hybridtest
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks=8 # Number of MPI ranks
#SBATCH --ntasks-per-node=4 # Number of MPI ranks per node
#SBATCH --cpus-per-task=3 # Number of OpenMP threads for each MPI process/rank
#SBATCH --mem-per-cpu=2000mb # Per processor memory request
#SBATCH --time=2-00:00:00 # Walltime in hh:mm:ss or d-hh:mm:ss (2 days in the example)
#SBATCH --output=hybridtest.%j.out # Standard output log
#SBATCH --error=hybridtest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load OpenMPI/4.1.4-GCC-11.3.0

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun ./myhybridprog.exe

</pre>

====Array job====

Sample job submission script (sub.sh) to submit an array job with 10 elements. In this example, each array job element will run the a.out binary using an input file called input_0, input_1, ..., input_9.
<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=arrayjobtest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --mem=1gb # Job Memory
#SBATCH --time=10:00:00 # Time limit hrs:min:sec
#SBATCH --output=array_%A-%a.out # Standard output log
#SBATCH --error=array_%A-%a.err # Standard error log
#SBATCH --array=0-9 # Array range

cd $SLURM_SUBMIT_DIR

module load foss/2022a # load any needed module files, e.g. foss/2022a

time ./a.out < input_$SLURM_ARRAY_TASK_ID

</pre>

For more information, please see [[Array Jobs]].

====GPU/CUDA====

Sample script to run Amber on a GPU node using one node, 2 CPU cores, and 1 GPU card:
<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=amber # Job name
#SBATCH --partition=gpu_p # Partition (queue) name
#SBATCH --gres=gpu:1 # Requests one GPU device
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=2 # Number of CPU cores per task
#SBATCH --mem=40gb # Job memory request
#SBATCH --time=10:00:00 # Time limit hrs:min:sec
#SBATCH --output=amber.%j.out # Standard output log
#SBATCH --error=amber.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

ml Amber/22.0-foss-2021b-AmberTools-22.3-CUDA-11.4.1

$AMBERHOME/bin/pmemd.cuda -O -i ./prod.in -o prod.out -p ./dimerFBP_GOL.prmtop -c ./restart.rst -r prod.rst -x prod.mdcrd
</pre>
You can use the option <code>#SBATCH --gres=gpu:P100:1</code> or <code>#SBATCH --gres=gpu:A100:1</code> to specify using a P100 or an A100 GPU device, respectively. To use an A100 GPU device, please explicitly request it with <code>#SBATCH --gres=gpu:A100:1</code>. Jobs that request a GPU, but that do not specify the device type (that is, jobs that use <code>#SBATCH --gres=gpu:1</code>) will get allocated a P100 device and will not get allocated an A100 device.

====Singularity job====

Sample job submission script (sub.sh) to run a program (e.g. sortmerna) using a singularity container:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=j_sortmerna # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=sortmerna.%j.out # Standard output log
#SBATCH --error=sortmerna.%j.err # Standard error log
#SBATCH --cpus-per-task=4 # Number of CPU cores per task
#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

singularity exec /apps/singularity-images/sortmerna-3.0.3.simg sortmerna \
--threads 4 --ref db.fasta,db.idx --reads file.fa --aligned base_name_output
</pre>
For more information about software installed as singularity containers on the cluster, please see [[Software_on_Sapelo2#Singularity_Containers]]

To run a GPU-enabled singularity container on the GPU, please submit the job to the gpu_p partition, request a GPU device and add the '''--nv''' option to the singularity command.

Sample job submission script (sub.sh) to run a program using a singularity container (e.g. gpuapp.sif) on the GPU device:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=myjobname # Job name
#SBATCH --partition=gpu_p # Partition (queue) name
#SBATCH --gres=gpu:1 # Requests one GPU device
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=10gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --cpus-per-task=1 # Number of CPU cores per task

cd $SLURM_SUBMIT_DIR

singularity exec --nv /apps/singularity-images/gpuapp.sif prog.x
</pre>
For more information about software installed as singularity containers on the cluster, please see [[Software_on_Sapelo2#Singularity_Containers]]

----
[[#top|Back to Top]]

===How to submit a batch job===

With the resource requirements specified in the job submission script (sub.sh), submit your job with
<pre class="gcommand">
sbatch <scriptname>
</pre>
For example
<pre class="gcommand">
sbatch sub.sh
</pre>
Once the job is submitted, the Job ID of the job (e.g. 12345) will be printed on the screen.

----
[[#top|Back to Top]]

===Discovering if a partition (queue) is busy===
The nodes allocated to each partition (queue) and their state can be view with the command
<pre class="gcommand">
sinfo
</pre>

Sample output of the '''sinfo''' command:

<pre class="gcommand">
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 7-00:00:00 1 drain* ra4-2
batch* up 7-00:00:00 3 down* d4-7,ra3-19,ra4-12
batch* up 7-00:00:00 1 mix b1-2
batch* up 7-00:00:00 1 alloc b1-3
batch* up 7-00:00:00 53 idle b1-[4-24],c1-3,c5-19,d4-[5-6,8-12],ra3-[1-18,20-24]
gpu_p up 7-00:00:00 1 mix c4-23
highmem_p up 7-00:00:00 6 idle d4-[11-12],ra4-[21-24]
inter_p up 2-00:00:00 2 idle ra4-[16-17]
</pre>
where some common values of STATE are:
*STATE=idle indicates that those nodes are completely free.
*STATE=mix indicates that some cores on those nodes are in use (and some are free).
*STATE=alloc indicates that all cores on those nodes are in use.
*STATE=drain indicates that nodes are draining, not accepting new jobs
*STATE=down indicates that nodes are not running or accepting new jobs

This command can be used with many options. We have configured one option that shows some quantities that are commonly of interest, including node feature defined for each node. This command is
<pre class="gcommand">
sinfo-gacrc
</pre>
You can also specify the number of characters displayed in the NODELIST column (e.g. 40) and in the AVAIL_FEATURES column (e.g. 50), with
<pre class="gcommand">
sinfo-gacrc 40 50
</pre>

Sample output of the '''sinfo-gacrc''' command:

<pre class="gcommand">
PARTITION NODELIST STATE CPUS MEMORY AVAIL_FEATURES GRES
batch* ra4-2 drained* 32 126000 AMD,Opteron,QDR lscratch:230
batch* ra3-19 down* 32 126000 AMD,Opteron,QDR lscratch:230
batch* ra4-12 down* 32 126000 AMD,Opteron,QDR lscratch:230
batch* b1-3 mixed 64 126976 AMD,EPYC,Rome,EDR lscratch:890
batch* b1-2 allocated 64 126976 AMD,EPYC,Rome,EDR lscratch:890
batch* b1-[4-24] idle 64 126976 AMD,EPYC,Rome,EDR lscratch:890
batch* c1-3 idle 28 59127 Intel,Broadwell,EDR lscratch:890
batch* c5-19 idle 32 187868 Intel,Skylake,EDR lscratch:890
batch* d4-[5-6] idle 32 126976 AMD,EPYC,Naples,EDR lscratch:890
batch* d4-[8-12] idle 32 126976+ AMD,EPYC,Naples,EDR lscratch:890
batch* ra3-[1-18,20-24] idle 32 126000 AMD,Opteron,QDR lscratch:230
gpu_p c4-23 idle 32 187868 Intel,Skylake,EDR gpu:P100:1,lscratch:890
highmem_p d4-[11-12] idle 32 514048 AMD,EPYC,Naples,EDR lscratch:890
highmem_p ra4-[21-24] idle 32 126000 AMD,Opteron,QDR lscratch:230
inter_p ra4-[16-17] idle 32 126000 AMD,Opteron,QDR lscratch:230
scavenge_p rb7-18 idle 28 515780 Intel,Broadwell,QDR lscratch:180
</pre>

----
[[#top|Back to Top]]

===What is the scavenge_p partition===

A portion of the Sapelo2 compute nodes were purchased by UGA PIs and their group members have priority in using those resources (also referred to as buyin nodes). The GACRC purchased the rest on UGA's behalf. The agreement for the PI-owned nodes allows "other users" to also run jobs on owned nodes, as long as those jobs don't cause that lab group to wait over two hours for access to its nodes. We have implemented a partition called scavenge_p and short jobs (for example, jobs that request less than 4h) submitted to the 'batch' partition might be automatically moved into the scavenge_p partition if the 'batch' partition is busy. This is a way to reduce the wait time of the short jobs, while making use of the buyin nodes that are not in use. Jobs running on the buyin nodes (or any nodes) cannot be dynamically migrate to other nodes, so buyin-group users might have to wait up to 4h to access their nodes, if there are jobs running in the scavenge_p partition.

Users cannot submit jobs directly to the scavenge_p partition, but if you submitted short jobs to the batch partition, you might see them running on the scavenge_p partition.

----
[[#top|Back to Top]]

===How to request a specific node feature===

Each compute node has a set of features, such as shown with the sinfo-gacrc command above. Common features are Intel (if the node has Intel processors), AMD (if the node has AMD processors), EPYC (if the node has AMD EPYC processors), or specific EPYC processor types, such as Rome, Milan, etc. You can request using nodes with a specific feature by adding the following header line in your job submission script:

<pre class="gscript">
#SBATCH --constraint=featurename
</pre>
where '''featurename''' needs to be replaced by the feature you want to use. For example, to request that the job goes to a node that has a Milan processor, use
<pre class="gscript">
#SBATCH --constraint=Milan
</pre>

----
[[#top|Back to Top]]

===How to run Intel- or AMD-specific applications===

Most of the applications that GACRC installs centrally can be run on Intel and on AMD processors, but some exceptions do exist. Also, some third-party applications that you are using might have been pre-compiled for a given processor type and would fail if run on a different processor architecture If an application that you are using if only compatible with one type of processor (e.g. Intel), you can request that node feature by adding the following line in your job submission script
<pre class="gscript">
#SBATCH --constraint=Intel
</pre>
or
<pre class="gscript">
#SBATCH --constraint=EPYC
</pre>
or
<pre class="gscript">
#SBATCH --constraint=Milan
</pre>

----
[[#top|Back to Top]]

=== How to run a job using the local scratch /lscratch on a compute node ===
The IO performance of the local scratch file system /lscratch is much faster than the IO performance of the network file system /scratch. '''Please note''' that the local scratch file system can only be used for running single-node jobs, i.e., single-core jobs or multi-thread jobs. In general, MPI parallel jobs that use more than one node cannot use the local scratch file system. Detailed information and instructions about /lscratch can be found at [[Disk_Storage#lscratch_file_system]] .

To use /lscratch to run a batch job, you need a few additional steps in your job submission script to ask your job to:

# Create a job working folder in /lscratch on the compute node where your job is dispatched
# Copy any input files required to run the job from your current working space, e.g., /scratch/MyID, to the folder created in step 1
# Change directory from your current working space /scratch/MyID to the folder created in step 1 and run the software from there, i.e. from the local scratch file system /lscratch
# Copy output results from /lscratch back to your /scratch/MyID, before job finishes and exits from the node
# Clean up in /lscratch, before job finishes and exits from the node
To use /lscratch to run a batch job, you also need to:

1. Make sure that your job will use a single node by using the following line in your job submission script:

<pre class="gscript">
#SBATCH --nodes=1
</pre>

2. Request an appropriate amount of disk storage from the local scratch file system by adding the following line in your job submission script:

<pre class="gscript">
#SBATCH --gres=lscratch:200
</pre>
The above header requests 200GB local storage on the compute node where your job is dispatched.

Below is a sample job submission script (sub.sh) to run a batch job using /lscratch:
<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=RM_job
#SBATCH --partition=batch
#SBATCH --nodes=1
#SBATCH --gres=lscratch:200
#SBATCH --ntasks=12
#SBATCH --mem=36G
#SBATCH --time=7-00:00:00
#SBATCH --output=log.%j.out
#SBATCH --error=log.%j.err

cd $SLURM_SUBMIT_DIR

# Step 1
mkdir -p /lscratch/${USER}/${SLURM_JOB_ID}

# Step 2
cp ./Hawaii_H3_Final.fa /lscratch/${USER}/${SLURM_JOB_ID}

# Step 3
cd /lscratch/${USER}/${SLURM_JOB_ID}

ml RepeatModeler/2.0.4-foss-2022a

BuildDatabase -name E4 -engine ncbi Hawaii_H3_Final.fa
RepeatModeler -engine ncbi -pa 3 -database E4 > E4-repeat.out

# Step 4
cp ./E4* ${SLURM_SUBMIT_DIR}
cp -r ./RM_* ${SLURM_SUBMIT_DIR}

# Step 5
rm -rf /lscratch/${USER}/${SLURM_JOB_ID}
</pre>

Then submit sub.sh from your current working space /scratch/MyID with:

<pre class="gcommand">
sbatch sub.sh
</pre>

Since you submit the job from /scratch/MyID, the value stored in SLURM_SUBMIT_DIR in the above sub.sh will be /scratch/MyID.

To learn the total amount of local disk storage installed in compute nodes on Sapelo2, you can use '''sinfo-gacrc''' command. The '''GRES''' column reported is the information about the total amount of local disk storage in GB, for example, '''lscratch:890''' means total 890GB local disk storage is installed in the compute node(s). Detailed instructions about gacrc-sinfo can be found at [[Running_Jobs_on_Sapelo2#Discovering_if_a_partition_.28queue.29_is_busy]]

----
[[#top|Back to Top]]

===How to open an interactive session===

An interactive session on a compute node can be started with the command
<pre class="gcommand">
interact
</pre>
This command, invoked without any arguments, will start an interactive session with one core on one of the interactive nodes, and allocate 2GB of memory for a maximum walltime of 12h. It is equivalent to the <code>qlogin</code> command that we used previously, and it runs
<pre class="gcommand">
srun --pty --cpus-per-task=1 --job-name=interact --ntasks=1 --nodes=1 --partition=inter_p --time=12:00:00 --mem=2GB /bin/bash -l
</pre>
When the <code>interact</code> command is run, it will echo the equivalent srun command, so you can easily check the resources associated to your interactive session.

The <code>interact</code> command takes arguments that allow you to request cores, memory, walltime limit, specific node features, or a different partition and other resources.

The options that can be used with <code>interact</code> are diplayed when this command is run with the -h or --help option:

<pre class="gcomment">
[shtsai@ss-sub2 ~]$ interact -h

Usage: interact [OPTIONS]

Description: Start an interactive job

-c, --cpus-per-task CPU cores per task (default: 1)
-J, --job-name Job name (default: interact)
-n, --ntasks Number of tasks (default: 1)
-N, --nodes Number of nodes (default: 1)
-p, --partition Partition for interactive job (default: inter_p)
-q, --qos Request a quality of service for the job.
-t, --time Maximum run time for interactive job (default: 12:00:00)
-w, --nodelist List of node name(s) on which your job should run
--constraint Job constraints
--gres Generic consumable resources
--mem Memory per node (default 2GB)
--shell Absolute path to the shell to be used in your interactive job (default: /bin/bash)
--wckey Wckey to be used with job
--x11 Start an interactive job with X Forwarding
-h, --help Display this help output
</pre>

'''Examples:'''

To start an interactive session with 4 cores and 10GB of memory:
<pre class="gcommand">
interact -c 4 --mem=10G
</pre>

To start an interactive session with 1 core, 10GB of memory and a walltime limit of 18 hours:
<pre class="gcommand">
interact --mem=10G --time=18:00:00
</pre>

To start an interactive session with 1 core, 2GB of memory, on a node that has an AMD EPYC Milan processor in the batch partition:
<pre class="gcommand">
interact --constraint=Milan -p batch
</pre>

To start an interactive session with 1 core, 50GB of memory, and a A100 GPU device:
<pre class="gcommand">
interact -p gpu_p --gres=gpu:A100:1 --mem=50G
</pre>

----
[[#top|Back to Top]]

===How to run an interactive job with Graphical User Interface capabilities===

A number of software installed on GACRC clusters have X Window (GUI) front ends. Examples of such applications are Matlab, Mathematica, some text editors and debuggers, etc. The best way to run such applications is using the Open OnDemand (OOD) interface to Sapelo2, either by running an interactive application in OOD or by starting an X Desktop session on the cluster and running the application therein. More information is available at [[OnDemand]].

If using OnDemand is not an option, and you want to run an application as an interactive job and have its graphical
user interface displayed on the terminal of your local machine, you need to
enable X-forwarding when you ssh into the login node. This can be done in Linux
by simply adding the -X option when ssh-ing into Sapelo2. For information on how
to do this on windows and mac, please see questions 10 and 11 in the [[Frequently Asked Questions]]
page.

Then start an interactive session, but add the option --x11 to the <code>interact</code> command.

An interactive session on a compute node, with X forwarding enabled, can be started with the command
<pre class="gcommand">
interact --x11
</pre>
This command will start an interactive session, with X forwarding enabled, with one core on one of the interactive nodes, and allocate 2GB of memory for a maximum walltime of 12h.

The <code>interact --x11</code> command is an alias for
<pre class="gcommand">
srun --pty --x11 --cpus-per-task=1 --job-name=interact --ntasks=1 --nodes=1 --partition=inter_p --time=12:00:00 --mem=2GB /bin/bash -l
</pre>

The options available to <code>interact</code>, described in the previous section, can be used along with the <code>--x11</code> option.

----
[[#top|Back to Top]]



===How to check on running or pending jobs===

To list all running and pending jobs (by all users), use the command
<pre class="gcommand">
squeue
</pre>
or
<pre class="gcommand">
squeue -l
</pre>
This command can be used with many options. We have wrapper to this command, called <code>sq</code> that shows some quantities that are commonly of interest. To use the <code>sq</code> command to list all of your running and pending jobs, use
<pre class="gcommand">
sq --me
</pre>

For detailed information on how to monitor your jobs, please see [[Monitoring Jobs on Sapelo2]].

----
[[#top|Back to Top]]

===How to cancel (delete) a running or pending job===

To cancel one of your running or pending job, use the command
<pre class="gcommand">
scancel <jobid>
</pre>
For example, to cancel a job with Job ID 12345 use
<pre class="gcommand">
scancel 12345
</pre>

To cancel all of your jobs, use the command
<pre class="gcommand">
scancel -u MyID
</pre>

To cancel all of your pending jobs, use the command
<pre class="gcommand">
scancel -t PENDING -u MyID
</pre>

To cancel one or more jobs by job name, use the command
<pre class="gcommand">
scancel --name <myJobName>
</pre>

To cancel an element (index) of an array job
<pre class="gcommand">
scancel <jobid>_<index>
</pre>
For example, to cancel array job element 4 of an array job whose Job ID is 12345 use
<pre class="gcommand">
scancel 12345_4
</pre>

----
[[#top|Back to Top]]

===How to check resource utilization of a running or finished job===

The following command can be used to show resource utilization by a running job or a job that has already completed:
<pre class="gcommand">
sacct
</pre>

This command can be used with many options. We have configured one option that shows some quantities that are commonly of interest, including the amount of memory used and the cputime used by the jobs:
<pre class="gcommand">
sacct-gacrc
</pre>

For detailed information on how to monitor your jobs, please see [[Monitoring Jobs on Sapelo2]].

----
[[#top|Back to Top]]

Running Jobs on Sapelo2

2024-07-05T15:00:32Z

Jerky: Increased '2TB' to '3TB' in hugemem desc due to new nodes RA4-[3-5].

[[Category:Sapelo2]]

===Using the Queueing System===

The login node for the Sapelo2 cluster should be used for text editing, and job submissions. '''No jobs should be run directly on the login node.'''
Processes that use too much CPU or RAM on the login node may be terminated by GACRC staff, or automatically, in order to keep the cluster running properly. Jobs should
be run using the Slurm queueing system. The queueing system should be used to run both interactive and batch jobs.

----
[[#top|Back to Top]]

===Batch partitions (queues) defined on the Sapelo2===

There are different partitions defined on Sapelo2. The Slurm queueing system refers to queues as partition. Users are required to specify, in the job submission script or as job submission command line arguments, the partition and the resources needed by the job in order for it to be assigned to compute node(s) that have enough available resources (such as number of cores, amount of memory, GPU cards, etc). Please note, Slurm will not allow a job to be submitted if there are no resources matching your request. Please refer to [[Migrating from Torque to Slurm]] for more info about Slurm queueing system.

The following partitions are defined on the Sapelo2 cluster:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | Time limit
! scope="col" | Max jobs
! scope="col" | Notes
|-
|-
| batch || 7 days || || Regular nodes.
|-
| batch-30d || 30 days || 2 || Regular nodes. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| highmem_p || 7 days || || For high memory jobs
|-
| highmem_30d_p || 30 days || 2 || For high memory jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
|hugemem_p || 7 days ||4 || For jobs needing up to 3TB of memory.
|-
|hugemem_30d_p || 30 days || 4 || For jobs needing up to 3TB of memory.
|-
| gpu_p || 7 days || || For GPU-enabled jobs.
|-
| gpu_30d_p || 30 days || 2 || For GPU-enabled jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| inter_p || 2 days || || Regular nodes, for interactive jobs.
|-
| '''name'''_p || variable || || Partitions that target different groups' buy-in nodes. The '''name''' string is specific to each group.
|-
| scavenge_p || 2 hours || || Partition that targets the buy-in nodes. When there are no available resources in the batch partition, short jobs submitted there might be automatically transferred into scavenge_p, to run on idle buy-in resources. Jobs cannot be submitted directly to this partition.
|-
|}
For more detailed information about the partitions, please see [[Job Submission partitions on Sapelo2]].

The table below summarizes the partitions (queues) defined and the compute nodes that they target:
{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | Node Features
! scope="col" | Node Number
! scope="col" | Description
! scope="col" | Memory for jobs
! scope="col" | Notes
|-
|-
| batch, batch_30d || AMD, Opteron, QDR || || 48-core, 128GB RAM, AMD Opteron, QDR IB interconnect || 122GB || Regular nodes.
|-
| batch, batch_30d || AMD, EPYC, EDR || || 64-core, 128GB RAM, AMD EPYC, IB EDR interconnect || 120GB || Regular nodes
|-
| batch, batch_30d || AMD, EPYC, EDR || || 32-core, 128GB RAM, AMD EPYC, IB EDR interconnect || 120GB || Regular nodes
|-
| batch, batch_30d || AMD, Opteron, QDR || || 48-core, 256GB RAM, AMD Opteron, QDR IB interconnect || 250GB || Regular nodes.
|-
| batch, batch_30d || Intel, Skylake, EDR || || 32-core, 192GB RAM, Intel Xeon Skylake, IB EDR interconnect || 180GB || Regular nodes
|-
| batch, batch_30d || Intel, Broadwell, EDR || || 28-core, 64GB RAM, Intel Xeon Broadwell, IB EDR interconnect || 58GB || Regular nodes
|-
| highmem_p, highmem_30d_p || AMD, EPYC, EDR || || 64-core, 1TB RAM, AMD EPYC, IB EDR interconnect || 950GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || Intel, EDR || || 32-core, 1TB RAM, Intel, IB EDR interconnect || 950GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || AMD, Opteron, EDR || || 48-core, 1TB RAM, AMD Opteron, IB EDR interconnect || 950GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || AMD, Opteron, QDR || || 48-core, 512GB, AMD Opteron, IB QDR interconnect || 500GB || For high memory jobs
|-
| highmem_p, highmem_30d_p || AMD, EPYC, EDR || || 32-core, 512GB RAM, AMD EPYC, IB EDR interconnect || 490GB || For high memory jobs
|-
| hugemem_p, hugemem_30d_p || AMD, EPYC, EDR || || 32-core, 2TB RAM, AMD EPYC, IB EDR interconnect || 2000GB || For high memory jobs
|-
| gpu_p, gpu_30d_p || GPU, A100, EDR || || 64-core, 1000GB RAM, AMD EPYC, 4 NVIDIA A100 GPUs, EDR IB interconnect || 1000GB || For GPU-enabled jobs.
|-
| gpu_p, gpu_30d_p || GPU, P100, EDR || || 32-core, 192GB RAM, Intel Xeon Skylake, 1 NVIDIA P100 GPUs, EDR IB interconnect || 180GB || For GPU-enabled jobs.
|-
| gpu_p, gpu_30d_p || GPU, K40, QDR || || 16-core, 128GB RAM, Intel Xeon , 8 NVIDIA K40 GPUs, QDR IB interconnect || 120GB || For GPU-enabled jobs.
|-
| gpu_p, gpu_30d_p || GPU, K20, QDR || || 12-core, 96GB RAM, Intel Xeon , 7 NVIDIA K20Xm GPUs, QDR IB interconnect || 70GB || For GPU-enabled jobs.
|-

|}

You can check all partitions (queues) defined in the cluster with the command
<pre class="gcommand">
sinfo
</pre>

----
[[#top|Back to Top]]

===Job submission Scripts===

Users are required to specify the number of cores, the amount of memory, the partition (queue) name, and the maximum wallclock time needed by the job.

====Header lines====

'''Basic job submission script'''

At a minimum, the job submission script needs to have the following header lines:
<pre class="gscript">
#!/bin/bash
#SBATCH --partition=batch
#SBATCH --job-name=test
#SBATCH --ntasks=1
#SBATCH --time=4:00:00
#SBATCH --mem=10G
</pre>

Commands to run your application should be added after these header lines.

'''Header lines explained:'''

* '''#!/bin/bash''': specify Linux default shell bash
* '''#SBATCH --partition=batch''' : specify the partition (queue) to run on, e.g. ''batch''
* '''#SBATCH --job-name=test''' : specify the job name, e.g. ''test''
* '''#SBATCH --ntasks=1''' : specify the number of tasks (e.g. 1)
* '''#SBATCH --time=4:00:00''' : specify the maximum walltime of the job in the format D-HH:MM:SS (e.g. --time=1- for one day or --time=4:00:00 for 4 hours)
* '''#SBATCH --mem=10G''' : specify the maximum memory per node required by the job (e.g. 10GB)

Below are some of the most commonly used queueing system options to configure the job.

====Options to request resources for the job====

* -t, --time=time
Wall clock time limit of a job running on cluster. Acceptable formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes", and "days-hours:minutes:seconds". '''This is a required option.'''

* --mem=num
Maximum amount of memory in MegaBytes per node required by the job. Different units can be specified using the suffix [K|M|G|T].

* --mem-per-cpu=num
Minimum amount of memory in MegaBytes per allocated CPU. Different units can be specified using the suffix [K|M|G|T].

* -n, --ntasks=num
Number of tasks to run. The default is one task per node. For use with distributed parallelism. See below.

* -N, --nodes=num
Number of nodes allocated to the job. Default is one node.

* --ntasks-per-node=num
Number of tasks invoked on each node. Meant to be used with the --nodes option. For use with distributed parallelism. See below.

* -c, --cpus-per-task=ncpus
Number of CPUs allocated to each task. For use with shared memory parallelism. See below.

* -C, --constraint=<list>
List of node features required by the job. Only nodes having features matching the job constraints will be used to satisfy the request. Multiple constraints may be specified with AND, OR, matching OR, resource counts, etc.

* --gres=<list>
A comma delimited list of generic consumable resources. For example, to request one P100 GPU card: --gres=gpu:P100:1

Please try to request resources for your job as accurately as possible, because this allows your job to be dispatched to run at the earliest opportunity and it helps the system allocate resources efficiently to start as many jobs as possible, benefiting all users.

====Options to manage job notification and output====

* -J, --job-name jobname
Specify a name for the job. The specified name will appear along with the job id number when querying running jobs on the system. The default is the supplied executable program's name. Within the job, $SBATCH_JOB_NAME expands to the job name.

* -o, --output=path/for/stdout
Send stdout to path/for/stdout. The default filename is slurm-${SLURM_JOB_ID}.out, e.g. slurm-12345.out, in the directory from which the job was submitted.

* -e, --error=path/for/stderr
Send stderr to path/for/stderr. If --error is not specified, both stdout and stderr will directed to the file specified by --output.

* --mail-user=username@uga.edu
Send email notification to the address you specified when certain events occur.

* --mail-type=type
Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL, TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 and TIME_LIMIT_50.

By default, email notifications set for an array job will generate one email message for the array job. If you would like to receive an email message for individual array job elements (up to a certain limit), please add ARRAY_TASKS to the --mail-type option.

====Options to set Array Jobs====
If you wish to run an application binary or script using e.g. different input files, then you might find it convenient to use an array job. To create an array job with e.g. 10 elements, use
<pre class="gscript">
#SBATCH -a 0-9
</pre>
or
<pre class="gscript">
#SBATCH --array=0-9
</pre>

Each array job element runs as an independent job, so multiple elements can run concurrently if resources are available. For this reason, the job ID which is stored in SLURM_JOB_ID for each element in an array job will be different and unique. The ID of each element in an array job, i.e., array element index value, is stored in SLURM_ARRAY_TASK_ID. The ID of an array job as whole is stored in SLURM_ARRAY_JOB_ID. For this reason, it will be the same for all elements in an array job. The JodID reported by sq command is a combination of SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID connected by "_".

<pre class="gscript">
sbatch --array=1-3 -N1 sub.sh

will generate a job array containing three jobs. If the sbatch command responds
Submitted batch job 36
then the environment variables will be set as follows:

SLURM_JOB_ID=36
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=37
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=38
SLURM_ARRAY_JOB_ID=36
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1
</pre>

Most Slurm commands recognize the SLURM_ARRAY_JOB_ID plus SLURM_ARRAY_TASK_ID values separated by an underscore as identifying an element of a job array, for example, 36_2 would be equivalent ways to identify the second array element of array job 36.

For more information, please see [[Array Jobs]].

====Option to set job dependency====
You can set job dependency with the option -d or --dependency=''dependency-list''. For example, if you want to specify that one job starts to run after the job 1234 and 1235 have successfully executed (ran to completion with an exit code of zero), you can add the following header line in the job submission script of the job:
<pre class="gscript">
#SBATCH --dependency=afterok:1234:1235
</pre>
Having this header line in the job submission script will ensure that the job is only dispatched to run after job 1234 and 1235 have completed successfully.

You can also use the following header line to specify that one job starts to run after the job 1236 and 1237 start or are cancelled:
<pre class="gscript">
#SBATCH --dependency=after:1236:1237
</pre>

====Options to requeue or not requeue a job when a node crashes====

If a job is running and one or more nodes that it is using crash, the job will stop running and, by default, it will get requeued. When resources become available, the job will start running again, from the beginning, unless the program saves intermediate results and it is able to automatically pick up from where it stopped. The files with the standard error and standard output of the job will get rewritten once the job restarts. Often other output files will get rewritten as well.

If you are running a program that cannot restart, e.g. the program will fail if a certain output file or directory has already been created, or if you would like to preserve the partial results, you can use the following option to prevent the job from being requeued:
<pre class="gscript">
#SBATCH --no-requeue
</pre>
When this option is used, the job will simply stop if a node crashes, it will not be requeued. In this case partial results and the standard error and output of the job will not get overwritten.

Although requeueing jobs is enabled by default now, you can also add the option below if you would like to ensure a job is requeued in the event of a node crash:
<pre class="gscript">
#SBATCH --requeue
</pre>

====Other content of the script====

Following the header lines, users can include commands to change to the working directory, to load the modules needed to run the application, and to invoke the application. For example, to use the directory from which the job is submitted as the working directory (where to find input files or binaries), add the line
<pre class="gscript">
cd $SLURM_SUBMIT_DIR
</pre>
(Note that Slurm jobs start from the submit directory by default, so adding the line above might not be necessary.)

You can then load the needed modules. For example, if you are running an R program, then include the line
<pre class="gscript">
module load R/4.3.1-foss-2022a
</pre>

Then invoke your application. For example, if you are running an R program called add.R which is in your job submission directory, use
<pre class="gscript">
R CMD BATCH add.R
</pre>

====Environment Variables exported by batch jobs====

When a batch job is started, a number of variables are introduced into the job's environment that can be used by the batch script in making decisions, creating output files, and so forth. Some of these variables are listed in the following table:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Variable
! scope="col" | Description
|-
| SLURM_ARRAY_JOB_ID || Job array's master job ID number, i.e., the first Slurm job id of a job array
|-
| SLURM_ARRAY_TASK_COUNT || Total number of tasks (elements) in a job array
|-
| SLURM_ARRAY_TASK_ID || Job array ID (index) number
|-
| SLURM_ARRAY_TASK_MAX || Job array's maximum ID (index) number
|-
| SLURM_ARRAY_TASK_MIN || Job array's minimum ID (index) number
|-
| SLURM_CPUS_ON_NODE || Number of CPUS on the allocated node
|-
| SLURM_CPUS_PER_TASK || Number of cpus requested per task. Only set if the --cpus-per-task option is specified
|-
| SLURM_JOB_ID || Unique Slurm job id
|-
| SLURM_JOB_NAME || Job name
|-
| SLURM_JOB_CPUS_PER_NODE || Count of processors available to the job on this node
|-
| SLURM_JOB_NODELIST || List of nodes allocated to the job
|-
| SLURM_JOB_NUM_NODES ||Total number of nodes in the job's resource allocation
|-
| SLURM_JOB_PARTITION || Name of the partition (i.e. queue) in which the job is running
|-
| SLURM_MEM_PER_NODE || Same as --mem
|-
| SLURM_MEM_PER_CPU || Same as --mem-per-cpu
|-
| SLURM_NTASKS || Same as -n, --ntasks
|-
| SLURM_NTASKS_PER_NODE || Number of tasks requested per node. Only set if the --ntasks-per-node option is specified
|-
| SLURM_SUBMIT_DIR || The directory from which sbatch was invoked
|-
| SLURM_SUBMIT_HOST || The hostname of the computer from which sbatch was invoked
|-
| SLURM_TASK_PID || The process ID of the task being started
|-
| SLURMD_NODENAME || Name of the node running the job script
|-
| CUDA_VISIBLE_DEVICES || GPU devide ID that assigned to the job to use
|-
|}

----
[[#top|Back to Top]]

===Sample job submission scripts===

====Serial (single-processor) Job====

Sample job submission script (sub.sh) to run an R program called add.R using a single core:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=testserial # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=testserial.%j.out # Standard output log
#SBATCH --error=testserial.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load R/4.3.1-foss-2022a

R CMD BATCH add.R
</pre>

In this sample script, the standard output and error of the job will be saved into a file called testserial.o%j, where %j will be automatically replaced by the job id of the job.

====Serial (single-processor) Job on an AMD EPYC Milan processor====

Sample job submission script (sub.sh) to run an R program called add.R using a single core:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=testserial # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --constraint=Milan # node feature
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=testserial.%j.out # Standard output log
#SBATCH --error=testserial.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load R/4.3.1-foss-2022a

R CMD BATCH add.R
</pre>

In this sample script, the standard output and error of the job will be saved into a file called testserial.%j.out and testserial.%j.err, where %j will be automatically replaced by the job id of the job.

====MPI Job====

Sample job submission script (sub.sh) to run an OpenMPI application. In this example the job requests 16 cores and further specifies that these 16 cores need to be divided equally on 2 nodes (8 cores per node) and the binary is called mympi.exe:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=mpitest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks=16 # Number of MPI ranks
#SBATCH --ntasks-per-node=8 # How many tasks on each node
#SBATCH --cpus-per-task=1 # Number of cores per MPI rank
#SBATCH --mem-per-cpu=600mb # Memory per processor
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=mpitest.%j.out # Standard output log
#SBATCH --error=mpitest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load OpenMPI/4.1.4-GCC-11.3.0

srun ./mympi.exe
</pre>

Please note that you need to start the application with '''srun''' and not with '''mpirun''' or '''mpiexec'''.

'''Important note:''' MPI jobs need to be submitted from a Sapelo2 login node, not from an interactive session, in order to get the correct core allocation for the MPI processes.

====MPI Job on nodes connected via the EDR IB fabric====

Sample job submission script (sub.sh) to run an OpenMPI application. In this example the job requests 16 cores and further specifies that these 16 cores need to be divided equally on 2 nodes (8 cores per node) and the binary is called mympi.exe:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=mpitest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --constraint=EDR # node feature
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks=16 # Number of MPI ranks
#SBATCH --ntasks-per-node=8 # How many tasks on each node
#SBATCH --cpus-per-task=1 # Number of cores per MPI rank
#SBATCH --mem-per-cpu=600mb # Memory per processor
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=mpitest.%j.out # Standard output log
#SBATCH --error=mpitest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load OpenMPI/4.1.4-GCC-11.3.0

srun ./mympi.exe
</pre>

Please note that you need to start the application with '''srun''' and not with '''mpirun''' or '''mpiexec'''.

'''Important note:''' MPI jobs need to be submitted from a Sapelo2 login node, not from an interactive session, in order to get the correct core allocation for the MPI processes.

====OpenMP (Multi-Thread) Job====

Sample job submission script (sub.sh) to run a program that uses OpenMP with 6 threads. Please set '''--ntasks=1''' and set '''--cpus-per-task''' to the number of threads you wish to use. The name of the binary in this example is a.out.

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=mctest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=6 # Number of CPU cores per task
#SBATCH --mem=4gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=mctest.%j.out # Standard output log
#SBATCH --error=mctest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=6

module load foss/2022a # load the appropriate module file, e.g. foss/2022a

time ./a.out
</pre>

====High Memory Job====

Sample job submission script (sub.sh) to run a velvet application that needs to use 200GB of memory and 4 threads:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=highmemtest # Job name
#SBATCH --partition=highmem_p # Partition (queue) name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=4 # Number of CPU cores per task
#SBATCH --mem=200gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=highmemtest.%j.out # Standard output log
#SBATCH --error=highmemtest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

export OMP_NUM_THREADS=4

module load Velvet

velvetg [options]
</pre>

====Hybrid MPI/shared-memory using OpenMPI====

Sample job submission script (sub.sh) to run a parallel job that uses 4 MPI processes with OpenMPI and each MPI process runs with 3 threads:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=hybridtest
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --nodes=2 # Number of nodes
#SBATCH --ntasks=8 # Number of MPI ranks
#SBATCH --ntasks-per-node=4 # Number of MPI ranks per node
#SBATCH --cpus-per-task=3 # Number of OpenMP threads for each MPI process/rank
#SBATCH --mem-per-cpu=2000mb # Per processor memory request
#SBATCH --time=2-00:00:00 # Walltime in hh:mm:ss or d-hh:mm:ss (2 days in the example)
#SBATCH --output=hybridtest.%j.out # Standard output log
#SBATCH --error=hybridtest.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

module load OpenMPI/4.1.4-GCC-11.3.0

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun ./myhybridprog.exe

</pre>

====Array job====

Sample job submission script (sub.sh) to submit an array job with 10 elements. In this example, each array job element will run the a.out binary using an input file called input_0, input_1, ..., input_9.
<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=arrayjobtest # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --mem=1gb # Job Memory
#SBATCH --time=10:00:00 # Time limit hrs:min:sec
#SBATCH --output=array_%A-%a.out # Standard output log
#SBATCH --error=array_%A-%a.err # Standard error log
#SBATCH --array=0-9 # Array range

cd $SLURM_SUBMIT_DIR

module load foss/2022a # load any needed module files, e.g. foss/2022a

time ./a.out < input_$SLURM_ARRAY_TASK_ID

</pre>

For more information, please see [[Array Jobs]].

====GPU/CUDA====

Sample script to run Amber on a GPU node using one node, 2 CPU cores, and 1 GPU card:
<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=amber # Job name
#SBATCH --partition=gpu_p # Partition (queue) name
#SBATCH --gres=gpu:1 # Requests one GPU device
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=2 # Number of CPU cores per task
#SBATCH --mem=40gb # Job memory request
#SBATCH --time=10:00:00 # Time limit hrs:min:sec
#SBATCH --output=amber.%j.out # Standard output log
#SBATCH --error=amber.%j.err # Standard error log

#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

ml Amber/22.0-foss-2021b-AmberTools-22.3-CUDA-11.4.1

$AMBERHOME/bin/pmemd.cuda -O -i ./prod.in -o prod.out -p ./dimerFBP_GOL.prmtop -c ./restart.rst -r prod.rst -x prod.mdcrd
</pre>
You can use the option <code>#SBATCH --gres=gpu:P100:1</code> or <code>#SBATCH --gres=gpu:A100:1</code> to specify using a P100 or an A100 GPU device, respectively. To use an A100 GPU device, please explicitly request it with <code>#SBATCH --gres=gpu:A100:1</code>. Jobs that request a GPU, but that do not specify the device type (that is, jobs that use <code>#SBATCH --gres=gpu:1</code>) will get allocated a P100 device and will not get allocated an A100 device.

====Singularity job====

Sample job submission script (sub.sh) to run a program (e.g. sortmerna) using a singularity container:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=j_sortmerna # Job name
#SBATCH --partition=batch # Partition (queue) name
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --output=sortmerna.%j.out # Standard output log
#SBATCH --error=sortmerna.%j.err # Standard error log
#SBATCH --cpus-per-task=4 # Number of CPU cores per task
#SBATCH --mail-type=END,FAIL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=username@uga.edu # Where to send mail (change username@uga.edu to your email address)

cd $SLURM_SUBMIT_DIR

singularity exec /apps/singularity-images/sortmerna-3.0.3.simg sortmerna \
--threads 4 --ref db.fasta,db.idx --reads file.fa --aligned base_name_output
</pre>
For more information about software installed as singularity containers on the cluster, please see [[Software_on_Sapelo2#Singularity_Containers]]

To run a GPU-enabled singularity container on the GPU, please submit the job to the gpu_p partition, request a GPU device and add the '''--nv''' option to the singularity command.

Sample job submission script (sub.sh) to run a program using a singularity container (e.g. gpuapp.sif) on the GPU device:

<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=myjobname # Job name
#SBATCH --partition=gpu_p # Partition (queue) name
#SBATCH --gres=gpu:1 # Requests one GPU device
#SBATCH --ntasks=1 # Run on a single CPU
#SBATCH --mem=10gb # Job memory request
#SBATCH --time=02:00:00 # Time limit hrs:min:sec
#SBATCH --cpus-per-task=1 # Number of CPU cores per task

cd $SLURM_SUBMIT_DIR

singularity exec --nv /apps/singularity-images/gpuapp.sif prog.x
</pre>
For more information about software installed as singularity containers on the cluster, please see [[Software_on_Sapelo2#Singularity_Containers]]

----
[[#top|Back to Top]]

===How to submit a batch job===

With the resource requirements specified in the job submission script (sub.sh), submit your job with
<pre class="gcommand">
sbatch <scriptname>
</pre>
For example
<pre class="gcommand">
sbatch sub.sh
</pre>
Once the job is submitted, the Job ID of the job (e.g. 12345) will be printed on the screen.

----
[[#top|Back to Top]]

===Discovering if a partition (queue) is busy===
The nodes allocated to each partition (queue) and their state can be view with the command
<pre class="gcommand">
sinfo
</pre>

Sample output of the '''sinfo''' command:

<pre class="gcommand">
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
batch* up 7-00:00:00 1 drain* ra4-2
batch* up 7-00:00:00 3 down* d4-7,ra3-19,ra4-12
batch* up 7-00:00:00 1 mix b1-2
batch* up 7-00:00:00 1 alloc b1-3
batch* up 7-00:00:00 53 idle b1-[4-24],c1-3,c5-19,d4-[5-6,8-12],ra3-[1-18,20-24]
gpu_p up 7-00:00:00 1 mix c4-23
highmem_p up 7-00:00:00 6 idle d4-[11-12],ra4-[21-24]
inter_p up 2-00:00:00 2 idle ra4-[16-17]
</pre>
where some common values of STATE are:
*STATE=idle indicates that those nodes are completely free.
*STATE=mix indicates that some cores on those nodes are in use (and some are free).
*STATE=alloc indicates that all cores on those nodes are in use.
*STATE=drain indicates that nodes are draining, not accepting new jobs
*STATE=down indicates that nodes are not running or accepting new jobs

This command can be used with many options. We have configured one option that shows some quantities that are commonly of interest, including node feature defined for each node. This command is
<pre class="gcommand">
sinfo-gacrc
</pre>
You can also specify the number of characters displayed in the NODELIST column (e.g. 40) and in the AVAIL_FEATURES column (e.g. 50), with
<pre class="gcommand">
sinfo-gacrc 40 50
</pre>

Sample output of the '''sinfo-gacrc''' command:

<pre class="gcommand">
PARTITION NODELIST STATE CPUS MEMORY AVAIL_FEATURES GRES
batch* ra4-2 drained* 32 126000 AMD,Opteron,QDR lscratch:230
batch* ra3-19 down* 32 126000 AMD,Opteron,QDR lscratch:230
batch* ra4-12 down* 32 126000 AMD,Opteron,QDR lscratch:230
batch* b1-3 mixed 64 126976 AMD,EPYC,Rome,EDR lscratch:890
batch* b1-2 allocated 64 126976 AMD,EPYC,Rome,EDR lscratch:890
batch* b1-[4-24] idle 64 126976 AMD,EPYC,Rome,EDR lscratch:890
batch* c1-3 idle 28 59127 Intel,Broadwell,EDR lscratch:890
batch* c5-19 idle 32 187868 Intel,Skylake,EDR lscratch:890
batch* d4-[5-6] idle 32 126976 AMD,EPYC,Naples,EDR lscratch:890
batch* d4-[8-12] idle 32 126976+ AMD,EPYC,Naples,EDR lscratch:890
batch* ra3-[1-18,20-24] idle 32 126000 AMD,Opteron,QDR lscratch:230
gpu_p c4-23 idle 32 187868 Intel,Skylake,EDR gpu:P100:1,lscratch:890
highmem_p d4-[11-12] idle 32 514048 AMD,EPYC,Naples,EDR lscratch:890
highmem_p ra4-[21-24] idle 32 126000 AMD,Opteron,QDR lscratch:230
inter_p ra4-[16-17] idle 32 126000 AMD,Opteron,QDR lscratch:230
scavenge_p rb7-18 idle 28 515780 Intel,Broadwell,QDR lscratch:180
</pre>

----
[[#top|Back to Top]]

===What is the scavenge_p partition===

A portion of the Sapelo2 compute nodes were purchased by UGA PIs and their group members have priority in using those resources (also referred to as buyin nodes). The GACRC purchased the rest on UGA's behalf. The agreement for the PI-owned nodes allows "other users" to also run jobs on owned nodes, as long as those jobs don't cause that lab group to wait over two hours for access to its nodes. We have implemented a partition called scavenge_p and short jobs (for example, jobs that request less than 4h) submitted to the 'batch' partition might be automatically moved into the scavenge_p partition if the 'batch' partition is busy. This is a way to reduce the wait time of the short jobs, while making use of the buyin nodes that are not in use. Jobs running on the buyin nodes (or any nodes) cannot be dynamically migrate to other nodes, so buyin-group users might have to wait up to 4h to access their nodes, if there are jobs running in the scavenge_p partition.

Users cannot submit jobs directly to the scavenge_p partition, but if you submitted short jobs to the batch partition, you might see them running on the scavenge_p partition.

----
[[#top|Back to Top]]

===How to request a specific node feature===

Each compute node has a set of features, such as shown with the sinfo-gacrc command above. Common features are Intel (if the node has Intel processors), AMD (if the node has AMD processors), EPYC (if the node has AMD EPYC processors), or specific EPYC processor types, such as Rome, Milan, etc. You can request using nodes with a specific feature by adding the following header line in your job submission script:

<pre class="gscript">
#SBATCH --constraint=featurename
</pre>
where '''featurename''' needs to be replaced by the feature you want to use. For example, to request that the job goes to a node that has a Milan processor, use
<pre class="gscript">
#SBATCH --constraint=Milan
</pre>

----
[[#top|Back to Top]]

===How to run Intel- or AMD-specific applications===

Most of the applications that GACRC installs centrally can be run on Intel and on AMD processors, but some exceptions do exist. Also, some third-party applications that you are using might have been pre-compiled for a given processor type and would fail if run on a different processor architecture If an application that you are using if only compatible with one type of processor (e.g. Intel), you can request that node feature by adding the following line in your job submission script
<pre class="gscript">
#SBATCH --constraint=Intel
</pre>
or
<pre class="gscript">
#SBATCH --constraint=EPYC
</pre>
or
<pre class="gscript">
#SBATCH --constraint=Milan
</pre>

----
[[#top|Back to Top]]

=== How to run a job using the local scratch /lscratch on a compute node ===
The IO performance of the local scratch file system /lscratch is much faster than the IO performance of the network file system /scratch. '''Please note''' that the local scratch file system can only be used for running single-node jobs, i.e., single-core jobs or multi-thread jobs. In general, MPI parallel jobs that use more than one node cannot use the local scratch file system. Detailed information and instructions about /lscratch can be found at [[Disk_Storage#lscratch_file_system]] .

To use /lscratch to run a batch job, you need a few additional steps in your job submission script to ask your job to:

# Create a job working folder in /lscratch on the compute node where your job is dispatched
# Copy any input files required to run the job from your current working space, e.g., /scratch/MyID, to the folder created in step 1
# Change directory from your current working space /scratch/MyID to the folder created in step 1 and run the software from there, i.e. from the local scratch file system /lscratch
# Copy output results from /lscratch back to your /scratch/MyID, before job finishes and exits from the node
# Clean up in /lscratch, before job finishes and exits from the node
To use /lscratch to run a batch job, you also need to:

1. Make sure that your job will use a single node by using the following line in your job submission script:

<pre class="gscript">
#SBATCH --nodes=1
</pre>

2. Request an appropriate amount of disk storage from the local scratch file system by adding the following line in your job submission script:

<pre class="gscript">
#SBATCH --gres=lscratch:200
</pre>
The above header requests 200GB local storage on the compute node where your job is dispatched.

Below is a sample job submission script (sub.sh) to run a batch job using /lscratch:
<pre class="gscript">
#!/bin/bash
#SBATCH --job-name=RM_job
#SBATCH --partition=batch
#SBATCH --nodes=1
#SBATCH --gres=lscratch:200
#SBATCH --ntasks=12
#SBATCH --mem=36G
#SBATCH --time=7-00:00:00
#SBATCH --output=log.%j.out
#SBATCH --error=log.%j.err

cd $SLURM_SUBMIT_DIR

# Step 1
mkdir -p /lscratch/${USER}/${SLURM_JOB_ID}

# Step 2
cp ./Hawaii_H3_Final.fa /lscratch/${USER}/${SLURM_JOB_ID}

# Step 3
cd /lscratch/${USER}/${SLURM_JOB_ID}

ml RepeatModeler/2.0.4-foss-2022a

BuildDatabase -name E4 -engine ncbi Hawaii_H3_Final.fa
RepeatModeler -engine ncbi -pa 3 -database E4 > E4-repeat.out

# Step 4
cp ./E4* ${SLURM_SUBMIT_DIR}
cp -r ./RM_* ${SLURM_SUBMIT_DIR}

# Step 5
rm -rf /lscratch/${USER}/${SLURM_JOB_ID}
</pre>

Then submit sub.sh from your current working space /scratch/MyID with:

<pre class="gcommand">
sbatch sub.sh
</pre>

Since you submit the job from /scratch/MyID, the value stored in SLURM_SUBMIT_DIR in the above sub.sh will be /scratch/MyID.

To learn the total amount of local disk storage installed in compute nodes on Sapelo2, you can use '''sinfo-gacrc''' command. The '''GRES''' column reported is the information about the total amount of local disk storage in GB, for example, '''lscratch:890''' means total 890GB local disk storage is installed in the compute node(s). Detailed instructions about gacrc-sinfo can be found at [[Running_Jobs_on_Sapelo2#Discovering_if_a_partition_.28queue.29_is_busy]]

----
[[#top|Back to Top]]

===How to open an interactive session===

An interactive session on a compute node can be started with the command
<pre class="gcommand">
interact
</pre>
This command, invoked without any arguments, will start an interactive session with one core on one of the interactive nodes, and allocate 2GB of memory for a maximum walltime of 12h. It is equivalent to the <code>qlogin</code> command that we used previously, and it runs
<pre class="gcommand">
srun --pty --cpus-per-task=1 --job-name=interact --ntasks=1 --nodes=1 --partition=inter_p --time=12:00:00 --mem=2GB /bin/bash -l
</pre>
When the <code>interact</code> command is run, it will echo the equivalent srun command, so you can easily check the resources associated to your interactive session.

The <code>interact</code> command takes arguments that allow you to request cores, memory, walltime limit, specific node features, or a different partition and other resources.

The options that can be used with <code>interact</code> are diplayed when this command is run with the -h or --help option:

<pre class="gcomment">
[shtsai@ss-sub2 ~]$ interact -h

Usage: interact [OPTIONS]

Description: Start an interactive job

-c, --cpus-per-task CPU cores per task (default: 1)
-J, --job-name Job name (default: interact)
-n, --ntasks Number of tasks (default: 1)
-N, --nodes Number of nodes (default: 1)
-p, --partition Partition for interactive job (default: inter_p)
-q, --qos Request a quality of service for the job.
-t, --time Maximum run time for interactive job (default: 12:00:00)
-w, --nodelist List of node name(s) on which your job should run
--constraint Job constraints
--gres Generic consumable resources
--mem Memory per node (default 2GB)
--shell Absolute path to the shell to be used in your interactive job (default: /bin/bash)
--wckey Wckey to be used with job
--x11 Start an interactive job with X Forwarding
-h, --help Display this help output
</pre>

'''Examples:'''

To start an interactive session with 4 cores and 10GB of memory:
<pre class="gcommand">
interact -c 4 --mem=10G
</pre>

To start an interactive session with 1 core, 10GB of memory and a walltime limit of 18 hours:
<pre class="gcommand">
interact --mem=10G --time=18:00:00
</pre>

To start an interactive session with 1 core, 2GB of memory, on a node that has an AMD EPYC Milan processor in the batch partition:
<pre class="gcommand">
interact --constraint=Milan -p batch
</pre>

To start an interactive session with 1 core, 50GB of memory, and a A100 GPU device:
<pre class="gcommand">
interact -p gpu_p --gres=gpu:A100:1 --mem=50G
</pre>

----
[[#top|Back to Top]]

===How to run an interactive job with Graphical User Interface capabilities===

A number of software installed on GACRC clusters have X Window (GUI) front ends. Examples of such applications are Matlab, Mathematica, some text editors and debuggers, etc. The best way to run such applications is using the Open OnDemand (OOD) interface to Sapelo2, either by running an interactive application in OOD or by starting an X Desktop session on the cluster and running the application therein. More information is available at [[OnDemand]].

If using OnDemand is not an option, and you want to run an application as an interactive job and have its graphical
user interface displayed on the terminal of your local machine, you need to
enable X-forwarding when you ssh into the login node. This can be done in Linux
by simply adding the -X option when ssh-ing into Sapelo2. For information on how
to do this on windows and mac, please see questions 10 and 11 in the [[Frequently Asked Questions]]
page.

Then start an interactive session, but add the option --x11 to the <code>interact</code> command.

An interactive session on a compute node, with X forwarding enabled, can be started with the command
<pre class="gcommand">
interact --x11
</pre>
This command will start an interactive session, with X forwarding enabled, with one core on one of the interactive nodes, and allocate 2GB of memory for a maximum walltime of 12h.

The <code>interact --x11</code> command is an alias for
<pre class="gcommand">
srun --pty --x11 --cpus-per-task=1 --job-name=interact --ntasks=1 --nodes=1 --partition=inter_p --time=12:00:00 --mem=2GB /bin/bash -l
</pre>

The options available to <code>interact</code>, described in the previous section, can be used along with the <code>--x11</code> option.

----
[[#top|Back to Top]]



===How to check on running or pending jobs===

To list all running and pending jobs (by all users), use the command
<pre class="gcommand">
squeue
</pre>
or
<pre class="gcommand">
squeue -l
</pre>
This command can be used with many options. We have wrapper to this command, called <code>sq</code> that shows some quantities that are commonly of interest. To use the <code>sq</code> command to list all of your running and pending jobs, use
<pre class="gcommand">
sq --me
</pre>

For detailed information on how to monitor your jobs, please see [[Monitoring Jobs on Sapelo2]].

----
[[#top|Back to Top]]

===How to cancel (delete) a running or pending job===

To cancel one of your running or pending job, use the command
<pre class="gcommand">
scancel <jobid>
</pre>
For example, to cancel a job with Job ID 12345 use
<pre class="gcommand">
scancel 12345
</pre>

To cancel all of your jobs, use the command
<pre class="gcommand">
scancel -u MyID
</pre>

To cancel all of your pending jobs, use the command
<pre class="gcommand">
scancel -t PENDING -u MyID
</pre>

To cancel one or more jobs by job name, use the command
<pre class="gcommand">
scancel --name <myJobName>
</pre>

To cancel an element (index) of an array job
<pre class="gcommand">
scancel <jobid>_<index>
</pre>
For example, to cancel array job element 4 of an array job whose Job ID is 12345 use
<pre class="gcommand">
scancel 12345_4
</pre>

----
[[#top|Back to Top]]

===How to check resource utilization of a running or finished job===

The following command can be used to show resource utilization by a running job or a job that has already completed:
<pre class="gcommand">
sacct
</pre>

This command can be used with many options. We have configured one option that shows some quantities that are commonly of interest, including the amount of memory used and the cputime used by the jobs:
<pre class="gcommand">
sacct-gacrc
</pre>

For detailed information on how to monitor your jobs, please see [[Monitoring Jobs on Sapelo2]].

----
[[#top|Back to Top]]

Systems

2024-07-05T14:57:40Z

Jerky: Added the 3TB/node entry for ra4-[3-5].

[[Category:Sapelo2]]
[[Category:Teaching]]



=== Sapelo2 ===

Sapelo2 is a Linux cluster that runs a 64-bit Rocky 8.8 operating system and it is managed using Warewulf. Several virtual login nodes are available, with Intel Xeon Gold 6230 processors, 32GB of RAM, and 16 cores per node. The queueing system on Sapelo2 is Slurm.

Internodal communication among the compute nodes and between these nodes and the storage systems serving the home directories and the scratch directories is provided by an EDR Infiniband network (100Gbps).

The cluster is currently comprised of the following resources:

'''Regular nodes'''

* 119 compute nodes with AMD EPYC (Milan 3rd gen) processors (128 cores and 512GB of RAM per node)
* 4 compute nodes with AMD EPYC (Milan 3rd gen) processors (64 cores and 256GB of RAM per node)
* 2 compute nodes with AMD EPYC (Milan 3rd gen) processors (64 cores and 128GB of RAM per node)
* 123 compute nodes with AMD EPYC (Rome 2nd gen) processors (64 cores and 128GB of RAM per node)
* 64 compute nodes with AMD EPYC (Naples 1st gen) processors (32 cores and 128GB of RAM per node)
* 42 compute nodes with Intel Xeon Skylake processors (32 cores and 192GB of RAM per node)

'''High memory nodes (3TB/node)'''

* 3 compute nodes with AMD EPYC (Genoa 4th gen) processors (48 cores and 3TB of RAM per node)

'''High memory nodes (2TB/node)'''

* 2 compute nodes with AMD EPYC (Rome 2nd gen) processors (32 cores and 2TB of RAM per node)

'''High memory nodes (1TB/node)'''

* 2 compute nodes with AMD EPYC (Milan 3rd gen) processors (128 cores and 1TB of RAM per node)
* 5 compute nodes with AMD EPYC (Milan 3rd gen) processors (32 cores and 1TB of RAM per node)
* 4 compute nodes with AMD EPYC (Naples 1st gen) processors (64 cores and 1TB of RAM per node)
* 4 compute nodes with Intel Xeon Broadwell processors (28 cores and 1TB of RAM per node)

'''High memory nodes (512GB/node)'''

* 18 compute nodes with AMD EPYC (Naples 1st gen) processors (32 cores and 512GB of RAM per node)


'''GPU nodes'''

* 12 compute nodes with AMD EPYC (Milan 3rd gen) processors (64 cores and 1TB of RAM) and 4x NVIDIA A100 GPU cards.
* 2 compute nodes with Intel Xeon Skylake processors (32 cores and 187GB of RAM) and 1x NVIDIA P100 GPU card per node


'''Buy-in nodes'''

* Various configurations


====[[Connecting#Connecting_to_Sapelo2 |Connecting to Sapelo2]]====

====[[Transferring Files]]====

====[[Disk Storage]]====

====[[Software on Sapelo2]]====

====[[Available Toolchains and Toolchain Compatibility]]====

====[[Code Compilation on Sapelo2]]====

====[[Running Jobs on Sapelo2]]====

====[[Monitoring Jobs on Sapelo2]]====

====[[Migrating from Torque to Slurm]]====

'''Training material'''

To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the GACRC's Kaltura channel at https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required). Training sessions and slides are available at https://wiki.gacrc.uga.edu/wiki/Training

----
[[#top|Back to Top]]



=== Teaching cluster ===

The teaching cluster is a Linux cluster that runs a 64-bit Linux, with Rocky 8.8. The login node is a VM that has 4 cores (Intel Xeon Gold 6230 processor) and 16GB of RAM. An EDR Infiniband network (100Gbps) provides internodal communication among compute nodes, and between the compute nodes and the storage systems serving the home directories and the work directories.

The cluster is currently comprised of the following resources:

'''Regular nodes:'''

* 10 compute nodes with AMD EPYC (Naples 1st gen) processors (32 cores and 128GB or RAM per node)

'''High-memory nodes:'''

* 2 compute nodes with AMD EPYC (Naples 1st gen) processors (64 cores and 1TB of RAM per node)

'''GPU nodes:'''

* 1 compute node with Intel Skylake processors (32 cores, 192GB RAM per node) and a P100 GPU card


The queueing system on the teaching cluster is Slurm.

====[[Connecting#Connecting_to_the_teaching_cluster |Connecting to the teaching cluster]]====

====[[Transferring Files]]====

====Software Installed on the teaching cluster====

The teaching cluster has access to the same software stack installed on Sapelo2.

====[[Code Compilation on the teaching cluster]]====

====[[Running Jobs on the teaching cluster]]====

====[[Monitoring Jobs on the teaching cluster]]====

Job Submission partitions on Sapelo2

2024-07-05T14:25:44Z

Jerky: Changed RAM limit in 'hugemem*p' descriptions due to addition of 3TB nodes.

[[category:sapelo2]]

===Batch partitions (queues) defined on the Sapelo2===

There are different partitions defined on Sapelo2. The Slurm queueing system refers to queues as partition. Users are required to specify, in the job submission script or as job submission command line arguments, the partition and the resources needed by the job in order for it to be assigned to compute node(s) that have enough available resources (such as number of cores, amount of memory, GPU cards, etc). Please note, Slurm will not allow a job to be submitted if there are no resources matching your request. Please refer to [[Migrating from Torque to Slurm]] for more info about Slurm queueing system.

The following partitions are defined on the Sapelo2 cluster:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | Time limit
! scope="col" | Max jobs running
! scope="col" | Max jobs able to be submitted
! scope="col" | Notes
|-
|-
| batch || 7 days || 250 || 10,000 || Regular nodes.
|-
| batch_30d || 30 days || 1 || 2 || Regular nodes. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| highmem_p || 7 days || 15 || 100 || For high memory jobs
|-
| highmem_30d_p || 30 days || 1 || 2 || For high memory jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
|hugemem_p
|7 days
|4
|4
|For jobs needing up to 3TB of memory
|-
|hugemem_30d_p
|30 days
|4
|4
|For jobs needing up to 3TB of memory
|-
| gpu_p || 7 days || 6 || 20 || For GPU-enabled jobs.
|-
| gpu_30d_p || 30 days || 2 || 2 || For GPU-enabled jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| inter_p || 2 days || 3 || 20 || Regular nodes, for interactive jobs.
|-
| '''name'''_p || style="text-align: center" colspan="2"| variable || Partitions that target different groups' buy-in nodes. The '''name''' string is specific to each group.
|-
|}

When defining the resources for your job, you'll want to make sure you stay within the bounds of the resources available for the partition that you're using. The below table outlines the resources available per type of node, with the red values being the maximum for that corresponding partition.

{| width="75%" border="1" cellspacing="0" cellpadding=0" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | # of Nodes
! scope="col" | Max Mem(GB)/Node
! scope="col" | Max Cores/Node
! scope="col" | Processor Type
! scope="col" | GPU Cards/Node
|-
| rowspan="7" style="text-align: center" | batch, batch_30d
|-
| 119 || style="color:red" |'''500''' || style="color:red"| '''128''' || AMD EPYC Milan (3rd gen) || rowspan="12" style="text-align: center" | N/A
|-
|4
|250
|64
|AMD EPYC Milan (3rd gen)
|-
| 2 || rowspan="3" | 120 || 64 || AMD EPYC Milan (3rd gen)
|-
| 123 || 64 || AMD EPYC Rome (2nd gen)
|-
| 64
| 32
| AMD EPYC Naples (1st gen)
|-
| 42 || 180 || 32 || Intel Xeon Skylake
|-
| rowspan="5" style="text-align: center" | highmem_p, highmem_30d_p
| 18 || 500 || 32 || AMD EPYC Naples (1st gen)
|-
| 2 || rowspan="4" style="color:red" |'''990'''|| style="color:red" |'''128'''|| AMD EPYC Milan (3rd gen)
|-
| 5 || 32 || AMD EPYC Milan (3rd gen)
|-
| 4 || 64 || AMD EPYC Naples (1st gen)
|-
| 4 || 28 || Intel Xeon Broadwell
|-
| style="text-align: center"|hugemem_p, hugemem_30d_p
| 2
| style="color:red"|'''2000'''
| style="color:red"|'''32'''
|AMD EPYC Rome (2nd gen)
|-
| rowspan="2" style="text-align: center" | gpu_p, gpu_30d_p || 4 || 180 || 32 || Intel Xeon Skylake || 1 NVDIA P100
|-
|12
|style="color:red" |'''1000'''
|style="color:red" |'''64'''
|AMD EPYC Milan (3rd gen)
|4 NVIDIA A100
|-
| style="text-align: center" | '''name'''_p || style="text-align: center" colspan="5" | variable
|-
|}

Job Submission partitions on Sapelo2

2024-07-02T13:36:37Z

Jerky: Changed MaxJobsPU gpu_p from '18' to '6'.

[[category:sapelo2]]

===Batch partitions (queues) defined on the Sapelo2===

There are different partitions defined on Sapelo2. The Slurm queueing system refers to queues as partition. Users are required to specify, in the job submission script or as job submission command line arguments, the partition and the resources needed by the job in order for it to be assigned to compute node(s) that have enough available resources (such as number of cores, amount of memory, GPU cards, etc). Please note, Slurm will not allow a job to be submitted if there are no resources matching your request. Please refer to [[Migrating from Torque to Slurm]] for more info about Slurm queueing system.

The following partitions are defined on the Sapelo2 cluster:

{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | Time limit
! scope="col" | Max jobs running
! scope="col" | Max jobs able to be submitted
! scope="col" | Notes
|-
|-
| batch || 7 days || 250 || 10,000 || Regular nodes.
|-
| batch_30d || 30 days || 1 || 2 || Regular nodes. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| highmem_p || 7 days || 15 || 100 || For high memory jobs
|-
| highmem_30d_p || 30 days || 1 || 2 || For high memory jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
|hugemem_p
|7 days
|4
|4
|For jobs needing up to 2TB of memory
|-
|hugemem_30d_p
|30 days
|4
|4
|For jobs needing up to 2TB of memory
|-
| gpu_p || 7 days || 6 || 20 || For GPU-enabled jobs.
|-
| gpu_30d_p || 30 days || 2 || 2 || For GPU-enabled jobs. A given user can have up to one job running at a time here, plus one pending, or two pending and none running. A user's attempt to submit a third job into this partition will be rejected.
|-
| inter_p || 2 days || 3 || 20 || Regular nodes, for interactive jobs.
|-
| '''name'''_p || style="text-align: center" colspan="2"| variable || Partitions that target different groups' buy-in nodes. The '''name''' string is specific to each group.
|-
|}

When defining the resources for your job, you'll want to make sure you stay within the bounds of the resources available for the partition that you're using. The below table outlines the resources available per type of node, with the red values being the maximum for that corresponding partition.

{| width="75%" border="1" cellspacing="0" cellpadding=0" align="center" class="wikitable unsortable"
|-
! scope="col" | Partition Name
! scope="col" | # of Nodes
! scope="col" | Max Mem(GB)/Node
! scope="col" | Max Cores/Node
! scope="col" | Processor Type
! scope="col" | GPU Cards/Node
|-
| rowspan="7" style="text-align: center" | batch, batch_30d
|-
| 119 || style="color:red" |'''500''' || style="color:red"| '''128''' || AMD EPYC Milan (3rd gen) || rowspan="12" style="text-align: center" | N/A
|-
|4
|250
|64
|AMD EPYC Milan (3rd gen)
|-
| 2 || rowspan="3" | 120 || 64 || AMD EPYC Milan (3rd gen)
|-
| 123 || 64 || AMD EPYC Rome (2nd gen)
|-
| 64
| 32
| AMD EPYC Naples (1st gen)
|-
| 42 || 180 || 32 || Intel Xeon Skylake
|-
| rowspan="5" style="text-align: center" | highmem_p, highmem_30d_p
| 18 || 500 || 32 || AMD EPYC Naples (1st gen)
|-
| 2 || rowspan="4" style="color:red" |'''990'''|| style="color:red" |'''128'''|| AMD EPYC Milan (3rd gen)
|-
| 5 || 32 || AMD EPYC Milan (3rd gen)
|-
| 4 || 64 || AMD EPYC Naples (1st gen)
|-
| 4 || 28 || Intel Xeon Broadwell
|-
| style="text-align: center"|hugemem_p, hugemem_30d_p
| 2
| style="color:red"|'''2000'''
| style="color:red"|'''32'''
|AMD EPYC Rome (2nd gen)
|-
| rowspan="2" style="text-align: center" | gpu_p, gpu_30d_p || 4 || 180 || 32 || Intel Xeon Skylake || 1 NVDIA P100
|-
|12
|style="color:red" |'''1000'''
|style="color:red" |'''64'''
|AMD EPYC Milan (3rd gen)
|4 NVIDIA A100
|-
| style="text-align: center" | '''name'''_p || style="text-align: center" colspan="5" | variable
|-
|}

Rocky 8 Transition Guide

2023-07-28T16:59:11Z

Jerky: some words changed

==Introduction==

As part of our August 29-31,2023 maintenance window, the GACRC will be upgrading the Sapelo2 cluster operating system from CentOS 7 to Rocky 8.

==Why is a major Operating System (OS) update necessary?==

* Existing RHEL-7-based OS is End of Life - There are no more full version updates being released for the existing operating system and newer versions of some software applications are not supported by the current OS version.
* Hardware Support for new nodes and processors - As development within the existing OS has stopped, some of the latest generation of compute node hardware cannot use it, needing driver types newer than what this OS has. New hardware and architecture that we will be bringing online soon requires this OS update.
* Security - to retain compliance with current and future security requirements, we must keep using a supported version of the operating system.
* Why Rocky 8? - A good portion of the HPC centers is adopting it, which means there is a good amount of community support. PB proposes: "The community around the development and support of this RHEL-based distribution is primarily HPC-oriented, making it a good fit for HPC centers."

==What does this mean to you and your workflows?==

===Overview===

* We are not changing anything from the data storage standpoint. All existing /home, /scratch, /work, and /project spaces will retain their existing data.
* The compiler toolchains and many software packages will be updated to newer versions.
* Because this is a major OS update, we need to recompile all the applications and ensure that they work with the new version of OS.
* We will have as comprehensive a software suite available on the new OS as possible, but some less widely used applications and older version software will not be immediately available.


* As software modules will be reinstalled and updated, all pending jobs will be canceled during the maintenance window, to prevent job failure due to changes in the module names post maintenance.

===Storage===

There will be no changes to the storage system at this maintenance window. All existing /home, /scratch, /work, /project, and /db spaces will be available after the maintenance and they will retain their existing data.

===Queueing System===

The Slurm queueing system will be updated from version 21.08.8 to version 23.02.2. Most compute nodes available on the CentOS 7 system will continue to be available after the transition to Rocky 8, and the Slurm partitions will remain the same.

===Software===

====Warning====
Because this is a major change in the operating system, most user software built on CentOS 7 will not work and will need to be rebuilt. Even if the programs run without being rebuilt, the change in the underlying libraries may impact code execution and results. Therefore, users should test and verify that their codes are producing the expected results on the new operating system.

====Compiler toolchains====

The base compiler toolchains used to build software libraries and applications on the cluster will be updated, as newer versions are able to generate more optimized code for newer computer hardware and newer software versions.

<u>Base compiler toolchains on CentOS 7 (the current Sapelo2): </u>
*GCCcore/8.3.0, GCC/8.3.0, gompi/2019b, foss/2019b
*GCCcore/10.2.0, GCC/10.2.0, gompi/2020b, foss/2020b
*CUDA versions 10.2 and 11.1
*OpenMPI versions 3.1.4 and 4.0.5

<u>Base compiler toolchains on Rocky 8: </u>
*GCCcore/11.2.0, GCC/11.2.0, gompi/2021b, foss/2021b
*GCCcore/11.3.0, GCC/11.3.0, gompi/2022a, foss/2022a
*CUDA versions 11.4, 11.7, and 12.0
*OpenMPI versions 4.1.2 and 4.1.4

====Centrally installed modules====

Centrally installed software modules will continue to have the format <b>Name/Version-Toolchain</b>, but for most software packages the <b>Version</b> and <b>Toolchain</b> will updated. Some module names have an optional <b>Versionsuffix</b> and it might change or be dropped on the new system. There are modules whose names will remain the same on the Rocky 8 system. Some examples:

{| class="wikitable"
! Software !! Module name on CentOS 7 !! Module name on Rocky 8 !! Changes
|-
| ABySS || ABySS/2.3.1-foss-2019b || ABySS/2.3.5-foss-2021b || version, toolchain
|-
| BLAST+ || BLAST+/2.12.0-gompi-2020b || BLAST+/2.13.0-gompi-2022a || version, toolchain
|-
| BWA || BWA/0.7.17-GCC-10.3.0 || BWA/0.7.17-GCCcore-11.2.0 || toolchain
|-
| DeepAffinity || DeepAffinity/0.1 || || not available (yet)
|-
| SAMtools || SAMtools/1.16.1-GCC-11.3.0 || SAMtools/1.16.1-GCC-11.3.0 || no changes
|-
| STAR || STAR/2.7.10a-GCC-8.3.0 || STAR/2.7.10b-GCC-11.3.0 || version, toolchain
|-
| Trinity || Trinity/2.10.0-foss-2019b-Python-3.7.4 || Trinity/2.15.1-foss-2022a || version, toolchain, versionsuffix
|}

====Conda environments====

Some users have conda environments installed in their home directory or group shared directories. These environments should be reinstalled on the Rocky 8 system, using versions of Miniconda or Anaconda available there. Documentation on how to install conda environments on the cluster is available at https://wiki.gacrc.uga.edu/wiki/Installing_Applications_on_Sapelo2

====Python packages====

Python libraries and virtual environments need to be reinstalled as well, using versions of Python, Miniconda, or Anaconda available there.

====R packages====

We recommend that user reinstall any R packages that they have installed in their own directories, to make sure they are compatible with the new OS version and with the versions of R available there.

====Singularity containers====

Singularity containers that you used on CentOS 7 should continue to work on the Rocky 8 system. The containers installed centrally in /apps/singularity-images will be available after the maintenance.

==Potential issues==

===Error connecting to Sapelo2===

Because Sapelo2 was reinstalled, you might encounter a "host key" or "host id" error when you connect to Sapelo2 for the first time after the maintenance.

'''Connecting from MacOS or Linux'''

Users connecting from a MacOS or a Linux system might see an error like this:

<pre class="gcomment">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for sapelo2 has changed,
and the key for the corresponding IP address 128.192.75.18
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /Users/jsmith/.ssh/known_hosts:76
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:E1ovq19vLNYNF1eFiOQ91tc1EPtbHcMhML2I45UrJrE.
Please contact your system administrator.
Add correct host key in /Users/jsmith/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/jsmith/.ssh/known_hosts:25
ECDSA host key for sapelo2 has changed and you have requested strict checking.
Host key verification failed.
</pre>

To fix this problem, open the known_hosts file on '''your local machine''' (in the example above the full path to this file is '''/Users/jsmith/.ssh/known_hosts''', as shown in the error message above). Then delete the line that has sapelo2.gacrc.uga.edu and save the file.

Once you have done this, you should be able to ssh into sapelo2.gacrc.uga.edu. You might still get a message like this:

<pre class="gcomment">
[jsmith@laptop]$ ssh jsmith@sapelo2.gacrc.uga.edu
The authenticity of host 'sapelo2.gacrc.uga.edu' can't be established.
ECDSA key fingerprint is SHA256:ikdjggjeorjgnkresitnsgjsms
ECDSA key fingerprint is MD5:be:1xxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)?
</pre>

You can type '''yes''' and your connection should work.

'''Connecting from Windows'''

When connecting from Windows for the first time after the maintenance, users might encounter an error like '''POTENTIAL SECURITY BREACH''' or '''HOST IDENTIFICATION HAS CHANGED'''. Users can click '''Yes''' to continue the connection and have a new host key saved on their local machines.

===Modules in your .bashrc no longer work or give errors on login===

If you have edited your .bashrc file to include commands to load modules automatically when you login, you may find that some CentOS 7 modules will not be found or may not work on Rocky 8. You will need to edit your .bashrc and comment out or remove any such lines. You can also replace the module load commands in your .bashrc file with new module names. If you can no longer log in because of something in your .bashrc, contact us and we can rename your .bashrc and copy in a default version for you.

If you’d like to start from scratch, a default .bashrc contains the following:

<pre class="gscript">
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

# User specific aliases and functions below
</pre>

===My job gets module not found errors, the same script used to work on Sapelo2===

Many software modules have been updated with a new version and/or a new toolchain version. The modules your jobs loaded on the CentOS 7 system might not be available on Rocky 8. Please check the name of the modules on the updated cluster. You can search for a module using the <code>ml spider NAME</code> command, where NAME needs to be replaced by the software package name that you are searching for. You can also see a list of all installed software with the command <code>ml avail</code>.

===My job gets command not found errors, but I did load the module===

If your are attempting to load a module that was available on CentOS 7, but no longer available on Rocky 8, the module will not be loaded, and the commands provided by that module will not be available for the job. Please check the correct name of the modules on the Rocky 8 system. If the software is not available on the updated cluster, please feel free to [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25850 submit a software installation request ticket] and we will try to get it installed for you.

Rocky 8 Transition Guide

2023-07-28T16:57:40Z

Jerky: Proposed "why rocky 8".

==Introduction==

As part of our August 29-31,2023 maintenance window, the GACRC will be upgrading the Sapelo2 cluster operating system from CentOS 7 to Rocky 8.

==Why is a major Operating System (OS) update necessary?==

* Existing RHEL-7-based OS is End of Life - There are no more full version updates being released for the existing operating system and newer versions of some software applications are not supported by the current OS version.
* Hardware Support for new nodes and processors - As development within the existing OS has stopped, some of the latest generation of compute node hardware cannot use it, needing driver types newer than what this OS has. New hardware and architecture that we will be bringing online soon requires this OS update.
* Security - to retain compliance with current and future security requirements, we must keep using a supported version of the operating system.
* Why Rocky 8? - A good portion of the HPC centers is adopting it, which means there is a good amount of community support. PB proposes: "The community around this RHEL-based distribution (its development and support) is primarily HPC-oriented, making it a good fit for HPC centers."

==What does this mean to you and your workflows?==

===Overview===

* We are not changing anything from the data storage standpoint. All existing /home, /scratch, /work, and /project spaces will retain their existing data.
* The compiler toolchains and many software packages will be updated to newer versions.
* Because this is a major OS update, we need to recompile all the applications and ensure that they work with the new version of OS.
* We will have as comprehensive a software suite available on the new OS as possible, but some less widely used applications and older version software will not be immediately available.


* As software modules will be reinstalled and updated, all pending jobs will be canceled during the maintenance window, to prevent job failure due to changes in the module names post maintenance.

===Storage===

There will be no changes to the storage system at this maintenance window. All existing /home, /scratch, /work, /project, and /db spaces will be available after the maintenance and they will retain their existing data.

===Queueing System===

The Slurm queueing system will be updated from version 21.08.8 to version 23.02.2. Most compute nodes available on the CentOS 7 system will continue to be available after the transition to Rocky 8, and the Slurm partitions will remain the same.

===Software===

====Warning====
Because this is a major change in the operating system, most user software built on CentOS 7 will not work and will need to be rebuilt. Even if the programs run without being rebuilt, the change in the underlying libraries may impact code execution and results. Therefore, users should test and verify that their codes are producing the expected results on the new operating system.

====Compiler toolchains====

The base compiler toolchains used to build software libraries and applications on the cluster will be updated, as newer versions are able to generate more optimized code for newer computer hardware and newer software versions.

<u>Base compiler toolchains on CentOS 7 (the current Sapelo2): </u>
*GCCcore/8.3.0, GCC/8.3.0, gompi/2019b, foss/2019b
*GCCcore/10.2.0, GCC/10.2.0, gompi/2020b, foss/2020b
*CUDA versions 10.2 and 11.1
*OpenMPI versions 3.1.4 and 4.0.5

<u>Base compiler toolchains on Rocky 8: </u>
*GCCcore/11.2.0, GCC/11.2.0, gompi/2021b, foss/2021b
*GCCcore/11.3.0, GCC/11.3.0, gompi/2022a, foss/2022a
*CUDA versions 11.4, 11.7, and 12.0
*OpenMPI versions 4.1.2 and 4.1.4

====Centrally installed modules====

Centrally installed software modules will continue to have the format <b>Name/Version-Toolchain</b>, but for most software packages the <b>Version</b> and <b>Toolchain</b> will updated. Some module names have an optional <b>Versionsuffix</b> and it might change or be dropped on the new system. There are modules whose names will remain the same on the Rocky 8 system. Some examples:

{| class="wikitable"
! Software !! Module name on CentOS 7 !! Module name on Rocky 8 !! Changes
|-
| ABySS || ABySS/2.3.1-foss-2019b || ABySS/2.3.5-foss-2021b || version, toolchain
|-
| BLAST+ || BLAST+/2.12.0-gompi-2020b || BLAST+/2.13.0-gompi-2022a || version, toolchain
|-
| BWA || BWA/0.7.17-GCC-10.3.0 || BWA/0.7.17-GCCcore-11.2.0 || toolchain
|-
| DeepAffinity || DeepAffinity/0.1 || || not available (yet)
|-
| SAMtools || SAMtools/1.16.1-GCC-11.3.0 || SAMtools/1.16.1-GCC-11.3.0 || no changes
|-
| STAR || STAR/2.7.10a-GCC-8.3.0 || STAR/2.7.10b-GCC-11.3.0 || version, toolchain
|-
| Trinity || Trinity/2.10.0-foss-2019b-Python-3.7.4 || Trinity/2.15.1-foss-2022a || version, toolchain, versionsuffix
|}

====Conda environments====

Some users have conda environments installed in their home directory or group shared directories. These environments should be reinstalled on the Rocky 8 system, using versions of Miniconda or Anaconda available there. Documentation on how to install conda environments on the cluster is available at https://wiki.gacrc.uga.edu/wiki/Installing_Applications_on_Sapelo2

====Python packages====

Python libraries and virtual environments need to be reinstalled as well, using versions of Python, Miniconda, or Anaconda available there.

====R packages====

We recommend that user reinstall any R packages that they have installed in their own directories, to make sure they are compatible with the new OS version and with the versions of R available there.

====Singularity containers====

Singularity containers that you used on CentOS 7 should continue to work on the Rocky 8 system. The containers installed centrally in /apps/singularity-images will be available after the maintenance.

==Potential issues==

===Error connecting to Sapelo2===

Because Sapelo2 was reinstalled, you might encounter a "host key" or "host id" error when you connect to Sapelo2 for the first time after the maintenance.

'''Connecting from MacOS or Linux'''

Users connecting from a MacOS or a Linux system might see an error like this:

<pre class="gcomment">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for sapelo2 has changed,
and the key for the corresponding IP address 128.192.75.18
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /Users/jsmith/.ssh/known_hosts:76
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:E1ovq19vLNYNF1eFiOQ91tc1EPtbHcMhML2I45UrJrE.
Please contact your system administrator.
Add correct host key in /Users/jsmith/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/jsmith/.ssh/known_hosts:25
ECDSA host key for sapelo2 has changed and you have requested strict checking.
Host key verification failed.
</pre>

To fix this problem, open the known_hosts file on '''your local machine''' (in the example above the full path to this file is '''/Users/jsmith/.ssh/known_hosts''', as shown in the error message above). Then delete the line that has sapelo2.gacrc.uga.edu and save the file.

Once you have done this, you should be able to ssh into sapelo2.gacrc.uga.edu. You might still get a message like this:

<pre class="gcomment">
[jsmith@laptop]$ ssh jsmith@sapelo2.gacrc.uga.edu
The authenticity of host 'sapelo2.gacrc.uga.edu' can't be established.
ECDSA key fingerprint is SHA256:ikdjggjeorjgnkresitnsgjsms
ECDSA key fingerprint is MD5:be:1xxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)?
</pre>

You can type '''yes''' and your connection should work.

'''Connecting from Windows'''

When connecting from Windows for the first time after the maintenance, users might encounter an error like '''POTENTIAL SECURITY BREACH''' or '''HOST IDENTIFICATION HAS CHANGED'''. Users can click '''Yes''' to continue the connection and have a new host key saved on their local machines.

===Modules in your .bashrc no longer work or give errors on login===

If you have edited your .bashrc file to include commands to load modules automatically when you login, you may find that some CentOS 7 modules will not be found or may not work on Rocky 8. You will need to edit your .bashrc and comment out or remove any such lines. You can also replace the module load commands in your .bashrc file with new module names. If you can no longer log in because of something in your .bashrc, contact us and we can rename your .bashrc and copy in a default version for you.

If you’d like to start from scratch, a default .bashrc contains the following:

<pre class="gscript">
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

# User specific aliases and functions below
</pre>

===My job gets module not found errors, the same script used to work on Sapelo2===

Many software modules have been updated with a new version and/or a new toolchain version. The modules your jobs loaded on the CentOS 7 system might not be available on Rocky 8. Please check the name of the modules on the updated cluster. You can search for a module using the <code>ml spider NAME</code> command, where NAME needs to be replaced by the software package name that you are searching for. You can also see a list of all installed software with the command <code>ml avail</code>.

===My job gets command not found errors, but I did load the module===

If your are attempting to load a module that was available on CentOS 7, but no longer available on Rocky 8, the module will not be loaded, and the commands provided by that module will not be available for the job. Please check the correct name of the modules on the Rocky 8 system. If the software is not available on the updated cluster, please feel free to [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25850 submit a software installation request ticket] and we will try to get it installed for you.

Rocky 8 Transition Guide

2023-07-28T16:47:58Z

Jerky: Cosmetic changes (commas, etc.)

==Introduction==

As part of our August 29-31,2023 maintenance window, the GACRC will be upgrading the Sapelo2 cluster operating system from CentOS 7 to Rocky 8.

==Why is a major Operating System (OS) update necessary?==

* Existing RHEL-7-based OS is End of Life - There are no more full version updates being released for the existing operating system and newer versions of some software applications are not supported by the current OS version.
* Bringing New Nodes Online - As development within the existing OS has stopped, some of the latest generation of compute node hardware cannot use it, needing driver types newer than what this OS has. New hardware and architecture that we will be bringing online soon requires this OS update.
* Security Improvements - In order to keep our cluster as up to date as possible, these kinds of big OS updates need to happen.
* Why Rocky 8? - A good portion of the HPC centers is adopting it, which means there is a good amount of community support.

==What does this mean to you and your workflows?==

===Overview===

* We are not changing anything from the data storage standpoint. All existing /home, /scratch, /work, and /project spaces will retain their existing data.
* The compiler toolchains and many software packages will be updated to newer versions.
* Because this is a major OS update, we need to recompile all the applications and ensure that they work with the new version of OS.
* We will have as comprehensive a software suite available on the new OS as possible, but some less widely used applications and older version software will not be immediately available.


* As software modules will be reinstalled and updated, all pending jobs will be canceled during the maintenance window, to prevent job failure due to changes in the module names post maintenance.

===Storage===

There will be no changes to the storage system at this maintenance window. All existing /home, /scratch, /work, /project, and /db spaces will be available after the maintenance and they will retain their existing data.

===Queueing System===

The Slurm queueing system will be updated from version 21.08.8 to version 23.02.2. Most compute nodes available on the CentOS 7 system will continue to be available after the transition to Rocky 8, and the Slurm partitions will remain the same.

===Software===

====Warning====
Because this is a major change in the operating system, most user software built on CentOS 7 will not work and will need to be rebuilt. Even if the programs run without being rebuilt, the change in the underlying libraries may impact code execution and results. Therefore, users should test and verify that their codes are producing the expected results on the new operating system.

====Compiler toolchains====

The base compiler toolchains used to build software libraries and applications on the cluster will be updated, as newer versions are able to generate more optimized code for newer computer hardware and newer software versions.

<u>Base compiler toolchains on CentOS 7 (the current Sapelo2): </u>
*GCCcore/8.3.0, GCC/8.3.0, gompi/2019b, foss/2019b
*GCCcore/10.2.0, GCC/10.2.0, gompi/2020b, foss/2020b
*CUDA versions 10.2 and 11.1
*OpenMPI versions 3.1.4 and 4.0.5

<u>Base compiler toolchains on Rocky 8: </u>
*GCCcore/11.2.0, GCC/11.2.0, gompi/2021b, foss/2021b
*GCCcore/11.3.0, GCC/11.3.0, gompi/2022a, foss/2022a
*CUDA versions 11.4, 11.7, and 12.0
*OpenMPI versions 4.1.2 and 4.1.4

====Centrally installed modules====

Centrally installed software modules will continue to have the format <b>Name/Version-Toolchain</b>, but for most software packages the <b>Version</b> and <b>Toolchain</b> will updated. Some module names have an optional <b>Versionsuffix</b> and it might change or be dropped on the new system. There are modules whose names will remain the same on the Rocky 8 system. Some examples:

{| class="wikitable"
! Software !! Module name on CentOS 7 !! Module name on Rocky 8 !! Changes
|-
| ABySS || ABySS/2.3.1-foss-2019b || ABySS/2.3.5-foss-2021b || version, toolchain
|-
| BLAST+ || BLAST+/2.12.0-gompi-2020b || BLAST+/2.13.0-gompi-2022a || version, toolchain
|-
| BWA || BWA/0.7.17-GCC-10.3.0 || BWA/0.7.17-GCCcore-11.2.0 || toolchain
|-
| DeepAffinity || DeepAffinity/0.1 || || not available (yet)
|-
| SAMtools || SAMtools/1.16.1-GCC-11.3.0 || SAMtools/1.16.1-GCC-11.3.0 || no changes
|-
| STAR || STAR/2.7.10a-GCC-8.3.0 || STAR/2.7.10b-GCC-11.3.0 || version, toolchain
|-
| Trinity || Trinity/2.10.0-foss-2019b-Python-3.7.4 || Trinity/2.15.1-foss-2022a || version, toolchain, versionsuffix
|}

====Conda environments====

Some users have conda environments installed in their home directory or group shared directories. These environments should be reinstalled on the Rocky 8 system, using versions of Miniconda or Anaconda available there. Documentation on how to install conda environments on the cluster is available at https://wiki.gacrc.uga.edu/wiki/Installing_Applications_on_Sapelo2

====Python packages====

Python libraries and virtual environments need to be reinstalled as well, using versions of Python, Miniconda, or Anaconda available there.

====R packages====

We recommend that user reinstall any R packages that they have installed in their own directories, to make sure they are compatible with the new OS version and with the versions of R available there.

====Singularity containers====

Singularity containers that you used on CentOS 7 should continue to work on the Rocky 8 system. The containers installed centrally in /apps/singularity-images will be available after the maintenance.

==Potential issues==

===Error connecting to Sapelo2===

Because Sapelo2 was reinstalled, you might encounter a "host key" or "host id" error when you connect to Sapelo2 for the first time after the maintenance.

'''Connecting from MacOS or Linux'''

Users connecting from a MacOS or a Linux system might see an error like this:

<pre class="gcomment">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for sapelo2 has changed,
and the key for the corresponding IP address 128.192.75.18
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /Users/jsmith/.ssh/known_hosts:76
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:E1ovq19vLNYNF1eFiOQ91tc1EPtbHcMhML2I45UrJrE.
Please contact your system administrator.
Add correct host key in /Users/jsmith/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/jsmith/.ssh/known_hosts:25
ECDSA host key for sapelo2 has changed and you have requested strict checking.
Host key verification failed.
</pre>

To fix this problem, open the known_hosts file on '''your local machine''' (in the example above the full path to this file is '''/Users/jsmith/.ssh/known_hosts''', as shown in the error message above). Then delete the line that has sapelo2.gacrc.uga.edu and save the file.

Once you have done this, you should be able to ssh into sapelo2.gacrc.uga.edu. You might still get a message like this:

<pre class="gcomment">
[jsmith@laptop]$ ssh jsmith@sapelo2.gacrc.uga.edu
The authenticity of host 'sapelo2.gacrc.uga.edu' can't be established.
ECDSA key fingerprint is SHA256:ikdjggjeorjgnkresitnsgjsms
ECDSA key fingerprint is MD5:be:1xxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)?
</pre>

You can type '''yes''' and your connection should work.

'''Connecting from Windows'''

When connecting from Windows for the first time after the maintenance, users might encounter an error like '''POTENTIAL SECURITY BREACH''' or '''HOST IDENTIFICATION HAS CHANGED'''. Users can click '''Yes''' to continue the connection and have a new host key saved on their local machines.

===Modules in your .bashrc no longer work or give errors on login===

If you have edited your .bashrc file to include commands to load modules automatically when you login, you may find that some CentOS 7 modules will not be found or may not work on Rocky 8. You will need to edit your .bashrc and comment out or remove any such lines. You can also replace the module load commands in your .bashrc file with new module names. If you can no longer log in because of something in your .bashrc, contact us and we can rename your .bashrc and copy in a default version for you.

If you’d like to start from scratch, a default .bashrc contains the following:

<pre class="gscript">
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

# User specific aliases and functions below
</pre>

===My job gets module not found errors, the same script used to work on Sapelo2===

Many software modules have been updated with a new version and/or a new toolchain version. The modules your jobs loaded on the CentOS 7 system might not be available on Rocky 8. Please check the name of the modules on the updated cluster. You can search for a module using the <code>ml spider NAME</code> command, where NAME needs to be replaced by the software package name that you are searching for. You can also see a list of all installed software with the command <code>ml avail</code>.

===My job gets command not found errors, but I did load the module===

If your are attempting to load a module that was available on CentOS 7, but no longer available on Rocky 8, the module will not be loaded, and the commands provided by that module will not be available for the job. Please check the correct name of the modules on the Rocky 8 system. If the software is not available on the updated cluster, please feel free to [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25850 submit a software installation request ticket] and we will try to get it installed for you.

File System Purging

2019-01-31T22:03:14Z

Jerky:

==SCRATCH Filesystem==
The SCRATCH filesystem (mounted under the /scratch directory) is subject to the 30-day file purge rule. Any file that has not been accessed in over 30 days will be purged. The purge process deletes files but not directories.

In order to help users identify old files in their /scratch directory, we generate one file per user, every morning. This file, namely /usr/local/var/lustre_stats/$USER.over30d.files.lst (where $USER refers to the user's UGA MyID), contains a list of all the files that this user has in his/her /scratch directory and that have not been accessed in over 30 days. This file provides the full path, the last accessed date, and the size of the files that have not been accessed in over 30 days. Because the /scratch filesystem only updates on-disk file access times every 5 days, the last accessed time reported in that list file is approximate and could be off by up to 5 days. The purging system understands this and will make sure the access time falls within the purge window before deleting the file.

File System Purging

2019-01-31T22:02:54Z

Jerky:

==SCRATCH Filesystem==
The SCRATCH filesystem (mounted under the /scratch directory) is subject to the 30-day file purge rule. Any file that has not been accessed in over 30 days will be purged. The purge process deletes files but not directories.
In order to help users identify old files in their /scratch directory, we generate one file per user, every morning. This file, namely /usr/local/var/lustre_stats/$USER.over30d.files.lst (where $USER refers to the user's UGA MyID), contains a list of all the files that this user has in his/her /scratch directory and that have not been accessed in over 30 days. This file provides the full path, the last accessed date, and the size of the files that have not been accessed in over 30 days. Because the /scratch filesystem only updates on-disk file access times every 5 days, the last accessed time reported in that list file is approximate and could be off by up to 5 days. The purging system understands this and will make sure the access time falls within the purge window before deleting the file.

File System Purging

2019-01-31T21:58:52Z

Jerky:

==SCRATCH Filesystem==
The SCRATCH filesystem (mounted under the /scratch directory) is subject to the 30-day file purge rule. Any file that has not been accessed in over 30 days will be purged. In order to help users identify old files in their /scratch directory, we generate one file per user, every morning. This file, namely /usr/local/var/lustre_stats/$USER.over30d.files.lst (where $USER refers to the user's UGA MyID), contains a list of all the files that this user has in his/her /scratch directory and that have not been accessed in over 30 days. This file provides the full path, the last accessed date, and the size of the files that have not been accessed in over 30 days. Because the /scratch filesystem only updates on-disk file access times every 5 days, the last accessed time reported in that list file is approximate and could be off by up to 5 days. The purging system understands this and will make sure the access time falls within the purge window before deleting the file.
During the purge only files are deleted. Directories will be left alone. Please let us know if you have any questions.

Policies

2019-01-28T22:43:40Z

Jerky:

==Introduction to GACRC Policies==

The following policies are subject to revision, especially as the GACRC grows in scope and services. Your comments and questions will be useful to our policy formulation and refinement and are actively solicited by the GACRC Advisory Committee.

The GACRC computational infrastructure, including its servers, clusters, data stores, and other related devices are for the exclusive use of authorized users only.

Anyone using these systems expressly consents to abide by the policies of the University of Georgia and the Georgia Advanced Computing Resource Center and, accordingly, is subject to account termination and/or immediate disconnection from GACRC resources.

==GACRC Resource Usage==

The computational resources of the Georgia Advanced Computing Resource Center are to be used in direct support of research programs at the University of Georgia. Support is also provided for classes that teach computational methods, and provide training for high performance computing. The GACRC reserves the right to restrict access to its resources for course work if such work is deemed to present a negative impact to authorized research activities.

GACRC policies supplement UGA’s Policies on the Use of Computers, found at:. http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup

==GACRC Eligibility and Access ==

Access to and use of the computing facilities managed by the Georgia Advanced Computing Resource Center are limited to persons affiliated with the University of Georgia or associated with research projects sponsored by UGA.

Affiliation in this context means faculty, research staff and supervised students of the University of Georgia. Faculty includes persons holding permanent or temporary appointments as well as adjunct faculty, instructors and visiting faculty while in residence at the University. It also includes those persons with faculty status such as research associates, research scientists, post-doctoral researchers and academic and service professionals. Staff includes all those non-faculty persons employed directly by the University in a research-support role. Graduate and undergraduate students who are members of faculty research labs are eligible for accounts as well. For directly affiliated users, accounts on the GACRC computers will remain active as long as the individuals hold the above status.

Access by non-UGA researchers and their students, affiliated to higher-education institutions or non-profit research organizations, for work on research projects conducted in collaboration with UGA Faculty is possible under the guidelines established by the Office of the Vice President for Research and the Office of International Education. A request for access can be forwarded to the GACRC by the UGA Faculty, providing details of the collaboration on a joint research project. Such Affiliate users will be considered part of the UGA Faculty’s group and will be under the Faculty’s responsibility. Affiliate users’ access will be granted for a fixed period of time, according to the expected length of the collaborative project. Renewal of affiliate accounts will be required annually.

All accounts will remain active no more than 30 days following a status change (i.e., leaving the university). Graduate instructional accounts will only remain active for the duration of the semester in which they are actually needed. HOME and PROJECT directories will be archived for at least 90 days, but no longer than 180 days after an account becomes inactive.

Requests for access by individuals other than those listed above should be directed to the Director of the Center for consideration with the GACRC Advisory Committee.

Access will be granted to a specific GACRC resource after appropriate training is undertaken with GACRC staff. Existing access to other GACRC resources is not a sufficient criteria for access to a new resource. No exceptions will be given to the training requirement.

==GACRC Identity Management==

Below are described the procedures for validating the identity of account users.

===UGA Users and Faculty Lab Groups===

*A UGA Faculty must first establish a GACRC group account using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The UGA Faculty can choose or not to obtain a GACRC user account affiliated with his/her group account.

*All directly affiliated persons wanting an account must apply for access to the GACRC using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The applicant must authenticate to the form using his/her MyID. The applicant must specify to which group he/she belongs. Verification with the UGA Faculty responsible for the group’s account will be used in case identity or affiliation needs to be verified.

*Upon acceptance of the application, the user will be notified via e-mail. The applicant’s UGA MyID and password will be used to log into the requested GACRC resources.

===Affiliate Users===

*A recognized Affiliate user must be sponsored by a UGA Faculty member through an established GACRC group account. The UGA Faculty involved in an established collaboration with the Affiliate user, must apply on behalf of the applicant by contacting the GACRC staff.

*A request will be made by the GACRC to EITS to allocate to the Affiliate a UGA MyID.

*Upon acceptance of the Affiliate user application, the Affiliate will be notified via e-mail. The Affiliate’s UGA MyID and password will be used to log into the requested GACRC resources.

===Protection of Passwords===

As described in UGA’s Password Policy, an account holder must never divulge their MyID and password to a third party. Only authorized account holders may access the resources of the GACRC. If a third party is found to be using an account holder’s login with or without the permission of the account holder, the account holder’s access privileges may be revoked at the sole discretion of the GACRC Manager or Director. Enforcement of this policy is under the responsibility of the Office of the Vice President for Information Technology’s Division of Information Security.

More information is found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/passwords/

==GACRC Storage Usage ==

===Some working definitions===

'''Snapshot''' - Copies of files that are stored on the same storage system as the original files. Snapshots are primarily used to recover files that have been accidentally deleted or corrupted within the recent past. Users are able to manage the file recovery tasks. Snapshots are not maintained beyond a defined rotation schedule, i.e., some number of hourly, daily, weekly, and monthly snapshots are kept on the storage system.

'''Backup''' - Copies of files and/or snapshots kept on a storage system (disk/tape) other than the one that the original files reside on. Backups are primarily used to recover files following a catastrophic failure of the original file or storage system. Backups require administrators to perform file system recovery tasks. Like snapshots, backups have a defined rotation schedule.

'''Archive''' - Copies of files that are not currently being accessed, on a resilient storage system dedicated to reliable long-term storage. Archives can be tape-based or disk-based, and typically part of a disaster recovery plan. The files may be copies of original data which is stored elsewhere (individual groups having their own copies), or the archive storage system may be fed by a dedicated "backup" storage system.

'''Active Projects''' – Projects that have on-going computational work being performed with files that are regularly created, accessed or modified.

===Policy Statement for SCRATCH File System===

The SCRATCH file system resides on a high-performance storage device and is to be used uniquely for temporary storage of files in use by actively running compute jobs. Files are to be removed from SCRATCH when a job completes, e.g. can be copied to the PROJECT file system. The SCRATCH file system is not backed up in any way and no snapshots are taken. The SCRATCH filesystem is mounted under /scratch on all the compute nodes, login nodes and data transfer nodes.

Any file that is not accessed or modified by a compute job in a time period no longer than 30 days will be automatically deleted off the SCRATCH file system. Measures circumventing this policy will be monitored and actively discouraged.

There is no storage size quota for SCRATCH usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, the GACRC may request/force users to clean up their SCRATCH directories or reduce temporarily the 30 day limit to a lower limit.

===Policy Statement for WORK File System===

The WORK file system resides on a high-performance storage device and is to be used for storing files that are frequently used by the group for computation. The WORK file system is '''NOT''' subject to the 30 day purge policy. The filesystem usage is controlled using a quota on the size and number of files that can be stored in a lab group's WORK area. Initially each group is given a 500GB and 100,000-file quota. The WORK file system is not backed up in any way and no snapshots are taken. The WORK filesystem is mounted under /work on all the compute nodes, login nodes and data transfer nodes. Each lab group has a directory under the /work directory.

The WORK file system is '''NOT''' subject to the 30 day purge policy. But if there is sufficient space consumption on the storage appliance, we reserve the right to ask users to clean up their WORK area. If the users do not respond in a timely fashion we will purge files beginning with the oldest ones. Please do not user WORK area to store files long term.

===Policy Statement for HOME File System===

The HOME file system resides on a high-performance storage device and is used for long-term storage of files, typically programs and scripts, needed for analysis on the GACRC computing cluster.

All users have 100GB allocated for their HOME usage. Groups may request a separate 100GB allocation for a directory under /usr/local/lab/, for shared use of common applications, libraries, and scripts.

HOME directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the HOME file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with HOME usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the HOME service.

===Policy Statement for PROJECT File System===

The PROJECT file system resides on lower-performance/higher-capacity storage devices, accessible by all GACRC login and data transfer nodes. PROJECT will not be accessible on Sapelo2's compute nodes. This space is to be used by groups for storage of active projects using Sapelo2. PROJECT should not be seen as a long-term repository, as it is not designed as such. Once a project is completed, data should be moved from the PROJECT space to user-managed storage, freeing up capacity for the next active project.

Access to the PROJECT file system is not supported through NFS to a destination outside of the Boyd Data Center, or through the use of the Samba or CIFS protocols. Transfer protocols available through the data transfer nodes are secure ftp, scp, rsync, GridFTP, amongst others.

More info is found at https://wiki.gacrc.uga.edu/wiki/Transferring_Files.

Each group can request a PROJECT volume with an initial 1TB allocation, accessible by all users ascribed to the group, where the sharing of files will be enabled. Users are encouraged to consider their PROJECT space as the primary area to transfer compute job inputs/outputs. Additional space can be requested by a Faculty on behalf of his/her group, in increments of 1TB.

The GACRC reserves the right to establish a cost-recovery rate for PROJECT storage beyond the initial 1TB allocation. Appropriate communications will take place in such an event.

PROJECT directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the PROJECT file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with PROJECT usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the PROJECT service.

==GACRC Software Policy==

The GACRC maintains a collection of program libraries and software packages to support research computing activities across diverse research domains. While a user can install a software package in their own environment, for the sake of general access across groups, and an appropriate deployment with current libraries, compilers and other dependencies, we strongly recommend that GACRC staff be asked to perform the installation or upgrade.

Any software that requires a signed license or contract, even if it is a click-through agreement, must absolutely be reviewed and handled by the Office of Legal Affairs before being signed by an appropriate signature authority. After the license or contract is accepted and the software is made available, GACRC users must fully comply and use the software in a way that does not violate any terms of the license or contract. Further information on licensing issues can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup/eula

As a matter of policy, the GACRC will not purchase any commercial software for the use of a single group or a small number of groups. Commercial software currently purchased and maintained by the GACRC are of general interest and applicability to the whole UGA research community. The GACRC will however install and maintain a group-purchased commercial software, which complies with the above comments on licenses and contracts.

==Security==

To minimize disruption of service, protect data integrity, conserve facility resources and maximize the effectiveness of staff support, the GACRC maintains strict security requirements for access to GACRC resources. Over time, the enforcement of these requirements will become increasingly strict, with the goal of preventing any access to the GACRC resources by any person or any device that is not in strict compliance with these requirements.

===Operating Systems===

Any computer accessing the GACRC for any purpose must run a currently supported operating system, updated to the latest version and update (patch) levels.

===Anti-Virus Software===

Any computer accessing the GACRC for any purpose must meet minimum levels of anti-virus protection. Any computer used by an account holder must have anti-virus software from a source approved by UGA’s Office of Information Security must have that virus protection activated, and must have automatic updates activated.

More information can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/protect_your_computer

===Suspiciously Behaving Software===

Any software that behaves in a suspicious manner may at any time be terminated and/or deleted from GACRC resources at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Suspiciously Behaving Networks and Devices===

Any connection from any device to the GACRC may be terminated at any time, if the device or the connection or a network to which the device is attached appears to be not in compliance with UGA’s security requirements, is behaving suspiciously, or if a threat emerges requiring termination for intrusion prevention at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Account Holder Responsibility===

The account holder is responsible for diligently monitoring their account and compliance with the GACRC’s operating system, intrusion and virus protection standards. The account holder will be duly notified if GACRC personnel determine that minimum security requirements are not met. Specific actions will be requested of the account holder and compliance to these will be expected in a timely fashion. An account holder’s privileges to use GACRC facilities may be terminated by the GACRC Manager or Director at any time, without notice if, in the opinion of either, the account holder is reluctant or averse to practicing diligence in meeting the GACRC’s minimum requirements for intrusion and/or anti-viral protection.

==Resolving Disagreements about Revocation of Privileges or Provisioning Resources==

The Director of the Georgia Advanced Computing Resource Center has full authority to revoke a user's privileges or deny the request of a new resource allocation. The decision to revoke a user's privileges will be based on, but not limited to, abuses of the UGA Policies on the Use of Computers and/or abuses of the UGA Password Policy.

If an account holder is denied a request for provisioning of GACRC resources or resource privileges are revoked, the account holder’s Department Head may appeal to the Vice President for Research and the Vice President for Information Technology. Their decision will be informed by the Director of the GACRC, the Chief Technology Officer as well as the Associate Chief Information Officer for Information Security. The decision of the Vice President for Research and Vice President for Information Technology is final.

==System Maintenance and Downtime==

===Planned Maintenance===

The GACRC instituted monthly maintenance windows in order to perform maintenance operations requiring system operations to be reduced or interrupted.

The schedule will be as follows:

*The last Wednesday of each month from 10AM to 4PM will be reserved for partial cluster maintenance.

*Twice a year, a two-day shut-down of GACRC services will be scheduled for more complex maintenance operations. These will occur on the last Tuesday and Wednesday of the months of January and July.

These maintenance windows represent periods when the GACRC may choose to drain the queues of running jobs and suspend access to the Sapelo2 cluster, as well as storage devices for maintenance purposes. Interruptions will be kept as brief as possible.

The GACRC will notify all users at least 10 days in advance that a maintenance window will be in effect. The notification will describe the nature and extent (partial or full) of the interruptions of cluster and or storage services. In case a maintenance window has to be extended due to unavoidable technical reasons, adequate communications will be made to all users.

The impact of the outages will vary, and the GACRC will do its best to preserve pending and running jobs, which is often very doable. Nevertheless, users will need to plan their job submissions around the maintenance windows.

===Unplanned Maintenance and System Outage===

From time to time, hardware, software, and/or environmental factors may cause a system or subsystem to malfunction, causing disruption to service. Also, there may be circumstances or events related to possible security issues or intrusions which will cause GACRC staff to take systems offline while the nature of the apparent breach is analyzed and appropriate action is taken.

Whenever possible, account holders will be notified by e-mail of these outages in advance, but that may not always be possible. Account holders will be notified by e-mail if the disruption should last more than 30 minutes.

GACRC staff will strive to preserve the work and/or prevent disruption of jobs in process during such outages. However, there may be circumstances which cause disruption of jobs and loss of data. Users are encouraged to implement methods in their code which minimize the effect of unplanned interruption of a job’s execution, such as checkpoints. Users are also strongly encouraged to maintain copies of files of importance.

==Regulatory Compliance==

The GACRC as an infrastructure and service provider does NOT currently warrant that its practices or facilities meet government-mandated requirements for the storage and protection of sensitive, private or classified information. Users may not store such information on GACRC facilities. In other words, data that falls under HIPAA, FERPA, FISMA or similar regulatory requirements, may not be stored, computed against or otherwise transacted through, or with, GACRC infrastructure.

The GACRC and its users must comply with all existing Federal export control regulations for services and infrastructure. Research groups must agree to NOT install or use any software or data that falls under Export Control regulations. More information on the subject of Export Control is available at the following OVPR website:

http://research.uga.edu/export-control/

Copyrighted materials are prohibited without proper authorization. Additionally, illegal content is prohibited.

Non-compliance with any such Federal requirements might impact GACRC operations or delivery of services and could place the GACRC and UGA at risk. If a research group is found to be in non-compliance, then account access will be immediately suspended, while an investigation by EITS’s Information Security division is instigated.

===Disclosure===

Research groups that are involved in activities that store protected data on GACRC infrastructure must contact immediately the GACRC Director in order to address the issue. Depending on circumstances, accommodations might be possible for such activities.

===Research Data Compliance===

Research Data Management is a critical factor in both obtaining federal funding from the NSF, NIH, DoD and other federal agencies, and in the conduct of funded research. Responsibility in maintaining and preserving research data, as detailed in data management plans submitted to Federal funding agencies, is entirely placed upon the research faculty, post docs, and graduate students conducting the research. The GACRC will help by providing information and assistance, but will not be responsible to ensure compliance with a project’s data management plan.

During the phase of proposal writing, arrangements can be discussed and agreed upon as to the GACRC playing an active role, and ensuring the provision of specific services towards the compliance of a data management plan. Depending on the complexity or the nature of the proposed services, the GACRC might require the purchase of specific hardware/software and/or the availability of a %FTE salary and benefits.

More information on data management plans can be found [http://guides.libs.uga.edu/c.php?g=349946&p=2363161 here].

Georgia Advanced Computing Resource Center

2018-05-11T16:34:53Z

Jerky:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"









<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Online </span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> IMPORTANT NEWS </div>
The following is an important notice for all of our current zcluster users.
* [[zcluster Decommissioning]]

* [[The Sapelo2 Project]]

* [[Sapelo2 Frequently Asked Questions]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]
* [[Training]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Sapelo and Sapelo2 comparison]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Job and Data Management </div>
Information on how to run jobs and data management.
* [[Running Jobs]]
* [[Monitoring Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]
* [[Best Practices]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]
* [[Build Applications]]
* [[Zaney]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>

Georgia Advanced Computing Resource Center

2016-12-09T21:08:43Z

Jerky:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"









<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Online </span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]
* [[Training]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Running Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>