Research Computing Center Wiki - User contributions [en]

Training

2023-02-15T16:29:08Z

Ben: March training schedule

==GACRC Training==

The GACRC regularly hosts training sessions on a number of subjects relevant to the use of our computational and storage resources. Scheduled trainings will be announced through the GACRC mailing list.

'''NOTE: New users are required to attend a Sapelo2 cluster introductory training session and information about that will be sent once an account is requested.'''

==Regular Training Announcement==
In '''March''' 2022, the GACRC is hosting 7 training sessions (Linux basics, Sapelo2 cluster new user training, and Using Sapelo2 Cluster at the GACRC, Part II).

We will offer:

1. Linux training for Linux-inexperienced cluster new users (3 sessions)

2. Sapelo2 cluster new user training (3 sessions)

3. Using Sapelo2 Cluster at the GACRC, Part II (1 session)

'''Please Note:''' The training workshops will be offered remotely via Zoom Meeting. Detailed information on how to join the Zoom meeting will be sent to your UGA email account prior to each training session.

==Event Schedule==

===Sapelo2 Cluster New User Training===

Our Sapelo2 training consists of 1 hr 30 mins of instructional videos, followed by a 1 hr 30 min workshop. '''The instructional videos are required to be viewed prior to the training workshop, and they can be found [https://kaltura.uga.edu/playlist/dedicated/176125031/1_4o12v8b4/1_9zi68rgi here]'''

'''Prerequisites:'''

* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.

'''Video Playlist Training Goals:'''

* Understand the layout of Sapelo2

* Understand the Sapelo2 file systems

* Understand the Sapelo2 partitions

* Understand the Sapelo2 software environment

'''Workshop Training Goals:'''

* Understand how to request computing resources and submit a computational batch job following the Sapelo2 cluster general workflow

* Understand how to initiate an interactive job

* Understand how to transfer files to and from the cluster

* Understand how to get support from GACRC support team when you have any issues on cluster

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC
|February 24th, Friday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|March 8th, Wednesday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|March 17th, Friday, 1:00 PM - 3:00 PM
|-
|Using Sapelo2 Cluster at the GACRC
|March 30th, Thursday, 1:00 PM - 3:00 PM
|-
|}

===Linux Training for Linux-inexperienced Cluster New Users===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamental concepts of Linux working environment (filesystem hierarchy, path, PATH, etc.)

2. Know how to use Linux common commands (ls, cd, pwd, cat, more, nano, mkdir, rm, cp, mv, etc.)

3. Understand what is Linux bash shell and know how to make a simple Linux script and run it in Linux environment

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Use Linux on Cluster
|February 22nd, Wednesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|March 6th, Monday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|March 15th, Wednesday, 1:00 PM - 3:00 PM
|-
|Use Linux on Cluster
|March 28th, Tuesday, 1:00 PM - 3:00 PM
|-
|}

===Using Sapelo2 Cluster at the GACRC, Part II===
'''Prerequisites:'''
* Linux basics. A Linux-inexperienced user must complete a prerequisite Linux training for Linux-inexperienced cluster new users.
* Sapelo2 cluster new user training. Fundamental HPC and Sapelo2 knowledge is required for this advanced Sapelo2 workshop.

'''Training Goals:'''

1. Learn about high-performance computing framework

2. Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

3. How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

4. How can I make my job run more efficiently (through the correct use of software and hardware)?

5. A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Using Sapelo2 Cluster at the GACRC, Part II||March 24th, Friday, 1:00 PM - 3:00 PM
|-
|}

===Python Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand Python scientific modules and distributions

2. Understand Python general lexical conventions; Python built-in data types, like string, list, tuple, dictionary, etc.

3. Understand Python programming structures and procedural programming using functions

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Title
! scope="col" |Date/Time
|-
|Python Basics I||Not scheduled in March
|-
|Python Basics II||Not scheduled in March
|-
|}

===R Basics===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of R language, e.g. R general lexical conventions, data types, functions, and packages. Part 2 will introduce loops and functions.

2. Be able to manipulate and create data frames using built in functions and the dplyr package.

3. Interact with your file system and submit R code as a batch job to Sapelo 2.

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
| R Basics I||Not scheduled in March
|-
|R Basics II||Not scheduled in March
|-
|}

===Conda===
'''Prerequisite:''' No prerequisites

'''Training Goals:'''

1. Understand fundamentals of conda environment

2. Use conda to create and configure your own virtual environments

3. Activate your environments to run python apps from your home directory on Sapelo2

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" |Title
! scope="col" |Date/Time
|-
|Conda Basics||Not scheduled in March
|-
|}

==How to Register==

'''Please Note''', the training workshops '''Using Sapelo2 Cluster at the GACRC''' and '''Use Linux on Cluster''' are '''ONLY''' offered to '''new users''' who need computing user accounts on the GACRC Sapelo2 cluster, or any current users who have never attended the GACRC Sapelo2 cluster new user training before. Please ask your group PI/UGA faculty member to send us a request for you, using the GACRC User Account Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25839

If you want to attend '''Python Basics''', '''R''', and '''Conda basics''' training sessions, please send us a request using the GACRC Training Request form at https://uga.teamdynamix.com/TDClient/Requests/ServiceDet?ID=25852 . In your request, please tell us which session(s) you want to attend.

The GACRC is going to host other training workshops and seminars covering various HPC topics, including HPC fundamental introduction, Linux introductory III (Linux working environment and utilities), Bioinfomatics applications on Sapelo cluster, Perl, R, C/C++/Fortran programming, etc., in the near future. We will announce those events when they are scheduled.

The GACRC Web Training page can be found at https://gacrc.uga.edu/training/ and the GACRC Wiki Training page can be found at https://wiki.gacrc.uga.edu/wiki/Training, from which you can find detailed information about upcoming and past training sessions from GACRC and download training materials.

== Topic Introduction==

Title: '''Sap2test cluster migration training'''

Focus: Slurm queueing system, including Slurm job commands, job environment variables, and job submission headers, etc.

The new software environment on Sap2test

Other important topics related to Sap2test working environment

Title: '''Using Sapelo2 Cluster at the GACRC'''

Focus: Sapelo2 HPC cluster and computational batch job submission workflow

Cluster's storage environment

Computational queues on cluster

Software environment

How to submit computational batch jobs

Other tips and guidelines for users

Title: '''Using Sapelo2 Cluster at the GACRC, Part II'''

Focus: More topics on how to use Sapelo2 cluster

Learn about high-performance computing framework

Why is my job pending? How can I get my job to start sooner? How to find available computing resources on Sapelo2?

How to request computing resources such as nodes, CPU cores, memory, GPU device, etc. to run serial, threaded, MPI, and GPU jobs on Sapelo2?

How can I make my job run more efficiently (through the correct use of software and hardware)?

A quick intro to MPI library and how to compile/run MPI jobs on Sapelo2

Title: '''Use Linux on Cluster'''

Focus: Linux OS fundamentals

Linux common commands, filesystem, and shell

Linux shell scripting basics

Common Linux utilities, e.g., grep, sed, find, sort, and awk, etc.

Linux Hands-on practice

Title: '''Python Basics I, II'''

Focus of I: Python language overview, scientific modules and distributions

Python general lexical conventions

Basic built-in data types, like string, list, tuple, dictionary, etc.

Focus of II: Programming structures: control flow and loop

Function: procedural programming with examples, lambda expression, factory function and generator

Title: '''R Basics I, II'''

Focus of I: R language overview,general lexical conventions, data types, functions, and packages.

Basic built-in data types, like string, numeric, list, dataframe etc. Using the dplyr package.

Focus of II: Programming structures: control flow, loops and functions

Title: '''Python on GACRC Sapelo2 Cluster'''

Focus: Install Python packages/modules in a user's home directory on Sapelo2 cluster

Python versions installed on Sapelo2

Python environment details on Sapelo2

How to know a Python package is installed or not on Sapelo2

How to install a Python package in user's home directory on Sapelo2

Title: '''Do It Yourself: Using Conda to create and run python environments to suit your computing needs effortlessly!'''

Focus: Use conda to create and configure your own python virtual environments; Activate your environments to run python apps from your home directory on Sapelo2

What is Conda and its environment

Conda on Sapelo2

Use conda to create and configure your own python virtual environments

Activate your environments to run python apps from your home directory on Sapelo2

Title: '''How to submit and run jobs efficiently and correctly on Sapelo2'''

Focus: Sapelo2 cluster general workflow and correct computing resource requesting

Overview of Sapelo2 cluster with reference tables and operational diagrams

Sapelo2 batch job submission workflow taking global scratch as job working space

How to request computing resources correctly

How to run pipeline tasks and what are advantages/disadvantages of different options

Sapelo2 cluster guideline and practical tips

Title: '''GACRC Storage Environment'''

Focus: Overview of Linux common commands related to file and folder operations

Overview of the storage enviornment of zcluster and Sapelo cluster at GACRC

How to transfer data between local and GACRC storage

New file transfer node xfer2 and how to use it to transfer data between zcluster and the new cluster

GACRC suggestions on good practices on GACRC storage, etc;

Title: '''NCBI Blast application on sapelo'''

Focus: Introduction to BLAST

BLAST job submission to sapelo

Advantages & Disadvantages: NCBI website vs run at sapelo.

Understand BLAST output

Troubleshooting the BLAST results

Title: '''NGS application overview at GACRC'''

Focus: Overview of Bioinformatics software available on HPC clusters at GACRC

It’s a brave new world – NGS and its Applications

Hardware, Software, Databases available at GACRC

NGS project: Logistics and resource considerations

Best practices, common mistakes, troubleshooting and getting help from GACRC

Title: '''Perl Language Basics I, II'''

Focus of I: Overview of Perl language,

Perl general scripting style

Perl fundamental data types

Focus of II: Program structure: control flow and loop

Perl subroutine

Perl I/O

==Download==

====Sapelo2 Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:GACRC_Sapelo2_cluster_new_user_training_workshop_v10.7.pdf]]
|-
|}

====Sap2test Migration Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Migrating_to_Slurm_and_new_software_environment.pdf]]
|}

'''Please note:''' To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the '''GACRC's Kaltura channel''' at
https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required).

==== Teaching Cluster Training====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:GACRC-Teaching-cluster-new-user-training-workshop.pdf]]
|-
|}

====Linux Training for New Cluster Users====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Linux_Training_For_New_Users_Of_Cluster_Suchi_04252019.pdf]]
|-
|}

==== Python Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Python_Language_Basics_I_v5.1.pdf]]
|-
|[[Media:Python_Language_Basics_II_v5.1.pdf]]
|-
|[[Media:Python_Basics_v6.1.pdf]]
|-
|}

====R Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:R Language Basics PowerPoint v2.0.1.pdf|Media:R_Language_Basics_PowerPoint_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_Document_v2.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Powerpoint_v1.0.pdf]]
|-
|[[Media:R_Language_Basics_part_2_Document_v1.0.pdf]]
|-
|}

====Perl Basics====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
|[[Media:Perl_Language_Basics_I_Workshop_v1.pdf]]
|-
|}

====Topical Sessions====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|
|-
| [[Media:Using_Sapelo2_Cluster_at_the_GACRC_Part_II.pdf]]
|-
| [[Media:Using_Conda_on_the_GACRC_Sap2test_cluster_v1.pdf]]
|-
|[[Media:Blast_Workshop_GACRC_02012017.pdf]]
|-
|[[Media:Next-Generation_Sequencing_Applications_at_GACRC_10282016.pdf]]
|-
|}

==== Out-Reach/On-Class Talk====

{| class="wikitable unsortable" width="100%" border="1" cellspacing="0" cellpadding="2" align="center"
|-
! scope="col" | Dept./Center/Institute
! scope="col" | Type
! scope="col" |Workshop PDF
|-
|BCMB8330 - Spring2023||On-Class||[[Media:GACRC-Teaching-cluster-new-user-training-workshop_bcmb8330.pdf]]
|-
|PHYS4601/6601 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2023_phys4601.pdf]]
|-
|PHYS8602 - Spring2023||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] ; [[Media:Gacrc_handout2023_phys8602.pdf]]
|-
|ILS GradFIRST course - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220901-ILS.pdf]]
|-
|FYOS1001 - Fall 2022||Out-Reach||[[Media:High_Performance_Computing_(HPC)_on_GACRC_Sapelo2_Cluster.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:GACRC_overview_20220830-CSP.pdf]]
|-
|CSP seminar - Fall 2022||Out-Reach||[[Media:Compile_and_Run_HPC_code_on_Sapelo2.pdf]]
|-
|Terry College IT - Spring2022||Out-Reach||[[Media:GACRC_overview_20220506-Terry.pdf]]
|-
|PHYS8601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]]
|-
|PHYS4601/6601 - Spring2022||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] ; [[Media:Gacrc_handout2021_phys4601.pdf]]
|-
|PHYS8602 - Spring2021||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602-2021.pdf]] ; [[Media:Gacrc_handout2021_phys8602.pdf]]
|-
|GENE4220 - Fall2020||On-Class||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_GENE4220_Fall2020.pdf]]
|-
|College of Veterinary Medicine - Spring2020||Out-Reach (jlslab) ||[[Media:Using_GACRC_Sapelo2_Cluster-Advanced_Topics(1).pdf]]
|-
|Byod Data Center - Fall2019||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Cluster.pdf]]
|-
|Department of Linguistics - Fall2019||On-class (LING6570)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop_LING6570_Part2.pdf]]
|-
|The Center for Simulational Physics - Fall2019 ||Out-Reach (Seminar Talk 20190820)||[[Media:Introduction_to_GACRC_Computing_Facility_-_Sapelo2_Cluster_CSP-Fall2019.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS4601/6601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys4601.pdf]] [[Media:Gacrc_handout2019_phys4601.pdf]]
|-
|The Center for Simulational Physics||On-Class (PHYS8601)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8601.pdf]] [[Media:Gacrc_handout2020_phys8601.pdf]]
|-
|The Center for Simulational Physics ||On-Class (PHYS8602)||[[Media:GACRC_Teaching_cluster_new_user_training_workshop-phys8602.pdf]] [[Media:Gacrc_handout2019_phys8602.pdf]]
|-
|Food Science - Fall2018 ||On-Class (FYOS1001)||[[Media:High_Performance_Computing_(HPC)_on_Sapelo2_Cluster_at_GACRC.pdf]]
|-
|The Center for Simulational Physics - Summer2018||Out-Reach (Seminar Talk 20180821)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
| Miller plant science - Summer2018 ||Out-Reach (jlmlab)||[[Media:Introduction_to_GACRC_Sapelo2_cluster.pdf]]
|-
|Biochemistry and Molecular Biology - Spring2018||On-Class (BCMB8330)||[[Media:GACRC_zcluster_Class_Training_BCMB8330_Spring_2018.pdf]]
|-
| The Center for Simulational Physics - Summer2017||Out-Reach (Seminar Talk 20170831) ||[[Media:Introduction_on_HPC_Resources_at_the_GACRC.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS4601/6601) ||[[Media:Phys4601.pdf]]
|-
|Computational Physics - Spring2017||On-class (PHYS8602)||[[Media:Phys8602.pdf]]
|-
|The Institute of Bioinformatics and the Quantitative Biology Consulting Group|| Out-Reach||[[Media:Introduction_to_HPC_Resources_at_GACRC_BBB_Talk_20151014.pdf]]
|-
|The Center for Simulational Physics|| Out-Reach (Seminar Talk 20160906)||[[Media:Introduction_to_Sapelo_Computing_Resources_at_GACRC_Workshop20160906.pdf]]
|-
|Microbiology||On-Class (MIBO8150) ||[[Media:Introduction_to_HPC_Resources_at_GACRC_MIBO8150_20160926.pdf]]
|-
|Statistics||On-Class (STAT8060)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_Workshop_STAT8060_20150826.pdf]]
|-
|Biochemistry and Molecular Biology||On-Class (BCMB8211) ||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BCMB8211_20160114.pdf]]
|-
|Plant Biology||On-Class (PBIO/BINF8350)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO-BINF8350_20160115.pdf]]
|-
|Plant Biology - Bioinformatics Applications Fall2016||On-Class (PBIO4550)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_PBIO_4550_08182016.pdf]]
|-
| Bioinformatics - Essential Computing Skills for Biologists Fall2016||On-Class (BINF4005)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_BINF_4005_08312016.pdf]]
|-
|Computers in Experimental Genetics Fall2016||On-Class (GENE4220)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_GENE_4220_10192016.pdf]]
|-
|Statistics - Advanced Applications and Computing in R Fall2016||On-Class (STAT8330)||[[Media:Introduction_to_HPC_Using_zcluster_at_GACRC_STAT8330_11022016.pdf]]
|}

'''NOTE:''' The slides may become outdated and you should always check GACRC Wiki for up to date information.

== Past Sessions ==

[[Pass Sessions in 2021]]

[[Past Sessions in 2020]]

[[Past Sessions in 2019]]

[[Past Sessions in 2018]]

[[Past Sessions in 2017]]

[[Past Sessions in 2016]]

[[Past Sessions in 2015]]

Portal

2023-02-07T13:24:31Z

Ben:

[[Category:Sapelo2]]
__TOC__

==Introduction==

The GACRC Portal is a web interface for PIs to request Sapelo2 accounts for their lab members and view their current Sapelo2 lab group membership, and for anyone to register or unregister for or from GACRC training workshops.

==Training Registration==

To sign up for any GACRC training, navigate to the "Training" page of the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-home-user.png|700px]]

From the training page, simply click on the date of the workshop for which you would like to register:

[[File:Portal-workshop-2.png|700px]]

Upon clicking your desired training date, you will receive a confirmation email with the training Zoom link and other additional information.

----

[[File:Portal-training-confirmation.png|700px]]

You will see that your are registered for your desired training date as the date becomes green and the other dates are unselectable:

[[File:Portal-training-registered.png|700px]]

To unregister from a workshop, simply click your registered training date.

==New User Account Requests==

PIs who have already registered their lab group with GACRC are able to request Sapelo2 accounts for their lab members via the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-req-acc.png|700px]]

From this page the PI can search for the lab member for whom they would like to request a Sapelo2 account by MyID, email address, or employee number.

[[File:Req-acc-search.png|700px]]

[[File:Portal-search-result.png|700px]]

Once the new user has been found, the PI can click the "request" button to choose their lab on the cluster that the user should be added to and finalize the account request.

[[File:Send-req.png|700px]]

[[File:Acc-req-sent.png|700px]]

Upon clicking the "Send Request" button, an email will be sent to the new Sapelo2 user, inviting them to login to the portal, report their computing experience, and sign up for any GACRC training they may need.

Portal

2023-02-07T13:23:57Z

Ben:

Portal

2023-02-07T13:22:40Z

Ben:

[[Category:Sapelo2]]
__TOC__

==Introduction==

The GACRC Portal is a web interface for PIs to request Sapelo2 accounts for their lab members and view their current Sapelo2 lab group membership, and for anyone to register or unregister for or from GACRC training workshops.

==Training Registration==

To sign up for any GACRC training, navigate to the "Training" page of the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-home-user.png|700px]]

From the training page, simply click on the date of the workshop for which you would like to register:

[[File:Portal-workshop-2.png|700px]]

Upon clicking your desired training date, you will receive a confirmation email with the training Zoom link and other additional information.

[[File:Portal-training-confirmation.png|700px]]

You will see that your are registered for your desired training date as the date becomes green and the other dates are unselectable:

[[File:Portal-training-registered.png|700px]]

To unregister from a workshop, simply click your registered training date.

==New User Account Requests==

PIs who have already registered their lab group with GACRC are able to request Sapelo2 accounts for their lab members via the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-req-acc.png|700px]]

From this page the PI can search for the lab member for whom they would like to request a Sapelo2 account by MyID, email address, or employee number.

[[File:Req-acc-search.png|700px]]

[[File:Portal-search-result.png|700px]]

Once the new user has been found, the PI can click the "request" button to choose their lab on the cluster that the user should be added to and finalize the account request.

[[File:Send-req.png|700px]]

[[File:Acc-req-sent.png|700px]]

Upon clicking the "Send Request" button, an email will be sent to the new Sapelo2 user, inviting them to login to the portal, report their computing experience, and sign up for any GACRC training they may need.

Portal

2023-02-07T13:21:52Z

Ben:

[[Category:Sapelo2]]
__TOC__

==Introduction==

The GACRC Portal is a web interface for PIs to request Sapelo2 accounts for their lab members and view their current Sapelo2 lab group membership, and for anyone to register or unregister for or from GACRC training workshops.

==Training Registration==

To sign up for any GACRC training, navigate to the "Training" page of the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-home-user.png|700px]]

From the training page, simply click on the date of the workshop for which you would like to register:

[[File:Portal-workshop-2.png|700px]]

Upon clicking your desired training date, you will receive a confirmation email with the training Zoom link and other additional information.

[[File:Portal-training-confirmation.png|700px]]

You will see that your are registered for your desired training date as the date becomes green and the other dates are unselectable:

[[File:Portal-training-registered.png|700px]]

To unregister from a workshop, simply click your registered training date.

==New User Account Requests==

PIs who have already registered their lab group with GACRC are able to request Sapelo2 accounts for their lab members via the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-req-acc.png|700px]]

From this page the PI can search for the lab member for whom they would like to request a Sapelo2 account by MyID, email address, or employee number.

[[File:Req-acc-search.png|700px]]

[[File:Portal-search-result.png|700px]]

Once the new user has been found, the PI can click the "request" button to choose their lab on the cluster that the user should be added to and finalize the account request.

[[File:Send-req.png|700px]]

[[File:Acc-req-sent.png|700px]]

Upon clicking the "Send Request" button, an email will be sent to the new Sapelo2 user, inviting them to login to the portal, report their computing experience, and sign up for any GACRC training they may need.

Portal

2023-02-07T13:19:07Z

Ben:

[[Category:Sapelo2]]
__TOC__

==Introduction==

The GACRC Portal is a web interface for PIs to request Sapelo2 accounts for their lab members, and for anyone to sign up for GACRC training workshops.

==Training Registration==

To sign up for any GACRC training, navigate to the "Training" page of the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-home-user.png|700px]]

From the training page, simply click on the date of the workshop for which you would like to register:

[[File:Portal-workshop-2.png|700px]]

Upon clicking your desired training date, you will receive a confirmation email with the training Zoom link and other additional information.

[[File:Portal-training-confirmation.png|700px]]

You will see that your are registered for your desired training date as the date becomes green and the other dates are unselectable:

[[File:Portal-training-registered.png|700px]]

==New User Account Requests==

PIs who have already registered their lab group with GACRC are able to request Sapelo2 accounts for their lab members via the portal. This can be accessed via the sliding carousel on the home page, or the expandable sidebar on the left.

[[File:Portal-req-acc.png|700px]]

From this page the PI can search for the lab member for whom they would like to request a Sapelo2 account by MyID, email address, or employee number.

[[File:Req-acc-search.png|700px]]

[[File:Portal-search-result.png|700px]]

Once the new user has been found, the PI can click the "request" button to choose their lab on the cluster that the user should be added to and finalize the account request.

[[File:Send-req.png|700px]]

[[File:Acc-req-sent.png|700px]]

Upon clicking the "Send Request" button, an email will be sent to the new Sapelo2 user, inviting them to login to the portal, report their computing experience, and sign up for any GACRC training they may need.

File:Send-req.png

2023-02-07T13:14:37Z

Ben:

2023-02-06T13:39:31Z

Ben:

File:Portal-workshop-2.png

2023-02-06T13:35:33Z

Ben:

2022-07-28T13:58:21Z

Ben:

2022-01-14T18:34:31Z

Ben:

= Open OnDemand (OOD) =

Open OnDemand ("OOD") is a web-based service provided by GACRC which provides users an alternative and/or complimentary method to access the HPC clusters. It helps computational researchers and students efficiently utilize remote computing resources by making them easy to access from any device, via a convenient web interface. Through Open OnDemand, graphical applications can be utilized on the cluster far more easily and smoothly than using X11 forwarding. It also helps computer center staff support a wide range of clients by simplifying the user interface and experience. A video introduction to Kaltura at UGA can be found '''[https://kaltura.uga.edu/media/t/1_u9d1xrpp/176125031 here]'''.

OOD Features:

* Job submission and monitoring
* File transfer and management
* File editing
* Shell access
* Graphical Desktop on the cluster
* Interactive Applications

{| class="wikitable"
|+Open OnDemand Integration with GACRC Clusters
!Availability
!Cluster
!URL
|-
|Production
|Sapelo2
|https://ondemand.gacrc.uga.edu
|-
|Development Testing
|Sapelo2
|https://oodev.gacrc.uga.edu
|-
|Future
|Teach
|N/A
|}

----
[[#top|Back to Top]]

= How to access Open OnDemand =

'''In order to access the GACRC OOD login page, You must be connected to the UGA Network, either directly or by [https://eits.uga.edu/access_and_security/infosec/tools/vpn/ VPN].'''

'''Logging in requires a [https://wiki.gacrc.uga.edu/wiki/User_Accounts GACRC Cluster Account].'''

'''GACRC OOD uses UGA Single Sign-On Service (SSO) for authentication.'''

# Open a web browser to the cluster's Open OnDemand URL: https://ondemand.gacrc.uga.edu
# Login with your UGA MyID credentials
===Browser Requirements/Notes===
To have the best experience using OnDemand, GACRC recommends using the latest version of '''Google Chrome, Mozilla Firefox,''' or '''Microsoft Edge'''. Any modern browser that supports ECMAScript 2016 ''should'' work, but GACRC will not support it.

'''Individual browser notes'''

* Google Chrome is currently the ''only'' web browser that natively supports the copy and paste functionality in <code>noVNC</code>. Other browsers can do copy and pasting manually through the <code>noVNC</code> tool drawer.

* Safari users '''''may''''' have issues when using the Shell App or <code>noVNC</code>. If you have issues, make sure your application is up-to-date.
* Internet Explorer 11 is '''NOT''' supported.

----
[[#top|Back to Top]]

= Using Open OnDemand =

Upon logging into Open OnDemand, all of its tools can be accessed via the links in the top menu bar.

[[File: Oodheader.png| OOD Menu Bar]]



== Files ==

Clicking "Files" in the top menu bar will open a drop menu that displays yours and your lab's directories on the cluster. Clicking any of those displayed directories will take you to a new page that allows you to manage the directories contents, as well as navigate to other directories on the cluster to which you have access. In this file browsing interface, there is a menu above the file browsing area that allows you to:

*Open an interactive shell in the current directory
*Create a new file
*Create a new directory
*Upload files
*Download files
*Copy/move files
*Delete files

[[File:Oodfiles2.png |900px]]

Clicking on a directory will navigate the OOD file browser to that directory. Clicking on a text file will open a new read-only window, displaying the file. If you would like to edit a file from within OOD, click the three-vertical-dots button to the right of a file name, and then click "Edit". This will open a new tab that allows you to edit and save changes to a file. The OOD file editor has a plethora of different syntax highlighting options. If the file you're editing has a file extension in its name for which OOD has syntax highlighting, it will pre-select that option, however you can change the syntax highlighting by selecting a different option from the "Mode" drop down menu at the top. Additionally in the top menu of the OOD file editor, you can change the font size, as well as select a different display theme.

=== File Transfer Limitations ===

'''Copy and Moving files'''
When it comes to copying and moving files within the cluster, it is recommended that you only use OOD for smaller files. For anything else, please use the [[Transferring Files|transfer nodes]]. This is due to the OOD server acting as a "man-in-the-middle" for transferring files between the different file systems. For example, if you move something from /home to /scratch, in the background the file first has to be transferred to a temporary storage on the OOD server, before it is then transferred to the destination.

'''Uploading files'''
GACRC '''highly''' recommends using the [[Transferring Files|transfer nodes]] or [[Globus]] for larger file transfers. If you must use OOD for large file transfers, please be aware there is a hard-limit in place of ~10.0 Gib. GACRC reserves the right to lower this limit in the future.

[[#top|Back to Top]]

== Jobs ==

Clicking "Jobs" will open a drop down menu allowing you to navigate to "Active Jobs" or "Job Composer".

=== Active Jobs ===

In the Active Jobs interface, you're able to view information about your currently running jobs. Clicking the arrow on the left will expand this interface displaying further information about the job, as seen below.

[[File:oodjobs2.png | 1200px]] [[File:Oodjobs.png | 1200px]]

[[#top|Back to Top]]

== Clusters ==

Clicking "Clusters" will open a drop down menu giving you an option to open a shell for the cluster in your browser, on one of the login nodes. This functions just as if you had connected to a Sapelo2 login node via PuTTY (Windows) or Terminal (Mac/Linux). In the top right of this interface you are able to select a different theme for your shell. Please see [[#Browser | Browser Requirements/Notes]] for possible limitations.

[[File:Oodcli1.png | 600 px]]

[[#top|Back to Top]]

== Interactive Apps ==

One of the best features of Open OnDemand is the ability to directly launch interactive graphical applications. "Interactive Applications" or "Apps" are wrappers that will schedule a Slurm job with a particular application, and give you direct web based access into the interface.

These apps can be accessed near the bottom of the Open OnDemand main page called "Pinned Apps" or from the "Interactive Apps" in the top menu bar. ''Note that the image below is only an example, as the list of interactive applications will grow over time.''

[[File:Ood-interactive-apps-3.png|600px]]

===Launching an application===
Clicking the icon of the interactive application that you would like to use will take you to a new page in which you define the resources allocated to interactive application.

If the GUI application you want to use has its own interactive app in Open OnDemand, you should use that launcher. If the GUI application you want to use does not have a launcher yet you have two options; You an open a ticket with GACRC and request the application be added to OnDemand (Link TBD), or you can use the "[[#X_Desktop |X Desktop Session]]".

====Configuring the resources====
Resources must be defined like this because an Open OnDemand interactive app is happening within the context of a Slurm job. Each application may allow for the customization of different resources, while some may have static defined resources that cannot be changed. Almost all of the resources available to be configured should be familiar to a user who has submitted a job before.

'''Please note''' that the <code>Number of hours</code> option is equivalent to "wall time". If you do not terminate the job before this limit is reached, Slurm will terminate the job for you.

Once you have defined the resources for your job, click "Launch" to be taken to a new page ([[#Inter |My Interactive Sessions]]). After a small delay, the page will refresh showing the status of the job that your app has scheduled. See [[#Inter |My Interactive Sessions]] section of the wiki for more information.

===X Desktop Session (A.K.A. The Interactive X login app)===
The "X Desktop Session on Sapelo2" is an interactive app that will start an X Desktop GUI session as a job. Launching the VNC session will connect you to an actual X desktop running on a Sapelo2 compute node, in your browser. This desktop is currently configured to run the XFCE lightweight graphical manager.

The Sapelo2 X desktop can be used for graphical applications that do not have a dedicated Open OnDemand interactive app. Please be aware that while it is possible within a Sapelo2 Desktop session to open a terminal, load a module, and run a graphical application, for example grace or gnuplot, some of the computing resources requested for this job would be used by the desktop itself. If you find that you have an application that you do this with regularly, please consider opening a ticket with GACRC requesting an interactive app.

====Copy/Paste in VNC====
If launching an interactive session in the portal, there are a few extra steps that need to be taken. Please reference the media below, or the summary of steps below that for more information.

#Open the toolbar on the left of the screen and select "Clipboard".
#If you want to paste text from your host computer to the remote session, paste the text in the clipboard box. You can then use the middle-mouse button (MMB) to paste it in your terminal.
#If you want to copy text from the remote session to your host computer's clipboard, simply highlight the text in the terminal. It will appear in the Clipboard toolbar pop-out where you can copy it to your host clipboard.

[[#top|Back to Top]]

==My Interactive Sessions==

The last item in Open OnDemand top menu bar is "My Interactive Sessions". This interface will list your current and recent interactive app sessions.

[[File:Interactive Session Example.png|frameless|693x693px]]

'''Notable items:'''

*'''Host''': Click the name of the host to open a shell to the compute node that is running this job.
*'''Time Remaining''': This is how long the job will remain running before terminating. (Can be thought of as remaining wall time.)
*'''Session ID''': The unique ID of this session. Clicking the link will open a file app window to this Ondemand sessions working directory. (Useful for troubleshooting issues with application launch)
*'''Compression/Image Quality''': These sliders allow you to change the amount of graphical compression and quality of the VNC session. If you are on a particularly slow connection, or find that the 'redraw' time is slow, try lowering the Image Quality, increasing the Compression, or both, and relaunch the VNC window.
*'''Launch Button''': This button will open a new tab with the web based VNC viewer application. Closing the VNC window will NOT terminate the job, only the VNC connection to it. You may return at any time (as long as the job has time remaining) and click the Launch button to reconnect to your session.
*'''View Only''': This will open a new web based VNC viewer application just like the Launch button, with the caveat that no user input is allowed. You can then copy the URL for this window and send it to others, who will be able to view the VNC session, but not interact with it. (e.g. move the mouse, etc) (This is a great function if you want to show multiple people something in a controlled session!)
*'''Delete''': This button will terminate the job, and delete the session.

[[#top|Back to Top]]

=Additional Notes=

==Interactive applications and GPUs (Hardware accelerated rendering)==
Sapelo2 is currently not configured for applications to use hardware GPU for accelerated graphical rendering. This is currently on our "To Do" list.

Requesting a GPU for an interactive application will assign a GPU to the job just as it would requesting one from a normal command line submit script, but the Xserver will not use this GPU for graphical acceleration. (e.g. you can still use the GPU for computation, but it will not accelerate your frame rate.)

==Per-User Experience==
OnDemand will spawn a Per User Nginx session upon successful login. This means that each connection on the OOD server is running as that particular user. Each session is completely separated from other users sessions.

==Troubleshooting==

For any assistance with Open OnDemand, please reach out to us in via the "General Support" form of the "For Users" section at help.gacrc.uga.edu.

[[#top|Back to Top]]

Installing Applications on Sapelo2

2021-12-24T13:51:44Z

Ben: qlogin > interact in all the code blocks

[[Category:Sapelo2]]

==Introduction==

We introduce here some guidance on how to install applications and libraries on Sapelo2.

In general, users can build applications under their [https://wiki.gacrc.uga.edu/wiki/Disk_Storage#Home_file_system home] directory or other space owned by the users.

If an application will be used by several members of a group, the application can be installed in the user's group [https://wiki.gacrc.uga.edu/wiki/Disk_Storage#Work_file_system work] space.

GACRC team takes requests to install applications at a central place (/apps) if the application satisfies the following conditions:

* The application is Linux compatible.
* The application has general interest among users.
* The application needs to be built with special settings, such as root privilege, shared common data set, complex structures, or other requirements.
* The application needs to be configured to integrate into the GACRC environment, such as queue settings or database connections.

Please use the [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25850 GACRC Software Installation/Update] online form to submit a support ticket to GACRC team if you need any help from us.

----
[[#top|Back to Top]]

==General Guidlines==

===DO NOT install applications at login node===
Sapelo2 login node (sapelo2.gacrc.uga.edu) has limited memory. Most of the time software installation fails due to insufficient memory. More importantly, the process of installing applications on the login node can degrade the performance of the cluster for everyone. So '''please do not install any applications on the cluster while you are on the login node'''. We strongly advise that you build or install applications from an interactive session that you can start with '''interact''' command from the login node:

<pre class="gcommand">
interact
</pre>

All of the following installation examples were executed in a '''interact''' environment.

===How to check if an application is installed ===
To find if an application, e.g. Trinity, is already installed on Sapelo2, use the following command (the application name is NOT case sensitive):
<pre class="gcommand">
module avail trinity
</pre>
or
<pre class="gcommand">
module spider trinity
</pre>

To see a description of the module (the application name is case sensitive)
<pre class="gcommand">
module whatis Trinity
</pre>

To see configuration details, e.g., environment variables, of the module (the application name is case sensitive)
<pre class="gcommand">
module show Trinity
</pre>

----
[[#top|Back to Top]]

===How to check if a library is installed===

'''Perl'''

To check if a perl library is installed, e.g. DBI, first load the Perl package of interest, for example Perl/5.30.0-GCCcore-8.3.0:

<pre class="gcommand">
module load Perl/5.30.0-GCCcore-8.3.0
perl -MDBI -e 'print "OK\n"'
</pre>

If the library is not installed, it will warn that the library is not in the path. Otherwise, it will print OK.

'''Python'''

To check if a Python library is installed, e.g. numpy, first load the Python package of interest, then run "pip show" or "pip list" commands. The Linux command "which" is to identify if pip is in place. In the following example, we use Python/2.7.16-GCCcore-8.3.0:

<pre class="gcommand">
module load Python/2.7.16-GCCcore-8.3.0
which pip
pip show numpy
</pre>

To list all installed libraries with their versions:

<pre class="gcommand">
module load Python/2.7.16-GCCcore-8.3.0
pip list
</pre>

To generate an alphabetical list of the libraries installed under a Python version:

<pre class="gcommand">
module load Python/2.7.16-GCCcore-8.3.0
pip freeze | sort
</pre>

For Python3, some versions have both pip3 and pip. pip3 is the Python3 version of pip. If you load Python3, run pip3, otherwise run pip. You can use "which" command to identify if pip3 is in place. For example:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
which pip3
</pre>

Then check if a Python3 library is installed, e.g. using pip3:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
pip3 show numpy
</pre>

To list all installed libraries with their versions:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
pip3 list
</pre>

To generate an alphabetical list of the libraries installed under a Python version:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
pip3 freeze | sort
</pre>

If the library is not installed, it will not be shown by pip or pip3 commands. Otherwise, pip or pip3 will give you information about libraries' version and installation location.

'''Note:''' Some Python libraries are installed outside of the Python default location and are provided as a separate module file that needs to be loaded separately. To check if a python library is installed as a separate module, please use the '''module spider''' command. For example, to check if matplotlib is installed as a separate module:

<pre class="gcommand">
module spider matplotlib
</pre>

This command will return all the matplotlib versions currently installed on the cluster, along with the version of Python that it uses. For example, the module named matplotlib/3.1.1-foss-2019b-Python-3.8.2 provides matplotlib version 3.1.1 for Python 3.8.2 and it uses the foss-2019b toolchain. To use this version of matplotlib, please load the module matplotlib/3.1.1-foss-2019b-Python-3.8.2. Similarly, you can load other module files that provide other Python libraries.

'''R'''

To check if a R library is installed, e.g. xtable, first load the R version of interest, for example R/4.0.0-foss-2019b:

<pre class="gcommand">
interact
module load R/4.0.0-foss-2019b
R
require("xtable")
exit
</pre>

If the library is not installed, it will warn that the library is not in the path. Otherwise, it will print "Loading required package: xtable".

----
[[#top|Back to Top]]

===How to structure directories===

Common practice is to set up the following three directories using "mkdir" command in the user's home directory /home/MyID:

*'''apps''': directory where the applications will be installed
*'''src''': directory under which you can store the source files and build the applications
*'''modulefiles''': directory under which you can put your own module files

These directories can be created with
<pre class="gcommand">
mkdir ~/apps
mkdir ~/src
mkdir ~/modulefiles
</pre>

----
[[#top|Back to Top]]

===How to use local Lmod modules===

To use your own modules, e.g. trinity/1.0:

<pre class="gcommand">
module use ~/modulefiles
module load trinity/1.0
</pre>

If there are module files with the same name as in our central place, your local ones will take precedence over the central ones.

----
[[#top|Back to Top]]

==How to Install Software Packages (general)==

If you are interested in using an application, check the website of this application to see if (a) it is compatible with Linux, (b) it is distributed as binaries or source code, (c) and the kind of package that it is (Python library and scripts, Perl scripts, Java, C, C++, Fortran code, etc).

Perl scripts and Java jar files can be downloaded into the users' home directory and run from there.

If the application is distributed as pre-compiled binaries, check if binaries are available for the Linux OS that our cluster run (Sapelo2 runs CentOS 7.8). Binaries compiled for Window and for Mac OSX cannot be run on Sapelo2. Also, binaries generated in other Linux distributions (e.g. Ubuntu, Debian, etc) will in general not work on Sapelo2.

If pre-compiled binaries are not available for the OS on Sapelo2 (CentOS 7.8), then you can compile the code yourself, if the source code is available.

Another option for running binaries compatible with other Linux distributions is to create a Singularity container and run it on Sapelo2 (more information below).

----
[[#top|Back to Top]]

==Installing Software Packages==

===How to build C, C++ applications===

Here is an example on how to install an application called GERUDsim3, which comprise a single C++ program.

1. Login to sapelo2 login node. If you do not have a directory called '''apps''', create one with

<pre class="gcommand">
mkdir ~/apps
</pre>

2. Download the application into the ~/apps directory. For the example used here, you can download it with
<pre class="gcommand">
cd ~/apps

git clone https://github.com/JonesLabIdaho/GERUDsim3.git
</pre>

3. Start an interactive session with
<pre class="gcommand">
qlogin
</pre>

4. compile the code on the interactive node with:
<pre class="gcommand">
cd ~/apps/GERUDsim3/GERUDsim3/Source_files

g++ GerudSim3.cpp -o GerudSim3
</pre>

The g++ command that is on user's default path is version 4.8.5. If you wish/need to use a different GNU compiler version, or a different compiler (e.g. Intel), then first load the corresponding module and then compile the code. For example, to use g++ 8.3.0:

<pre class="gcommand">
cd ~/apps/GERUDsim3/GERUDsim3/Source_files

module load foss/2019b

g++ GerudSim3.cpp -o GerudSim3
</pre>

5. The name of your executable is GerudSim3. You can run it with the full path

<pre class="gcommand">
~/apps/GERUDsim3/GERUDsim3/Source_files/GerudSim3
</pre>

or copy this binary to ~/bin (create this directory first, if it does not exist yet) and run it with

<pre class="gcommand">
~/bin/GerudSim3
</pre>

You can also add ~/bin to your default PATH. To do that, add the following in your .bashrc file:

<pre class="gcommand">
export PATH=/home/MYID/bin/:$PATH
</pre>

where MYID needs to be replaced by your UGA MyID. With this, any executable you put in ~/bin will be in your PATH and can be invoked without its full path.

For more information on compilers available on Sapelo2, please see [[Code Compilation on Sapelo2]].

----
[[#top|Back to Top]]

===Cross build for mixed processor architecture===
At Sapelo2, we have two types of processors: '''Intel''' and '''AMD'''.

In some C or C++ applications, compiler performance-optimization flags, such as -march=native, may be introduced in the configuration step or in the Makefile. Executables compiled with such options on one type of processor might not run on a different processor type. To enable the compiled application to run on both types of processors, these processor specific compiler optimization options should be removed. One option is to set the flags as
'''-mtune=generic -march=x86-64'''.

If you want to compile or test your program on different types of processors to ensure that they work there, you could start an interactive session on each of the processor types. For example, to start an interactive session on an AMD EPYC node, use

<pre class="gcommand">
interact --constraint EPYC
</pre>

You could choose a different processor type with the '''--constraint''' option. For more information on how to start interactive sessions on Sapelo2, please see [https://wiki.gacrc.uga.edu/wiki/Running_Jobs_on_Sapelo2#How_to_open_an_interactive_session Running interactive jobs on Sapelo2]

[[#top|Back to Top]]

===How to install a Perl module===

A convenient way to build and install perl modules is to use CPAN, which can be used to install Perl modules in the user's home directory. We suggest that you do this installation in an interactive session.

1. Start an interactive session with
<pre class="gcommand">
interact
</pre>

2. Decide where you want the perl modules to be installed, e.g. ~/perlmods, and create this directory if it does not exist yet
<pre class="gcommand">
mkdir ~/perlmods
</pre>

3. Load the perl module that you want to use, e.g. Perl/5.30.0-GCCcore-8.3.0
<pre class="gcommand">
module load Perl/5.30.0-GCCcore-8.3.0
</pre>

4. Start the CPAN shell with
<pre class="gcommand">
cpan
</pre>

When you start cpan for the first time, some configurations will be set (unless you already have configurations set in ~/.cpan/CPAN/MyConfig.pm):

<pre class="gcommand">
cpan

CPAN.pm requires configuration, but most of it can be done automatically.
If you answer 'no' below, you will enter an interactive dialog for each
configuration option instead.

Would you like to configure as much as possible automatically? [yes]

Autoconfiguration complete.

commit: wrote '/home/MyID/.cpan/CPAN/MyConfig.pm'

You can re-run configuration any time with 'o conf init' in the CPAN shell

cpan shell -- CPAN exploration and modules installation (v2.14)
Enter 'h' for help.

cpan[1]>

</pre>

Note that in the above commit output: 'wrote /home/MyID/.cpan/CPAN/MyConfig.pm', MyID is replaced by your own MyID.

5. From within the CPAN shell, enter the following two (2) commands to specify the installation directory:

<pre class="gcommand">
cpan[1]> o conf mbuildpl_arg "--install_base /home/MyID/perlmods"

mbuildpl_arg [--install_base /home/MyID/perlmods]
Please use 'o conf commit' to make the config permanent!

cpan[2]> o conf makepl_arg "PREFIX=/home/MyID/perlmods"

makepl_arg [PREFIX=/home/MyID/perlmods]
Please use 'o conf commit' to make the config permanent!

cpan[3]>
</pre>

Note that in the two commands above you will need to replace MyID by your own MyID and change perlmods if you are using a different directory name. If you want to set the above installation path as the default one, you can make the settings above permanent by entering "o conf commit". If this is not done, you will need to reset this value every time you restart CPAN. If you do make the settings permanent, you can always change them later and re-commit as shown above.

6. To install a module (for example, if you want to install "IO::CaptureOutput") enter:
<pre class="gcommand">
cpan[3]> install IO::CaptureOutput
</pre>

Respond to any prompts for information that might be requested. When you are finished, enter:
<pre class="gcommand">
cpan[4]> quit
</pre>

7. After you have successfully installed a local Perl module, set the PERL5LIB environmental variable to tell Perl where to find the module. For example:

<pre class="gcommand">
export PERL5LIB=/home/MyID/perlmods:$PERL5LIB
</pre>

where /home/MyID/perlmods has to be replaced by the path to your installation directory.

You can add this export command to your .bashrc file if you'd like to ensure that this PERL5LIB environment variable is always set when you login and for your non-interactive scripts. If this export line is not added in your .bashrc file, you can add it to your job submission scripts. You can also define a module file where this variable is defined.

----
[[#top|Back to Top]]

===How to install Python packages===
To check what Python versions are installed:

<pre class="gcommand">
module spider Python
</pre>

Make sure the right Python version you need is loaded, for example, Python/3.8.2-GCCcore-8.3.0:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
</pre>

Most Python versions have commands such as "python" and "pip", some versions have version concatenate after command, such as "python2.7", "python3", "pip3" etc.. So after Python module is loaded, usually we can check if it is the right command by:

<pre class="gcommand">
which python
which pip
which pip3 (Python3 only)
</pre>

or list the path to confirm the correct python commands, such as:

<pre class="gcommand">
ls /apps/eb/Python/3.8.2-GCCcore-8.3.0/bin
</pre>

The above Python binary path can be found by using "ml show" command:

<pre class="gcommand">
ml show Python/3.8.2-GCCcore-8.3.0
</pre>

From outputs, you will find the PATH environment variable pre-appended with the Python binary path, i.e., /apps/eb/Python/3.8.2-GCCcore-8.3.0/bin.

====How to install Python package tarball in Sapelo2 home directory====

There are several options you can take to install a Python package in your Sapelo2 home directory /home/MyID; MyID should be replaced by your UGA MyID.

Common steps are listed as below:

1. Start an interactive session with "interact" command from Sapelo2 login node:

<pre class="gcommand">
interact
</pre>

2. Suppose you have downloaded a Python package tarball file, e.g., myPackage-1.0.tar.gz, in the source folder (src) in your home directory with a full path as:

<pre class="gcommand">
/home/MyID/src/myPackage-1.0.tar.gz
</pre>

Change directory to /home/MyID/src/:

<pre class="gcommand">
cd ~/src
</pre>

Uncompress and untar the tarball file using "tar xzvf" command:

<pre class="gcommand">
tar xzvf myPackage-1.0.tar.gz
</pre>

3. Once the tarball is untarred, you will have a package source folder in your current directory, e.g., myPackage-1.0/. You need to change directory into it (full path is /home/MyID/src/myPackage-1.0):

<pre class="gcommand">
cd ./myPackage-1.0
</pre>

4. Decide which version of Python you want myPackage-1.0 to be installed with. In this example we use Python/3.8.2-GCCcore-8.3.0:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
</pre>

5. Install package using "python setup.py install" command

*'''Option 1:''' using "--user" option. The package will be installed into a default installation location, i.e., /home/MyID/'''.local'''/lib/python3.8/site-packages/:

<pre class="gcommand">
python setup.py install --user
</pre>

*'''Option 2 (recommended):''' using "--user" option with PYTHONUSERBASE environment variable defined and exported. The package will be installed into a location you specified, e.g., /home/MyID/'''python/3.8.2'''/lib/python3.8/site-packages/:

<pre class="gcommand">
mkdir -p /home/MyID/python/3.8.2
export PYTHONUSERBASE=/home/MyID/python/3.8.2
python setup.py install --user
</pre>

*'''option 3:''' using '--prefix' option with PYTHONPATH environment variable defined and exported. The package will be installed into a location you specified, e.g., /home/MyID/'''python/3.8.2'''/lib/python3.8/site-packages:

<pre class="gcommand">
mkdir -p /home/MyID/python/3.8.2/lib/python3.8/site-packages
export PYTHONPATH=/home/MyID/python/3.8.2/lib/python3.8/site-packages
python setup.py install --prefix="/home/MyID/python/3.8.2"
</pre>

6. Make sure to export the above library path when the package is needed. This can be included in your module file as well. For example:

<pre class="gcommand">
export PYTHONPATH=/home/MyID/.local/lib/python3.8/site-packages/:$PYTHONPATH
</pre>

or

<pre class="gcommand">
export PYTHONPATH=/home/MyID/python/3.8.2/lib/python3.8/site-packages:$PYTHONPATH
</pre>

====How to install Python package using pip/pip3 in Sapelo2 home directory====

The pip/pip3 is a package management system used to install and manage Python2/Python3 packages, such as those found in the Python Package Index (PyPI). PyPI is a software repository for the Python language. It helps to find and install software developed and shared by the Python community. In this example, let us try scikit-learn-0.23.2, which is the most recent version of scikit-learn package as of 10-26-2020, at PyPI.

Common steps are listed as below:

1. Start an interactive session with '''interact''' command from Sapelo2 login node:

<pre class="gcommand">
interact
</pre>

2. Decide which version of Python you want myPackage-1.0 to be installed with. In this example we load Python/3.8.2-GCCcore-8.3.0, then use "which" command to identify if pip3 is in place:

<pre class="gcommand">
module load Python/3.8.2-GCCcore-8.3.0
which pip3
</pre>

"which" command shows pip3 is from 3.8.2-GCCcore-8.3.0 module:

<pre class="gcommand">
/apps/eb/Python/3.8.2-GCCcore-8.3.0/bin/pip3
</pre>

3. Install package using "pip3 install" command

*'''Option 1:''' using "--user" option. The package will be installed into a default installation location, i.e., /home/MyID/'''.local'''/lib/python3.8/site-packages/:

<pre class="gcommand">
pip3 install --user scikit-learn
</pre>

"pip3 install" command will output and show you that scikit-learn-0.23.2 is successfully downloaded and installed. Then you can use "ls" command to show the package is indeed installed in '''.local'''/lib/python3.8/site-packages/, in your home directory:

<pre class="gcommand">
ls ~/.local/lib/python3.8/site-packages/
scikit_learn-0.23.2.dist-info scikit_learn.libs
</pre>

*'''Option 2 (recommended):''' using "--user" option with PYTHONUSERBASE environment variable defined and exported. The package will be installed into a location you specified, e.g., /home/MyID/'''python/3.8.2'''/lib/python3.8/site-packages/:

<pre class="gcommand">
mkdir -p /home/MyID/python/3.8.2
export PYTHONUSERBASE=/home/MyID/python/3.8.2
pip3 install --user scikit-learn
</pre>

"pip3 install" command will output and show you that scikit-learn-0.23.2 is successfully downloaded and installed. Then you can use "ls" command to show the package is indeed installed in '''/python/3.8.2'''/lib/python3.8/site-packages/, in your home directory:

<pre class="gcommand">
ls ~/python/3.8.2/lib/python3.8/site-packages/
scikit_learn-0.23.2.dist-info scikit_learn.libs
</pre>

4. Make sure to export the above library path when the package is needed. This can be included in your module file as well. For example:

<pre class="gcommand">
export PYTHONPATH=/home/MyID/.local/lib/python3.8/site-packages/:$PYTHONPATH
</pre>

or

<pre class="gcommand">
export PYTHONPATH=/home/MyID/python/3.8.2/lib/python3.8/site-packages:$PYTHONPATH
</pre>

----
[[#top|Back to Top]]

===How to install Conda packages===
To install a software package as a conda environment in user's Sapelo2 home directory, please use the option "-p" to define your environment installation path, e.g., /home/MyID/busco_conda.

For example, to install busco v3.0.2 conda environment at /home/MyID/busco_conda:

<pre class="gcommand">
module load Miniconda3/4.7.10
conda create -p /home/MyID/busco_conda -c bioconda busco=3.0.2
</pre>

where '''bioconda''' is the channel where busco v3.0.2 source is downloaded. Other common conda channels are, for example, '''conda-forge''', '''r''', '''defaults''', and '''qiime2''', etc.. Please refer to [https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html# Conda channels]

If the busco version 3.0.2 is not specified, the most recent version of busco will be downloaded and installed, i.e.,

<pre class="gcommand">
module load Miniconda3/4.7.10
conda create -p /home/zhuofei/busco_conda -c bioconda busco
</pre>

To list conda environments you installed:

<pre class="gcommand">
module load Miniconda3/4.7.10
conda env list
</pre>

To activate your conda environment, you need to give your environment path to '''source activate''' command:

<pre class="gcommand">
module load Miniconda3/4.7.10
source activate /home/MyID/busco_conda/
</pre>

Once a conda environment is activated, you can list packages installed inside, with their names, versions, and downloading channels. For example, to list packages installed in busco v3.0.2 conda environment:

<pre class="gcommand">
module load Miniconda3/4.7.10
source activate /home/MyID/busco_conda/
conda list
</pre>

You can add or install new packages to an existing environment. For example, to install scipy v1.5.0 package to busco v3.0.2 conda environment:

<pre class="gcommand">
module load Miniconda3/4.7.10
source activate /home/MyID/busco_conda/
conda install scipy=1.5.0
</pre>

Likewise, if the scipy version 1.5.0 is not specified, the most recent version of scipy will be downloaded and installed for you.

To remove an installed package, e.g., the scipy package, from your conda environment:

<pre class="gcommand">
module load Miniconda3/4.7.10
source activate /home/MyID/busco_conda/
conda remove scipy
</pre>

To deactivate a conda environment:

<pre class="gcommand">
module load Miniconda3/4.7.10
conda deactivate /home/MyID/busco_conda/
</pre>

Once a conda environment is deactivated, its conda layer will be removed from your Linux shell environment.

----
[[#top|Back to Top]]

===How to install R packages===

If you wish to install an R package in your home directory, first decide the directory where you will install it. For example, to install it in ~/Rlibs, first create this dir if it does not exist yet:

<pre class="gcommand">
mkdir ~/Rlibs
</pre>

Then, create a file called '''~/.Renviron''' containing the following line:
<pre class="gscript">
R_LIBS_USER=/path/to/Rlibs
</pre>
replacing /path/to/Rlibs with the path that you want to use. For example, to /home/MyID/Rlibs, where MyID needs to be replaced by your UGA MyID.

Start an interactive session and load the module for the version of R you want to use (e.g. R/4.0.0-foss-2019b)
<pre class="gcommand">
interact
module load R/4.0.0-foss-2019b
</pre>

====Install R package using R command line====
If you download the R package tarball (e.g. brocolors_0.1.tar.gz) to your home dir, you can install it with R CMD INSTALL at the command line, but use the flag '''--library=/path/to/Rlibs''', as follows:

<pre class="gcommand">
R CMD INSTALL --library=/path/to/Rlibs brocolors_0.1.tar.gz
</pre>

Note that the path for --library flag should be the same path contained in '''~/.Renviron''' as described above.

====Install R package in an interactive R session====

You can install a package in an interactive R session using install.packages(). To use this method, you need to create the '''~/.Renviron''' file as described above. For example, to open an interactive R session to install ggplot2 package into the path given in ~/.Renviron:

<pre class="gcommand">
$ R
R version 4.0.0 (2020-04-24) -- "Arbor Day"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
......
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> install.packages('ggplot2')
</pre>

You will be prompted to enter your selection of a CRAN mirror site for downloading package source. Usually we select #75: USA (TX 1) [https]. If one site does not work well for you, you could try others.

----
[[#top|Back to Top]]

===How to install Java applications===

Most third-party Java applications are distributed as pre-compiled binaries (jar file), which users can download into their own home directories. For example, to install picard 2.4.1 in your home directory.

1. Create a directory where you want to install the java application (e.g. picard). For example
<pre class="gcommand">
mkdir -p ~apps/picard
</pre>

2. Download the package (e.g. picard-tools-2.4.1.zip) from their website, put into your chosen directory (e.g. ~apps/picard) and extract the file. For example
<pre class="gcommand">
cd ~apps/picard

unzip picard-tools-2.4.1.zip
</pre>

This will create a directory called picard-tools-2.4.1 that contains picard.jar. You can rename this dir e.g.
<pre class="gcommand">
cd ~apps/picard

mv picard-tools-2.4.1 2.4.1
</pre>

3. To run this application, load a java module and invoke this application in your job submission script. For example
<pre class="gcommand">
module load Java/1.8.0_144

java -Xmx20g -classpath "/home/MyID/apps/picard/2.4.1" -jar /home/MyID/apps/picard/2.4.1/picard.jar [options]
</pre>

where MyID needs to be replaced by your own MyID.

----
[[#top|Back to Top]]

===How to build complex applications===

Many applications use a ''configure'' step to check on system libraries and path to dependencies in order to create Makefiles. The Makefiles are then used to build and install the application. To illustrate how this process is typically set up, we will describe how to install the GNU Scientific Libraries (GSL) v2.6 using the GNU 8.3.0 compilers.

1. Create a directory to use for building the application, e.g. ~/src/gsl
<pre class="gcommand>
mkdir -p ~/src/gsl
</pre>

2. Download the source tarball into your chosen directory, e.g.
<pre class="gcommand>
cd ~/src/gsl

wget http://mirror.sbb.rs/gnu/gsl/gsl-2.6.tar.gz
</pre>

3. Start an interactive session
<pre class="gcommand">
interact
</pre>

4. Load the module for the compiler suite that you want to use, e.g. GCC 8.3.0:
<pre class="gcommand">
module load foss/2019b
</pre>

5. Change into your working directory and check the tarball with
<pre class="gcommand">
cd ~/src/gsl

tar ztvf gsl-2.6.tar.gz
</pre>
and extract the files with
<pre class="gcommand">
tar zxvf gsl-2.6.tar.gz
</pre>

6. Change into the extracted directory
<pre class="gcommand">
cd gsl-2.6
</pre>

7. Create a subdirectory to build the application
<pre class="gcommand">
mkdir build_gcc830

cd build_gcc830
</pre>

8. Check all the configure options with
<pre class="gcommand">
../configure --help
</pre>

9. Configure the application with e.g.
<pre class="gcommand">
../configure --prefix=/home/MyID/apps/gsl/2.6/gcc830
</pre>

where /home/MyID/apps/gsl/2.6/gcc830 should be replaced by the directory where you want to install GSL.

You can capture the standard error and standard output of the configure step into a file, to help troubleshoot the step if it encounters any issues. This can be done with
<pre class="gcommand">
../configure --prefix=/home/MyID/apps/gsl/2.6/gcc830 2>&1 | tee my.config.log
</pre>

10. If the configure step worked, you can build the application with
<pre class="gcommand">
make 2>&1 | tee my.make.log
</pre>

11. Some applications, including GSL, provide some tests that can be run after the build step to ensure the application was built correctly. For example, for GSL you can run
<pre class="gcommand">
make check 2>&1 | tee my.check.log
</pre>

12. Install the application with
<pre class="gcommand">
make install 2>&1 | tee my.install.log
</pre>

----
[[#top|Back to Top]]

===How to link dependencies===
Some applications need third party program, library, header files to build. There are various ways to introduce these to the build command.

Usually the manual/README of the application would elaborate more about the needed configuration variables. To explore these configuration variables from command is to issue --help options following the main build command.

Such as cmake build,
<pre class="gcommand">
cmake --help
</pre>

Or if it is a configuration build,
<pre class="gcommand">
./configure -h
</pre>

To set needed components in build environment, this could simply be done by loading modules prior to build.


<pre class="gcommand">
module load CMake/3.15.3-GCCcore-8.3.0 zlib/1.2.11-GCCcore-8.3.0
cmake -DCMAKE_INSTALL_PREFIX:PATH=/home/MyID/app/diamond/1.0 ...[skipped]
make install
</pre>

Line 1: load CMake module and zlib module.
Line 2: set prefix, there are other variable could be defined here.

Or set the variables explicitly by export command

<pre class="gcommand">
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/apps/eb/GSL/2.6-GCC-8.3.0/lib
export CFLAGS="-I/apps/eb/GSL/2.6-GCC-8.3.0/include"
export LDFLAGS="-L/apps/eb/GSL/2.6-GCC-8.3.0/lib"
export LIBS="-lgsl"
python setup.py build_ext --inplace
</pre>
Line 1: specify the location of gsl shared dynamic lib, pass this to shared library loader at runtime

Line 2: specify the location of gsl c header file, pass this to GNU C compiler at compilation time

Line 3 and 4: specify the location of gsl shared dynamic lib and the lib, pass this to GNU linker at linkage time

Another way is to define variables at configuration line.

<pre class="gcommand">
./configure --prefix=/home/MyID/app/diamond/1.0 --with-jemalloc=/apps/eb/jemalloc/5.2.1-GCCcore-8.3.0/lib
</pre>

----
[[#top|Back to Top]]

===How to download Singularity images===
Singularity image can be searched and downloaded from [https://cloud.sylabs.io/library Singularity Container Library]. For example, to pull Trinity v2.9.1 Singularity image built for amd64 (default) architecture:

<pre class="gcommand">
singularity pull --arch amd64 library://colinsauze/default/trinity:v2.9.1
</pre>

[https://www.docker.com/ Docker container] is the most well-known container system. Docker also has a bigger ecosystem than Singularity. However, Docker was initially designed for ephemeral servers; by default Docker tries to isolate the running container as much as possible, which makes it not suitable for running in a HPC environment. Like Docker, Singularity is a container runtime too. But it starts from a very different place. It favors integration rather than isolation. Singularity is also the best friend of Docker and can import images from Docker registries. You can search a docker image at [https://hub.docker.com/ Docker Hub]. If the image you need is on Docker Hub, you can pull and build it into a Singularity image.

For example, to pull and build Trinity latest version Singularity image from Docker Hub:

<pre class="gcommand">
singularity pull docker://trinityrnaseq/trinityrnaseq:latest
</pre>

Detailed instructions on how to build Singularity container can be found at [https://sylabs.io/guides/3.6/user-guide/build_a_container.html# Singularity Build a Container]

----
[[#top|Back to Top]]