CryoSPARC-Sapelo2: Difference between revisions

From Research Computing Center Wiki
Jump to navigation Jump to search
No edit summary
 
(47 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Category:Sapelo2]][[Category:Software]][[Category:Engineering]]
[[Category:Sapelo2]][[Category:Software]][[Category:Engineering]]
 
=== Category ===
=== Category ===


Line 10: Line 11:
=== Version ===
=== Version ===
   
   
3.3.1
4.3.1


=== Author / Distributor ===
=== Author / Distributor ===
Line 20: Line 21:
"CryoSPARC (Cryo-EM Single Particle Ab-Initio Reconstruction and Classification) is a state of the art HPC software solution for complete processing of single-particle cryo-electron microscopy (cryo-EM) data. CryoSPARC is useful for solving cryo-EM structures of membrane proteins, viruses, complexes, flexible molecules, small particles, phase plate data and negative stain data." For more information, please see https://guide.cryosparc.com/.
"CryoSPARC (Cryo-EM Single Particle Ab-Initio Reconstruction and Classification) is a state of the art HPC software solution for complete processing of single-particle cryo-electron microscopy (cryo-EM) data. CryoSPARC is useful for solving cryo-EM structures of membrane proteins, viruses, complexes, flexible molecules, small particles, phase plate data and negative stain data." For more information, please see https://guide.cryosparc.com/.


'''NOTE''': Users are required to be added into GACRC '''cryosparc''' group before being allowed to run this software. Please fill out the [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25844 GACRC General Support form] to request. We will reach out to you once we received your request.
'''NOTE''': Users are required to be added into GACRC '''cryosparc''' group before they can run this software from Sapelo2. Please fill out the [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25844 GACRC General Support form] to request. We will reach out to you after we received your request.


=== Configurations ===
=== Configurations ===
'''Master node VM:'''
'''Master node VM:'''


* Host name: '''ss-cryo.gacrc.uga.edu'''
* Host name: '''cryosparc.gacrc.uga.edu'''
* Intel Xeon processors (8 cores) and 24GB of RAM
* Intel Xeon processors (16 cores) and 64GB of RAM
* mongodb is installed.
* mongodb is installed and run on the master node
'''Worker nodes:'''
'''Worker nodes:'''
* Two NVIDIA Tesla K40m nodes, Intel Xeon processors (16 cores and 128GB of RAM) and 8 NVIDIA K40m GPU cards per node.
* One NVIDIA Tesla A100 node: Intel Xeon processors (64 cores), 1TB host RAM memory, 4 NVIDIA Tesla A100 GPU cards (80GiB device memory per card), and NVMe SSD 3570GB local drive.
* cryoSPARC recommends using SSD or caching particle data. /lscratch/gacrc-cryo is set up on worker nodes for this purpose.
* cryoSPARC recommends using SSD for caching particle data. /lscratch/gacrc-cryo is set up on the worker node for this purpose.
* The amount of space that cryoSPARC can use in /lscratch/gacrc-cryo is capped at 100GB.
* The amount of space that cryoSPARC can use in /lscratch/gacrc-cryo is capped at 100GB.
'''cryoSPARC group:''' cryosparc
'''cryoSPARC group:''' '''cryosparc'''


'''cryoSPARC service account:''' gacrc-cryo  
'''cryoSPARC service account:''' '''gacrc-cryo'''


* gacrc-cryo is the service user account that will run the cryoSPARC workflow jobs for all regular cryoSPARC users on the master node and each worker node that will be used for computation.
* gacrc-cryo is the '''service account''' that will run cryoSPARC workflow jobs for all cryoSPARC users.
* Some tasks can only be performed by gacrc-cryo, like start or stop cryosparcm from the master node, user management, connect or update worker nodes to master, etc..
* Some tasks can only be handled by gacrc-cryo, like make a new lane of worker node(s), user management, and connect or update worker node(s) to master, etc..
* Regular cryoSPARC users can still run cryosparcm on the master node to check status of the master and its database, using cryosparcm status and cryosparcm checkdb.
* Regular CryoSPARC users can run cryosparcm on the master node to check cryoSPARC status, using cryosparcm status or cryosparcm checkdb.
 
'''cryoSPARC group space:''' /work/cryosparc/, with a per group quota 500GB and a maximum of 100,000 files.
 
There are 6 sub folders in /work/cryosparc/:
 
* cryosparc_master/ , cryosparc_worker/ : Master and worker installation folders
* database/ : cryoSPARC database folder
* users/ : cryoSPARC user project folder
* cryosparc_cluster/ : The folder storing cluster integration scripts
* testdataset/ : The folder storing cryoSPARC test data
* src_v3.3.1/ : The folder storing cryoSPARC sources v3.3.1


'''cryoSPARC project space:''' '''/scratch/gacrc-cryo'''
#  
#  


=== Running cryoSPARC from Sapelo2 ===
=== How to run cryoSPARC from Sapelo2 ===


===== User login =====
===== User login =====
Line 59: Line 50:
User needs to establish a SSH tunnel to expose the port 39000 from the master node to a local computer.   
User needs to establish a SSH tunnel to expose the port 39000 from the master node to a local computer.   


If you are using a Linux or Apple desktop or laptop, you can use the following command in Terminal to establish the ssh tunnel: <blockquote>'''ssh -N -L 39000:128.192.75.59:39000 username@ss-cryo.gacrc.uga.edu'''</blockquote>If you are using a Windows desktop or laptop, please download the '''[https://the.earth.li/~sgtatham/putty/latest/x86/plink.exe plink program]''' to use in place of the ssh client:<blockquote>'''plink -ssh -N -L 39000:128.192.75.59:39000 username@ss-cryo.gacrc.uga.edu'''</blockquote>
If you are using a Linux or Apple desktop or laptop, you can use the following command in Terminal to establish the ssh tunnel: <blockquote>'''ssh -N -L 39000:10.2.0.60:39000 username@cryosparc.gacrc.uga.edu'''</blockquote>If you are using a Windows desktop or laptop, please download the '''[https://the.earth.li/~sgtatham/putty/latest/x86/plink.exe plink program]''' to use in place of the ssh client:<blockquote>'''plink -ssh -N -L 39000:10.2.0.60:39000 username@cryosparc.gacrc.uga.edu'''</blockquote>
'''Note:''' Please put the plink.exe in the current directory where you have a command window open.
'''Note:''' Please put the plink.exe in the current directory where you have a command window open.


Line 66: Line 57:
Once you established the ssh tunnel by running the above command, you can open a browser (Chrome) on the local machine and navigate to<code><nowiki>http://localhost:39000</nowiki></code>. The cryoSPARC User Interface should be presented with the cryoSPARC login page.
Once you established the ssh tunnel by running the above command, you can open a browser (Chrome) on the local machine and navigate to<code><nowiki>http://localhost:39000</nowiki></code>. The cryoSPARC User Interface should be presented with the cryoSPARC login page.


Please refer to https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc
===== Run cryoSPARC workflow jobs =====
 
===== How to run cryoSPARC workflow jobs =====


====== Project space selection ======
====== Project space selection ======
Project in cryoSPARC is a high level container corresponding with a project directory on the file system, which stores all associated Jobs of a project. Each project in cryoSPARC is entirely contained within a file system directory. All the jobs and their respective intermediate and output data created within a project will be stored within the project directory
A project in cryoSPARC is a high level container corresponding with a project directory on the file system, which stores all associated Jobs of a project. Each project in cryoSPARC is entirely contained within a file system directory. All the jobs and their respective intermediate and output data created within a project will be stored within the project directory.
 
* In cryoSPARC group space /work/cryosparc/, for each '''cryoSPARC''' user, a default user project folder is created at /work/cryosparc/users/<username>. '''Please note''', this folder is not suitable for running a project with large data, since /work/cryosparc has a per group quota 500GB and a maximum limit of 100,000 files. If you want to process large EM data, we highly recommend that you use your own scratch space for running the project.
* You can use the scratch space to run a project with large data '''(recommended)'''. Steps to set up a project folder, for example '''cryo_project/''', in your scratch space are shown below:
*# cd /scratch/username
*# mkdir ./cryo_project
*# chgrp cryosparc ./cryo_project
*# chmod g+rwx ./cryo_project
*# chmod o+rx /scratch/username


* gacrc-cryo is the service user account to launch and run jobs for all regular cryoSPARC users. The above steps 3, 4, and 5 will enable gacrc-cryo to write/read into/from /scratch/username/cryo_project. Once a project is completed, we suggest that you turn off the rx permission on your scratch folder by '''chmod o-rx /scratch/username'''
* When you start a new project in cryoSPARC GUI, please select and use '''/scratch/gacrc-cryo''' as the cryoSPARC project space.
* When you start a new project in cryoSPARC, please select the appropriate project space.
* This folder is owned by gacrc-cryo. Regular cryoSPARC users have access and read permissions which allow them to browse files in this folder and copy files from this folder to their own storage spaces on Sapelo2.


====== Launch job on the mater node ======
====== Run job on the master node ======
cryoSPARC will decide on its own to run some types of workflow jobs on the master node, like "Import Movies", "Inspect Picks", and the interactive job "Select 2D Classes". It will notify you that the job will be running on the master node when job is created.  
cryoSPARC will use the master node to run some types of workflow jobs, for example, "Import Movies", "Inspect Picks", and the interactive job "Select 2D Classes". When a job is created, if cryoSPARC will use the master node to run the job, you will be notified about this in cryoSPARC GUI.  


====== Launch job via Slurm (highly recommended) ======
====== Run job using "Lane Sapelo2 Default (cluster)" ======


* In cryoSPARC, queue a job to "'''Lane Sapelo2 (cluster)'''"; The job will be dispatched to a worker node via Sapelo2 Slurm queueing system and will run on the node using gacrc-cryo account. We highly recommend you to use "Lane Sapelo2 (cluster)" to run cryoSPARC workflow jobs on Sapelo2.
* In cryoSPARC, queue a job to "'''Lane Sapelo2 Default (cluster)'''"; The job will be dispatched to the worker node via Slurm. Please note, gacrc-cryo, instead of your own Sapelo2 user account, is the user account owning and running the job. We highly recommend you to use this method to run cryoSPARC workflow jobs on Sapelo2.
* cryoSPARC will decide on its own on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. At this moment, we give a fixed amount of 64GB memory on each worker node for cryoSPARC to use.
* cryoSPARC will decide on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. Currently, we configured that from each worker node cryoSPARC can use up to '''20GB''' memory and '''4''' CPU cores.
* If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices for you, for example 1 or 4. You can change this number by yourself. Please note that the maximum number of GPU devices on each worker node is 8.
* If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices , for example 1. You can change this number by yourself in cryoSPARC GUI when you create the job. Please note that the maximum number of GPU devices installed on the worker node is '''4'''.


====== Launch and run job on a worker node ======
===Documentation (v4.0+)===
You can run cryoSPARC workflow jobs on a specific worker node. '''You need to follow steps given below carefully''':
 
# From the login node, open an interactive session to reserve a whole worker node via Slurm, for example, '''interact -p gpu_p --gres gpu:K40:8 -c 16 . ''Please request a whole node with the options "--gres gpu:K40:8" and "-c 16".'''''
# Once an interactive session is opened on a worker node, please run this command to know its short hostname: '''hostname -s'''
# In cryoSPARC, firstly queue a job to "'''Lane default (node)'''" , then click "'''Run on Specific GPU'''" to select the worker node that you reserved in step 1 (rb6-3 or rb6-4, whose name you got from step 2). '''''Please perform this step very carefully and do not select the other worker node that is not reserved in step 1!'''''
# cryoSPARC will decide on its own on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. At this moment, we give a fixed amount of 64GB memory on each worker node for cryoSPARC to use.
# If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices for you, for example 1 or 4. You can change this number by yourself up to 8.
 
===Documentation===
   
   
About cryoSPARC: https://guide.cryosparc.com/
About cryoSPARC: https://guide.cryosparc.com/


User Interface and Usage Guide: https://guide.cryosparc.com/processing-data/user-interface-and-usage-guide
Get Started with CryoSPARC: Introductory Tutorial: https://guide.cryosparc.com/processing-data/get-started-with-cryosparc-introductory-tutorial
 
Accessing the cryoSPARC User Interface https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc
 
All Job Types in cryoSPARC: https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc


Management and Monitoring: https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring
A Tour of the CryoSPARC Interface: https://guide.cryosparc.com/application-guide-v4.0+/a-tour-of-the-cryosparc-interface


Cluster (Slurm) integration: <nowiki>https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connect-a-cluster-to-cryosparc</nowiki>
Using the CryoSPARC Interface: https://guide.cryosparc.com/application-guide-v4.0+/using-the-cryosparc-interface


Introductory Tutorial: https://guide.cryosparc.com/processing-data/cryo-em-data-processing-in-cryosparc-introductory-tutorial
Creating and Running Jobs: https://guide.cryosparc.com/application-guide-v4.0+/creating-and-running-jobs


Tutorials and Usage Guides: https://guide.cryosparc.com/processing-data/tutorials-and-case-studies
Tutorial videos: https://guide.cryosparc.com/processing-data/tutorial-videos


===Installation===
===Installation===
   
   
*Version 3.3.1 master is installed on the master node (ss-cryo.gacrc.uga.edu). Source codes are downloaded in /work/cryosparc/cryosparc_master on the master node.
*Version 4.3.1 master is installed on the master node (cryosparc.gacrc.uga.edu).  
*Version 3.3.1 workers are installed on two worker nodes (NVIDIA Tesla K40m GPU nodes rb6-[3-4]). Source codes are downloaded in /work/cryosparc/cryosparc_worker on the master ndoe.
*Version 4.3.1 workers are installed on one worker GPU node (NVIDIA Tesla A100 GPU node).


===System===
===System===
64-bit Linux
64-bit Linux

Latest revision as of 13:00, 12 October 2023


Category

Engineering

Program On

Sapelo2

Version

4.3.1

Author / Distributor

See https://guide.cryosparc.com/

Description

"CryoSPARC (Cryo-EM Single Particle Ab-Initio Reconstruction and Classification) is a state of the art HPC software solution for complete processing of single-particle cryo-electron microscopy (cryo-EM) data. CryoSPARC is useful for solving cryo-EM structures of membrane proteins, viruses, complexes, flexible molecules, small particles, phase plate data and negative stain data." For more information, please see https://guide.cryosparc.com/.

NOTE: Users are required to be added into GACRC cryosparc group before they can run this software from Sapelo2. Please fill out the GACRC General Support form to request. We will reach out to you after we received your request.

Configurations

Master node VM:

  • Host name: cryosparc.gacrc.uga.edu
  • Intel Xeon processors (16 cores) and 64GB of RAM
  • mongodb is installed and run on the master node

Worker nodes:

  • One NVIDIA Tesla A100 node: Intel Xeon processors (64 cores), 1TB host RAM memory, 4 NVIDIA Tesla A100 GPU cards (80GiB device memory per card), and NVMe SSD 3570GB local drive.
  • cryoSPARC recommends using SSD for caching particle data. /lscratch/gacrc-cryo is set up on the worker node for this purpose.
  • The amount of space that cryoSPARC can use in /lscratch/gacrc-cryo is capped at 100GB.

cryoSPARC group: cryosparc

cryoSPARC service account: gacrc-cryo

  • gacrc-cryo is the service account that will run cryoSPARC workflow jobs for all cryoSPARC users.
  • Some tasks can only be handled by gacrc-cryo, like make a new lane of worker node(s), user management, and connect or update worker node(s) to master, etc..
  • Regular CryoSPARC users can run cryosparcm on the master node to check cryoSPARC status, using cryosparcm status or cryosparcm checkdb.

cryoSPARC project space: /scratch/gacrc-cryo

How to run cryoSPARC from Sapelo2

User login

User needs to establish a SSH tunnel to expose the port 39000 from the master node to a local computer.

If you are using a Linux or Apple desktop or laptop, you can use the following command in Terminal to establish the ssh tunnel:

ssh -N -L 39000:10.2.0.60:39000 username@cryosparc.gacrc.uga.edu

If you are using a Windows desktop or laptop, please download the plink program to use in place of the ssh client:

plink -ssh -N -L 39000:10.2.0.60:39000 username@cryosparc.gacrc.uga.edu

Note: Please put the plink.exe in the current directory where you have a command window open.

Unless you have SSH public key configured, you will be prompted for your MyID password and for Archpass Duo authentication. Once authentication is established, this session prompt will hang and you are ready to go to access the cryoSPARC User Interface.

Once you established the ssh tunnel by running the above command, you can open a browser (Chrome) on the local machine and navigate tohttp://localhost:39000. The cryoSPARC User Interface should be presented with the cryoSPARC login page.

Run cryoSPARC workflow jobs
Project space selection

A project in cryoSPARC is a high level container corresponding with a project directory on the file system, which stores all associated Jobs of a project. Each project in cryoSPARC is entirely contained within a file system directory. All the jobs and their respective intermediate and output data created within a project will be stored within the project directory.

  • When you start a new project in cryoSPARC GUI, please select and use /scratch/gacrc-cryo as the cryoSPARC project space.
  • This folder is owned by gacrc-cryo. Regular cryoSPARC users have access and read permissions which allow them to browse files in this folder and copy files from this folder to their own storage spaces on Sapelo2.
Run job on the master node

cryoSPARC will use the master node to run some types of workflow jobs, for example, "Import Movies", "Inspect Picks", and the interactive job "Select 2D Classes". When a job is created, if cryoSPARC will use the master node to run the job, you will be notified about this in cryoSPARC GUI.

Run job using "Lane Sapelo2 Default (cluster)"
  • In cryoSPARC, queue a job to "Lane Sapelo2 Default (cluster)"; The job will be dispatched to the worker node via Slurm. Please note, gacrc-cryo, instead of your own Sapelo2 user account, is the user account owning and running the job. We highly recommend you to use this method to run cryoSPARC workflow jobs on Sapelo2.
  • cryoSPARC will decide on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. Currently, we configured that from each worker node cryoSPARC can use up to 20GB memory and 4 CPU cores.
  • If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices , for example 1. You can change this number by yourself in cryoSPARC GUI when you create the job. Please note that the maximum number of GPU devices installed on the worker node is 4.

Documentation (v4.0+)

About cryoSPARC: https://guide.cryosparc.com/

Get Started with CryoSPARC: Introductory Tutorial: https://guide.cryosparc.com/processing-data/get-started-with-cryosparc-introductory-tutorial

A Tour of the CryoSPARC Interface: https://guide.cryosparc.com/application-guide-v4.0+/a-tour-of-the-cryosparc-interface

Using the CryoSPARC Interface: https://guide.cryosparc.com/application-guide-v4.0+/using-the-cryosparc-interface

Creating and Running Jobs: https://guide.cryosparc.com/application-guide-v4.0+/creating-and-running-jobs

Tutorial videos: https://guide.cryosparc.com/processing-data/tutorial-videos

Installation

  • Version 4.3.1 master is installed on the master node (cryosparc.gacrc.uga.edu).
  • Version 4.3.1 workers are installed on one worker GPU node (NVIDIA Tesla A100 GPU node).

System

64-bit Linux

Retrieved from "https://wiki.gacrc.uga.edu/index.php?title=CryoSPARC-Sapelo2&oldid=21578"