CryoSPARC-Sapelo2: Difference between revisions
No edit summary |
No edit summary |
||
| Line 20: | Line 20: | ||
"CryoSPARC (Cryo-EM Single Particle Ab-Initio Reconstruction and Classification) is a state of the art HPC software solution for complete processing of single-particle cryo-electron microscopy (cryo-EM) data. CryoSPARC is useful for solving cryo-EM structures of membrane proteins, viruses, complexes, flexible molecules, small particles, phase plate data and negative stain data." For more information, please see https://guide.cryosparc.com/. | "CryoSPARC (Cryo-EM Single Particle Ab-Initio Reconstruction and Classification) is a state of the art HPC software solution for complete processing of single-particle cryo-electron microscopy (cryo-EM) data. CryoSPARC is useful for solving cryo-EM structures of membrane proteins, viruses, complexes, flexible molecules, small particles, phase plate data and negative stain data." For more information, please see https://guide.cryosparc.com/. | ||
'''NOTE''': Users are required to be added into GACRC '''cryosparc''' group before | '''NOTE''': Users are required to be added into GACRC '''cryosparc''' group before they can run this software from Sapelo2. Please fill out the [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25844 GACRC General Support form] to request. We will reach out to you after we received your request. | ||
=== Configurations === | === Configurations === | ||
| Line 38: | Line 38: | ||
* gacrc-cryo is the '''service account''' that will run cryoSPARC workflow jobs for all cryoSPARC users. | * gacrc-cryo is the '''service account''' that will run cryoSPARC workflow jobs for all cryoSPARC users. | ||
* Some tasks can only be handled by gacrc-cryo, like start or stop cryosparcm from the master node, user management, connect or update worker nodes to master, etc.. | * Some tasks can only be handled by gacrc-cryo, like start or stop cryosparcm from the master node, user management, connect or update worker nodes to master, etc.. | ||
* Regular | * Regular CryoSPARC users can run cryosparcm from the master node to check cryoSPARC status, using cryosparcm status and cryosparcm checkdb. | ||
'''cryoSPARC project space:''' '''/scratch/gacrc-cryo''' | '''cryoSPARC project space:''' '''/scratch/gacrc-cryo''' | ||
| Line 55: | Line 55: | ||
Once you established the ssh tunnel by running the above command, you can open a browser (Chrome) on the local machine and navigate to<code><nowiki>http://localhost:39000</nowiki></code>. The cryoSPARC User Interface should be presented with the cryoSPARC login page. | Once you established the ssh tunnel by running the above command, you can open a browser (Chrome) on the local machine and navigate to<code><nowiki>http://localhost:39000</nowiki></code>. The cryoSPARC User Interface should be presented with the cryoSPARC login page. | ||
===== Run cryoSPARC workflow jobs ===== | ===== Run cryoSPARC workflow jobs ===== | ||
| Line 72: | Line 70: | ||
* In cryoSPARC, queue a job to "'''Lane Sapelo2 (cluster)'''"; The job will be dispatched to a worker node via Slurm. Please note, gacrc-cryo, instead of your own Sapelo2 user name, is the user name running the job. We highly recommend you to use this method to run cryoSPARC workflow jobs on Sapelo2. | * In cryoSPARC, queue a job to "'''Lane Sapelo2 (cluster)'''"; The job will be dispatched to a worker node via Slurm. Please note, gacrc-cryo, instead of your own Sapelo2 user name, is the user name running the job. We highly recommend you to use this method to run cryoSPARC workflow jobs on Sapelo2. | ||
* cryoSPARC will decide on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. Currently, we | * cryoSPARC will decide on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. Currently, we configured that from each worker node cryoSPARC can use up to '''20GB''' memory and '''4''' CPU cores. | ||
* If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices , for example 1 or 4. You can change this number by yourself. Please note that the maximum number of GPU devices on | * If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices , for example 1 or 4. You can change this number by yourself in cryoSPARC GUI when you create the job. Please note that the maximum number of GPU devices installed on worker node is '''8'''. | ||
====== Run job using "Lane default (node)" ====== | ====== Run job using "Lane default (node)" ====== | ||
You can select and use a worker node to run cryoSPARC workflow jobs. This method is | You can select and use a worker node to run cryoSPARC workflow jobs. This method is convenient, for example, when you want to debug your cryoSPARC workflow. However, if you want to use this method, you are required to reserve a whole worker node at first via Sapelo2 queueing system. '''Please follow the following steps carefully:''' | ||
'''Please follow the following steps carefully:''' | |||
# From the login node, open an interactive session to reserve a | # From the login node, use interact command to open an interactive session to reserve a worker node via Slurm '''interact -p gpu_p --gres gpu:K40:8 -c 16''' | ||
# Once an interactive session is opened on a worker node, please run this command to know its short hostname: '''hostname -s''' | # Once an interactive session is opened on a worker node, please run this command to know its short hostname: '''hostname -s''' | ||
# In cryoSPARC, firstly queue a job to "'''Lane default (node)'''" , then click "'''Run on Specific GPU'''" to select the worker node that you reserved in step 1 (rb6-3 or rb6-4, the hostname you get in step 2). '''''Please do this step carefully and do NOT select and use the other worker node which you didn't reserve in step 1. If you selected and used a wrong node which you didn't reserve in step 1, the node could be crashed and all jobs running on the node will be lost.''''' | # In cryoSPARC, firstly queue a job to "'''Lane default (node)'''" , then click "'''Run on Specific GPU'''" to select the worker node that you reserved in step 1 (rb6-3 or rb6-4, the hostname you get in step 2). '''''Please do this step carefully and do NOT select and use the other worker node which you didn't reserve in step 1. If you selected and used a wrong node which you didn't reserve in step 1, the node could be crashed and all jobs running on the node will be lost.''''' | ||
Revision as of 09:09, 16 February 2022
ategory
Engineering
Program On
Sapelo2
Version
3.3.1
Author / Distributor
See https://guide.cryosparc.com/
Description
"CryoSPARC (Cryo-EM Single Particle Ab-Initio Reconstruction and Classification) is a state of the art HPC software solution for complete processing of single-particle cryo-electron microscopy (cryo-EM) data. CryoSPARC is useful for solving cryo-EM structures of membrane proteins, viruses, complexes, flexible molecules, small particles, phase plate data and negative stain data." For more information, please see https://guide.cryosparc.com/.
NOTE: Users are required to be added into GACRC cryosparc group before they can run this software from Sapelo2. Please fill out the GACRC General Support form to request. We will reach out to you after we received your request.
Configurations
Master node VM:
- Host name: ss-cryo.gacrc.uga.edu
- Intel Xeon processors (8 cores) and 24GB of RAM
- mongodb is installed and run from the master node
Worker nodes:
- Two NVIDIA Tesla K40m nodes, Intel Xeon processors (16 cores and 128GB of RAM) and 8 NVIDIA K40m GPU cards per node.
- cryoSPARC recommends using SSD for caching particle data. /lscratch/gacrc-cryo is set up on worker nodes for this purpose.
- The amount of space that cryoSPARC can use in /lscratch/gacrc-cryo is capped at 100GB.
cryoSPARC group: cryosparc
cryoSPARC service account: gacrc-cryo
- gacrc-cryo is the service account that will run cryoSPARC workflow jobs for all cryoSPARC users.
- Some tasks can only be handled by gacrc-cryo, like start or stop cryosparcm from the master node, user management, connect or update worker nodes to master, etc..
- Regular CryoSPARC users can run cryosparcm from the master node to check cryoSPARC status, using cryosparcm status and cryosparcm checkdb.
cryoSPARC project space: /scratch/gacrc-cryo
How to run cryoSPARC from Sapelo2
User login
User needs to establish a SSH tunnel to expose the port 39000 from the master node to a local computer.
If you are using a Linux or Apple desktop or laptop, you can use the following command in Terminal to establish the ssh tunnel:
ssh -N -L 39000:128.192.75.59:39000 username@ss-cryo.gacrc.uga.edu
If you are using a Windows desktop or laptop, please download the plink program to use in place of the ssh client:
plink -ssh -N -L 39000:128.192.75.59:39000 username@ss-cryo.gacrc.uga.edu
Note: Please put the plink.exe in the current directory where you have a command window open.
Unless you have SSH public key configured, you will be prompted for your MyID password and for Archpass Duo authentication. Once authentication is established, this session prompt will hang and you are ready to go to access the cryoSPARC User Interface.
Once you established the ssh tunnel by running the above command, you can open a browser (Chrome) on the local machine and navigate tohttp://localhost:39000. The cryoSPARC User Interface should be presented with the cryoSPARC login page.
Run cryoSPARC workflow jobs
Project space selection
A project in cryoSPARC is a high level container corresponding with a project directory on the file system, which stores all associated Jobs of a project. Each project in cryoSPARC is entirely contained within a file system directory. All the jobs and their respective intermediate and output data created within a project will be stored within the project directory.
- When you start a new project in cryoSPARC GUI, please select and use /scratch/gacrc-cryo as the cryoSPARC project space.
- This folder is owned by gacrc-cryo. Regular cryoSPARC users have access and read permissions which allow them to browse files in this folder and copy files from this folder to their own storage spaces on Sapelo2.
Run job on the master node
cryoSPARC will use the master node to run some types of workflow jobs, like "Import Movies", "Inspect Picks", and the interactive job "Select 2D Classes", etc.. It will notify you that the job will be running on the master node when the job is created.
Run job using "Lane Sapelo2 (cluster)" (highly recommended)
- In cryoSPARC, queue a job to "Lane Sapelo2 (cluster)"; The job will be dispatched to a worker node via Slurm. Please note, gacrc-cryo, instead of your own Sapelo2 user name, is the user name running the job. We highly recommend you to use this method to run cryoSPARC workflow jobs on Sapelo2.
- cryoSPARC will decide on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size. Currently, we configured that from each worker node cryoSPARC can use up to 20GB memory and 4 CPU cores.
- If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices , for example 1 or 4. You can change this number by yourself in cryoSPARC GUI when you create the job. Please note that the maximum number of GPU devices installed on worker node is 8.
Run job using "Lane default (node)"
You can select and use a worker node to run cryoSPARC workflow jobs. This method is convenient, for example, when you want to debug your cryoSPARC workflow. However, if you want to use this method, you are required to reserve a whole worker node at first via Sapelo2 queueing system. Please follow the following steps carefully:
- From the login node, use interact command to open an interactive session to reserve a worker node via Slurm interact -p gpu_p --gres gpu:K40:8 -c 16
- Once an interactive session is opened on a worker node, please run this command to know its short hostname: hostname -s
- In cryoSPARC, firstly queue a job to "Lane default (node)" , then click "Run on Specific GPU" to select the worker node that you reserved in step 1 (rb6-3 or rb6-4, the hostname you get in step 2). Please do this step carefully and do NOT select and use the other worker node which you didn't reserve in step 1. If you selected and used a wrong node which you didn't reserve in step 1, the node could be crashed and all jobs running on the node will be lost.
- cryoSPARC will decide on how many CPU cores and how much memory it will use to run a workflow job, depend on the type of the job and your data size.
- If the job needs to run on GPU devices, cryoSPARC will queue the job with a default number of GPU devices for you, for example 1 or 4. You can change this number by yourself up to 8.
Documentation
About cryoSPARC: https://guide.cryosparc.com/
User Interface and Usage Guide: https://guide.cryosparc.com/processing-data/user-interface-and-usage-guide
Accessing the cryoSPARC User Interface https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc
All Job Types in cryoSPARC: https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc
Management and Monitoring: https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring
Cluster (Slurm) integration: https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connect-a-cluster-to-cryosparc
Introductory Tutorial: https://guide.cryosparc.com/processing-data/cryo-em-data-processing-in-cryosparc-introductory-tutorial
Tutorials and Usage Guides: https://guide.cryosparc.com/processing-data/tutorials-and-case-studies
Installation
- Version 3.3.1 master is installed on the master node (ss-cryo.gacrc.uga.edu). Source codes are downloaded in /work/cryosparc/cryosparc_master on the master node.
- Version 3.3.1 workers are installed on two worker nodes (NVIDIA Tesla K40m GPU nodes rb6-[3-4]). Source codes are downloaded in /work/cryosparc/cryosparc_worker on the master ndoe.
System
64-bit Linux