Research Computing Center Wiki - User contributions [en]

Transferring Files

2021-07-13T22:26:08Z

Raj76:

[[Category:Sapelo2]][[Category:Teaching]]

===The File Transfer nodes (xfer.gacrc.uga.edu)===

Users can transfer files between their local machines and GACRC storage using various programs, such as (a) Filezilla, (b) WinSCP, and (c) a secure copy (scp). To transfer files, you must have a file transfer program installed on your local machine and a connection to the UGA campus network.

All Sapelo2 users, as well as all GACRC PIs, can access the xfer nodes using the hostname '''xfer.gacrc.uga.edu''' and using their UGA MyID (not the 810 or 811 number) and the MyID password to authenticate. Two-factor authentication using Archpass Duo is also necessary. For more details of Archpass Duo, please refer to
[https://eits.uga.edu/access_and_security/infosec/tools/archpass_duo archpass_duo].

The GACRC file transfer nodes (xfer) are configured to facilitate file transfer. These nodes, which have a hostname of '''xfer.gacrc.uga.edu''', are connected via a higher-bandwidth network connection. Hence, maximum transfer speed between a user's local machine and the GACRC systems can be achieved by transferring files to/from the host xfer.gacrc.uga.edu (instead of the login nodes of the clusters).

It is not necessary to connect to the UGA VPN when connecting to a file transfer node from off-campus. If you are transferring data from/to a server outside of campus (such as your local computer), you will get much faster transfer speeds if you are not connected to the VPN.

Note that a user's home directory on the xfer nodes is the same as the user's Sapelo2 home directory, which is '''not''' the same as the user's home directory on the teaching cluster.

The xfer node can access the following file systems using the full path:

1. Your home directory on an xfer node is the same as your Sapelo2 home directory, and the path is /home/username

2. The Sapelo2 scratch directory: /scratch/username

3. The Sapelo2 work directory: /work/groupname

4. The project file system for the lab: /project/groupname

===Transferring Files from your local storage device to Sapelo2 or vice-versa===

To transfer files from your local storage device to Sapelo2 or from Sapelo2 to your local storage device, please connect to xfer.gacrc.uga.edu (use your UGA MyID, the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.

====Using FileZilla====

In order to use [http://filezilla-project.org/ FileZilla] to transfer files between a local machine and Sapelo2, users will need to install the [http://filezilla-project.org/ FileZilla] software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the '''General''' tab select

<pre class="gcomment">

Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://xfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User: [your UGA MyID]
</pre>

In the '''Transfer Settings''' tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

Once the connection is established you can upload files from your local machine to Sapelo2 or download files from Sapelo2 to your local machine.

Here is the PDF to check step-by-step screen shots:
{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
| [[ media:Xfer Filezilla 12032018.pdf]]
|-
|}

====Using WinSCP====

In order to use [http://winscp.net/ WinSCP] to transfer files between a local Windows machine and Sapelo2, users will need to install the [http://winscp.net/ WinSCP] software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:

<pre class="gcomment">

Session:

File Protocol: SFTP

Host name: xfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

</pre>

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.


====Using scp====

To transfer files using scp you must have scp on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X).

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to Sapelo2:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
</pre>

where ''filename'' is the name of the file to be transferred, ''yourUGAMyID'' is your UGA MyID, and ''subdirectory'' is the subdirectory in your Sapelo2 home directory to which files are being transferred. You will then be asked to enter your UGA MyID password. You will also be prompted for two-factor authentication with Archpass Duo.

To transfer files from Sapelo2 to your local machine, use the full path to your file on Sapelo2 followed by the location you would like to transfer the files locally. The following example transfers a file called "filename" to the local current working directory.

<pre class="gcommand">
scp yourUGAMyID@xfer.gacrc.uga.edu:full/path/to/filename ./
</pre>

(Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:

<pre class="gcommand">
scp file1 file2 file3 yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
</pre>

Wildcards can be used for multiple file transfer, for example, to upload all files with .dat extension to your sapelo2 home directory, to a subdirectory called subdir:

<pre class="gcommand">
scp *.dat yourUGAMyID@xfer.gacrc.uga.edu:subdir
</pre>

An example that uses a wildcard to download all files with .dat extension in your Sapelo2 home dir, in a subdirectory called subdir to your local machine:

<pre class="gcommand">
scp yourUGAMyID@xfer.gacrc.uga.edu:subdir/\*.dat ./
</pre>

(Note the backslash “ \ ” preceeding * in the last example.)

A directory and all its contents can be transferred using the scp option '''-r''', for recursive file transferring. For example, to transfer a directory on your local machine called programs and all files in it to your Sapelo2 home directory, use:

<pre class="gcommand">
scp -r programs yourUGAMyID@xfer.gacrc.uga.edu:
</pre>

You can also use the scp option '''-p''' to preserve the file features, such as last modification time, file permissions, etc.

To transfer files directly to your Sapelo2 /scratch area (such as /scratch/johndoe/):

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/scratch/johndoe/
</pre>

To transfer files directly to your Sapelo2 work directory:

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/work/groupname/subdirname
</pre>

To transfer files directly to your lab's project file system (such as /project/groupname):

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/project/groupname
</pre>

====Download directly from internet to Sapelo2====

Here is an example on how to download [http://hannonlab.cshl.edu/fastx_toolkit/index.html FastaX] software from internet directly to Sapelo2

<pre class="gcommand">
wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
</pre>

Some websites use indirect connections, it could be downloaded as
<pre class="gcommand">
curl -OL https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
</pre>


===Transferring Files between two file systems on Sapelo2===

Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and password (and two-factor authentication via Archpass Duo) to transfer files between different file systems on Sapelo2, including the project area. To transfer data between two file systems that are available on the xfer node, you can use the '''cp''' or the '''rsync''' commands. If you have many files to copy, then a good option is to use the '''fpsync''' command, as that will allow the use of multiple cores.

For example, if you want to use 4 cores to transfer all files from /scratch/myid/mydata to /project/mylab/myid/mydata, you can use the following command on an xfer node shell prompt:

<pre class="gcomman">
fpsync -n 4 -t $HOME/fpsync /scratch/myid/mydata /project/mylab/myid/mydata
</pre>

When using the '''fpysnc''' command, please use the <code> -t $HOME/fpsync</code> option to set the fpsync's temporary directory to be in your home directory.



===The File Transfer node for the teaching cluster (txfer.gacrc.uga.edu)===

Users can transfer files between their local machines and the teaching cluster using (a) Filezilla, (b) a secure copy (scp), or (c) WinSCP. To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X).

Access to txfer.gacrc.uga.edu requires an active MyID and password, and two-factor authentication with Archpass Duo. For more details of Archpass Duo, please refer to
[https://eits.uga.edu/access_and_security/infosec/tools/archpass_duo archpass_duo] .

===Transferring Files from your local storage device to the teaching cluster===

To transfer files from your local storage device to the teaching cluster, please connect to txfer.gacrc.uga.edu (use your UGA MyID and the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.

====Using FileZilla====

In order to use [http://filezilla-project.org/ FileZilla] to transfer files between a local machine and the teaching cluster, users will need to install the [http://filezilla-project.org/ FileZilla] software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the '''General''' tab select

<pre class="gcomment">

Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://txfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User: [your UGA MyID]
</pre>

In the '''Transfer Settings''' tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

====Using WinSCP====

In order to use [http://winscp.net/ WinSCP] to transfer files between a local Windows machine and the teaching cluster, users will need to install the [http://winscp.net/ WinSCP] software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:

<pre class="gcomment">

Session:

File Protocol: SFTP

Host name: txfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

</pre>

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.


====Using scp====

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to the teaching cluster:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:

<pre class="gcommand">
scp filename yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
</pre>

where ''filename'' is the name of the file to be transferred, ''yourUGAMyID'' is your UGA MyID, and ''subdirectory'' is the subdirectory in your teaching cluster home directory to which files are being transferred. You will then be asked to enter your UGA MyID password.

(Note: yourUGAMyID@ can be omitted if your username on the teaching cluster is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:

<pre class="gcommand">
scp file1 file2 file3 yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
</pre>

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension):

<pre class="gcommand">
scp *.dat yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
</pre>

===Transferring Files between your Sapelo2 directories and the teaching cluster===

To transfer files between your Sapelo2 directories (e.g. your Sapelo2 home directory or /scratch directory) and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to copy files to/from your home directory at txfer.gacrc.uga.edu.

For example, to transfer a directory called subdir2 in your /scratch directory to your teaching cluster home directory. On xfer.gacrc.uga.edu issue the command:
<pre class="gcommand">
scp -r -p /scratch/yourUGAMyID/subdir2 txfer.gacrc.uga.edu:

UGA DUO authentication is required for SSH/SCP access to
GACRC systems.

UGA DUO is a two-factor authentication service which
requires a password (one factor) and a code, phone,
or device (second factor) to successfully authenticate.

If you are not enrolled in the UGA DUO service please
visit the UGA DUO service self-service portal to enroll
and configure or manage your DUO enabled devices.

https://eits.uga.edu/access_and_security/infosec/tools/duo/portal/

For additional help with UGA DUO authentication or to
report an issue please visit:

https://eits.uga.edu/access_and_security/infosec/tools/archpass/

Password:

Duo two-factor login for yourUGAMyID

Enter a passcode or select one of the following options:

1. Duo Push to XXX-XXX-4304
2. Phone call to XXX-XXX-4304
3. SMS passcodes to XXX-XXX-4304

Passcode or option (1-3):

</pre>
When you enter your password, you will be prompted for the two-factor authentication step. Once you enter a passcode or select an option, the file transfer will start.

===Transferring Files between your project area and the teaching cluster===

To transfer files between your /project area and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to e.g. copy files from /project to your home directory at txfer.gacrc.uga.edu (or to copy files from your home directory at txfer.gacrc.uga.edu to your /project area).

For example, to copy a file called species.fa from your teaching cluster home directory to a subdirectory called binf1234 in your /project area, use the following command on xfer.gacrc.uga.edu:

<pre class="gcommand">
scp txfer.gacrc.uga.edu:species.fa /project/abclab/binf1234
</pre>
where ''abclab'' needs to be changed to the name of your group.

Bioinformatics Databases

2021-07-13T22:25:14Z

Raj76:

As part of our services, the GACRC builds and hosts local copies of frequently cited application data, and provides assistance for sharing data among GACRC members. Datasets are located in the commonly shared "/db" filesystem. NCBI BLAST datasets are located in "/db/ncbiblast/" and are organized by date.

NCBI BLAST Datasets can be loaded in a similar way to software modules. This allows users to replicate results by always being able to use a time stamped version of a database. The time stamp corresponds to the download date of the dataset. To search for available NCBI BLAST dataset modules run the command:

<code>ml spider ncbiblastdb</code>

You can then load a database by running the command module load. For example to load the databases timestamped at 06/16/2021:

<code>module load ncbiblastdb/20210616</code>

Loading this module sets the environment variable BLASTDB to /db/ncbiblast/20210616. This allows BLAST+ to search that directory for databases. You can then use the name of the database you would like to use.(nt, nr, etc.) Here is an example of a shell script to run BLAST+ on the batch queue using the database nt:

<div class="gscript2">
<nowiki>#</nowiki>!/bin/bash<br>
<nowiki>#</nowiki>SBATCH --job-name=j_BLAST+<br>
<nowiki>#</nowiki>SBATCH --partition=batch<br>
<nowiki>#</nowiki>SBATCH --mail-type=ALL<br>
<nowiki>#</nowiki>SBATCH --mail-user=<u>username@uga.edu</u><br>
<nowiki>#</nowiki>SBATCH --ntasks=<u>1</u><br>
<nowiki>#</nowiki>SBATCH --mem=<u>10gb</u><br>
<nowiki>#</nowiki>SBATCH --time=<u>08:00:00</u><br>
<nowiki>#</nowiki>SBATCH --output=BLAST+.%j.out<br>
<nowiki>#</nowiki>SBATCH --error=BLAST+.%j.err<br>

cd $SLURM_SUBMIT_DIR<br>
module load BLAST+/2.9.0-gompi-2019b<br>
module load ncbiblastdb/20210616<br>
blastn -query example.fasta -out results.out -db nt<br>
</div>

In your actual submission script, use your own discretion for the Slurm header values.

NCBI's nr and nt datasets are the most commonly used at the GACRC. GACRC staff will update these datasets every month as well as other NCBI BLAST datasets: cdd_delta, human_genome, mouse_genome, nrte, refseq_protein, refseq_rna, swissprot, and taxdb datasets.

Various subject datasets, e.g. pfam, bowtie indexes of human and mouse, and NCBI bacterial datasets, are hosted as well. However, these datasets are not accessed with a module. These datasets either don’t need updating, the source does not update frequently, or the datasets are not used frequently by our users. These datasets will only be updated by user request.
Instead of using a module, to use these databases, use the path to the databases in your script. Below is an example using BLAST+ with a dataset not available as a module.

<div class="gscript2">
<nowiki>#</nowiki>!/bin/bash<br>
<nowiki>#</nowiki>SBATCH --job-name=j_BLAST+<br>
<nowiki>#</nowiki>SBATCH --partition=batch<br>
<nowiki>#</nowiki>SBATCH --mail-type=ALL<br>
<nowiki>#</nowiki>SBATCH --mail-user=<u>username@uga.edu</u><br>
<nowiki>#</nowiki>SBATCH --ntasks=<u>1</u><br>
<nowiki>#</nowiki>SBATCH --mem=<u>10gb</u><br>
<nowiki>#</nowiki>SBATCH --time=<u>08:00:00</u><br>
<nowiki>#</nowiki>SBATCH --output=BLAST+.%j.out<br>
<nowiki>#</nowiki>SBATCH --error=BLAST+.%j.err<br>

cd $SLURM_SUBMIT_DIR<br>
module load BLAST+/2.9.0-gompi-2019b<br>
blastn -query example.fasta -out results.out -db /db/ncbiblast/refseq_microbial/fasta/09162020/ <br>
</div>

For datasets requested by individual lab groups, GACRC encourages users to maintain their own copies of databases, whose files can be installed in their group's shared "/work" area.
Similarly for datasets frequently updated by their source, the GACRC encourages users to maintain their own copies of these public databases.

The set of data which are regularly updated is open to review, and can be expanded based on available GACRC resources.

==Installed Bioinformatics Databases ==
{| border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable sortable"
|-
! scope="col" | Name
! scope="col" class="unsortable" | Version
! scope="col" | Module Available
|-
| akblab || 01/28/2018 || no
|-
| [[Bacteria NCBI-sapelo2|Bacteria NCBI]] || 12/21/2017 || no
|-
| [[bowtie2|hg19,mm10 & mm9]] || 07/29/2016 || no
|-
| cellranger || 10/29/2019 || mo
|-
| conspred_ressources || 02/02/2017 || no
|-
| dammit || 03/18/2021 || no
|-
| deconseq || 12/01/2016 || no
|-
| decontaMiner-Sapelo2|decontaMiner || 08/29/2019 || no
|-
| eggnog-mapper || 04/16/2019 || no
|-
| funannotate || 08/15/2019 || no
|-
| DTDB-Tk || 01/15/2021 || no
|-
| hg || 09/20/2016 || no
|-
| kegg || 09/09/2016 || no
|-
| maker || 05/22/2018 || no
|-
| MetaCLADE || 08/29/2019 || no
|-
| [[BLAST Databases-Sapelo2 | NCBI BLAST Databases]] || beginning of every month || yes
|-
| [[NCBI Fasta-Sapelo2|NCBI Fasta]] || beginning of every month || yes
|-
| ngs || 11/03/2016 || no
|-
| nndb || 04/29/2020 || no
|-
| PB || 04/24/2019 || no
|-
| [[PfamDB-Sapelo2|pfam]] || 03/02/2020 || no
|-
| pylosift || 07/20/2020 || no
|-
| [[Refseq-Sapelo2|Refseq_genomic]] || || no
|-
| repbase || 01/09/2017 || no
|-
| rfam || 02/09/2017 || no
|-
| seqdb || 05/23/2019 || no
|-
| sortmerna || 02/02/2021 || no
|-
| [[TaxDB-Sapelo2|TaxDB]] || 02/26/2019 || no
|-
| topcons2 || 02/18/2021 || no
|-
| [[Uniprot-Sapelo2|Uniprot]] || 11/19/2020 || no
|-
| [[Uniref-Sapelo2|Uniref]] || 06/28/2018 || no
|-
| [[wublast-Sapelo2|wublast]] || 03/01/2020 || no
|-
|}

Systems

2021-07-13T22:24:13Z

Raj76:

[[Category:Sapelo2]]
[[Category:Teaching]]



=== Sapelo2 ===

Sapelo2 is a Linux cluster that runs a 64-bit CentOS 7.8 operating system and it is managed using xCAT and Puppet. Several virtual login nodes are available, with Intel Xeon Gold 6230 processors, 32GB of RAM, and 16 cores per node. The queueing system on Sapelo2 is Slurm.

For a subset of compute nodes, internodal communication among them and between these nodes and the storage systems serving the home directories and the scratch directories is provided by a QDR Infiniband network(40Gbps). For another subset of compute nodes, these communications are provided by an EDR Infiniband network (100Gbps).

The cluster is currently comprised of the following resources:

'''Regular nodes'''

* 105 compute nodes with AMD EPYC (Rome) processors (64 cores and 128GB of RAM per node)
* 49 compute nodes with AMD EPYC (Naples)processors (32 cores and 128GB of RAM per node)
* 42 compute nodes with Intel Xeon Skylake processors (32 cores and 192GB of RAM per node)
* 32 compute nodes with Intel Xeon Broadwell processors (28 cores and 64GB of RAM per node)
* 2 compute nodes with AMD Opteron processors (48 cores and 256GB of RAM per node)
* 68 compute nodes with AMD Opteron processors (48 cores and 128GB of RAM per node)

'''High memory nodes (1TB/node)'''

* 4 compute nodes with AMD EPYC processors (64 cores and 1TB of RAM per node)
* 4 compute nodes with Intel Xeon Broadwell processors (28 cores and 1TB of RAM per node)
* 1 compute node with AMD Opteron processors (48 cores and 1TB of RAM per node)

'''High memory nodes (512GB/node)'''

* 18 compute nodes with AMD EPYC (Naples)processors (32 cores and 512GB of RAM per node)
* 6 compute nodes with AMD Opteron processors (48 cores and 512GB of RAM per node)


'''GPU nodes'''

* 4 compute nodes with Intel Xeon Skylake processors (32 cores and 187GB of RAM) and 1 NVIDIA P100 GPU card per node
* 2 compute nodes with Intel Xeon processors (16 cores and 128GB of RAM) and 8 NVIDIA K40m GPU cards per node
* 2 compute node with Intel Xeon processors (12 cores and 96GB of RAM) and 7 NVIDIA K20Xm GPU cards per node

'''Buy-in nodes'''

* Various configurations


====[[Connecting#Connecting_to_Sapelo2 |Connecting to Sapelo2]]====

====[[Transferring Files]]====

====[[Disk Storage]]====

====[[Software on Sapelo2]]====

====[[Available Toolchains and Toolchain Compatibility]]====

====[[Code Compilation on Sapelo2]]====

====[[Running Jobs on Sapelo2]]====

====[[Monitoring Jobs on Sapelo2]]====

====[[Migrating from Torque to Slurm]]====

'''Training material'''

To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the GACRC's Kaltura channel at https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required). Training sessions and slides are available at https://wiki.gacrc.uga.edu/wiki/Training

----
[[#top|Back to Top]]



=== Teaching cluster ===

The teaching cluster is a Linux cluster that runs a 64-bit Linux, with Centos 7.8. The login node is a VM that has 4 cores (Intel Xeon Gold 6230 processor) and 16GB of RAM. An Ethernet network (1Gbps) provides internodal communication among compute nodes, and between the compute nodes and the storage systems serving the home directories and the work directories.

The cluster is currently comprised of the following resources:

*30 compute nodes with Intel Xeon X5650 2.67GHz processors (12 cores and 48GB of RAM per node)
* 2 compute nodes with Intel Xeon L7555 1.87GHz processors (32 cores and 512GB of RAM per node)
* 4 NVIDIA Tesla (Kepler) K20Xm GPU cards. These cards are installed on one host that has dual 6-core Intel Xeon CPUs and 48GB of RAM

The queueing system on the teaching cluster is Slurm.

====[[Connecting#Connecting_to_the_teaching_cluster |Connecting to the teaching cluster]]====

====[[Transferring Files]]====

====[[Disk Storage]]====

====Software Installed on the teaching cluster====

The list of installed application is available at [[Software]] page.

====[[Code Compilation on the teaching cluster]]====

====[[Running Jobs on the teaching cluster]]====

====[[Monitoring Jobs on the teaching cluster]]====

Data transfer

2021-05-07T17:22:06Z

Raj76:

== F*EX Service ==
The F*EX service (Referred to as FEX) allows you to share large datasets with collaborators. You can automatically register for this service as long as you have a uga.edu active email address and can sign-on to the UGA VPN.

=== Registration ===
* You have to have an active uga.edu email address
* You have to sign on to the UGA archpass VPN (The registration is only allowed when you are connected to the VPN, you can still send and receive data when you are not connected ot the VPN)
* Go to [https://fex.gacrc.uga.edu/fur FEX Registration] and sign up for an account. Use your uga.edu email address
* You will get an email from the service with a link to confirm your request.
* Visit the confirmation link sent in the email to confirm your request.
* After confirmation is successful, you will see the following info '''''Save the info that you see in on the webpage''''':
** '''FEX Account direct URL''' for your account
** FEX Server '''URL''' (Always https://fex.gacrc.uga.edu/fup/)
** '''Sender''' (Your email address)
** '''auth-ID''' (Keep this to yourself)
***If you decide to change this auth-ID at a later data, '''''PLEASE, DO NOT USE YOUR MYID PASSWORD. USE A DIFFERENT PASSWORD UNIQUE TO THIS SERVICE.'''''

=== Accessing the FEX service from a web browser ===
* You can visit your unique '''FEX Account direct URL''' that you saved from the request confirmation page when you registered.
* Or, you can go to [https://fex.gacrc.uga.edu/fup F*EX Service] and enter your email and auth-ID.

=== Accessing the FEX service from GACRC (Xfer nodes) ===
The FEX command line clients are installed on all GACRC xfer nodes.
Make sure that you use your email and auth-ID instead of mine at the highlighted spots below.
<nowiki>raj76@xfer4:~$ fexsend -I
F*EX server URL: https://fex.gacrc.uga.edu
proxy address (hostname:port or empty if none):
Your e-mail address as registered at https://fex.gacrc.uga.edu: YOUR_EMAIL_ADDRESS
Your auth-ID for raj76@uga.edu at https://fex.gacrc.uga.edu: MY_PRIVATE_AUTH_ID
data written to /home/raj76/.fex/id
</nowiki>

You can then send any file or directory using the fexsend command.
To send a file you use the command:
fexsend file_to_send email_of_recipients
''email_of_recipients - Can be a comma separated list for multiple recipients.''

To send a directory as a archive, use the following command:
fexsend -a archive.tgz Directory_to_send email_of_recipients

<nowiki>
>$ fexsend -h
usage:
fexsend [options] FILE[...] [@] RECIPIENT[,...]
fexsend [special option]
fexsend -l ['RECIPIENT-REGEXP']
fexsend -f NUMBER RECIPIENT[,...]
fexsend -x NUMBER [-C -k -D -K]
options:
-v verbose mode
-a ARCHIVE put files in archive (.zip .7z .tar .tgz) without .* files
-A ARCHIVE put files in archive (.zip .7z .tar .tgz) with .* files
-# FILELIST exclude files (# is list separator) from archive
-0 do not compress files for .zip and .7z archives
-d delete file on fex server
-o overwrite mode, do not try to resume
-m KBS limit throughput (KBS kB/s)
-p SIZE:WAIT send file in chunks (SIZE MB, WAIT seconds)
-i ACCOUNT use ID data [ACCOUNT] from ID file
-C 'COMMENT' add COMMENT to notification email
-k MAX keep file MAX days on fex server
-D delay auto-delete after download
-K no auto-delete after download
-M MIME-file (to be displayed in recipient's webbrowser)
-s STREAM read data from pipe and upload it with STREAM name
-r ADDRESS use Reply-To ADDRESS in notification email
-n send no notification email, just show the download URL
-/ do not upload the file, but tell the server to link it
-= ALIAS use ALIAS as filename
-^ add date_time to filename
-q quiet mode
-+ undocumented feature - test it :-)
special options:
-I initialize ID file or show ID
-I ACCOUNT add alternate ID data (secondary logins) to ID file
-l list sent files (numbers needed for -f -x -d -N)
-u list download URLs of sent files
-f NUMBER forward already uploaded file to another recipient
-x NUMBER use -C -k -D -K for already uploaded file
-d NUMBER delete file on fex server
-N NUMBER resend notification email
-Q check quotas
-U generate upload URL with bash function (-U+ for archives)
-T UP:DOWN test internet speed with upload and download MBs
-@ edit server address book (aliases)
-S show server/user settings and auth-ID
-V show version and installation/upgrade hint
-H show hints and examples and more options
examples:
fexsend visualization.mp4 framstag@rus.uni-stuttgart.de
fexsend -a images.zip *.jpg webmaster@belwue.de,metoo
lshw | fexsend -s hardware.list admin@belwue.de
</nowiki>

Data transfer

2021-05-07T17:19:58Z

Raj76:

Data transfer

2021-05-07T15:54:10Z

Raj76:

Data transfer

2021-05-07T15:52:53Z

Raj76:

Data transfer

2021-05-07T15:47:17Z

Raj76:

Data transfer

2021-05-07T15:35:04Z

Raj76:

Data transfer

2021-05-07T15:32:58Z

Raj76:

Data transfer

2021-05-07T15:24:47Z

Raj76: Wiki page to document the different data transfer methods available to GACRC users.

Transferring Files

2020-01-28T15:12:36Z

Raj76:

[[Category:Sapelo2]][[Category:Teaching]]

===The File Transfer nodes (xfer.gacrc.uga.edu)===

Users can transfer files between their local machines and GACRC storage using various programs, such as (a) Filezilla, (b) WinSCP, (c) a secure copy (scp), and (d) an SSH secure File Transfer software (SSH file transfer). To transfer files, you must have a file transfer program installed on your local machine and a connection to the UGA campus network.

All Sapelo2 users, as well as all GACRC PIs, can access the xfer nodes using the hostname '''xfer.gacrc.uga.edu''' and using their UGA MyID (not the 810 or 811 number) and the MyID password to authenticate. Two-factor authentication using Archpass Duo is also necessary. For more details of Archpass Duo, please refer to
[https://eits.uga.edu/access_and_security/infosec/tools/archpass_duo archpass_duo].

For connecting from off-campus, please first connect to the UGA VPN and then connect to xfer.gacrc.uga.edu. Information on how to use the VPN is available at [https://eits.uga.edu/access_and_security/infosec/tools/vpn/ UGA_VPN].

To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure File Transfer Utility is available for Windows on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page (you will need to enter your UGA myID and password to download the software).

The GACRC file transfer nodes (xfer) are configured to facilitate file transfer. These nodes, which have a hostname of '''xfer.gacrc.uga.edu''', are connected via a higher-bandwidth network connection. Hence, maximum transfer speed between a user's local machine and the GACRC systems can be achieved by transferring files to/from the host xfer.gacrc.uga.edu (instead of the login nodes of the clusters).

Note that a user's home directory on the xfer nodes is the same as the user's Sapelo2 home directory, which is '''not''' the same as the user's home directory on the teaching cluster.

The xfer node can access the following file systems using the full path:

1. Your home directory on an xfer node is the same as your Sapelo2 home directory, and the path is /home/username

2. The zcluster home directory: /panfs/pstor.storage/home/groupname/username '''(for former zcluster users only)'''

3. The zcluster escratch4 directory: /escratch4/username '''(for former zcluster users only)'''

4. The Sapelo2 scratch directory: /scratch/username

5. The Sapelo2 work directory: /work/groupname

6. The project file system for the lab: /project/groupname

'''Note:''' Starting Nov. 16th, the old /lustre1 file system was replaced by a /scratch file system. All files that were present in /lustre1 on Nov. 16th have been copied to /scratch. When submitting new jobs, please use /scratch and do not use /lustre1 anymore.

===Transferring Files from your local storage device to Sapelo2 or vice-versa===

To transfer files from your local storage device to Sapelo2 or from Sapelo2 to your local storage device, please connect to xfer.gacrc.uga.edu (use your UGA MyID, the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.

====Using FileZilla====

In order to use [http://filezilla-project.org/ FileZilla] to transfer files between a local machine and Sapelo2, users will need to install the [http://filezilla-project.org/ FileZilla] software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the '''General''' tab select

<pre class="gcomment">

Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://xfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User: [your UGA MyID]
</pre>

In the '''Transfer Settings''' tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

Once the connection is established you can upload files from your local machine to Sapelo2 or download files from Sapelo2 to your local machine.

Here is the PDF to check step-by-step screen shots:
{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
| [[ media:Xfer Filezilla 12032018.pdf]]
|-
|}

====Using WinSCP====

In order to use [http://winscp.net/ WinSCP] to transfer files between a local Windows machine and Sapelo2, users will need to install the [http://winscp.net/ WinSCP] software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:

<pre class="gcomment">

Session:

File Protocol: SFTP

Host name: xfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

</pre>

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.

====Using SSH Secure File Transfer====

In order to use the SSH Secure File Transfer software that is installed as part of SSH (Secure Shell Utilities for Windows) available on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page, users need to start the Secure File Transfer application and open a connection to host xfer.gacrc.uga.edu. You will have to enter your UGA MyID as the username, but other fields (port number, etc) can be left empty (or leave the default value, if any). You will be prompted for your MyID password and after that you will be prompted to enter DUO two factor authentication code. Please choose one of the options like DUO push, Phone call or SMS passcodes. Once the connection is established, you can upload or download files.

Here is the PDF to check step-by-step screen shots:
{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
| [[ media:Xfer SSH File Transfer Nov292018.pdf]]
|-
|}

====Using scp====

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to Sapelo2:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
</pre>

where ''filename'' is the name of the file to be transferred, ''yourUGAMyID'' is your UGA MyID, and ''subdirectory'' is the subdirectory in your Sapelo2 home directory to which files are being transferred. You will then be asked to enter your UGA MyID password. You will also be prompted for two-factor authentication with Archpass Duo.

(Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:

<pre class="gcommand">
scp file1 file2 file3 yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
</pre>

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension):

<pre class="gcommand">
scp *.dat yourUGAMyID@xfer.gacrc.uga.edu:subdirectory
</pre>

A directory and all its contents can be transferred using the scp option '''-r''', for recursive file transferring. For example, to transfer a directory on your local machine called programs and all files in it to your Sapelo2 home directory, use:

<pre class="gcommand">
scp -r programs yourUGAMyID@xfer.gacrc.uga.edu:
</pre>

You can also use the scp option '''-p''' to preserve the file features, such as last modification time, file permissions, etc.

To transfer files directly to your Sapelo2 /scratch area (such as /scratch/johndoe/):

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/scratch/johndoe/subdirectory
</pre>

To transfer files directly to your Sapelo2 work directory:

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/work/groupname/subdirname
</pre>

To transfer files directly to your lab's project file system (such as /project/groupname):

<pre class="gcommand">
scp filename yourUGAMyID@xfer.gacrc.uga.edu:/project/groupname
</pre>

To transfer files from the decommissioned zcluster home dir to your local machine

From your local machine, issue the command:

<pre class="gcommand">
scp yourUGAMyID@xfer.gacrc.uga.edu:/panfs/pstor.storage/home/groupname/username/subdirectory/filename ./
</pre>

The transferred file will be at your working directory. (Note: yourUGAMyID@ can be omitted if your username on Sapelo2 is the same as on your local machine)

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension in your Sapelo2 home dir, in a subdirectory called subdir):

<pre class="gcommand">
scp yourUGAMyID@xfer.gacrc.uga.edu:subdir/\*.dat ./
</pre>

(Note the backslash “ \ ” preceeding * in the last example.)

====Download directly from internet to Sapelo2====

Here is an example on how to download [http://hannonlab.cshl.edu/fastx_toolkit/index.html FastaX] software from internet directly to Sapelo2

<pre class="gcommand">
wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
</pre>

Some websites use indirect connections, it could be downloaded as
<pre class="gcommand">
curl -OL https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2
</pre>



===Transferring Files from the decommissioned zcluster to Sapelo2===

Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and MyID password. You will also be prompted for two-factor authentication with Archpass Duo. This file transfer node has the following file systems mounted:

* zcluster /escratch4 file system. To access it, use e.g.
<pre class="gcommand">
cd /escratch4/username
</pre>

* zcluster home file systems. To access your zcluster home directory, use the following
<pre class="gcommand">
cd /panfs/pstor.storage/home/groupname/username
</pre>

For example, if your group is named abclab and your username is jsmith use
<pre class="gcommand">
cd /panfs/pstor.storage/home/abclab/jsmith
</pre>

* Sapelo2 home file systems. To access your Sapelo2 home directory, use the following
<pre class="gcommand">
cd /home/username
</pre>

* Sapelo2 scratch file system on the Lustre file system. To access your /scratch space use the following
<pre class="gcommand">
cd /scratch/username
</pre>

* Sapelo2 work file system on the Lustre file system. To access your group's /work space use the following
<pre class="gcommand">
cd /work/abclab
</pre>

* /project file system. To access your group's /project space use the following (please note that this file system is mounted when it is first accessed, so you need to access it with the full path):
<pre class="gcommand">
cd /project/abclab
</pre>

Because the xfer node mounts all these file systems, users can ssh into an xfer node and use '''cp''' or '''rsync''' to copy the files between any two of these file systems. Users can also copy files from their zcluster home dirs or /escratch4 to their project file system (/project/groupname).

===Transferring Files between two file systems on Sapelo2===

Use ssh to login to xfer.gacrc.uga.edu using your UGA MyID username and password (and two-factor authentication via Archpass Duo) to transfer files between different file systems on Sapelo2, including the project area.



===The File Transfer node for the teaching cluster (txfer.gacrc.uga.edu)===

Users can transfer files between their local machines and the teaching cluster using (a) Filezilla, (b) a secure copy (scp), or (c) an SSH secure File Transfer software (SSH file transfer). To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure File Transfer Utility is available for Windows on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page (you will need to enter your UGA myID and password to download the software).

Access to txfer.gacrc.uga.edu requires an active MyID and password, and two-factor authentication with Archpass Duo. For more details of Archpass Duo, please refer to
[https://eits.uga.edu/access_and_security/infosec/tools/archpass_duo archpass_duo] .

===Transferring Files from your local storage device to the teaching cluster===

To transfer files from your local storage device to the teaching cluster, please connect to txfer.gacrc.uga.edu (use your UGA MyID and the MyID password to authenticate). You will also be prompted for two-factor authentication with Archpass Duo.

====Using FileZilla====

In order to use [http://filezilla-project.org/ FileZilla] to transfer files between a local machine and the teaching cluster, users will need to install the [http://filezilla-project.org/ FileZilla] software on the local machine. After installation, start FileZilla and follow these steps:

1. Go to File -> Site Manager

2. Add a New Site with the following configuration.

In the '''General''' tab select

<pre class="gcomment">

Protocol: SFTP - SSH File Transfer Protocol

Host: sftp://txfer.gacrc.uga.edu

Port: 22

Logon Type: Interactive

User: [your UGA MyID]
</pre>

In the '''Transfer Settings''' tab, check the box for "Limit number of simultaneous connections" and set "Maximum number of connections" to 1.

If this step is omitted, then you will need to enter your password and Duo authentication for each file that you upload or download. By limiting the number of simultaneous connections to 1, you only need to enter your password and Duo authentication once per session.

With the above settings saved, you should be able to transfer files to txfer using filezilla. When you open the connection, you will be prompted for your MyID password (unless you use key based ssh). If the password authenticates successfully, then another pop-up window will prompt for Duo. Note that the Duo window looks very similar to the password window, but the text in the box will describe the options you can use for Duo (for example, a push, enter a passcode, etc).

====Using WinSCP====

In order to use [http://winscp.net/ WinSCP] to transfer files between a local Windows machine and the teaching cluster, users will need to install the [http://winscp.net/ WinSCP] software on the local machine. After installation, when you click the WinSCP shortcut button, Login window pops up, in that you select new site and enter the following information to start the new session:

<pre class="gcomment">

Session:

File Protocol: SFTP

Host name: txfer.gacrc.uga.edu

Port number: 22

Username: [your UGA MyID]

Password: [your MyID password]

</pre>

If the password authenticates successfully, then another pop-up window will prompt for Duo two-factor authentication. It says, Using Keyboard-interactive authentication and lists the Duo options you can choose from: Duo push, Phone Call, or SMS passcode, etc. Once the connection is established you can upload or download files from Local Machine to Cluster or vice-versa. Note: Under Advanced, kept the "Default" option for Transfer settings Rule. With this default option, you will be able to transfer multiple files or folders during the session.

====Using SSH Secure File Transfer====

In order to use the SSH Secure File Transfer software that is installed as part of SSH (Secure Shell Utilities for Windows) available on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page, users need to start the Secure File Transfer application and open a connection to host txfer.gacrc.uga.edu. You will have to enter your UGA MyID as the username, but other fields (port number, etc) can be left empty (or leave the default value, if any). You will be prompted for your MyID password and after that you will be prompted to enter DUO two factor authentication code. Please choose one of the options like DUO push, Phone call or SMS passcodes. Once the connection is established, you can upload or download files.

Here is the PDF to check step-by-step screen shots:
{| width="100%" border="1" cellspacing="0" cellpadding="2" align="center" class="wikitable unsortable"
|-
| [[ media: Txfer SSH File Transfer Nov302018.pdf]]
|-
|}

====Using scp====

Sample commands to transfer files from your local Unix/Linux/Mac OS X machine to the teaching cluster:

Open a terminal in your local machine, change directory (cd) to where the files are located in your local machine, and at the command prompt type:

<pre class="gcommand">
scp filename yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
</pre>

where ''filename'' is the name of the file to be transferred, ''yourUGAMyID'' is your UGA MyID, and ''subdirectory'' is the subdirectory in your teaching cluster home directory to which files are being transferred. You will then be asked to enter your UGA MyID password.

(Note: username@ can be omitted if your username on the teaching cluster is the same as on your local machine). Multiple files (e.g. file1, file2, and file3) can be transferred with a single command:

<pre class="gcommand">
scp file1 file2 file3 yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
</pre>

Wildcards can be used for multiple file transfer (e.g. all files with .dat extension):

<pre class="gcommand">
scp *.dat yourUGAMyID@txfer.gacrc.uga.edu:subdirectory
</pre>

===Transferring Files between your Sapelo2 directories and the teaching cluster===

To transfer files between your Sapelo2 directories (e.g. your Sapelo2 home directory or /lustre1 directory) and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to copy files to/from your home directory at txfer.gacrc.uga.edu.

For example, to transfer a directory called subdir2 in your /lustre1 directory to your teaching cluster home directory. On xfer.gacrc.uga.edu issue the command:
<pre class="gcommand">
scp -r -p /lustre1/yourUGAMyID/subdir2 yourUGAMyID@txfer.gacrc.uga.edu:

UGA DUO authentication is required for SSH/SCP access to
GACRC systems.

UGA DUO is a two-factor authentication service which
requires a password (one factor) and a code, phone,
or device (second factor) to successfully authenticate.

If you are not enrolled in the UGA DUO service please
visit the UGA DUO service self-service portal to enroll
and configure or manage your DUO enabled devices.

https://eits.uga.edu/access_and_security/infosec/tools/duo/portal/

For additional help with UGA DUO authentication or to
report an issue please visit:

https://eits.uga.edu/access_and_security/infosec/tools/archpass/

Password:

Duo two-factor login for yourUGAMyID

Enter a passcode or select one of the following options:

1. Duo Push to XXX-XXX-4304
2. Phone call to XXX-XXX-4304
3. SMS passcodes to XXX-XXX-4304

Passcode or option (1-3):

</pre>
When you enter your password, you will be prompted for the two-factor authentication step. Once you enter a passcode or select an option, the file transfer will start.

===Transferring Files between your project area and the teaching cluster===

To transfer files between your /project area and your teaching cluster home directory, please first use ssh to login into xfer.gacrc.uga.edu. Then use the scp command to e.g. copy files from /project to your home directory at txfer.gacrc.uga.edu (or to copy files from your home directory at txfer.gacrc.uga.edu to your /project area).

For example, to copy a file called species.fa from your teaching cluster home directory to a subdirectory called binf1234 in your /project area, use the following command on xfer.gacrc.uga.edu:

<pre class="gcommand">
scp txfer.gacrc.uga.edu:species.fa /project/abclab/binf1234
</pre>
where ''abclab'' needs to be changed to the name of your group.

Disk Storage

2020-01-13T17:34:49Z

Raj76: /* Scratch file system */

[[Category:Sapelo2]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch" and "work", "home", and "offline" storage.

The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The work file system is a group-shared space that can be used to store common files needed by jobs. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

Each compute node has local physical hard drives that the user can utilize as temporary storage, aka lscratch. The lscratch device is a very fast storage device compared to the network attached storage systems. The drawback is that the capacity is low and it cannot be accessed from outside the compute node. The data in lscratch is not backed up and it can be deleted anytime after the job on the compute node is finished.

=== Home file system ===

When you login into a system (e.g. sapelo2 or xfer nodes), you will land on your home directory. Home directories are "auto mounted" on the login nodes and xfer nodes when you login. Your home directory on the xfer nodes is the same as your home directory on sapelo2. Sapelo2 interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens and compute nodes will mount a user's home directory when a job submitted by this user is dispatched to those compute nodes. Users of the teaching cluster have a separate home directory, which is not the same as on Sapelo2.

Home directories have a per user quota and have snapshots. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten. A user's snapshot is stored within his/her home file system, thus snapshots consume a user's home directory quota. If files are created and deleted with frequency, the snapshots will grow and might end up using a large fraction (or all) the space available within a user's home file system.

The recommended data workflow is to have files in the home directory *change* as little as possible. These should be databases, applications that you use frequently but do not need to modify that often and other things that you, primarily, *read from*. Think of snapshots as the memory of the files that were stored there - no matter if you add, change or delete the files, the total sum of that activity will build up over time and may exceed your quota.

Summary of the home directory characteristics for a sample user 'jsmith' in 'abclab':
<pre>
sapelo2
home dir quota = 100GB
home dir path = /home/jsmith
snapshots = yes
subject to 30-day purge = no
</pre>

=== Scratch file system ===

The scratch file system resides on a high-speed storage device and it should be used to store temporary files needed for current jobs. Files that are not needed for current jobs should not be left on the scratch file system. This file system is mounted on the login nodes, xfer nodes, and compute nodes.

The scratch file system is mounted on the login nodes, xfer nodes, and compute nodes.

The recommended data workflow will have jobs write output files, including intermediate data, such as checkpoint files, and final results into the scratch file system. Final results, intermediate files, and other data should then be transferred out of and immediately deleted from the scratch file system, if these are not needed for other jobs that are being submitted soon.

Because the scratch file system stores large amounts of data that change a lot, it is does not have snapshots turned on and it is not backed up in anyway. Files deleted from a scratch directory cannot be recovered.

There is no per user quota in the scratch file system, but a file retention policy is implemented to help prevent this file system from filling up.

'''Scratch file system "30-day purge" policy'''

<blockquote style="background-color: lightyellow; border: solid thin grey;">
Any file that is not accessed or modified by a compute job in a time period no longer than 30 days will be automatically deleted off the /scratch file system. Measures circumventing this policy will be monitored and actively discouraged.

There is no storage size quota for /scratch usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their /scratch directories or reduce temporarily the 30 day limit to a lower limit.

Please see [[Policies#Policy_Statement_for_SCRATCH_File_System|purge policy]] for more info.
</blockquote>

Summary of the scratch directory characteristics for a sample user 'jsmith' in 'abclab':
<pre>
sapelo2
scratch dir quota = Currently no per user quota
scratch dir path = /scratch/jsmith
snapshots = no
subject to 30-day purge = yes
</pre>

=== Work file system ===

The work file system resides on a high-speed storage device and it should be used to store files needed for jobs. Each group has a directory in the work file system and this space can be used to store files needed by multiple users within a group. The work file system has a per group quota and files stored there are not subject to the auto-purge policy that is applied to the scratch file system.

The work file system is mounted on the login nodes, xfer nodes, and compute nodes.

The recommended data workflow is to have files that are often needed for repeated jobs, possibly by multiple users within a group, such as reference data and model data, be stored in the group's work directory. This directory is not intended as a place for jobs to write output files.

The work file system does not have snapshots turned on and it is not backed up in anyway. Files deleted from a work directory cannot be recovered.

Summary of the work directory characteristics for a sample user 'jsmith' in 'abclab':
<pre>
sapelo2
work dir group quota = 500GB and a maximum of 100,000 files
work dir path = /work/abclab
snapshots = no
subject to 30-day purge = no
</pre>

=== lscratch file system ===

Each compute node has local physical hard drives that the user can utilize as temporary storage. The file system defined on the hard drives is called /lscratch. The lscratch device is a very fast storage device compared to the network attached storage systems. The drawback is that the capacity is low and it cannot be accessed from outside the compute node. This file system can be used for single-core jobs and for multi-thread jobs that run within a single node. In general, parallel jobs that use more than one node (e.g. MPI jobs) cannot use the /lscratch file system.

The data in lscratch is not backed up and it needs to be deleted when job on the compute node is finished.

Jobs that do not need to write large output files, but that need to access the files often (for example, to write small amounts of data into disk), can benefit from using /lscratch. Jobs that use /lscratch should request the amount of space in /lscratch. For information on how to request lscratch space for jobs, please refer to [https://wiki.gacrc.uga.edu/wiki/Running_Jobs_on_Sapelo2#How_to_run_a_job_from_the_compute_node.27s_local_disk_.28.2Flscratch.29 How to run a job from lscratch]

Summary of the lscratch directory characteristics for a sample user 'jsmith' in 'abclab':
<pre>
sapelo2
quota = Limited by device size (Approx. 210GB on the AMD nodes and 800GB on the Intel nodes)
path = /lscratch
snapshots = no
subject to purge = yes (files to be deleted when job exits the node)
</pre>

=== Project file system ===

The offline storage filesystem is named "project" and is configured for use by lab groups. By default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the "xfer" nodes when it is first accessed using its full path.

The project filesystem has snapshots turned on.

The recommended data workflow is to have data not needed for current jobs, but that are still needed for future jobs on the cluster, be transferred into the project file system and deleted from the scratch area.

Summary of the project directory characteristics for a sample group 'abclab':
<pre>
sapelo2
quota = default of 1TB per group
path = /project/abclab
snapshots = yes
subject to 30-day purge = no
</pre>

[[#top|Back to Top]]

== Storage Architecture Summary ==

Mount path for home, scratch, work, and lscratch filesystems using an example user 'jsmith' in a lab group 'abclab':
<pre>
sapelo2

home= /home/jsmith
scratch= /scratch/jsmith
work= /work/abclab
lscratch= /lscratch

</pre>

Quota for home, scratch, work, and lscratch filesystems:
<pre>
sapelo2

home= 100GB
scratch= Currently no quota
work= (to be added)
lscratch= Limited by device size (Approx. 210GB on the AMD nodes and 800GB on the Intel nodes)

</pre>

== Auto Mounting Filesystems ==

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo2 home directories and the project filesystems. Sapelo2 interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

== Snapshots ==

'''Home directories'''

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo2 have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. Weekly and monthly snapshots are also made and a few recent snapshots are stored.

'''Note: Users cannot recover files from their home directory snapshots by themselves.'''

If you would like to recover a file that you have deleted from your home directory, please submit a [[Getting Help|help request]] so the GACRC staff can recover the file for you. Please let us know the name(s) of the file(s) and the directory from which they were deleted.

'''Project file systems'''

The project file systems are also snapshotted. Each /project filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /project directories may retrieve files from these snapshots by changing into their snapshot directory /project/abclab/.snapshot and then cd into an appropriate snapshot directory and copying files from the that snapshot to any location they would like.

'''Note: ANY user in a lab can access the snapshots of his/her group project file system and restore files he/she has there.'''

Here is an example for user jsmith who is in the abclab group, on an xfer node:

<pre>
[jsmith@xfer1 ]$ pwd
/home/jsmith

[jsmith@xfer1 ]$ cd /project/abclab/.snapshot

[jsmith@xfer1 .snapshot]$ ls
2019.02.17.04.00.03.Weekly 2019.03.01.06.00.03.Daily
2019.02.26.06.00.03.Daily 2019.03.02.06.00.03.Daily
2019.02.27.06.00.03.Daily 2019.03.03.04.00.03.Weekly
2019.02.28.06.00.03.Daily 2019.03.03.06.00.03.Daily

[jsmith@xfer1 snapshot]$ cd 2019.03.03.06.00.03.Daily

[jsmith@xfer1 2019.03.03.06.00.03.Daily]$ cp my-to-restore-file /home/jsmith/test
</pre>

[[#top|Back to Top]]

== Current Storage Systems ==

(1) Seagate (Xyratex) ClusterStor1500 Lustre appliance (480TB) - $SCRATCH on Sapelo2

(2) DDN SFA14KX Lustre appliance (1.26PB) - $SCRATCH & $WORK on Sapelo2

(3) Penguin IceBreakers ZFS storage chains (84TB usable capacity) - $HOME on Sapelo2

(4) Penguin IceBreakers ZFS storage chains (374TB usable capacity) - $PROJECT research groups' long-term space - only for active projects requiring Sapelo2 access

(5) Panasas ActiveStor 100H (1PB) - $PROJECT research groups' long-term space - only for active projects requiring Sapelo2 access

(6) ZFS storage chains (720TB) - backup environment for $HOME and $PROJECT.

Policies

2020-01-13T17:30:12Z

Raj76: /* Policy Statement for SCRATCH File System */

==Introduction to GACRC Policies==

The following policies are subject to revision, especially as the GACRC grows in scope and services. Your comments and questions will be useful to our policy formulation and refinement and are actively solicited by the GACRC Advisory Committee.

The GACRC computational infrastructure, including its servers, clusters, data stores, and other related devices are for the exclusive use of authorized users only.

Anyone using these systems expressly consents to abide by the policies of the University of Georgia and the Georgia Advanced Computing Resource Center and, accordingly, is subject to account termination and/or immediate disconnection from GACRC resources.

==GACRC Resource Usage==

The computational resources of the Georgia Advanced Computing Resource Center are to be used in direct support of research programs at the University of Georgia. Support is also provided for classes that teach computational methods, and provide training for high performance computing. The GACRC reserves the right to restrict access to its resources for course work if such work is deemed to present a negative impact to authorized research activities.

GACRC policies supplement UGA’s Policies on the Use of Computers, found at:. http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup

==GACRC Eligibility and Access ==

Access to and use of the computing facilities managed by the Georgia Advanced Computing Resource Center are limited to persons affiliated with the University of Georgia or associated with research projects sponsored by UGA.

Affiliation in this context means faculty, research staff and supervised students of the University of Georgia. Faculty includes persons holding permanent or temporary appointments as well as adjunct faculty, instructors and visiting faculty while in residence at the University. It also includes those persons with faculty status such as research associates, research scientists, post-doctoral researchers and academic and service professionals. Staff includes all those non-faculty persons employed directly by the University in a research-support role. Graduate and undergraduate students who are members of faculty research labs are eligible for accounts as well. For directly affiliated users, accounts on the GACRC computers will remain active as long as the individuals hold the above status.

Access by non-UGA researchers and their students, affiliated to higher-education institutions or non-profit research organizations, for work on research projects conducted in collaboration with UGA Faculty is possible under the guidelines established by the Office of the Vice President for Research and the Office of International Education. A request for access can be forwarded to the GACRC by the UGA Faculty, providing details of the collaboration on a joint research project. Such Affiliate users will be considered part of the UGA Faculty’s group and will be under the Faculty’s responsibility. Affiliate users’ access will be granted for a fixed period of time, according to the expected length of the collaborative project. Renewal of affiliate accounts will be required annually.

All accounts will remain active no more than 30 days following a status change (i.e., leaving the university). Graduate instructional accounts will only remain active for the duration of the semester in which they are actually needed. HOME and PROJECT directories will be archived for at least 90 days, but no longer than 180 days after an account becomes inactive.

Requests for access by individuals other than those listed above should be directed to the Director of the Center for consideration with the GACRC Advisory Committee.

Access will be granted to a specific GACRC resource after appropriate training is undertaken with GACRC staff. Existing access to other GACRC resources is not a sufficient criteria for access to a new resource. No exceptions will be given to the training requirement.

==GACRC Identity Management==

Below are described the procedures for validating the identity of account users.

===UGA Users and Faculty Lab Groups===

*A UGA Faculty must first establish a GACRC group account using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The UGA Faculty can choose or not to obtain a GACRC user account affiliated with his/her group account.

*All directly affiliated persons wanting an account must apply for access to the GACRC using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The applicant must authenticate to the form using his/her MyID. The applicant must specify to which group he/she belongs. Verification with the UGA Faculty responsible for the group’s account will be used in case identity or affiliation needs to be verified.

*Upon acceptance of the application, the user will be notified via e-mail. The applicant’s UGA MyID and password will be used to log into the requested GACRC resources.

===Affiliate Users===

*A recognized Affiliate user must be sponsored by a UGA Faculty member through an established GACRC group account. The UGA Faculty involved in an established collaboration with the Affiliate user, must apply on behalf of the applicant by contacting the GACRC staff.

*A request will be made by the GACRC to EITS to allocate to the Affiliate a UGA MyID.

*Upon acceptance of the Affiliate user application, the Affiliate will be notified via e-mail. The Affiliate’s UGA MyID and password will be used to log into the requested GACRC resources.

===Protection of Passwords===

As described in UGA’s Password Policy, an account holder must never divulge their MyID and password to a third party. Only authorized account holders may access the resources of the GACRC. If a third party is found to be using an account holder’s login with or without the permission of the account holder, the account holder’s access privileges may be revoked at the sole discretion of the GACRC Manager or Director. Enforcement of this policy is under the responsibility of the Office of the Vice President for Information Technology’s Division of Information Security.

More information is found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/passwords/

==GACRC Storage Usage ==

===Some working definitions===

'''Snapshot''' - Copies of files that are stored on the same storage system as the original files. Snapshots are primarily used to recover files that have been accidentally deleted or corrupted within the recent past. Users are able to manage the file recovery tasks. Snapshots are not maintained beyond a defined rotation schedule, i.e., some number of hourly, daily, weekly, and monthly snapshots are kept on the storage system.

'''Backup''' - Copies of files and/or snapshots kept on a storage system (disk/tape) other than the one that the original files reside on. Backups are primarily used to recover files following a catastrophic failure of the original file or storage system. Backups require administrators to perform file system recovery tasks. Like snapshots, backups have a defined rotation schedule.

'''Archive''' - Copies of files that are not currently being accessed, on a resilient storage system dedicated to reliable long-term storage. Archives can be tape-based or disk-based, and typically part of a disaster recovery plan. The files may be copies of original data which is stored elsewhere (individual groups having their own copies), or the archive storage system may be fed by a dedicated "backup" storage system.

'''Active Projects''' – Projects that have on-going computational work being performed with files that are regularly created, accessed or modified.

===Policy Statement for SCRATCH File System===

The SCRATCH file system resides on a high-performance storage device and is to be used uniquely for temporary storage of files in use by actively running compute jobs. Files are to be removed from SCRATCH when a job completes, e.g. can be copied to the PROJECT file system. The SCRATCH file system is not backed up in any way and no snapshots are taken. The SCRATCH filesystem is mounted under /scratch on all the compute nodes, login nodes and data transfer nodes.

Any file that is not accessed or modified by a compute job in a time period no longer than 30 days will be automatically deleted off the SCRATCH file system. Measures circumventing this policy will be monitored and actively discouraged.

There is no storage size quota for SCRATCH usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, the GACRC may request/force users to clean up their SCRATCH directories or reduce temporarily the 30 day limit to a lower limit.

In order to help users identify old files in their /scratch directory, we generate one file per user, every morning. This file, namely '''/usr/local/var/lustre_stats/$USER.over30d.files.lst''' (where $USER refers to the user's UGA MyID), contains a list of all the files that this user has in his/her /scratch directory and that have not been accessed in over 30 days. This file provides the full path, the last accessed date, and the size of the files that have not been accessed in over 30 days. Because the /scratch filesystem only updates on-disk file access times every 5 days, the last accessed time reported in that list file is approximate and could be off by up to 5 days. The purging system understands this and will make sure the access time falls within the purge window before deleting the file.

===Policy Statement for WORK File System===

The WORK file system resides on a high-performance storage device and is to be used for storing files that are frequently used by the group for computation. The WORK file system is '''NOT''' subject to the 30-day purge policy. The filesystem usage is controlled using a quota on the size and number of files that can be stored in a lab group's WORK area. Initially each group is given a 500GB and 100,000-file quota. The WORK file system is not backed up in any way and no snapshots are taken. The WORK filesystem is mounted under /work on all the compute nodes, login nodes and data transfer nodes. Each lab group has a directory under the /work directory.

The WORK file system is '''NOT''' subject to the 30-day purge policy. But if there is sufficient space consumption on the storage appliance, we reserve the right to ask users to clean up their WORK area. If the users do not respond in a timely fashion we will purge files beginning with the oldest ones. Please do not use the WORK area to store files long term.

===Policy Statement for HOME File System===

The HOME file system resides on a high-performance storage device and is used for long-term storage of files, typically programs and scripts, needed for analysis on the GACRC computing cluster.

All users have 100GB allocated for their HOME usage. Groups may request a separate 100GB allocation for a directory under /usr/local/lab/, for shared use of common applications, libraries, and scripts.

HOME directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the HOME file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with HOME usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the HOME service.

===Policy Statement for PROJECT File System===

The PROJECT file system resides on lower-performance/higher-capacity storage devices, accessible by all GACRC login and data transfer nodes. PROJECT will not be accessible on Sapelo2's compute nodes. This space is to be used by groups for storage of active projects using Sapelo2. PROJECT should not be seen as a long-term repository, as it is not designed as such. Once a project is completed, data should be moved from the PROJECT space to user-managed storage, freeing up capacity for the next active project.

Access to the PROJECT file system is not supported through NFS to a destination outside of the Boyd Data Center, or through the use of the Samba or CIFS protocols. Transfer protocols available through the data transfer nodes are secure ftp, scp, rsync, GridFTP, amongst others.

More info is found at https://wiki.gacrc.uga.edu/wiki/Transferring_Files.

Each group can request a PROJECT volume with an initial 1TB allocation, accessible by all users ascribed to the group, where the sharing of files will be enabled. Users are encouraged to consider their PROJECT space as the primary area to transfer compute job inputs/outputs. Additional space can be requested by a Faculty on behalf of his/her group, in increments of 1TB.

The GACRC reserves the right to establish a cost-recovery rate for PROJECT storage beyond the initial 1TB allocation. Appropriate communications will take place in such an event.

PROJECT directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the PROJECT file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with PROJECT usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the PROJECT service.

==GACRC Software Policy==

The GACRC maintains a collection of program libraries and software packages to support research computing activities across diverse research domains. While a user can install a software package in their own environment, for the sake of general access across groups, and an appropriate deployment with current libraries, compilers and other dependencies, we strongly recommend that GACRC staff be asked to perform the installation or upgrade.

Any software that requires a signed license or contract, even if it is a click-through agreement, must absolutely be reviewed and handled by the Office of Legal Affairs before being signed by an appropriate signature authority. After the license or contract is accepted and the software is made available, GACRC users must fully comply and use the software in a way that does not violate any terms of the license or contract. Further information on licensing issues can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup/eula

As a matter of policy, the GACRC will not purchase any commercial software for the use of a single group or a small number of groups. Commercial software currently purchased and maintained by the GACRC are of general interest and applicability to the whole UGA research community. The GACRC will however install and maintain a group-purchased commercial software, which complies with the above comments on licenses and contracts.

==Security==

To minimize disruption of service, protect data integrity, conserve facility resources and maximize the effectiveness of staff support, the GACRC maintains strict security requirements for access to GACRC resources. Over time, the enforcement of these requirements will become increasingly strict, with the goal of preventing any access to the GACRC resources by any person or any device that is not in strict compliance with these requirements.

===Operating Systems===

Any computer accessing the GACRC for any purpose must run a currently supported operating system, updated to the latest version and update (patch) levels.

===Anti-Virus Software===

Any computer accessing the GACRC for any purpose must meet minimum levels of anti-virus protection. Any computer used by an account holder must have anti-virus software from a source approved by UGA’s Office of Information Security must have that virus protection activated, and must have automatic updates activated.

More information can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/protect_your_computer

===Suspiciously Behaving Software===

Any software that behaves in a suspicious manner may at any time be terminated and/or deleted from GACRC resources at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Suspiciously Behaving Networks and Devices===

Any connection from any device to the GACRC may be terminated at any time, if the device or the connection or a network to which the device is attached appears to be not in compliance with UGA’s security requirements, is behaving suspiciously, or if a threat emerges requiring termination for intrusion prevention at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Account Holder Responsibility===

The account holder is responsible for diligently monitoring their account and compliance with the GACRC’s operating system, intrusion and virus protection standards. The account holder will be duly notified if GACRC personnel determine that minimum security requirements are not met. Specific actions will be requested of the account holder and compliance to these will be expected in a timely fashion. An account holder’s privileges to use GACRC facilities may be terminated by the GACRC Manager or Director at any time, without notice if, in the opinion of either, the account holder is reluctant or averse to practicing diligence in meeting the GACRC’s minimum requirements for intrusion and/or anti-viral protection.

==Resolving Disagreements about Revocation of Privileges or Provisioning Resources==

The Director of the Georgia Advanced Computing Resource Center has full authority to revoke a user's privileges or deny the request of a new resource allocation. The decision to revoke a user's privileges will be based on, but not limited to, abuses of the UGA Policies on the Use of Computers and/or abuses of the UGA Password Policy.

If an account holder is denied a request for provisioning of GACRC resources or resource privileges are revoked, the account holder’s Department Head may appeal to the Vice President for Research and the Vice President for Information Technology. Their decision will be informed by the Director of the GACRC, the Chief Technology Officer as well as the Associate Chief Information Officer for Information Security. The decision of the Vice President for Research and Vice President for Information Technology is final.

==System Maintenance and Downtime==

===Planned Maintenance===

The GACRC instituted monthly maintenance windows in order to perform maintenance operations requiring system operations to be reduced or interrupted.

The schedule will be as follows:

*The last Wednesday of each month from 10AM to 4PM will be reserved for partial cluster maintenance.

*Twice a year, a two-day shut-down of GACRC services will be scheduled for more complex maintenance operations. These will occur on the last Tuesday and Wednesday of the months of January and July.

These maintenance windows represent periods when the GACRC may choose to drain the queues of running jobs and suspend access to the Sapelo2 cluster, as well as storage devices for maintenance purposes. Interruptions will be kept as brief as possible.

The GACRC will notify all users at least 10 days in advance that a maintenance window will be in effect. The notification will describe the nature and extent (partial or full) of the interruptions of cluster and or storage services. In case a maintenance window has to be extended due to unavoidable technical reasons, adequate communications will be made to all users.

The impact of the outages will vary, and the GACRC will do its best to preserve pending and running jobs, which is often very doable. Nevertheless, users will need to plan their job submissions around the maintenance windows.

===Unplanned Maintenance and System Outage===

From time to time, hardware, software, and/or environmental factors may cause a system or subsystem to malfunction, causing disruption to service. Also, there may be circumstances or events related to possible security issues or intrusions which will cause GACRC staff to take systems offline while the nature of the apparent breach is analyzed and appropriate action is taken.

Whenever possible, account holders will be notified by e-mail of these outages in advance, but that may not always be possible. Account holders will be notified by e-mail if the disruption should last more than 30 minutes.

GACRC staff will strive to preserve the work and/or prevent disruption of jobs in process during such outages. However, there may be circumstances which cause disruption of jobs and loss of data. Users are encouraged to implement methods in their code which minimize the effect of unplanned interruption of a job’s execution, such as checkpoints. Users are also strongly encouraged to maintain copies of files of importance.

==Regulatory Compliance==

The GACRC as an infrastructure and service provider does NOT currently warrant that its practices or facilities meet government-mandated requirements for the storage and protection of sensitive, private or classified information. Users may not store such information on GACRC facilities. In other words, data that falls under HIPAA, FERPA, FISMA or similar regulatory requirements, may not be stored, computed against or otherwise transacted through, or with, GACRC infrastructure.

The GACRC and its users must comply with all existing Federal export control regulations for services and infrastructure. Research groups must agree to NOT install or use any software or data that falls under Export Control regulations. More information on the subject of Export Control is available at the following OVPR website:

http://research.uga.edu/export-control/

Copyrighted materials are prohibited without proper authorization. Additionally, illegal content is prohibited.

Non-compliance with any such Federal requirements might impact GACRC operations or delivery of services and could place the GACRC and UGA at risk. If a research group is found to be in non-compliance, then account access will be immediately suspended, while an investigation by EITS’s Information Security division is instigated.

===Disclosure===

Research groups that are involved in activities that store protected data on GACRC infrastructure must contact immediately the GACRC Director in order to address the issue. Depending on circumstances, accommodations might be possible for such activities.

===Research Data Compliance===

Research Data Management is a critical factor in both obtaining federal funding from the NSF, NIH, DoD and other federal agencies, and in the conduct of funded research. Responsibility in maintaining and preserving research data, as detailed in data management plans submitted to Federal funding agencies, is entirely placed upon the research faculty, post docs, and graduate students conducting the research. The GACRC will help by providing information and assistance, but will not be responsible to ensure compliance with a project’s data management plan.

During the phase of proposal writing, arrangements can be discussed and agreed upon as to the GACRC playing an active role, and ensuring the provision of specific services towards the compliance of a data management plan. Depending on the complexity or the nature of the proposed services, the GACRC might require the purchase of specific hardware/software and/or the availability of a %FTE salary and benefits.

More information on data management plans can be found [http://guides.libs.uga.edu/c.php?g=349946&p=2363161 here].

Policies

2020-01-13T17:29:54Z

Raj76: /* Policy Statement for SCRATCH File System */

==Introduction to GACRC Policies==

The following policies are subject to revision, especially as the GACRC grows in scope and services. Your comments and questions will be useful to our policy formulation and refinement and are actively solicited by the GACRC Advisory Committee.

The GACRC computational infrastructure, including its servers, clusters, data stores, and other related devices are for the exclusive use of authorized users only.

Anyone using these systems expressly consents to abide by the policies of the University of Georgia and the Georgia Advanced Computing Resource Center and, accordingly, is subject to account termination and/or immediate disconnection from GACRC resources.

==GACRC Resource Usage==

The computational resources of the Georgia Advanced Computing Resource Center are to be used in direct support of research programs at the University of Georgia. Support is also provided for classes that teach computational methods, and provide training for high performance computing. The GACRC reserves the right to restrict access to its resources for course work if such work is deemed to present a negative impact to authorized research activities.

GACRC policies supplement UGA’s Policies on the Use of Computers, found at:. http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup

==GACRC Eligibility and Access ==

Access to and use of the computing facilities managed by the Georgia Advanced Computing Resource Center are limited to persons affiliated with the University of Georgia or associated with research projects sponsored by UGA.

Affiliation in this context means faculty, research staff and supervised students of the University of Georgia. Faculty includes persons holding permanent or temporary appointments as well as adjunct faculty, instructors and visiting faculty while in residence at the University. It also includes those persons with faculty status such as research associates, research scientists, post-doctoral researchers and academic and service professionals. Staff includes all those non-faculty persons employed directly by the University in a research-support role. Graduate and undergraduate students who are members of faculty research labs are eligible for accounts as well. For directly affiliated users, accounts on the GACRC computers will remain active as long as the individuals hold the above status.

Access by non-UGA researchers and their students, affiliated to higher-education institutions or non-profit research organizations, for work on research projects conducted in collaboration with UGA Faculty is possible under the guidelines established by the Office of the Vice President for Research and the Office of International Education. A request for access can be forwarded to the GACRC by the UGA Faculty, providing details of the collaboration on a joint research project. Such Affiliate users will be considered part of the UGA Faculty’s group and will be under the Faculty’s responsibility. Affiliate users’ access will be granted for a fixed period of time, according to the expected length of the collaborative project. Renewal of affiliate accounts will be required annually.

All accounts will remain active no more than 30 days following a status change (i.e., leaving the university). Graduate instructional accounts will only remain active for the duration of the semester in which they are actually needed. HOME and PROJECT directories will be archived for at least 90 days, but no longer than 180 days after an account becomes inactive.

Requests for access by individuals other than those listed above should be directed to the Director of the Center for consideration with the GACRC Advisory Committee.

Access will be granted to a specific GACRC resource after appropriate training is undertaken with GACRC staff. Existing access to other GACRC resources is not a sufficient criteria for access to a new resource. No exceptions will be given to the training requirement.

==GACRC Identity Management==

Below are described the procedures for validating the identity of account users.

===UGA Users and Faculty Lab Groups===

*A UGA Faculty must first establish a GACRC group account using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The UGA Faculty can choose or not to obtain a GACRC user account affiliated with his/her group account.

*All directly affiliated persons wanting an account must apply for access to the GACRC using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The applicant must authenticate to the form using his/her MyID. The applicant must specify to which group he/she belongs. Verification with the UGA Faculty responsible for the group’s account will be used in case identity or affiliation needs to be verified.

*Upon acceptance of the application, the user will be notified via e-mail. The applicant’s UGA MyID and password will be used to log into the requested GACRC resources.

===Affiliate Users===

*A recognized Affiliate user must be sponsored by a UGA Faculty member through an established GACRC group account. The UGA Faculty involved in an established collaboration with the Affiliate user, must apply on behalf of the applicant by contacting the GACRC staff.

*A request will be made by the GACRC to EITS to allocate to the Affiliate a UGA MyID.

*Upon acceptance of the Affiliate user application, the Affiliate will be notified via e-mail. The Affiliate’s UGA MyID and password will be used to log into the requested GACRC resources.

===Protection of Passwords===

As described in UGA’s Password Policy, an account holder must never divulge their MyID and password to a third party. Only authorized account holders may access the resources of the GACRC. If a third party is found to be using an account holder’s login with or without the permission of the account holder, the account holder’s access privileges may be revoked at the sole discretion of the GACRC Manager or Director. Enforcement of this policy is under the responsibility of the Office of the Vice President for Information Technology’s Division of Information Security.

More information is found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/passwords/

==GACRC Storage Usage ==

===Some working definitions===

'''Snapshot''' - Copies of files that are stored on the same storage system as the original files. Snapshots are primarily used to recover files that have been accidentally deleted or corrupted within the recent past. Users are able to manage the file recovery tasks. Snapshots are not maintained beyond a defined rotation schedule, i.e., some number of hourly, daily, weekly, and monthly snapshots are kept on the storage system.

'''Backup''' - Copies of files and/or snapshots kept on a storage system (disk/tape) other than the one that the original files reside on. Backups are primarily used to recover files following a catastrophic failure of the original file or storage system. Backups require administrators to perform file system recovery tasks. Like snapshots, backups have a defined rotation schedule.

'''Archive''' - Copies of files that are not currently being accessed, on a resilient storage system dedicated to reliable long-term storage. Archives can be tape-based or disk-based, and typically part of a disaster recovery plan. The files may be copies of original data which is stored elsewhere (individual groups having their own copies), or the archive storage system may be fed by a dedicated "backup" storage system.

'''Active Projects''' – Projects that have on-going computational work being performed with files that are regularly created, accessed or modified.

===Policy Statement for SCRATCH File System===

The SCRATCH file system resides on a high-performance storage device and is to be used uniquely for temporary storage of files in use by actively running compute jobs. Files are to be removed from SCRATCH when a job completes, e.g. can be copied to the PROJECT file system. The SCRATCH file system is not backed up in any way and no snapshots are taken. The SCRATCH filesystem is mounted under /scratch on all the compute nodes, login nodes and data transfer nodes.

Any file that is not accessed or modified by a compute job in a time period no longer than 30 days will be automatically deleted off the SCRATCH file system. Measures circumventing this policy will be monitored and actively discouraged.

There is no storage size quota for SCRATCH usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, the GACRC may request/force users to clean up their SCRATCH directories or reduce temporarily the 30 day limit to a lower limit.

In order to help users identify old files in their /scratch directory, we generate one file per user, every morning. This file, namely /usr/local/var/lustre_stats/$USER.over30d.files.lst (where $USER refers to the user's UGA MyID), contains a list of all the files that this user has in his/her /scratch directory and that have not been accessed in over 30 days. This file provides the full path, the last accessed date, and the size of the files that have not been accessed in over 30 days. Because the /scratch filesystem only updates on-disk file access times every 5 days, the last accessed time reported in that list file is approximate and could be off by up to 5 days. The purging system understands this and will make sure the access time falls within the purge window before deleting the file.

===Policy Statement for WORK File System===

The WORK file system resides on a high-performance storage device and is to be used for storing files that are frequently used by the group for computation. The WORK file system is '''NOT''' subject to the 30-day purge policy. The filesystem usage is controlled using a quota on the size and number of files that can be stored in a lab group's WORK area. Initially each group is given a 500GB and 100,000-file quota. The WORK file system is not backed up in any way and no snapshots are taken. The WORK filesystem is mounted under /work on all the compute nodes, login nodes and data transfer nodes. Each lab group has a directory under the /work directory.

The WORK file system is '''NOT''' subject to the 30-day purge policy. But if there is sufficient space consumption on the storage appliance, we reserve the right to ask users to clean up their WORK area. If the users do not respond in a timely fashion we will purge files beginning with the oldest ones. Please do not use the WORK area to store files long term.

===Policy Statement for HOME File System===

The HOME file system resides on a high-performance storage device and is used for long-term storage of files, typically programs and scripts, needed for analysis on the GACRC computing cluster.

All users have 100GB allocated for their HOME usage. Groups may request a separate 100GB allocation for a directory under /usr/local/lab/, for shared use of common applications, libraries, and scripts.

HOME directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the HOME file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with HOME usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the HOME service.

===Policy Statement for PROJECT File System===

The PROJECT file system resides on lower-performance/higher-capacity storage devices, accessible by all GACRC login and data transfer nodes. PROJECT will not be accessible on Sapelo2's compute nodes. This space is to be used by groups for storage of active projects using Sapelo2. PROJECT should not be seen as a long-term repository, as it is not designed as such. Once a project is completed, data should be moved from the PROJECT space to user-managed storage, freeing up capacity for the next active project.

Access to the PROJECT file system is not supported through NFS to a destination outside of the Boyd Data Center, or through the use of the Samba or CIFS protocols. Transfer protocols available through the data transfer nodes are secure ftp, scp, rsync, GridFTP, amongst others.

More info is found at https://wiki.gacrc.uga.edu/wiki/Transferring_Files.

Each group can request a PROJECT volume with an initial 1TB allocation, accessible by all users ascribed to the group, where the sharing of files will be enabled. Users are encouraged to consider their PROJECT space as the primary area to transfer compute job inputs/outputs. Additional space can be requested by a Faculty on behalf of his/her group, in increments of 1TB.

The GACRC reserves the right to establish a cost-recovery rate for PROJECT storage beyond the initial 1TB allocation. Appropriate communications will take place in such an event.

PROJECT directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the PROJECT file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with PROJECT usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the PROJECT service.

==GACRC Software Policy==

The GACRC maintains a collection of program libraries and software packages to support research computing activities across diverse research domains. While a user can install a software package in their own environment, for the sake of general access across groups, and an appropriate deployment with current libraries, compilers and other dependencies, we strongly recommend that GACRC staff be asked to perform the installation or upgrade.

Any software that requires a signed license or contract, even if it is a click-through agreement, must absolutely be reviewed and handled by the Office of Legal Affairs before being signed by an appropriate signature authority. After the license or contract is accepted and the software is made available, GACRC users must fully comply and use the software in a way that does not violate any terms of the license or contract. Further information on licensing issues can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup/eula

As a matter of policy, the GACRC will not purchase any commercial software for the use of a single group or a small number of groups. Commercial software currently purchased and maintained by the GACRC are of general interest and applicability to the whole UGA research community. The GACRC will however install and maintain a group-purchased commercial software, which complies with the above comments on licenses and contracts.

==Security==

To minimize disruption of service, protect data integrity, conserve facility resources and maximize the effectiveness of staff support, the GACRC maintains strict security requirements for access to GACRC resources. Over time, the enforcement of these requirements will become increasingly strict, with the goal of preventing any access to the GACRC resources by any person or any device that is not in strict compliance with these requirements.

===Operating Systems===

Any computer accessing the GACRC for any purpose must run a currently supported operating system, updated to the latest version and update (patch) levels.

===Anti-Virus Software===

Any computer accessing the GACRC for any purpose must meet minimum levels of anti-virus protection. Any computer used by an account holder must have anti-virus software from a source approved by UGA’s Office of Information Security must have that virus protection activated, and must have automatic updates activated.

More information can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/protect_your_computer

===Suspiciously Behaving Software===

Any software that behaves in a suspicious manner may at any time be terminated and/or deleted from GACRC resources at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Suspiciously Behaving Networks and Devices===

Any connection from any device to the GACRC may be terminated at any time, if the device or the connection or a network to which the device is attached appears to be not in compliance with UGA’s security requirements, is behaving suspiciously, or if a threat emerges requiring termination for intrusion prevention at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Account Holder Responsibility===

The account holder is responsible for diligently monitoring their account and compliance with the GACRC’s operating system, intrusion and virus protection standards. The account holder will be duly notified if GACRC personnel determine that minimum security requirements are not met. Specific actions will be requested of the account holder and compliance to these will be expected in a timely fashion. An account holder’s privileges to use GACRC facilities may be terminated by the GACRC Manager or Director at any time, without notice if, in the opinion of either, the account holder is reluctant or averse to practicing diligence in meeting the GACRC’s minimum requirements for intrusion and/or anti-viral protection.

==Resolving Disagreements about Revocation of Privileges or Provisioning Resources==

The Director of the Georgia Advanced Computing Resource Center has full authority to revoke a user's privileges or deny the request of a new resource allocation. The decision to revoke a user's privileges will be based on, but not limited to, abuses of the UGA Policies on the Use of Computers and/or abuses of the UGA Password Policy.

If an account holder is denied a request for provisioning of GACRC resources or resource privileges are revoked, the account holder’s Department Head may appeal to the Vice President for Research and the Vice President for Information Technology. Their decision will be informed by the Director of the GACRC, the Chief Technology Officer as well as the Associate Chief Information Officer for Information Security. The decision of the Vice President for Research and Vice President for Information Technology is final.

==System Maintenance and Downtime==

===Planned Maintenance===

The GACRC instituted monthly maintenance windows in order to perform maintenance operations requiring system operations to be reduced or interrupted.

The schedule will be as follows:

*The last Wednesday of each month from 10AM to 4PM will be reserved for partial cluster maintenance.

*Twice a year, a two-day shut-down of GACRC services will be scheduled for more complex maintenance operations. These will occur on the last Tuesday and Wednesday of the months of January and July.

These maintenance windows represent periods when the GACRC may choose to drain the queues of running jobs and suspend access to the Sapelo2 cluster, as well as storage devices for maintenance purposes. Interruptions will be kept as brief as possible.

The GACRC will notify all users at least 10 days in advance that a maintenance window will be in effect. The notification will describe the nature and extent (partial or full) of the interruptions of cluster and or storage services. In case a maintenance window has to be extended due to unavoidable technical reasons, adequate communications will be made to all users.

The impact of the outages will vary, and the GACRC will do its best to preserve pending and running jobs, which is often very doable. Nevertheless, users will need to plan their job submissions around the maintenance windows.

===Unplanned Maintenance and System Outage===

From time to time, hardware, software, and/or environmental factors may cause a system or subsystem to malfunction, causing disruption to service. Also, there may be circumstances or events related to possible security issues or intrusions which will cause GACRC staff to take systems offline while the nature of the apparent breach is analyzed and appropriate action is taken.

Whenever possible, account holders will be notified by e-mail of these outages in advance, but that may not always be possible. Account holders will be notified by e-mail if the disruption should last more than 30 minutes.

GACRC staff will strive to preserve the work and/or prevent disruption of jobs in process during such outages. However, there may be circumstances which cause disruption of jobs and loss of data. Users are encouraged to implement methods in their code which minimize the effect of unplanned interruption of a job’s execution, such as checkpoints. Users are also strongly encouraged to maintain copies of files of importance.

==Regulatory Compliance==

The GACRC as an infrastructure and service provider does NOT currently warrant that its practices or facilities meet government-mandated requirements for the storage and protection of sensitive, private or classified information. Users may not store such information on GACRC facilities. In other words, data that falls under HIPAA, FERPA, FISMA or similar regulatory requirements, may not be stored, computed against or otherwise transacted through, or with, GACRC infrastructure.

The GACRC and its users must comply with all existing Federal export control regulations for services and infrastructure. Research groups must agree to NOT install or use any software or data that falls under Export Control regulations. More information on the subject of Export Control is available at the following OVPR website:

http://research.uga.edu/export-control/

Copyrighted materials are prohibited without proper authorization. Additionally, illegal content is prohibited.

Non-compliance with any such Federal requirements might impact GACRC operations or delivery of services and could place the GACRC and UGA at risk. If a research group is found to be in non-compliance, then account access will be immediately suspended, while an investigation by EITS’s Information Security division is instigated.

===Disclosure===

Research groups that are involved in activities that store protected data on GACRC infrastructure must contact immediately the GACRC Director in order to address the issue. Depending on circumstances, accommodations might be possible for such activities.

===Research Data Compliance===

Research Data Management is a critical factor in both obtaining federal funding from the NSF, NIH, DoD and other federal agencies, and in the conduct of funded research. Responsibility in maintaining and preserving research data, as detailed in data management plans submitted to Federal funding agencies, is entirely placed upon the research faculty, post docs, and graduate students conducting the research. The GACRC will help by providing information and assistance, but will not be responsible to ensure compliance with a project’s data management plan.

During the phase of proposal writing, arrangements can be discussed and agreed upon as to the GACRC playing an active role, and ensuring the provision of specific services towards the compliance of a data management plan. Depending on the complexity or the nature of the proposed services, the GACRC might require the purchase of specific hardware/software and/or the availability of a %FTE salary and benefits.

More information on data management plans can be found [http://guides.libs.uga.edu/c.php?g=349946&p=2363161 here].

Policies

2019-01-28T21:53:27Z

Raj76:

==Introduction to GACRC Policies==

The following policies are subject to revision, especially as the GACRC grows in scope and services. Your comments and questions will be useful to our policy formulation and refinement and are actively solicited by the GACRC Advisory Committee.

The GACRC computational infrastructure, including its servers, clusters, data stores, and other related devices are for the exclusive use of authorized users only.

Anyone using these systems expressly consents to abide by the policies of the University of Georgia and the Georgia Advanced Computing Resource Center and, accordingly, is subject to account termination and/or immediate disconnection from GACRC resources.

==GACRC Resource Usage==

The computational resources of the Georgia Advanced Computing Resource Center are to be used in direct support of research programs at the University of Georgia. Support is also provided for classes that teach computational methods, and provide training for high performance computing. The GACRC reserves the right to restrict access to its resources for course work if such work is deemed to present a negative impact to authorized research activities.

GACRC policies supplement UGA’s Policies on the Use of Computers, found at:. http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup

==GACRC Eligibility and Access ==

Access to and use of the computing facilities managed by the Georgia Advanced Computing Resource Center are limited to persons affiliated with the University of Georgia or associated with research projects sponsored by UGA.

Affiliation in this context means faculty, research staff and supervised students of the University of Georgia. Faculty includes persons holding permanent or temporary appointments as well as adjunct faculty, instructors and visiting faculty while in residence at the University. It also includes those persons with faculty status such as research associates, research scientists, post-doctoral researchers and academic and service professionals. Staff includes all those non-faculty persons employed directly by the University in a research-support role. Graduate and undergraduate students who are members of faculty research labs are eligible for accounts as well. For directly affiliated users, accounts on the GACRC computers will remain active as long as the individuals hold the above status.

Access by non-UGA researchers and their students, affiliated to higher-education institutions or non-profit research organizations, for work on research projects conducted in collaboration with UGA Faculty is possible under the guidelines established by the Office of the Vice President for Research and the Office of International Education. A request for access can be forwarded to the GACRC by the UGA Faculty, providing details of the collaboration on a joint research project. Such Affiliate users will be considered part of the UGA Faculty’s group and will be under the Faculty’s responsibility. Affiliate users’ access will be granted for a fixed period of time, according to the expected length of the collaborative project. Renewal of affiliate accounts will be required annually.

All accounts will remain active no more than 30 days following a status change (i.e., leaving the university). Graduate instructional accounts will only remain active for the duration of the semester in which they are actually needed. HOME and PROJECT directories will be archived for at least 90 days, but no longer than 180 days after an account becomes inactive.

Requests for access by individuals other than those listed above should be directed to the Director of the Center for consideration with the GACRC Advisory Committee.

Access will be granted to a specific GACRC resource after appropriate training is undertaken with GACRC staff. Existing access to other GACRC resources is not a sufficient criteria for access to a new resource. No exceptions will be given to the training requirement.

==GACRC Identity Management==

Below are described the procedures for validating the identity of account users.

===UGA Users and Faculty Lab Groups===

*A UGA Faculty must first establish a GACRC group account using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The UGA Faculty can choose or not to obtain a GACRC user account affiliated with his/her group account.

*All directly affiliated persons wanting an account must apply for access to the GACRC using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The applicant must authenticate to the form using his/her MyID. The applicant must specify to which group he/she belongs. Verification with the UGA Faculty responsible for the group’s account will be used in case identity or affiliation needs to be verified.

*Upon acceptance of the application, the user will be notified via e-mail. The applicant’s UGA MyID and password will be used to log into the requested GACRC resources.

===Affiliate Users===

*A recognized Affiliate user must be sponsored by a UGA Faculty member through an established GACRC group account. The UGA Faculty involved in an established collaboration with the Affiliate user, must apply on behalf of the applicant by contacting the GACRC staff.

*A request will be made by the GACRC to EITS to allocate to the Affiliate a UGA MyID.

*Upon acceptance of the Affiliate user application, the Affiliate will be notified via e-mail. The Affiliate’s UGA MyID and password will be used to log into the requested GACRC resources.

===Protection of Passwords===

As described in UGA’s Password Policy, an account holder must never divulge their MyID and password to a third party. Only authorized account holders may access the resources of the GACRC. If a third party is found to be using an account holder’s login with or without the permission of the account holder, the account holder’s access privileges may be revoked at the sole discretion of the GACRC Manager or Director. Enforcement of this policy is under the responsibility of the Office of the Vice President for Information Technology’s Division of Information Security.

More information is found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/passwords/

==GACRC Storage Usage ==

===Some working definitions===

'''Snapshot''' - Copies of files that are stored on the same storage system as the original files. Snapshots are primarily used to recover files that have been accidentally deleted or corrupted within the recent past. Users are able to manage the file recovery tasks. Snapshots are not maintained beyond a defined rotation schedule, i.e., some number of hourly, daily, weekly, and monthly snapshots are kept on the storage system.

'''Backup''' - Copies of files and/or snapshots kept on a storage system (disk/tape) other than the one that the original files reside on. Backups are primarily used to recover files following a catastrophic failure of the original file or storage system. Backups require administrators to perform file system recovery tasks. Like snapshots, backups have a defined rotation schedule.

'''Archive''' - Copies of files that are not currently being accessed, on a resilient storage system dedicated to reliable long-term storage. Archives can be tape-based or disk-based, and typically part of a disaster recovery plan. The files may be copies of original data which is stored elsewhere (individual groups having their own copies), or the archive storage system may be fed by a dedicated "backup" storage system.

'''Active Projects''' – Projects that have on-going computational work being performed with files that are regularly created, accessed or modified.

===Policy Statement for SCRATCH File System===

The SCRATCH file system resides on a high-performance storage device and is to be used uniquely for temporary storage of files in use by actively running compute jobs. Files are to be removed from SCRATCH when a job completes, e.g. can be copied to the PROJECT file system. The SCRATCH file system is not backed up in any way and no snapshots are taken. The SCRATCH filesystem is mounted under /scratch on all the compute nodes, login nodes and data transfer nodes.

Any file that is not accessed or modified by a compute job in a time period no longer than 30 days will be automatically deleted off the SCRATCH file system. Measures circumventing this policy will be monitored and actively discouraged.

There is no storage size quota for SCRATCH usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their SCRATCH directories or reduce temporarily the 30 day limit to a lower limit.

===Policy Statement for WORK File System===

The WORK file system resides on a high-performance storage device and is to be used for storing files that are frequently used by the group for computation. The WORK file system is '''NOT''' subject to the 30 day purge policy. The filesystem usage is controlled using a quota on the size and number of files that can be stored in a lab groups WORK area. Initially each group is given a 500GB and 100,000 files quota. The WORK file system is not backed up in any way and no snapshots are taken. The WORK filesystem is mounted under /work on all the compute nodes, login nodes and data transfer nodes. Each lab group has a directory under the /work directory.

The WORK file system is '''NOT''' subject to the 30 day purge policy. But, if there is sufficient space consumption on the storage appliance we reserve the right to ask users to cleanup their WORK area. If the users do not respond in a timely fashion we will purge starting from old files. Please do not user WORK area to store files long term.

===Policy Statement for HOME File System===

The HOME file system resides on a high-performance storage device and is used for long-term storage of files, typically programs and scripts, needed for analysis on the GACRC computing cluster.

All users have 100GB allocated for their HOME usage. Groups may request a separate 100GB allocation for a directory under /usr/local/lab/, for shared use of common applications, libraries, and scripts.

HOME directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the HOME file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with HOME usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the HOME service.

===Policy Statement for PROJECT File System===

The PROJECT file system resides on lower-performance/higher-capacity storage devices, accessible by all GACRC login and data transfer nodes. PROJECT will not be accessible on Sapelo2's compute nodes. This space is to be used by groups for storage of active projects using Sapelo2. PROJECT should not be seen as a long-term repository, as it is not designed as such. Once a project is completed, data should be moved from the PROJECT space to user-managed storage, freeing up capacity for the next active project.

Access to the PROJECT file system is not supported through NFS to a destination outside of the Boyd Data Center, or through the use of the Samba or CIFS protocols. Transfer protocols available through the data transfer nodes are secure ftp, scp, rsync, GridFTP, amongst others.

More info is found at https://wiki.gacrc.uga.edu/wiki/Transferring_Files.

Each group can request a PROJECT volume with an initial 1TB allocation, accessible by all users ascribed to the group, where the sharing of files will be enabled. Users are encouraged to consider their PROJECT space as the primary area to transfer compute job inputs/outputs. Additional space can be requested by a Faculty on behalf of his/her group, in increments of 1TB.

The GACRC reserves the right to establish a cost-recovery rate for PROJECT storage beyond the initial 1TB allocation. Appropriate communications will take place in such an event.

PROJECT directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the PROJECT file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with PROJECT usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the PROJECT service.

==GACRC Software Policy==

The GACRC maintains a collection of program libraries and software packages to support research computing activities across diverse research domains. While a user can install a software package in their own environment, for the sake of general access across groups, and an appropriate deployment with current libraries, compilers and other dependencies, we strongly recommend that GACRC staff be asked to perform the installation or upgrade.

Any software that requires a signed license or contract, even if it is a click-through agreement, must absolutely be reviewed and handled by the Office of Legal Affairs before being signed by an appropriate signature authority. After the license or contract is accepted and the software is made available, GACRC users must fully comply and use the software in a way that does not violate any terms of the license or contract. Further information on licensing issues can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup/eula

As a matter of policy, the GACRC will not purchase any commercial software for the use of a single group or a small number of groups. Commercial software currently purchased and maintained by the GACRC are of general interest and applicability to the whole UGA research community. The GACRC will however install and maintain a group-purchased commercial software, which complies with the above comments on licenses and contracts.

==Security==

To minimize disruption of service, protect data integrity, conserve facility resources and maximize the effectiveness of staff support, the GACRC maintains strict security requirements for access to GACRC resources. Over time, the enforcement of these requirements will become increasingly strict, with the goal of preventing any access to the GACRC resources by any person or any device that is not in strict compliance with these requirements.

===Operating Systems===

Any computer accessing the GACRC for any purpose must run a currently supported operating system, updated to the latest version and update (patch) levels.

===Anti-Virus Software===

Any computer accessing the GACRC for any purpose must meet minimum levels of anti-virus protection. Any computer used by an account holder must have anti-virus software from a source approved by UGA’s Office of Information Security must have that virus protection activated, and must have automatic updates activated.

More information can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/protect_your_computer

===Suspiciously Behaving Software===

Any software that behaves in a suspicious manner may at any time be terminated and/or deleted from GACRC resources at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Suspiciously Behaving Networks and Devices===

Any connection from any device to the GACRC may be terminated at any time, if the device or the connection or a network to which the device is attached appears to be not in compliance with UGA’s security requirements, is behaving suspiciously, or if a threat emerges requiring termination for intrusion prevention at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Account Holder Responsibility===

The account holder is responsible for diligently monitoring their account and compliance with the GACRC’s operating system, intrusion and virus protection standards. The account holder will be duly notified if GACRC personnel determine that minimum security requirements are not met. Specific actions will be requested of the account holder and compliance to these will be expected in a timely fashion. An account holder’s privileges to use GACRC facilities may be terminated by the GACRC Manager or Director at any time, without notice if, in the opinion of either, the account holder is reluctant or averse to practicing diligence in meeting the GACRC’s minimum requirements for intrusion and/or anti-viral protection.

==Resolving Disagreements about Revocation of Privileges or Provisioning Resources==

The Director of the Georgia Advanced Computing Resource Center has full authority to revoke a user's privileges or deny the request of a new resource allocation. The decision to revoke a user's privileges will be based on, but not limited to, abuses of the UGA Policies on the Use of Computers and/or abuses of the UGA Password Policy.

If an account holder is denied a request for provisioning of GACRC resources or resource privileges are revoked, the account holder’s Department Head may appeal to the Vice President for Research and the Vice President for Information Technology. Their decision will be informed by the Director of the GACRC, the Chief Technology Officer as well as the Associate Chief Information Officer for Information Security. The decision of the Vice President for Research and Vice President for Information Technology is final.

==System Maintenance and Downtime==

===Planned Maintenance===

The GACRC instituted monthly maintenance windows in order to perform maintenance operations requiring system operations to be reduced or interrupted.

The schedule will be as follows:

*The last Wednesday of each month from 10AM to 4PM will be reserved for partial cluster maintenance.

*Twice a year, a two-day shut-down of GACRC services will be scheduled for more complex maintenance operations. These will occur on the last Tuesday and Wednesday of the months of January and July.

These maintenance windows represent periods when the GACRC may choose to drain the queues of running jobs and suspend access to the Sapelo2 cluster, as well as storage devices for maintenance purposes. Interruptions will be kept as brief as possible.

The GACRC will notify all users at least 10 days in advance that a maintenance window will be in effect. The notification will describe the nature and extent (partial or full) of the interruptions of cluster and or storage services. In case a maintenance window has to be extended due to unavoidable technical reasons, adequate communications will be made to all users.

The impact of the outages will vary, and the GACRC will do its best to preserve pending and running jobs, which is often very doable. Nevertheless, users will need to plan their job submissions around the maintenance windows.

===Unplanned Maintenance and System Outage===

From time to time, hardware, software, and/or environmental factors may cause a system or subsystem to malfunction, causing disruption to service. Also, there may be circumstances or events related to possible security issues or intrusions which will cause GACRC staff to take systems offline while the nature of the apparent breach is analyzed and appropriate action is taken.

Whenever possible, account holders will be notified by e-mail of these outages in advance, but that may not always be possible. Account holders will be notified by e-mail if the disruption should last more than 30 minutes.

GACRC staff will strive to preserve the work and/or prevent disruption of jobs in process during such outages. However, there may be circumstances which cause disruption of jobs and loss of data. Users are encouraged to implement methods in their code which minimize the effect of unplanned interruption of a job’s execution, such as checkpoints. Users are also strongly encouraged to maintain copies of files of importance.

==Regulatory Compliance==

The GACRC as an infrastructure and service provider does NOT currently warrant that its practices or facilities meet government-mandated requirements for the storage and protection of sensitive, private or classified information. Users may not store such information on GACRC facilities. In other words, data that falls under HIPAA, FERPA, FISMA or similar regulatory requirements, may not be stored, computed against or otherwise transacted through, or with, GACRC infrastructure.

The GACRC and its users must comply with all existing Federal export control regulations for services and infrastructure. Research groups must agree to NOT install or use any software or data that falls under Export Control regulations. More information on the subject of Export Control is available at the following OVPR website:

http://research.uga.edu/export-control/

Copyrighted materials are prohibited without proper authorization. Additionally, illegal content is prohibited.

Non-compliance with any such Federal requirements might impact GACRC operations or delivery of services and could place the GACRC and UGA at risk. If a research group is found to be in non-compliance, then account access will be immediately suspended, while an investigation by EITS’s Information Security division is instigated.

===Disclosure===

Research groups that are involved in activities that store protected data on GACRC infrastructure must contact immediately the GACRC Director in order to address the issue. Depending on circumstances, accommodations might be possible for such activities.

===Research Data Compliance===

Research Data Management is a critical factor in both obtaining federal funding from the NSF, NIH, DoD and other federal agencies, and in the conduct of funded research. Responsibility in maintaining and preserving research data, as detailed in data management plans submitted to Federal funding agencies, is entirely placed upon the research faculty, post docs, and graduate students conducting the research. The GACRC will help by providing information and assistance, but will not be responsible to ensure compliance with a project’s data management plan.

During the phase of proposal writing, arrangements can be discussed and agreed upon as to the GACRC playing an active role, and ensuring the provision of specific services towards the compliance of a data management plan. Depending on the complexity or the nature of the proposed services, the GACRC might require the purchase of specific hardware/software and/or the availability of a %FTE salary and benefits.

More information on data management plans can be found [http://guides.libs.uga.edu/c.php?g=349946&p=2363161 here].

File System Purging

2019-01-28T21:33:10Z

Raj76: Created page with "==SCRATCH Filesystem== The SCRATCH filesystem (mounted under /scratch directory) is subject to the 30 day file purge rule. Any file that has not be accessed in over than 30 da..."

==SCRATCH Filesystem==
The SCRATCH filesystem (mounted under /scratch directory) is subject to the 30 day file purge rule. Any file that has not be accessed in over than 30 days will be purged. In order to help users identify old file we generate a file, one per user, every morning. This file /usr/local/var/lustre_stats/$USER.over30d.files.lst contains a list of all the files that haven't been accessed in over 30 days. This file provides the full path, last accessed date and file size of the files.
The filesystem updates the file access time once in 5 days. The last accessed time in the file is approximate and could be off by up to 5 days. The purging systems understands this and will make sure the access time falls within the purge window before deleting the file.
During the purge only files are deleted, the directories will be left alone. Please let us know if you have any questions.

Policies

2019-01-28T21:09:01Z

Raj76:

==Introduction to GACRC Policies==

The following policies are subject to revision, especially as the GACRC grows in scope and services. Your comments and questions will be useful to our policy formulation and refinement and are actively solicited by the GACRC Advisory Committee.

The GACRC computational infrastructure, including its servers, clusters, data stores, and other related devices are for the exclusive use of authorized users only.

Anyone using these systems expressly consents to abide by the policies of the University of Georgia and the Georgia Advanced Computing Resource Center and, accordingly, is subject to account termination and/or immediate disconnection from GACRC resources.

==GACRC Resource Usage==

The computational resources of the Georgia Advanced Computing Resource Center are to be used in direct support of research programs at the University of Georgia. Support is also provided for classes that teach computational methods, and provide training for high performance computing. The GACRC reserves the right to restrict access to its resources for course work if such work is deemed to present a negative impact to authorized research activities.

GACRC policies supplement UGA’s Policies on the Use of Computers, found at:. http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup

==GACRC Eligibility and Access ==

Access to and use of the computing facilities managed by the Georgia Advanced Computing Resource Center are limited to persons affiliated with the University of Georgia or associated with research projects sponsored by UGA.

Affiliation in this context means faculty, research staff and supervised students of the University of Georgia. Faculty includes persons holding permanent or temporary appointments as well as adjunct faculty, instructors and visiting faculty while in residence at the University. It also includes those persons with faculty status such as research associates, research scientists, post-doctoral researchers and academic and service professionals. Staff includes all those non-faculty persons employed directly by the University in a research-support role. Graduate and undergraduate students who are members of faculty research labs are eligible for accounts as well. For directly affiliated users, accounts on the GACRC computers will remain active as long as the individuals hold the above status.

Access by non-UGA researchers and their students, affiliated to higher-education institutions or non-profit research organizations, for work on research projects conducted in collaboration with UGA Faculty is possible under the guidelines established by the Office of the Vice President for Research and the Office of International Education. A request for access can be forwarded to the GACRC by the UGA Faculty, providing details of the collaboration on a joint research project. Such Affiliate users will be considered part of the UGA Faculty’s group and will be under the Faculty’s responsibility. Affiliate users’ access will be granted for a fixed period of time, according to the expected length of the collaborative project. Renewal of affiliate accounts will be required annually.

All accounts will remain active no more than 30 days following a status change (i.e., leaving the university). Graduate instructional accounts will only remain active for the duration of the semester in which they are actually needed. HOME and PROJECT directories will be archived for at least 90 days, but no longer than 180 days after an account becomes inactive.

Requests for access by individuals other than those listed above should be directed to the Director of the Center for consideration with the GACRC Advisory Committee.

Access will be granted to a specific GACRC resource after appropriate training is undertaken with GACRC staff. Existing access to other GACRC resources is not a sufficient criteria for access to a new resource. No exceptions will be given to the training requirement.

==GACRC Identity Management==

Below are described the procedures for validating the identity of account users.

===UGA Users and Faculty Lab Groups===

*A UGA Faculty must first establish a GACRC group account using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The UGA Faculty can choose or not to obtain a GACRC user account affiliated with his/her group account.

*All directly affiliated persons wanting an account must apply for access to the GACRC using the instructions provided on the GACRC website (http://gacrc.uga.edu/accounts). The applicant must authenticate to the form using his/her MyID. The applicant must specify to which group he/she belongs. Verification with the UGA Faculty responsible for the group’s account will be used in case identity or affiliation needs to be verified.

*Upon acceptance of the application, the user will be notified via e-mail. The applicant’s UGA MyID and password will be used to log into the requested GACRC resources.

===Affiliate Users===

*A recognized Affiliate user must be sponsored by a UGA Faculty member through an established GACRC group account. The UGA Faculty involved in an established collaboration with the Affiliate user, must apply on behalf of the applicant by contacting the GACRC staff.

*A request will be made by the GACRC to EITS to allocate to the Affiliate a UGA MyID.

*Upon acceptance of the Affiliate user application, the Affiliate will be notified via e-mail. The Affiliate’s UGA MyID and password will be used to log into the requested GACRC resources.

===Protection of Passwords===

As described in UGA’s Password Policy, an account holder must never divulge their MyID and password to a third party. Only authorized account holders may access the resources of the GACRC. If a third party is found to be using an account holder’s login with or without the permission of the account holder, the account holder’s access privileges may be revoked at the sole discretion of the GACRC Manager or Director. Enforcement of this policy is under the responsibility of the Office of the Vice President for Information Technology’s Division of Information Security.

More information is found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/passwords/

==GACRC Storage Usage ==

===Some working definitions===

'''Snapshot''' - Copies of files that are stored on the same storage system as the original files. Snapshots are primarily used to recover files that have been accidentally deleted or corrupted within the recent past. Users are able to manage the file recovery tasks. Snapshots are not maintained beyond a defined rotation schedule, i.e., some number of hourly, daily, weekly, and monthly snapshots are kept on the storage system.

'''Backup''' - Copies of files and/or snapshots kept on a storage system (disk/tape) other than the one that the original files reside on. Backups are primarily used to recover files following a catastrophic failure of the original file or storage system. Backups require administrators to perform file system recovery tasks. Like snapshots, backups have a defined rotation schedule.

'''Archive''' - Copies of files that are not currently being accessed, on a resilient storage system dedicated to reliable long-term storage. Archives can be tape-based or disk-based, and typically part of a disaster recovery plan. The files may be copies of original data which is stored elsewhere (individual groups having their own copies), or the archive storage system may be fed by a dedicated "backup" storage system.

'''Active Projects''' – Projects that have on-going computational work being performed with files that are regularly created, accessed or modified.

===Policy Statement for SCRATCH File System===

The SCRATCH file system resides on a high-performance storage device and is to be used uniquely for temporary storage of files in use by actively running compute jobs. Files are to be removed from SCRATCH when a job completes, e.g. can be copied to the PROJECT file system. The SCRATCH file system is not backed up in any way and no snapshots are taken. The SCRATCH filesystem is mounted under /scratch on all the compute nodes, login nodes and data transfer nodes.

Any file that is not accessed or modified by a compute job in a time period no longer than 30 days will be automatically deleted off the SCRATCH file system. Measures circumventing this policy will be monitored and actively discouraged.

There is no storage size quota for SCRATCH usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their SCRATCH directories or reduce temporarily the 30 day limit to a lower limit.

===Policy Statement for HOME File System===

The HOME file system resides on a high-performance storage device and is used for long-term storage of files, typically programs and scripts, needed for analysis on the GACRC computing cluster.

All users have 100GB allocated for their HOME usage. Groups may request a separate 100GB allocation for a directory under /usr/local/lab/, for shared use of common applications, libraries, and scripts.

HOME directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the HOME file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with HOME usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the HOME service.

===Policy Statement for PROJECT File System===

The PROJECT file system resides on lower-performance/higher-capacity storage devices, accessible by all GACRC login and data transfer nodes. PROJECT will not be accessible on Sapelo2's compute nodes. This space is to be used by groups for storage of active projects using Sapelo2. PROJECT should not be seen as a long-term repository, as it is not designed as such. Once a project is completed, data should be moved from the PROJECT space to user-managed storage, freeing up capacity for the next active project.

Access to the PROJECT file system is not supported through NFS to a destination outside of the Boyd Data Center, or through the use of the Samba or CIFS protocols. Transfer protocols available through the data transfer nodes are secure ftp, scp, rsync, GridFTP, amongst others.

More info is found at https://wiki.gacrc.uga.edu/wiki/Transferring_Files.

Each group can request a PROJECT volume with an initial 1TB allocation, accessible by all users ascribed to the group, where the sharing of files will be enabled. Users are encouraged to consider their PROJECT space as the primary area to transfer compute job inputs/outputs. Additional space can be requested by a Faculty on behalf of his/her group, in increments of 1TB.

The GACRC reserves the right to establish a cost-recovery rate for PROJECT storage beyond the initial 1TB allocation. Appropriate communications will take place in such an event.

PROJECT directories will have daily, weekly and up to 3 monthly snapshots kept on the same storage unit to protect against accidental file deletion. Currently, the GACRC is not able to perform any backup of the PROJECT file system onto another storage device. Users are strongly encouraged to make their own copies of critical files, while accepting any risks associated with PROJECT usage. Appropriate communications will take place once a backup service is enabled.

Snapshot retention, data purge and quota allocation policies are subject to change based on available storage capacity, users’ demand, equipment condition and availability, as well as any other conditions that might affect the provision of the PROJECT service.

==GACRC Software Policy==

The GACRC maintains a collection of program libraries and software packages to support research computing activities across diverse research domains. While a user can install a software package in their own environment, for the sake of general access across groups, and an appropriate deployment with current libraries, compilers and other dependencies, we strongly recommend that GACRC staff be asked to perform the installation or upgrade.

Any software that requires a signed license or contract, even if it is a click-through agreement, must absolutely be reviewed and handled by the Office of Legal Affairs before being signed by an appropriate signature authority. After the license or contract is accepted and the software is made available, GACRC users must fully comply and use the software in a way that does not violate any terms of the license or contract. Further information on licensing issues can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup/eula

As a matter of policy, the GACRC will not purchase any commercial software for the use of a single group or a small number of groups. Commercial software currently purchased and maintained by the GACRC are of general interest and applicability to the whole UGA research community. The GACRC will however install and maintain a group-purchased commercial software, which complies with the above comments on licenses and contracts.

==Security==

To minimize disruption of service, protect data integrity, conserve facility resources and maximize the effectiveness of staff support, the GACRC maintains strict security requirements for access to GACRC resources. Over time, the enforcement of these requirements will become increasingly strict, with the goal of preventing any access to the GACRC resources by any person or any device that is not in strict compliance with these requirements.

===Operating Systems===

Any computer accessing the GACRC for any purpose must run a currently supported operating system, updated to the latest version and update (patch) levels.

===Anti-Virus Software===

Any computer accessing the GACRC for any purpose must meet minimum levels of anti-virus protection. Any computer used by an account holder must have anti-virus software from a source approved by UGA’s Office of Information Security must have that virus protection activated, and must have automatic updates activated.

More information can be found at the following EITS website:

http://eits.uga.edu/access_and_security/infosec/protect_your_computer

===Suspiciously Behaving Software===

Any software that behaves in a suspicious manner may at any time be terminated and/or deleted from GACRC resources at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Suspiciously Behaving Networks and Devices===

Any connection from any device to the GACRC may be terminated at any time, if the device or the connection or a network to which the device is attached appears to be not in compliance with UGA’s security requirements, is behaving suspiciously, or if a threat emerges requiring termination for intrusion prevention at the sole discretion of the GACRC’s systems administrator(s), manager, Director, or EITS information security staff.

===Account Holder Responsibility===

The account holder is responsible for diligently monitoring their account and compliance with the GACRC’s operating system, intrusion and virus protection standards. The account holder will be duly notified if GACRC personnel determine that minimum security requirements are not met. Specific actions will be requested of the account holder and compliance to these will be expected in a timely fashion. An account holder’s privileges to use GACRC facilities may be terminated by the GACRC Manager or Director at any time, without notice if, in the opinion of either, the account holder is reluctant or averse to practicing diligence in meeting the GACRC’s minimum requirements for intrusion and/or anti-viral protection.

==Resolving Disagreements about Revocation of Privileges or Provisioning Resources==

The Director of the Georgia Advanced Computing Resource Center has full authority to revoke a user's privileges or deny the request of a new resource allocation. The decision to revoke a user's privileges will be based on, but not limited to, abuses of the UGA Policies on the Use of Computers and/or abuses of the UGA Password Policy.

If an account holder is denied a request for provisioning of GACRC resources or resource privileges are revoked, the account holder’s Department Head may appeal to the Vice President for Research and the Vice President for Information Technology. Their decision will be informed by the Director of the GACRC, the Chief Technology Officer as well as the Associate Chief Information Officer for Information Security. The decision of the Vice President for Research and Vice President for Information Technology is final.

==System Maintenance and Downtime==

===Planned Maintenance===

The GACRC instituted monthly maintenance windows in order to perform maintenance operations requiring system operations to be reduced or interrupted.

The schedule will be as follows:

*The last Wednesday of each month from 10AM to 4PM will be reserved for partial cluster maintenance.

*Twice a year, a two-day shut-down of GACRC services will be scheduled for more complex maintenance operations. These will occur on the last Tuesday and Wednesday of the months of January and July.

These maintenance windows represent periods when the GACRC may choose to drain the queues of running jobs and suspend access to the Sapelo2 cluster, as well as storage devices for maintenance purposes. Interruptions will be kept as brief as possible.

The GACRC will notify all users at least 10 days in advance that a maintenance window will be in effect. The notification will describe the nature and extent (partial or full) of the interruptions of cluster and or storage services. In case a maintenance window has to be extended due to unavoidable technical reasons, adequate communications will be made to all users.

The impact of the outages will vary, and the GACRC will do its best to preserve pending and running jobs, which is often very doable. Nevertheless, users will need to plan their job submissions around the maintenance windows.

===Unplanned Maintenance and System Outage===

From time to time, hardware, software, and/or environmental factors may cause a system or subsystem to malfunction, causing disruption to service. Also, there may be circumstances or events related to possible security issues or intrusions which will cause GACRC staff to take systems offline while the nature of the apparent breach is analyzed and appropriate action is taken.

Whenever possible, account holders will be notified by e-mail of these outages in advance, but that may not always be possible. Account holders will be notified by e-mail if the disruption should last more than 30 minutes.

GACRC staff will strive to preserve the work and/or prevent disruption of jobs in process during such outages. However, there may be circumstances which cause disruption of jobs and loss of data. Users are encouraged to implement methods in their code which minimize the effect of unplanned interruption of a job’s execution, such as checkpoints. Users are also strongly encouraged to maintain copies of files of importance.

==Regulatory Compliance==

The GACRC as an infrastructure and service provider does NOT currently warrant that its practices or facilities meet government-mandated requirements for the storage and protection of sensitive, private or classified information. Users may not store such information on GACRC facilities. In other words, data that falls under HIPAA, FERPA, FISMA or similar regulatory requirements, may not be stored, computed against or otherwise transacted through, or with, GACRC infrastructure.

The GACRC and its users must comply with all existing Federal export control regulations for services and infrastructure. Research groups must agree to NOT install or use any software or data that falls under Export Control regulations. More information on the subject of Export Control is available at the following OVPR website:

http://research.uga.edu/export-control/

Copyrighted materials are prohibited without proper authorization. Additionally, illegal content is prohibited.

Non-compliance with any such Federal requirements might impact GACRC operations or delivery of services and could place the GACRC and UGA at risk. If a research group is found to be in non-compliance, then account access will be immediately suspended, while an investigation by EITS’s Information Security division is instigated.

===Disclosure===

Research groups that are involved in activities that store protected data on GACRC infrastructure must contact immediately the GACRC Director in order to address the issue. Depending on circumstances, accommodations might be possible for such activities.

===Research Data Compliance===

Research Data Management is a critical factor in both obtaining federal funding from the NSF, NIH, DoD and other federal agencies, and in the conduct of funded research. Responsibility in maintaining and preserving research data, as detailed in data management plans submitted to Federal funding agencies, is entirely placed upon the research faculty, post docs, and graduate students conducting the research. The GACRC will help by providing information and assistance, but will not be responsible to ensure compliance with a project’s data management plan.

During the phase of proposal writing, arrangements can be discussed and agreed upon as to the GACRC playing an active role, and ensuring the provision of specific services towards the compliance of a data management plan. Depending on the complexity or the nature of the proposed services, the GACRC might require the purchase of specific hardware/software and/or the availability of a %FTE salary and benefits.

More information on data management plans can be found [http://guides.libs.uga.edu/c.php?g=349946&p=2363161 here].

Georgia Advanced Computing Resource Center

2018-11-23T21:27:35Z

Raj76:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"









<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Teaching Cluster Online </span></div>

<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Sapelo2 Cluster Online </span></div>

<div style="width=100%; margin:0; background:#FF9F40; font-size:120%; font-weight:bold; border:1px solid #FF9F40; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Sapelo decommissioned</span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> IMPORTANT NEWS </div>
The following is an important notice for all of our current users.




* ''' [[Upcoming Maintenance - November 16-22, 2018]]'''

* ''' [[Changes implemented during November 16-22, 2018 maintenance]]'''

* [[Sapelo2 Frequently Asked Questions]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]
* [[Training]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Sapelo and Sapelo2 comparison]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Job and Data Management </div>
Information on how to run jobs and data management.
* [[Running Jobs]]
* [[Monitoring Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]
* [[Troubleshooting]]
* [[Best Practices]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]
* [[Build Applications]]
* [[Zaney]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>

Lustre1 archive

2018-09-17T19:17:30Z

Raj76:

= List of archived lustre1 user directories and their location =
<pre>
/lustre_archive/ajmlab:
total 128
drwxr-x---. 3 root ajmlab 4096 Sep 16 23:23 benowitz
drwxr-x---. 3 root ajmlab 4096 Sep 16 23:23 cbc83

/lustre_archive/akblab:
total 64
drwxr-x---. 3 root akblab 4096 Sep 16 23:24 spakala

/lustre_archive/cjtlab:
total 128
drwxr-x---. 3 root cjtlab 4096 Sep 16 23:24 pbreen
drwxr-x---. 3 root cjtlab 4096 Sep 16 23:24 px45794

/lustre_archive/ddlab:
total 64
drwxr-x---. 3 root ddlab 4096 Sep 16 23:24 cmcknig

/lustre_archive/dhlab:
total 64
drwxr-x---. 3 root dhlab 4096 Sep 16 23:24 dwhill

/lustre_archive/drplab:
total 64
drwxr-x---. 3 root drplab 4096 Sep 16 23:24 sa22744

/lustre_archive/eaklab:
total 64
drwxr-x---. 3 root eaklab 4096 Sep 16 23:24 erinef

/lustre_archive/eflab:
total 64
drwxr-x---. 3 root eflab 4096 Sep 16 23:24 ac22816

/lustre_archive/gedlab:
total 64
drwxr-x---. 3 root gedlab 4096 Sep 16 23:24 rs07597

/lustre_archive/grblab:
total 256
drwxr-x---. 3 root grblab 4096 Sep 16 23:24 cv32244
drwxr-x---. 3 root grblab 4096 Sep 16 23:24 jb14470
drwxr-x---. 3 root grblab 4096 Sep 16 23:24 kw43875
drwxr-x---. 3 root grblab 4096 Sep 16 23:24 ml20051

/lustre_archive/jbglab:
total 64
drwxr-x---. 3 root jbglab 4096 Sep 17 10:22 vnflint

/lustre_archive/jlblab:
total 64
drwxr-x---. 3 root jlblab 4096 Sep 16 23:24 wanghao

/lustre_archive/jlmlab:
total 64
drwxr-x---. 3 root jlmlab 4096 Sep 16 23:24 spakala

/lustre_archive/jmblab:
total 128
drwxr-x---. 3 root jmblab 4096 Sep 16 23:24 ad42448
drwxr-x---. 3 root jmblab 4096 Sep 17 10:49 icalic

/lustre_archive/jmslab:
total 64
drwxr-x---. 3 root jmslab 4096 Sep 16 23:24 stewartm

/lustre_archive/kmdlab:
total 320
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:03 fp84120
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:02 hw85460
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:02 limabel
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:02 natgeo
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:03 schwoyer

/lustre_archive/krlab:
total 64
drwxr-x---. 3 root krlab 4096 Sep 16 23:25 dgotzek

/lustre_archive/kullab:
total 64
drwxr-x---. 3 root kullab 4096 Sep 16 23:25 lc82542

/lustre_archive/madlab:
total 128
drwxr-x---. 3 root madlab 4096 Sep 16 23:25 abrathwa
drwxr-x---. 3 root madlab 4096 Sep 16 23:25 sg79548

/lustre_archive/mblab:
total 128
drwxr-x---. 3 root mblab 4096 Sep 16 23:25 busara
drwxr-x---. 3 root mblab 4096 Sep 16 23:25 joykai

/lustre_archive/mwjlab:
total 64
drwxr-x---. 3 root mwjlab 4096 Sep 16 23:25 cmleyson

/lustre_archive/mzlab:
total 64
drwxr-x---. 3 root mzlab 4096 Sep 16 23:25 yhg926

/lustre_archive/nllab:
total 64
drwxr-x---. 3 root nllab 4096 Sep 16 23:25 apjaeger

/lustre_archive/qhlab:
total 64
drwxr-x---. 3 root qhlab 4096 Sep 16 23:25 xy96296

/lustre_archive/qzlab:
total 64
drwxr-x---. 3 root qzlab 4096 Sep 16 23:25 gas52607

/lustre_archive/rdalab:
total 64
drwxr-x---. 3 root rdalab 4096 Sep 16 23:25 sy49293

/lustre_archive/rjslab:
total 128
drwxr-x---. 3 root rjslab 4096 Sep 16 23:25 klcates1
drwxr-x---. 3 root rjslab 4096 Sep 16 23:25 rw37111

/lustre_archive/rmclab:
total 64
drwxr-x---. 3 root rmclab 4096 Sep 16 23:25 hluo76

/lustre_archive/sbjlab:
total 64
drwxr-x---. 3 root sbjlab 4096 Sep 16 23:25 yy87095

/lustre_archive/sewlab:
total 64
drwxr-x---. 3 root sewlab 4096 Sep 16 23:25 lmk56078

/lustre_archive/simulphy:
total 192
drwxr-x---. 3 root simulphy 4096 Sep 16 23:25 am1993
drwxr-x---. 3 root simulphy 4096 Sep 16 23:25 locrian5
drwxr-x---. 3 root simulphy 4096 Sep 16 23:25 sgjerry

/lustre_archive/slllab:
total 64
drwxr-x---. 3 root slllab 4096 Sep 16 23:25 rbeasley

/lustre_archive/smblab:
total 64
drwxr-x---. 3 root smblab 4096 Sep 16 23:25 jagadish

/lustre_archive/tcglab:
total 64
drwxr-x---. 3 root tcglab 4096 Sep 16 23:25 bgao

/lustre_archive/xqwlab:
total 64
drwxr-x---. 3 root xqwlab 4096 Sep 16 23:25 hengchen

/lustre_archive/xynlab:
total 192
drwxr-x---. 3 root xynlab 4096 Sep 16 23:25 cmzuo
drwxr-x---. 3 root xynlab 4096 Sep 16 23:25 hshcao
drwxr-x---. 3 root xynlab 4096 Sep 16 23:25 sl72649

/lustre_archive/zrxlab:
total 64
drwxr-x---. 3 root zrxlab 4096 Sep 16 23:25 ms75422
</pre>

Lustre1 archive

2018-09-17T19:11:20Z

Raj76: Created page with "= List of archived lustre1 user directories and their location = <pre> /lustre_archive/ajmlab: total 128 drwx------. 3 root root 4096 Sep 16 23:23 benowitz drwx------. 3 root..."

= List of archived lustre1 user directories and their location =
<pre>
/lustre_archive/ajmlab:
total 128
drwx------. 3 root root 4096 Sep 16 23:23 benowitz
drwx------. 3 root root 4096 Sep 16 23:23 cbc83

/lustre_archive/akblab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 spakala

/lustre_archive/cjtlab:
total 128
drwx------. 3 root root 4096 Sep 16 23:24 pbreen
drwx------. 3 root root 4096 Sep 16 23:24 px45794

/lustre_archive/ddlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 cmcknig

/lustre_archive/dhlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 dwhill

/lustre_archive/drplab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 sa22744

/lustre_archive/eaklab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 erinef

/lustre_archive/eflab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 ac22816

/lustre_archive/gedlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 rs07597

/lustre_archive/grblab:
total 256
drwx------. 3 root root 4096 Sep 16 23:24 cv32244
drwx------. 3 root root 4096 Sep 16 23:24 jb14470
drwx------. 3 root root 4096 Sep 16 23:24 kw43875
drwx------. 3 root root 4096 Sep 16 23:24 ml20051

/lustre_archive/jbglab:
total 64
drwx------. 3 root root 4096 Sep 17 10:22 vnflint

/lustre_archive/jlblab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 wanghao

/lustre_archive/jlmlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 spakala

/lustre_archive/jmblab:
total 128
drwx------. 3 root root 4096 Sep 16 23:24 ad42448
drwx------. 3 root root 4096 Sep 17 10:49 icalic

/lustre_archive/jmslab:
total 64
drwx------. 3 root root 4096 Sep 16 23:24 stewartm

/lustre_archive/kmdlab:
total 320
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:03 fp84120
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:02 hw85460
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:02 limabel
drwxr-x---. 3 root kmdlab 4096 Sep 17 12:02 natgeo
drwx------. 3 root root 4096 Sep 17 12:03 schwoyer

/lustre_archive/krlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 dgotzek

/lustre_archive/kullab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 lc82542

/lustre_archive/madlab:
total 128
drwx------. 3 root root 4096 Sep 16 23:25 abrathwa
drwx------. 3 root root 4096 Sep 16 23:25 sg79548

/lustre_archive/mblab:
total 128
drwx------. 3 root root 4096 Sep 16 23:25 busara
drwx------. 3 root root 4096 Sep 16 23:25 joykai

/lustre_archive/mwjlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 cmleyson

/lustre_archive/mzlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 yhg926

/lustre_archive/nllab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 apjaeger

/lustre_archive/qhlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 xy96296

/lustre_archive/qzlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 gas52607

/lustre_archive/rdalab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 sy49293

/lustre_archive/rjslab:
total 128
drwx------. 3 root root 4096 Sep 16 23:25 klcates1
drwx------. 3 root root 4096 Sep 16 23:25 rw37111

/lustre_archive/rmclab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 hluo76

/lustre_archive/sbjlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 yy87095

/lustre_archive/sewlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 lmk56078

/lustre_archive/simulphy:
total 192
drwx------. 3 root root 4096 Sep 16 23:25 am1993
drwx------. 3 root root 4096 Sep 16 23:25 locrian5
drwx------. 3 root root 4096 Sep 16 23:25 sgjerry

/lustre_archive/slllab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 rbeasley

/lustre_archive/smblab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 jagadish

/lustre_archive/tcglab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 bgao

/lustre_archive/xqwlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 hengchen

/lustre_archive/xynlab:
total 192
drwx------. 3 root root 4096 Sep 16 23:25 cmzuo
drwx------. 3 root root 4096 Sep 16 23:25 hshcao
drwx------. 3 root root 4096 Sep 16 23:25 sl72649

/lustre_archive/zrxlab:
total 64
drwx------. 3 root root 4096 Sep 16 23:25 ms75422
</pre>

Borg backup

2018-09-17T19:04:26Z

Raj76:

= Introduction =
The [https://borgbackup.readthedocs.io/en/stable/index.html Borg backup software] is a deduplicating backup program. It supports compression using various codecs as well.

The borg backup software is installed on all xfer nodes. You can use this software to archive your files. If you have a large amount of data this could possibly reduce storage by deduplicating and compressing your data. This is especially true for large genomic datasets.

== Example showing archiving lustre project directory ==
=== Initializing a borg repository ===
Borg uses a repository (a special directory in your filesystem) to store the backup data. You will have to initialize a repository before writing backup data to it.
You can create the repository in any filesystem. In this example I am storing the repository in my project filesystem.
Run the following command to initialize the repository.
<pre>
$ borg init --encryption none /project/gclab/raj76/my_project
</pre>
If successful the command will not return/print anything.
=== Creating a backup ===
The next step is to create a backup to the repository.
Run the following command to create a backup with deduplication and fast compression.
<pre>
$ borg create -s --compression auto,lz4 /project/gclab/raj76/my_project::lustre1-{now} /lustre1/raj76/my_project
</pre>
In the above command ::lustre1-{now} is the name of the archive that will be created by this backup. The {now} shorthand tells borg backup to use the current time stamp as part of the archive name. This will be useful later to identify archives.
The above command should be run in a screen or tmux session as this will take a while for large datasets.
=== Checking the repository ===
You can check the repository for consistency using the following command.
<pre>
$ borg check /project/gclab/raj76/my_project
</pre>
You can list the contents of the repository using the following command. This will lists the archives in the repo.
<pre>
$ borg list /project/gclab/raj76/my_project
lustre-2018-09-01T23:26:59 Sat, 2018-09-01 23:27:00 [d2365ef51f205d20428c3df74bc9ae9ffadb779283f806722acc6e282b6abd27]
</pre>
You can list the files in each archive using a similar command.
<pre>
$ borg list /project/gclab/raj76/my_project::lustre-2018-09-01T23:26:59
drwxrwx--- raj76 jlmlab 0 Wed, 2018-04-11 23:39:20 scratch/raj76/Dogwood
drwx------ raj76 jlmlab 0 Wed, 2017-08-30 08:40:08 scratch/raj76/Dogwood/PASA_runs
drwxr-xr-x raj76 jlmlab 0 Tue, 2017-09-05 01:37:31 scratch/raj76/Dogwood/PASA_runs/run_2
drwxr-xr-x raj76 jlmlab 0 Mon, 2017-09-04 15:54:19 scratch/raj76/Dogwood/PASA_runs/run_2/compreh_init_build
-rw-r--r-- raj76 jlmlab 141790367 Wed, 2017-08-30 22:53:47 scratch/raj76/Dogwood/PASA_runs/run_2/StringTie-Merged.gtf.fasta
-rw-r--r-- raj76 jlmlab 80133357 Mon, 2017-09-04 17:18:08 scratch/raj76/Dogwood/PASA_runs/run_2/pasa_spakala_201.assemblies.fasta.transdecoder.pep
-rw------- raj76 jlmlab 63993718 Tue, 2017-09-05 01:37:46 scratch/raj76/Dogwood/PASA_runs/run_2/pasa_spakala_201.assemblies.fasta.transdecoder.pep-longest-ORFs.fasta
drwxr-xr-x raj76 jlmlab 0 Wed, 2017-08-30 21:44:27 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir
-rw-r--r-- raj76 jlmlab 37565886 Wed, 2017-08-30 13:37:15 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.0.fa
-rw-r--r-- raj76 jlmlab 0 Wed, 2017-08-30 21:29:35 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.0.fa.pslx.completed
-rw-r--r-- raj76 jlmlab 51667326 Wed, 2017-08-30 21:36:05 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.0.fa.pslx.top_1
-rw-r--r-- raj76 jlmlab 0 Wed, 2017-08-30 21:36:48 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.0.fa.pslx.top_1.completed
-rw-r--r-- raj76 jlmlab 36687377 Wed, 2017-08-30 13:37:55 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.111180.fa
-rw-r--r-- raj76 jlmlab 0 Wed, 2017-08-30 21:29:35 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.111180.fa.pslx.completed
-rw-r--r-- raj76 jlmlab 49636612 Wed, 2017-08-30 21:36:01 scratch/raj76/Dogwood/PASA_runs/run_2/blat_out_dir/partition.111180.fa.pslx.top_1
</pre>
=== Accessing the data in the backup archives ===
One of the nice feature of borg backup is the ability to mount a backup archive as a FUSE filesystem and access the files in the backup as if they are in a real directory tree.
Create a mount point in your home directory and mount the borg backup repo using the following commands.
<pre>
$ mkdir /home/raj76/backup_mount
$ borg mount /project/gclab/raj76/my_project /home/raj76/backup_mount
$ cd /home/raj76/backup_mount
$ ls -l
total 0
drwxr-xr-x. 1 raj76 rccstaff 0 Sep 1 23:27 lustre-2018-09-01T23:26:59
</pre>
You can browse the archive directories and access the files in the backup and if need be copy the files to an area outside of the backup.
The borg backup mount is a read-only mount and files and directories cannot be changed or deleted from the archive.
You can unmount the backup using the following command.
<pre>
$ cd
$ borg umount /home/raj76/backup_mount
</pre>

=== More information ===
Please refer to the Borg backup manual for more information and detailed explanation of the available options.
https://borgbackup.readthedocs.io/en/stable/index.html

Borg backup

2018-09-17T19:01:58Z

Raj76:

Borg backup

2018-09-17T19:00:38Z

Raj76:

Borg backup

2018-09-17T18:41:58Z

Raj76: Created page with "= Introduction = The [https://borgbackup.readthedocs.io/en/stable/index.html Borg backup software] is a deduplicating backup program. It supports compression using various cod..."

Singularity

2017-12-22T20:35:07Z

Raj76: /* Preparing the database */

== Using Singularity containers on Sapelo2 ==

The Sapelo2 cluster has the ability to run Singularity containers. [http://singularity.lbl.gov/ Singularity] containers are docker like containers that are HPC friendly.

=== Loading Singularity ===
Singularity is installed on all compute nodes on Sapelo2. In order to access that executables you have to load the Singularity module.
<pre>
$ module load Singularity
$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...

GLOBAL OPTIONS:
-d|--debug Print debugging information
-h|--help Display usage summary
-s|--silent Only print errors
-q|--quiet Suppress all normal output
--version Show application version
-v|--verbose Increase verbosity +1
-x|--sh-debug Print shell wrapper debugging information

GENERAL COMMANDS:
help Show additional help for a command or container
selftest Run some self tests for singularity install

CONTAINER USAGE COMMANDS:
exec Execute a command within container
run Launch a runscript within container
shell Run a Bourne shell within container
test Launch a testscript within container

CONTAINER MANAGEMENT COMMANDS:
apps List available apps within a container
bootstrap *Deprecated* use build instead
build Build a new Singularity container
check Perform container lint checks
inspect Display container's metadata
mount Mount a Singularity container image
pull Pull a Singularity/Docker container to $PWD

COMMAND GROUPS:
image Container image command group
instance Persistent instance command group

CONTAINER USAGE OPTIONS:
see singularity help <command>

For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/

</pre>

=== Using BioContainer containers ===
[https://biocontainers.pro/ BioContainers] is an open source and community-driven framework which provides system-agnostic executable environments for bioinformatics software. BioContainers framework allows software to be installed and executed under an isolated and controllable environment.

==== Blast container example ====
This example is based on the BioContainer example at http://biocontainers.pro/docs/101/running-example/ webpage.

===== Pulling a container =====
In order to use a pre built container, one has to "pull" the container from the container registry.
The command to pull a blast container from BioContainer docker registry is as below:

<pre>
$ mkdir /home/raj76/test
$ cd /home/raj76/test
$ singularity pull docker://biocontainers/blast
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/biocontainers/blast:latest
Cache folder set to /home/raj76/.singularity/docker
[1/1] |===================================| 100.0%
Importing: base Singularity environment
Importing: /home/raj76/.singularity/docker/sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7513e23e94e042f38785a66fba7302331c39e92491ea8057a6d0ec2094fda7c0.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1f1169998bd08c33903c56348ad438f90e69d517d0507b7fba7432decee61eb8.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:3c20a4bba592a988108453b44eefbd6fe2022151542b9883221b658145311681.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:9bfd0812e2d6aae116904d6c301511ef441b26e766811bcb82528941b416ed1b.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1c6ae521538275a62e5e949c49ca7ca611477d8638df15b6c6ab83b76d01da2f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7d1b4609c9a59e46ac2c106aa79b4707bd905526df25288f595d90bf01b6e26c.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8022c247944f5c801b34c391e19aa1792d20b3d55d6a55821cf757c0df43e80.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:10c8f0ff4e5a382068d7480f76ce323b131721ae960b880c64ba94f6c856c77a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8823f25a74a25d2943097ead4b7a90fc1dadcdd40dce298097ea52fc617981e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:aedae76aecd1555418bdb6d202c3464aebad9c44d366533218fb21b883570d2a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:f3124b62dbfe35c1349d0c38f1df851c0e536a00f15d085e53e3031eff2a94a5.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:d3f0d697543a6e83f259022cfa1b74e324b523a7e02057774df35751a5881bc2.tar.gz
Importing: /home/raj76/.singularity/metadata/sha256:944beb8dfe363519e9e9ba35cc76ba618aa58d15d1ed941b6a9b0a74fef88746.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./blast.simg
Cleaning up...
Done. Container is at: ./blast.simg
$ ls
blast.simg
$
</pre>

You now have a blast.simg image that can be used to run blast using Singularity.

===== Accessing the cluster filesystem from inside the container =====
The Singularity application on Sapelo2 has been pre-configured to mount the /home/$USER, /db, /lscratch, and /lustre1 filesystems inside the container.
<pre>
$ singularity exec blast.simg df -h
Filesystem Size Used Avail Use% Mounted on
OverlayFS 1.0M 0 1.0M 0% /
/dev/mapper/vg00-lv.root 20G 7.3G 13G 37% /tmp
/dev/mapper/vg00-lv.lscratch 869G 33M 869G 1% /lscratch
10.55.49.1@o2ib:10.55.49.2@o2ib:/csx0009g 452T 360T 88T 81% /lustre1
sn0.storage:/storage/xcluster/db 27T 3.7T 23T 14% /db
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
sn0.storage:/storage/xcluster/home/raj76 97G 24G 74G 25% /home/raj76
tmpfs 16M 8.0K 16M 1% /etc/group
</pre>

===== Running Blast =====
<pre>
$ singularity exec blast.simg blastp -help
</pre>
This will print the help page for '''blastp''' tool. The command '''singularity exec blast.simg''' tells singularity that it is going to execute the command that follows inside the blast.simg container.
====== Downloading the example datasets ======
Let us download the example datasets to run blast.
<pre>
$ wget http://www.uniprot.org/uniprot/P04156.fasta
$ curl -O ftp://ftp.ncbi.nih.gov/refseq/D_rerio/mRNA_Prot/zebrafish.1.protein.faa.gz
$ gunzip zebrafish.1.protein.faa.gz
$ ls
blast.simg P04156.fasta zebrafish.1.protein.faa
</pre>
====== Preparing the database ======
We need to prepare the zebrafish database with '''makeblastdb''' for the search.
<pre>
$ singularity exec blast.simg makeblastdb -in zebrafish.1.protein.faa -dbtype prot
$ ls -lt
total 779935
-rw-r--r-- 1 raj76 jlmlab 38605311 Dec 22 15:25 zebrafish.1.protein.faa.psq
-rw-r--r-- 1 raj76 jlmlab 7101626 Dec 22 15:25 zebrafish.1.protein.faa.phr
-rw-r--r-- 1 raj76 jlmlab 424888 Dec 22 15:25 zebrafish.1.protein.faa.pin
-rw-r--r-- 1 raj76 jlmlab 42849679 Dec 22 11:47 zebrafish.1.protein.faa
-rwxr-xr-x 1 raj76 jlmlab 753582111 Dec 22 11:38 blast.simg
-rw-r--r-- 1 raj76 jlmlab 334 Nov 21 19:00 P04156.fasta
</pre>

====== Running blastp search ======
Now that we have a database created we can do an alignment against it as follows.
<pre>
$ singularity exec blast.simg blastp -query P04156.fasta -db zebrafish.1.protein.faa -out results.txt
$ ls -lt
total 779936
-rw-r--r-- 1 raj76 jlmlab 17515 Dec 22 15:28 results.txt
-rw-r--r-- 1 raj76 jlmlab 38605311 Dec 22 15:25 zebrafish.1.protein.faa.psq
-rw-r--r-- 1 raj76 jlmlab 7101626 Dec 22 15:25 zebrafish.1.protein.faa.phr
-rw-r--r-- 1 raj76 jlmlab 424888 Dec 22 15:25 zebrafish.1.protein.faa.pin
-rw-r--r-- 1 raj76 jlmlab 42849679 Dec 22 11:47 zebrafish.1.protein.faa
-rwxr-xr-x 1 raj76 jlmlab 753582111 Dec 22 11:38 blast.simg
-rw-r--r-- 1 raj76 jlmlab 334 Nov 21 19:00 P04156.fasta
</pre>

The files results.txt has the blastp output.

You could have used the existing blast databases in /db to perform the alignment by specifying
<pre>
$ singularity exec blast.simg blastp -query P04156.fasta -db /db/uniprot/latest/uniprot_sprot -out results2.txt
$ ls -lth
total 762M
-rw-r--r-- 1 raj76 jlmlab 110K Dec 22 15:34 results2.txt
-rw-r--r-- 1 raj76 jlmlab 18K Dec 22 15:28 results.txt
-rw-r--r-- 1 raj76 jlmlab 37M Dec 22 15:25 zebrafish.1.protein.faa.psq
-rw-r--r-- 1 raj76 jlmlab 6.8M Dec 22 15:25 zebrafish.1.protein.faa.phr
-rw-r--r-- 1 raj76 jlmlab 415K Dec 22 15:25 zebrafish.1.protein.faa.pin
-rw-r--r-- 1 raj76 jlmlab 41M Dec 22 11:47 zebrafish.1.protein.faa
-rwxr-xr-x 1 raj76 jlmlab 719M Dec 22 11:38 blast.simg
-rw-r--r-- 1 raj76 jlmlab 334 Nov 21 19:00 P04156.fasta
</pre>

Singularity

2017-12-22T20:34:01Z

Raj76: /* Using BioContainer containers */

== Using Singularity containers on Sapelo2 ==

The Sapelo2 cluster has the ability to run Singularity containers. [http://singularity.lbl.gov/ Singularity] containers are docker like containers that are HPC friendly.

=== Loading Singularity ===
Singularity is installed on all compute nodes on Sapelo2. In order to access that executables you have to load the Singularity module.
<pre>
$ module load Singularity
$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...

GLOBAL OPTIONS:
-d|--debug Print debugging information
-h|--help Display usage summary
-s|--silent Only print errors
-q|--quiet Suppress all normal output
--version Show application version
-v|--verbose Increase verbosity +1
-x|--sh-debug Print shell wrapper debugging information

GENERAL COMMANDS:
help Show additional help for a command or container
selftest Run some self tests for singularity install

CONTAINER USAGE COMMANDS:
exec Execute a command within container
run Launch a runscript within container
shell Run a Bourne shell within container
test Launch a testscript within container

CONTAINER MANAGEMENT COMMANDS:
apps List available apps within a container
bootstrap *Deprecated* use build instead
build Build a new Singularity container
check Perform container lint checks
inspect Display container's metadata
mount Mount a Singularity container image
pull Pull a Singularity/Docker container to $PWD

COMMAND GROUPS:
image Container image command group
instance Persistent instance command group

CONTAINER USAGE OPTIONS:
see singularity help <command>

For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/

</pre>

=== Using BioContainer containers ===
[https://biocontainers.pro/ BioContainers] is an open source and community-driven framework which provides system-agnostic executable environments for bioinformatics software. BioContainers framework allows software to be installed and executed under an isolated and controllable environment.

==== Blast container example ====
This example is based on the BioContainer example at http://biocontainers.pro/docs/101/running-example/ webpage.

===== Pulling a container =====
In order to use a pre built container, one has to "pull" the container from the container registry.
The command to pull a blast container from BioContainer docker registry is as below:

<pre>
$ mkdir /home/raj76/test
$ cd /home/raj76/test
$ singularity pull docker://biocontainers/blast
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/biocontainers/blast:latest
Cache folder set to /home/raj76/.singularity/docker
[1/1] |===================================| 100.0%
Importing: base Singularity environment
Importing: /home/raj76/.singularity/docker/sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7513e23e94e042f38785a66fba7302331c39e92491ea8057a6d0ec2094fda7c0.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1f1169998bd08c33903c56348ad438f90e69d517d0507b7fba7432decee61eb8.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:3c20a4bba592a988108453b44eefbd6fe2022151542b9883221b658145311681.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:9bfd0812e2d6aae116904d6c301511ef441b26e766811bcb82528941b416ed1b.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1c6ae521538275a62e5e949c49ca7ca611477d8638df15b6c6ab83b76d01da2f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7d1b4609c9a59e46ac2c106aa79b4707bd905526df25288f595d90bf01b6e26c.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8022c247944f5c801b34c391e19aa1792d20b3d55d6a55821cf757c0df43e80.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:10c8f0ff4e5a382068d7480f76ce323b131721ae960b880c64ba94f6c856c77a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8823f25a74a25d2943097ead4b7a90fc1dadcdd40dce298097ea52fc617981e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:aedae76aecd1555418bdb6d202c3464aebad9c44d366533218fb21b883570d2a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:f3124b62dbfe35c1349d0c38f1df851c0e536a00f15d085e53e3031eff2a94a5.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:d3f0d697543a6e83f259022cfa1b74e324b523a7e02057774df35751a5881bc2.tar.gz
Importing: /home/raj76/.singularity/metadata/sha256:944beb8dfe363519e9e9ba35cc76ba618aa58d15d1ed941b6a9b0a74fef88746.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./blast.simg
Cleaning up...
Done. Container is at: ./blast.simg
$ ls
blast.simg
$
</pre>

You now have a blast.simg image that can be used to run blast using Singularity.

===== Accessing the cluster filesystem from inside the container =====
The Singularity application on Sapelo2 has been pre-configured to mount the /home/$USER, /db, /lscratch, and /lustre1 filesystems inside the container.
<pre>
$ singularity exec blast.simg df -h
Filesystem Size Used Avail Use% Mounted on
OverlayFS 1.0M 0 1.0M 0% /
/dev/mapper/vg00-lv.root 20G 7.3G 13G 37% /tmp
/dev/mapper/vg00-lv.lscratch 869G 33M 869G 1% /lscratch
10.55.49.1@o2ib:10.55.49.2@o2ib:/csx0009g 452T 360T 88T 81% /lustre1
sn0.storage:/storage/xcluster/db 27T 3.7T 23T 14% /db
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
sn0.storage:/storage/xcluster/home/raj76 97G 24G 74G 25% /home/raj76
tmpfs 16M 8.0K 16M 1% /etc/group
</pre>

===== Running Blast =====
<pre>
$ singularity exec blast.simg blastp -help
</pre>
This will print the help page for '''blastp''' tool. The command '''singularity exec blast.simg''' tells singularity that it is going to execute the command that follows inside the blast.simg container.
====== Downloading the example datasets ======
Let us download the example datasets to run blast.
<pre>
$ wget http://www.uniprot.org/uniprot/P04156.fasta
$ curl -O ftp://ftp.ncbi.nih.gov/refseq/D_rerio/mRNA_Prot/zebrafish.1.protein.faa.gz
$ gunzip zebrafish.1.protein.faa.gz
$ ls
blast.simg P04156.fasta zebrafish.1.protein.faa
</pre>
====== Preparing the database ======
We need to prepare the zebrafish database with '''makeblastdb''' for the search.
<pre>
$ singularity exec blast.simg makeblastdb -in zebrafish.1.protein.faa -dbtype prot
$ ls -lt
total 779935
-rw-r--r-- 1 raj76 jlmlab 38605311 Dec 22 15:25 zebrafish.1.protein.faa.psq
-rw-r--r-- 1 raj76 jlmlab 7101626 Dec 22 15:25 zebrafish.1.protein.faa.phr
-rw-r--r-- 1 raj76 jlmlab 424888 Dec 22 15:25 zebrafish.1.protein.faa.pin
-rw-r--r-- 1 raj76 jlmlab 42849679 Dec 22 11:47 zebrafish.1.protein.faa
-rwxr-xr-x 1 raj76 jlmlab 753582111 Dec 22 11:38 blast.simg
-rw-r--r-- 1 raj76 jlmlab 334 Nov 21 19:00 P04156.fasta
</pre>

====== Preparing the database ======
Now that we have a database created we can do an alignment against it as follows.
<pre>
$ singularity exec blast.simg blastp -query P04156.fasta -db zebrafish.1.protein.faa -out results.txt
$ ls -lt
total 779936
-rw-r--r-- 1 raj76 jlmlab 17515 Dec 22 15:28 results.txt
-rw-r--r-- 1 raj76 jlmlab 38605311 Dec 22 15:25 zebrafish.1.protein.faa.psq
-rw-r--r-- 1 raj76 jlmlab 7101626 Dec 22 15:25 zebrafish.1.protein.faa.phr
-rw-r--r-- 1 raj76 jlmlab 424888 Dec 22 15:25 zebrafish.1.protein.faa.pin
-rw-r--r-- 1 raj76 jlmlab 42849679 Dec 22 11:47 zebrafish.1.protein.faa
-rwxr-xr-x 1 raj76 jlmlab 753582111 Dec 22 11:38 blast.simg
-rw-r--r-- 1 raj76 jlmlab 334 Nov 21 19:00 P04156.fasta
</pre>

The files results.txt has the blastp output.

You could have used the existing blast databases in /db to perform the alignment by specifying

Singularity

2017-12-22T20:20:20Z

Raj76: /* Using BioContainer containers */

== Using Singularity containers on Sapelo2 ==

The Sapelo2 cluster has the ability to run Singularity containers. [http://singularity.lbl.gov/ Singularity] containers are docker like containers that are HPC friendly.

=== Loading Singularity ===
Singularity is installed on all compute nodes on Sapelo2. In order to access that executables you have to load the Singularity module.
<pre>
$ module load Singularity
$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...

GLOBAL OPTIONS:
-d|--debug Print debugging information
-h|--help Display usage summary
-s|--silent Only print errors
-q|--quiet Suppress all normal output
--version Show application version
-v|--verbose Increase verbosity +1
-x|--sh-debug Print shell wrapper debugging information

GENERAL COMMANDS:
help Show additional help for a command or container
selftest Run some self tests for singularity install

CONTAINER USAGE COMMANDS:
exec Execute a command within container
run Launch a runscript within container
shell Run a Bourne shell within container
test Launch a testscript within container

CONTAINER MANAGEMENT COMMANDS:
apps List available apps within a container
bootstrap *Deprecated* use build instead
build Build a new Singularity container
check Perform container lint checks
inspect Display container's metadata
mount Mount a Singularity container image
pull Pull a Singularity/Docker container to $PWD

COMMAND GROUPS:
image Container image command group
instance Persistent instance command group

CONTAINER USAGE OPTIONS:
see singularity help <command>

For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/

</pre>

=== Using BioContainer containers ===
[https://biocontainers.pro/ BioContainers] is an open source and community-driven framework which provides system-agnostic executable environments for bioinformatics software. BioContainers framework allows software to be installed and executed under an isolated and controllable environment.

==== Blast container example ====
This example is based on the BioContainer example at http://biocontainers.pro/docs/101/running-example/ webpage.

===== Pulling a container =====
In order to use a pre built container, one has to "pull" the container from the container registry.
The command to pull a blast container from BioContainer docker registry is as below:

<pre>
$ mkdir /home/raj76/test
$ cd /home/raj76/test
$ singularity pull docker://biocontainers/blast
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/biocontainers/blast:latest
Cache folder set to /home/raj76/.singularity/docker
[1/1] |===================================| 100.0%
Importing: base Singularity environment
Importing: /home/raj76/.singularity/docker/sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7513e23e94e042f38785a66fba7302331c39e92491ea8057a6d0ec2094fda7c0.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1f1169998bd08c33903c56348ad438f90e69d517d0507b7fba7432decee61eb8.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:3c20a4bba592a988108453b44eefbd6fe2022151542b9883221b658145311681.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:9bfd0812e2d6aae116904d6c301511ef441b26e766811bcb82528941b416ed1b.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1c6ae521538275a62e5e949c49ca7ca611477d8638df15b6c6ab83b76d01da2f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7d1b4609c9a59e46ac2c106aa79b4707bd905526df25288f595d90bf01b6e26c.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8022c247944f5c801b34c391e19aa1792d20b3d55d6a55821cf757c0df43e80.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:10c8f0ff4e5a382068d7480f76ce323b131721ae960b880c64ba94f6c856c77a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8823f25a74a25d2943097ead4b7a90fc1dadcdd40dce298097ea52fc617981e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:aedae76aecd1555418bdb6d202c3464aebad9c44d366533218fb21b883570d2a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:f3124b62dbfe35c1349d0c38f1df851c0e536a00f15d085e53e3031eff2a94a5.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:d3f0d697543a6e83f259022cfa1b74e324b523a7e02057774df35751a5881bc2.tar.gz
Importing: /home/raj76/.singularity/metadata/sha256:944beb8dfe363519e9e9ba35cc76ba618aa58d15d1ed941b6a9b0a74fef88746.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./blast.simg
Cleaning up...
Done. Container is at: ./blast.simg
$ ls
blast.simg
$
</pre>

You now have a blast.simg image that can be used to run blast using Singularity.

===== Accessing the cluster filesystem from inside the container =====
The Singularity application on Sapelo2 has been pre-configured to mount the /home/$USER, /db, /lscratch, and /lustre1 filesystems inside the container.

===== Running Blast =====
<pre>
$ singularity exec blast.simg blastp -help
</pre>
This will print the help page for '''blastp''' tool. The command '''singularity exec blast.simg''' tells singularity that it is going to execute the command that follows inside the blast.simg container.
====== Downloading the example datasets ======
Let us download the example datasets to run blast.
<pre>
$ wget http://www.uniprot.org/uniprot/P04156.fasta
$ curl -O ftp://ftp.ncbi.nih.gov/refseq/D_rerio/mRNA_Prot/zebrafish.1.protein.faa.gz
$ gunzip zebrafish.1.protein.faa.gz
$ ls
blast.simg P04156.fasta zebrafish.1.protein.faa
</pre>
====== Preparing the database ======
We need to prepare the zebrafish database with '''makeblastdb''' for the search.

Singularity

2017-12-22T16:56:01Z

Raj76: /* Using BioContainer containers */

== Using Singularity containers on Sapelo2 ==

The Sapelo2 cluster has the ability to run Singularity containers. [http://singularity.lbl.gov/ Singularity] containers are docker like containers that are HPC friendly.

=== Loading Singularity ===
Singularity is installed on all compute nodes on Sapelo2. In order to access that executables you have to load the Singularity module.
<pre>
$ module load Singularity
$ singularity --help
USAGE: singularity [global options...] <command> [command options...] ...

GLOBAL OPTIONS:
-d|--debug Print debugging information
-h|--help Display usage summary
-s|--silent Only print errors
-q|--quiet Suppress all normal output
--version Show application version
-v|--verbose Increase verbosity +1
-x|--sh-debug Print shell wrapper debugging information

GENERAL COMMANDS:
help Show additional help for a command or container
selftest Run some self tests for singularity install

CONTAINER USAGE COMMANDS:
exec Execute a command within container
run Launch a runscript within container
shell Run a Bourne shell within container
test Launch a testscript within container

CONTAINER MANAGEMENT COMMANDS:
apps List available apps within a container
bootstrap *Deprecated* use build instead
build Build a new Singularity container
check Perform container lint checks
inspect Display container's metadata
mount Mount a Singularity container image
pull Pull a Singularity/Docker container to $PWD

COMMAND GROUPS:
image Container image command group
instance Persistent instance command group

CONTAINER USAGE OPTIONS:
see singularity help <command>

For any additional help or support visit the Singularity
website: http://singularity.lbl.gov/

</pre>

=== Using BioContainer containers ===
[https://biocontainers.pro/ BioContainers] is an open source and community-driven framework which provides system-agnostic executable environments for bioinformatics software. BioContainers framework allows software to be installed and executed under an isolated and controllable environment.

==== Blast container example ====
This example is based on the BioContainer example at http://biocontainers.pro/docs/101/running-example/ webpage.

===== Pulling a container =====
In order to use a pre built container, one has to "pull" the container from the container registry.
The command to pull a blast container from BioContainer docker registry is as below:

<pre>
$ mkdir /home/raj76/test
$ cd /home/raj76/test
$ singularity pull docker://biocontainers/blast
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/biocontainers/blast:latest
Cache folder set to /home/raj76/.singularity/docker
[1/1] |===================================| 100.0%
Importing: base Singularity environment
Importing: /home/raj76/.singularity/docker/sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7513e23e94e042f38785a66fba7302331c39e92491ea8057a6d0ec2094fda7c0.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1f1169998bd08c33903c56348ad438f90e69d517d0507b7fba7432decee61eb8.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:3c20a4bba592a988108453b44eefbd6fe2022151542b9883221b658145311681.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:9bfd0812e2d6aae116904d6c301511ef441b26e766811bcb82528941b416ed1b.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:1c6ae521538275a62e5e949c49ca7ca611477d8638df15b6c6ab83b76d01da2f.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:7d1b4609c9a59e46ac2c106aa79b4707bd905526df25288f595d90bf01b6e26c.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8022c247944f5c801b34c391e19aa1792d20b3d55d6a55821cf757c0df43e80.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:10c8f0ff4e5a382068d7480f76ce323b131721ae960b880c64ba94f6c856c77a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:a8823f25a74a25d2943097ead4b7a90fc1dadcdd40dce298097ea52fc617981e.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:aedae76aecd1555418bdb6d202c3464aebad9c44d366533218fb21b883570d2a.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:f3124b62dbfe35c1349d0c38f1df851c0e536a00f15d085e53e3031eff2a94a5.tar.gz
Importing: /home/raj76/.singularity/docker/sha256:d3f0d697543a6e83f259022cfa1b74e324b523a7e02057774df35751a5881bc2.tar.gz
Importing: /home/raj76/.singularity/metadata/sha256:944beb8dfe363519e9e9ba35cc76ba618aa58d15d1ed941b6a9b0a74fef88746.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: ./blast.simg
Cleaning up...
Done. Container is at: ./blast.simg
$ ls
blast.simg
$
</pre>

You now have a blast.simg image that can be used to run blast using Singularity.

===== Running Blast =====
<pre>
$ singularity exec blast.simg blastp -help
</pre>
This will print the help page for '''blastp''' tool. The command '''singularity exec blast.simg''' tells singularity that it is going to execute the command that follows inside the blast.simg container.
====== Downloading the example datasets ======
Let us download the example datasets to run blast.
<pre>
$ wget http://www.uniprot.org/uniprot/P04156.fasta
$ curl -O ftp://ftp.ncbi.nih.gov/refseq/D_rerio/mRNA_Prot/zebrafish.1.protein.faa.gz
$ gunzip zebrafish.1.protein.faa.gz
$ ls
blast.simg P04156.fasta zebrafish.1.protein.faa
</pre>
====== Preparing the database ======
We need to prepare the zebrafish database with '''makeblastdb''' for the search. Before we do that let's understand how to access the Sapelo2 file systems from inside the container.
Execute the following command:
<pre>
$ singularity exec blast.simg df -h
Filesystem Size Used Avail Use% Mounted on
OverlayFS 1.0M 0 1.0M 0% /
/dev/mapper/vg00-lv.root 234G 7.8G 227G 4% /tmp
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 428M 63G 1% /dev/shm
sn0.storage:/storage/xcluster/home/raj76 97G 24G 74G 25% /home/raj76
tmpfs 16M 8.0K 16M 1% /etc/group
</pre>
You will see in the output above that my home directory '''(/home/raj76)''' is mounted and accessible from inside the container. By default, Singularity mounts the users home directory inside the container. It will also let us mount other directories inside the container.
<pre>
$ singularity exec -B /db -B /lustre1 -B /lscratch blast.simg df -h
Filesystem Size Used Avail Use% Mounted on
OverlayFS 1.0M 0 1.0M 0% /
/dev/mapper/vg00-lv.root 20G 7.3G 13G 37% /tmp
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
sn0.storage:/storage/xcluster/home/raj76 97G 24G 74G 25% /home/raj76
sn0.storage:/storage/xcluster/db 27T 3.7T 23T 14% /db
10.55.49.1@o2ib:10.55.49.2@o2ib:/csx0009g 452T 361T 87T 81% /lustre1
/dev/mapper/vg00-lv.lscratch 869G 33M 869G 1% /lscratch
tmpfs 16M 8.0K 16M 1% /etc/group
</pre>
In the above example, the '''-B /db''' option to the exec command.

Singularity

2017-12-22T16:44:08Z

Raj76: /* Using BioContainer containers */

Singularity

2017-12-22T16:38:51Z

Raj76: /* Pulling containers from BioContainers */

Singularity

2017-12-22T16:26:00Z

Raj76:

Singularity

2017-12-22T16:23:27Z

Raj76: /* Using Singularity containers on Sapelo2 */

Singularity

2017-12-22T16:18:07Z

Raj76: Created page with "== Using Singularity containers on Sapelo2 == The Sapelo2 cluster has the ability to run Singularity containers. [http://singularity.lbl.gov/ Singularity] containers are dock..."

Disk Storage

2017-04-19T18:43:32Z

Raj76: /* Storage Architecture */

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

Each compute node has local physical hard drives that the user can utilize as temporary storage, aka lscratch. The lscratch device is a very fast storage device compared to the network attached storage systems. The drawback is that the capacity is low and it cannot be accessed from outside the compute node. The data in lscratch is not backed up and it can be deleted anytime after the job on the compute node is finished.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):
<pre>
zcluster
home= 100GB
scratch= 4TB
/lscratch= Limited by device size

sapelo
home= 100GB
scratch= Currently none
/lscratch= Limited by device size (Approx. 250GB)
</pre>
The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':
<pre>
zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day
lscratch= /lscratch/jsmith

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith
lscratch= /lscratch

Note that sapelo users already have a scratch directory. Users of the zcluster need to type

make_escratch
</pre>

while on the login node (not interactive nodes) to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | Transferring Files]]) are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

Here is the example for zcluster:

<pre>
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp my-to-restore-file /home/rccstaff/cecombs/test
</pre>

For Sapelo, please send in a ticket for such request. It is a different procedure at backend.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 480TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Disk Storage

2017-04-19T18:43:07Z

Raj76:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

Each compute node has local physical hard drives that the user can utilize as temporary storage, aka lscratch. The lscratch device is a very fast storage device compared to the network attached storage systems. The drawback is that the capacity is low and it cannot be accessed from outside the compute node. The data in lscratch is not backed up and it can be deleted anytime after the job on the compute node is finished.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):
<pre>
zcluster
home= 100GB
scratch= 4TB
/lscratch= Limited by device size

sapelo
home= 100GB
scratch= Currently none
/lscratch= Limited by device size (Approx. 250GB)
</pre>
The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':
<pre>
zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day
lscratch= /lscratch/jsmith

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith
lscratch= /lscratch
</pre>
Note that sapelo users already have a scratch directory. Users of the zcluster need to type

make_escratch

while on the login node (not interactive nodes) to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | Transferring Files]]) are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

Here is the example for zcluster:

<pre>
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp my-to-restore-file /home/rccstaff/cecombs/test
</pre>

For Sapelo, please send in a ticket for such request. It is a different procedure at backend.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 480TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Disk Storage

2017-04-19T18:41:18Z

Raj76:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

Each compute node has local physical hard drives that the user can utilize as temporary storage, aka lscratch. The lscratch device is a very fast storage device compared to the network attached storage systems. The drawback is that the capacity is low and it cannot be accessed from outside the compute node. The data in lscratch is not backed up and it can be deleted anytime after the job on the compute node is finished.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):

zcluster
home= 100GB
scratch= 4TB
/lscratch= Limited by device size

sapelo
home= 100GB
scratch= Currently none
/lscratch= Limited by device size (Approx. 250GB)

The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':

zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day
lscratch= /lscratch/jsmith

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith
lscratch= /lscratch

Note that sapelo users already have a scratch directory. Users of the zcluster need to type

make_escratch

while on the login node (not interactive nodes) to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | Transferring Files]]) are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

Here is the example for zcluster:

<pre>
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp my-to-restore-file /home/rccstaff/cecombs/test
</pre>

For Sapelo, please send in a ticket for such request. It is a different procedure at backend.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 480TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems