Research Computing Center Wiki - User contributions [en]

Disk Storage

2016-05-17T14:51:23Z

Derda: /* Storage Architecture */

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):

zcluster
home= 100GB
scratch= 4TB

sapelo
home= 100GB
scratch= Currently none

The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':

zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith

Note that sapelo users already have a scratch directory. Users of the zcluster need to type 'make_escratch' while on the login node (not interactive nodes) to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | Transferring Files]]) are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

Here is the example for zcluster:

<pre>
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp my-to-restore-file /home/rccstaff/cecombs/test
</pre>

For Sapelo, please send in a ticket for such request. It is a different procedure at backend.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 240TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Consulting

2016-05-05T18:39:06Z

Derda:

===Perl and UNIX-shell scripting===

Automating and simplifying tasks with command-scripts is a routine procedure in the Unix/Linux environment. We regularly help users develop scripts to suit their specific needs.

===Code Development===

We provide assistance with code compilation and debugging in serial and parallel environments.

===Computational Physics===

Computational physics methods are sometimes useful and applicable to problems in other fields. We provide some
consultation for computational physics algorithm selection and implementation.

===Bioinformatics===

From the installation of programs and data (e.g., genetic databases), to helping users get started, bioinformatics support is a key support area.

===General Usage===

GACRC staff are available to provide consultation in order to facilitate and optimize the use of GACRC’s resources. We are available to meet with small groups of users, and to provide guest lectures for courses that use GACRC resources.

==People==

===Administrators===

*'''Guy Cormier, Ph.D.''' <br />Director, UGA Georgia Advanced Computing Resource Center<br />Associate Research Scientist, Institute of Bioinformatics

===GACRC Consultants===

*'''Yecheng Huang, M.S.''' <br /> Bioinformatics consultant
*'''Shan-Ho Tsai, Ph.D.''' <br /> Computational Physics and High Performance Computing consultant
*'''Zhuofei Hou, Ph.D.''' <br /> High Performance Computing Consultant and Lead Trainer

===GACRC IT Staff===

*'''Paul Brunk''' <br /> Principal Unix System Administrator
*'''Lawrence Kearney''' <br /> Principal Unix System Administrator

===Emeritus===
*'''Greg Derda, Ph.D.''' <br /> Operations Manager Extraordinaire

==Office Hours==
GACRC staff are located in 4098C Stegeman Coliseum. Our office is open Monday though Friday from 9am until 5pm. You are welcome to stop by and ask questions, or you can schedule an appointment using the [http://help.gacrc.uga.edu/ GACRC Support Form].

==Directions==
As you face Stegeman Coliseum from Carlton St., there is a ramp that leads from the street up to the Coliseum, to the left of the new glass wall. From the top of that ramp, you will see a small stairwell leading down to a single glass door. Come in through that door and climb the steps to the 4th floor door, which opens into the GACRC workspace.

Disk Storage

2016-05-04T18:28:33Z

Derda:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):

zcluster
home= 100GB
scratch= 4TB

sapelo
home= 100GB
scratch= Currently none

The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':

zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith

Note that sapelo users already have a scratch directory. Users of the zcluster need to type 'make_escratch' to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | Transferring Files]]) are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

For example:

<pre>
[cecombs@sites test]$ pwd
/home/rccstaff/cecombs/test
[cecombs@sites ~]$ cd test
[cecombs@sites test]$ ls
Main.java
[cecombs@sites test]$ rm -rf Main.java
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ ls
Main.java
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp Main.java /home/rccstaff/cecombs/test
[cecombs@sites 2013.04.18.00.00.01.daily]$ cd /home/rccstaff/cecombs/test
[cecombs@sites test]$ ls
Main.java
</pre>

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 240TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Consulting

2016-05-04T18:13:03Z

Derda: Reverted edits by Derda (talk) to last revision by Shtsai

===Perl and UNIX-shell scripting===

Automating and simplifying tasks with command-scripts is a routine procedure in the Unix/Linux environment. We regularly help users develop scripts to suit their specific needs.

===Code Development===

We provide assistance with code compilation and debugging in serial and parallel environments.

===Computational Physics===

Computational physics methods are sometimes useful and applicable to problems in other fields. We provide some
consultation for computational physics algorithm selection and implementation.

===Bioinformatics===

From the installation of programs and data (e.g., genetic databases), to helping users get started, bioinformatics support is a key support area.

===General Usage===

GACRC staff are available to provide consultation in order to facilitate and optimize the use of GACRC’s resources. We are available to meet with small groups of users, and to provide guest lectures for courses that use GACRC resources.

==People==

===Administrators===

*'''Guy Cormier, Ph.D.''' <br />Director, UGA Georgia Advanced Computing Resource Center<br />Associate Research Scientist, Institute of Bioinformatics

===GACRC Consultants===

*'''Greg Derda, Ph.D.''' <br /> Operations Manager
*'''Yecheng Huang, M.S.''' <br /> Bioinformatics consultant
*'''Shan-Ho Tsai, Ph.D.''' <br /> Computational Physics and High Performance Computing consultant
*'''Zhuofei Hou, Ph.D.''' <br /> High Performance Computing Consultant and Lead Trainer

===GACRC IT Staff===

*'''Paul Brunk''' <br /> Principal Unix System Administrator
*'''Lawrence Kearney''' <br /> Principal Unix System Administrator

==Office Hours==
GACRC staff are located in 4098C Stegeman Coliseum. Our office is open Monday though Friday from 9am until 5pm. You are welcome to stop by and ask questions, or you can schedule an appointment using the [http://help.gacrc.uga.edu/ GACRC Support Form].

==Directions==
As you face Stegeman Coliseum from Carlton St., there is a ramp that leads from the street up to the Coliseum, to the left of the new glass wall. From the top of that ramp, you will see a small stairwell leading down to a single glass door. Come in through that door and climb the steps to the 4th floor door, which opens into the GACRC workspace.

Disk Storage

2016-05-04T18:08:26Z

Derda: Reverted edits by Derda (talk) to last revision by Jerky

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==
Network attached storage systems at the GACRC are tiered in three collections of systems based on speed and capacity. Our fastest network attached storage system is a Panasas ActiveStor 12 which exports 156TB of data, mounted on every node at /panfs and is divided into two categories: home and scratch.

== Home Directories ==

'''Home directories are for highly static datasets. This volume can not support a high degree of variability with the features (like snapshots) it provides. Please make sure that any data stored in your home directory is limited to applications and inputs that you use frequently.'''

All users have a default 300GB home quota (i.e., maximum limit) on their home directory; however, justifiable requests for quotas up to 2TB can be made by making a formal request through the [http://gacrc.uga.edu/help GACRC Website contact form]. '''Storage in the home directory to avoid archive storage fees is not a justifiable request.''' Requests for home quotas greater than 2TB must be submitted by the PI of a lab group, and approved by the GACRC advisory committee (via the IT Manager). Users may create lab directories for data that is shared by a lab group, but those directories count against the quota of the creating user. An example of this, for the “abclab” users, would be: /home/abclab/labdata. Home directories are backed up.

=== Snapshots ===
Lab home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be copied from to restore files that may have been accidentally deleted or overwritten.

Home directories have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

For example:

<pre>
[cecombs@sites test]$ pwd
/home/rccstaff/cecombs/test
[cecombs@sites ~]$ cd test
[cecombs@sites test]$ ls
Main.java
[cecombs@sites test]$ rm -rf Main.java
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ ls
Main.java
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp Main.java /home/rccstaff/cecombs/test
[cecombs@sites 2013.04.18.00.00.01.daily]$ cd /home/rccstaff/cecombs/test
[cecombs@sites test]$ ls
Main.java
</pre>

== Scratch ==

Scratch Directories are for dynamic data. Scratch space is specifically designed to handle datasets that grow and shrink on-demand. The GACRC does not and can not snapshot scratch directories because the amount of data which changes periodically is too great and snapshots would only serve to slow the file systems down.

=== eScratch ===
eScratch directories are for ephemeral datasets; most commonly, the output of large calculations that need to be stored in a temporary place for a short period of time. Any user can make escratch directories for their work. Ephemeral scratch directories on GACRC clusters reside on a Panasas ActiveStor 12 storage cluster.

==== Making an eScratch Directory ====
Researchers who need to use scratch space can type '''(''on the login node zcluster.rcc ONLY'')'''

<pre class="gcommand">
make_escratch
</pre>

a sub-directory will be created, and the user will be told the path to the sub-directory, e.g. /panfs/pstor.storage/escratch1/jsmith_Oct_22. The life span of the directory will be one week longer than the longest duration queue, which is currently 30 days (i.e., life span = 37 days). '''At that time, the directory and its contents will be systematically deleted. ''' Users can create one escratch directory per day if needed. The total space a user can use on escratch (all escratch directories '''combined''') is 4TB. The escratch directories are not backed up.

If a user needs to retain their self-created escratch directory for more than the 37 days which they were allocated, they may contact the GACRC staff for an extension through the [http://gacrc.uga.edu/help support form]. We will grant almost all requests, but escratch directories do use precious HPC storage space and we must ask that a new request be submitted for every 37 day period that they are needed to ensure that the space is freed as soon as possible.

=== lscratch ===

lscratch stands for local scratch and is available on every node in the zcluster.

==== lscratch information ====
All /lscratch filesystems on every node have these properties:
* lscratch is by far the fastest possible filesystem at the GACRC, however the lscratch directory is only available to the node that a job gets scheduled to.
* lscratch filesystem resides on the local hard drive of the node.
* Represents the remainder of unused disk after the OS is installed.
* Multiple different sizes for /lscratch; nodes have different sized disks.
* Not accessible from other nodes
* Every user has a directory on every node, /lscratch/<username>

==== lscratch Guidelines ====
This is a list of guidelines for /lscratch usage:
* Do not count on any lscratch sizes above 10G unless you know the size of the local hard drive and target that node specifically (e.g.: qsub -l h=compute-15-36)
* You will be responsible for migrating your data from the node after your job finishes. The job itself can transfer the data.
* Make sure that your output goes to: /lscratch/<username> (e.g: /lscratch/cecombs)

== Quotas ==

To see how much space you are consuming on the home and scratch file systems, please use the command

<pre class="gcommand">
quota_rep
</pre>

== Overflow/Archival Storage ==

Some labs also have a subscription archival storage space, which is mounted on the zcluster login node and on the [[Transferring Files | copy nodes]] as /oflow (note that /oflow is not mounted on the compute nodes). The archival storage system is for long-term storage of large, static datasets.

This filesystem is snapshotted. The snapshots are available only from the mount point under the hidden ".zfs" directory (e.g.: /oflow/jlmlab/.zfs). Overflow devices are snapshotted once every hour, day, week and month. 24 hourlies, 7 dailies, 4 weeklies and 4 monthlies are kept.

Please contact the GACRC staff to request Overflow storage.

Consulting

2016-04-29T13:07:15Z

Derda:

Consulting

2016-04-29T13:06:10Z

Derda:

Disk Storage

2016-04-27T13:21:22Z

Derda:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):

zcluster
home= 100GB
scratch= 4TB

sapelo
home= 100GB
scratch= Currently none

The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':

zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith

Note that sapelo users already have a scratch directory. Users of the zcluster need to type 'make_escratch' to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | Transferring Files]]) are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Contact the GACRC staff if you need to recover data from a snapshot.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 240TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Disk Storage

2016-04-27T13:19:05Z

Derda:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):

zcluster
home= 100GB
scratch= 4TB

sapelo
home= 100GB
scratch= Currently none

The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':

zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith

Note that sapelo users already have a scratch directory. Users of the zcluster need to type 'make_escratch' to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | copy nodes]] are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Contact the GACRC staff if you need to recover data from a snapshot.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 240TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Disk Storage

2016-04-27T13:18:49Z

Derda:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==

Network attached storage systems at the GACRC are tiered in three levels based on speed and capacity. Ranked in order of decreasing speed, the file systems are "scratch", "home", and "offline" storage. The home filesystem is the "landing zone" when users login, and the scratch filesystem is where jobs should be run. Scratch is considered temporary and files are not to be left on it long-term. The offline storage filesystem is where data that is currently being used should be stored when it is not being used on scratch.

For home and scratch directories, users are assigned the following quotas (maximum space allowed):

zcluster
home= 100GB
scratch= 4TB

sapelo
home= 100GB
scratch= Currently none

The offline storage filesystem is named "project" and is configured for use by lab groups, and by default, each lab group has a 1TB quota. Individual members of a lab group can create subdirectories under their lab's project directory. PI's of lab groups can request additional storage on project as needed. Please note that this storage is not meant for long-term (e.g., archive) storage of data. That type of storage is the responsibility of the user.

=== Storage Architecture ===

The home and scratch filesystems are mounted on the zcluster and the sapelo cluster as follows, using an example user 'jsmith' in a lab group 'abclab':

zcluster-

home= /home/abclab/jsmith
scratch= /escratch4/jsmith/jsmith_Month_Day

sapelo-

home= /home/jsmith
scratch= /lustre1/jsmith

Note that sapelo users already have a scratch directory. Users of the zcluster need to type 'make_escratch' to create a scratch directory - the command will return the name of the directory.

The project filesystem is not mounted on the compute nodes and cannot be accessed by running jobs. It is mounted on the zcluster login node, and on the file "copy" and "xfer" nodes. The copy and xfer nodes (discussed under [[Transferring Files | copy nodes]] are the preferred servers to use for copying and moving files between all of the filesystems, and to and from the outside world.

The project filesystem has a consistent mount point of:

/project/abclab

=== Auto Mounting Filesystems ===

Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo home directories and the project filesystems. For the zcluster copy nodes, this includes the project filesystems. Sapelo interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens.

=== Snapshots ===

Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

Home directories on sapelo have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. On the zcluster, some home directories have snapshots taken once a day, and some have snapshots taken once every 2 days; these are maintained for 4 days.

Contact the GACRC staff if you need to recover data from a snapshot.

=== Current Storage Systems ===

(1) Panasas ActiveStor 12 storage cluster with 133TB usable capacity, running PanFS parallel file system. Currently supporting the home filesystem on the zcluster

(1) Seagate (Xyratex) Lustre appliance with 240TB usable capacity. Currently supporting the scratch filesystem on sapelo

(3) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 84TB usable capacity. Currently supporting home directories on sapelo

(2) Penguin IceBreakers storage chains running ZFS mounted through NFS for a total of 374TB usable capacity. This storage is used as an active project repository

(1) Penguin IceBreaker storage chains running ZFS mounted through NFS for a total of 142TB usable capacity. This storage is used as a backup resource for the home and project filesystems

Policies

2016-02-25T18:54:44Z

Derda:

==Introduction to GACRC Policies==

The following policies are subject to revision, especially as the GACRC grows in scope and services. Your comments and questions will be useful to our policy formulation and refinement and are actively solicited (rcac@uga.edu).

The GACRC computational infrastructure, including its servers, clusters, data stores, and other related devices are for the exclusive use of authorized users only. Individuals using these computer systems without proper authority, or in excess of their authority, are subject to having all of their activities on these systems monitored and recorded by GACRC personnel. In the course of monitoring individuals improperly using these systems, or in the course of any system maintenance, the activities of authorized users may also be monitored.

Anyone using these systems expressly consents to such monitoring and is advised that if such monitoring reveals possible evidence of unauthorized activity, system personnel may provide the evidence of such monitoring to law enforcement officials.

Anyone using these systems expressly consents to abide by the policies of the University of Georgia and/or the Georgia Advanced Computing Resource Center and, accordingly, is subject to account termination and/or immediate disconnection from GACRC resources.

==Definitions==
===Account===

The collection of information related to an authorized user of resources, including resource usage statistics.
===Active Account===

An account belonging to a person currently authorized to access resources.
===Home Directory===

Disk storage space assigned to each user with an active account, used to store temporary or permanent files. At the GACRC, there is one and only one Home Directory per Active Account, regardless of the computational resource(s) used by the account holder.
===Account Holder===

The authorized person responsible for an Active Account.
===Archive===

A file which has been moved to offline or nearline storage because activity on the file has virtually ceased.

==GACRC Resource Usage==

The computational resources of the Georgia Advanced Computing Resource Center are to be used in direct support of research programs at the University of Georgia. Support is also provided for classes that teach computational methods, and provide training for high performance computing. The GACRC reserves the right to restrict access to its resources for course work if such work is deemed to present a negative impact to authorized research activities.

GACRC policies supplement UGA’s Policies on the Use of Computers, found at: http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup

==GACRC Eligibility and Access==

Access to and use of the computing facilities managed by the Georgia Advanced Computing Resource Center are limited to persons affiliated with the University of Georgia and associated with research projects sponsored by UGA.

Direct affiliation in this context means faculty, staff and students of the University of Georgia. Faculty includes persons holding permanent or temporary appointments as well as adjunct faculty, instructors and visiting faculty while in residence at the University. It also includes those persons with faculty status such as research associates, research scientists, post-doctoral researchers and academic and service professionals. Staff includes all those non-faculty persons employed directly by the University in a research-support role. Graduate and undergraduate students who are members of faculty research labs are eligible for accounts as well.

For directly affiliated users, accounts on the GACRC computers will remain active as long as the researchers hold the above status.
Access by researchers affiliated with the University of Georgia that do not meet the criteria above will be considered on a case-by-case basis, especially researchers not directly affiliated with the University of Georgia who are collaborating on research with researchers directly affiliated with UGA. Requests for access must be forwarded to the GACRC in such cases by a person directly affiliated with UGA.

For indirectly affiliated users, access will be granted for a fixed period of time, according to the expected length of the collaborative project, but no longer than one (1) year. Application for extensions will be considered.

Accounts will remain active no more than 30 days following a status change (i.e., leaving the university). Graduate instructional accounts will only remain active for the duration of the semester in which they are actually needed. Home directories will be archived for at least 90 days, but no longer than 180 days after an account becomes inactive.

Requests for access by individuals other than those listed above should first be directed to the Director of the Center using the form provided.

==GACRC Identity Management==

Below are described the procedures for validating the identity of account users.
===Directly Affiliated Users===

All directly affiliated persons wanting an account must apply for access to the GACRC using the instructions provided on the GACRC website (http://www.gacrc.uga.edu/accounts). The applicant must authenticate to the form using his/her MyID and password for identification.
Upon acceptance of the application, the user will be notified via e-mail. The applicant’s UGA MyID in conjunction with the temporary password will be used to initially log into the requested GACRC resources . After initial login, a new password should be provided, as noted in the emailed instructions. Please note that the GACRC will NOT record a user’s MyID password or his/her Social Security number.
===Indirectly Affiliated Users===

Indirectly affiliated users must be sponsored by a directly affiliated user. The directly affiliated user must apply on behalf of the applicant by contacting the GACRC staff.
===Protection of Passwords===

An account holder must never divulge their login ID and password to a third party. Only authorized account holders may access the resources of the GACRC. If a third party is found to be using an account holder’s login with or without the permission of the account holder, the account holder’s access privileges may be revoked at the sole discretion of the GACRC Manager or Director.

==GACRC Resource Allocation==
===''High-Performance Storage Provisioning''===
===Home File System===

The home file system resides on a high-performance storage device and is used for long-term storage of files needed for analyses on the GACRC computing clusters. All users have a default 100GB home quota (i.e., maximum limit) on their home directory; however, justifiable requests for quotas up to 2TB can be made by contacting the GACRC IT Manager (currently Greg Derda: derda@uga.edu). Storage in the home directory to avoid archive storage fees is not a justifiable request. Requests for home quotas greater than 2TB must be submitted by the PI of a lab group, and approved by the GACRC advisory committee (via the IT Manager). Users may create lab directories for data that is shared by a lab group, but those directories count against the quota of the creating user. An example of this, for the “abclab” users, would be: /home/abclab/labdata. Home directories are backed up.
===Scratch File System===

The scratch file system resides on a high-performance storage device and is to be used for temporary storage of files in use by actively running jobs. Files are to be removed from scratch when the job(s) complete. Scratch space is not backed up.

The current scratch file system is mounted on the compute clusters as escratch. Researchers who need to use scratch space can type ‘make_escratch’ and a sub-directory will be created, and the user will be told the path to the sub-directory e.g., /escratch/jsmith_Oct_22. The life span of the directory will be one week longer than the longest duration queue, which is currently 30 days (i.e., life span = 37 days). At that time, the directory and its contents will be deleted. Users can create one escratch directory per day if needed.
===Archive File System===

There is an archive file system available for long-term storage of data that users don’t actively need in their home directories. It is subscribed to by a PI on behalf of his/her lab group, and is mounted on the compute cluster’s login nodes (not on the compute nodes) under oflow e.g., /oflow/abclab. There is a fee for this storage, which is currently $10 / 1TB / month, with the smallest increment being 500GB @ $5 / month. Contact the GACRC staff if you would like more information on this resource. Archived files are backed up.

==Security==

To minimize disruption of service, protect data integrity, conserve facility resources and maximize the effectiveness of staff support, the GACRC maintains strict security requirements for access to GACRC resources. Over time, the enforcement of these requirements will become increasingly strict, with the goal of preventing any access to the GACRC resources by any person or any device that is not in strict compliance with these requirements.
===''User-Managed Servers, Clusters, Networks and Desktop Computers''===
===Operating Systems===

Any computer accessing the GACRC for any purpose must meet minimum levels of operating system versions and update (patch) levels. The GACRC will, from time to time, publish these minimum requirements on its website.
===Anti-Virus Software===

Any computer accessing the GACRC for any purpose must meet minimum levels of anti-virus protection. Any computer used by an account holder must have anti-virus software from a source approved by the GACRC, must have that virus protection activated, and must have automatic updates activated for the anti-virus software.
===Suspiciously Behaving Software===

Any software that behaves in a suspicious manner may at any time be terminated and/or deleted from GACRC resources at the sole discretion of the GACRC’s system administrator(s), manager, director, or security staff.
===Suspiciously Behaving Networks and Devices===

Any connection from any device to the GACRC may be terminated at any time, if the device or the connection or a network to which the device is attached appears to be incompliant with the GACRC’s security requirements, seems to be behaving suspiciously, or if a threat emerges requiring termination for intrusion prevention at the sole discretion of the GACRC’s system administrator(s), manager, director, or security staff.
===Account Holder Responsibility===

The account holder is responsible for diligently monitoring and meeting the GACRC’s operating system, intrusion and virus protection standards.
An account holder’s privileges to use GACRC facilities may be terminated by the GACRC Manager or Director at any time, without notice if, in the opinion of either, the account holder is reluctant or averse to practicing diligence in meeting the GACRC’s minimum requirements for intrusion and/or anti-viral protection.

==Storing Sensitive Information on GACRC Resources==
===Sensitive, Private, or Classified Information===

The GACRC does NOT currently warrant that its practices or facilities meet government-mandated requirements for the storage and protection of sensitive, private or classified information. Users may not store such information on GACRC facilities.
===Intellectual Property===

The GACRC strives to protect documents, code, and results data on behalf of account holders. However, the GACRC does not assume responsibility for unauthorized access or data loss due to human or system error.

===Resolving Disagreements about Revocation of Privileges or Provisioning===

If an account holder is denied a request for provisioning of GACRC resources or resource privileges are revoked, the user’s Department Head may appeal to the Vice President for Research. The decision of the Vice President for Research is final.

==System Maintenance and Downtime==
===Planned Maintenance===

Starting March 2016, the GACRC will institute monthly maintenance windows in order to perform maintenance operations requiring system operations to be reduced or interrupted.

The schedule will be as follows:

*The last Wednesday of each month from 10AM to 4PM will be reserved for partial cluster maintenance.
*Twice a year, a two-day shut-down of GACRC services will be scheduled for more complex maintenance operations. These will occur on the last Tuesday and Wednesday of the months of January and July.

These maintenance windows represent periods when the GACRC may choose to drain the queues of running jobs and suspend access to either or both clusters, as well as storage devices for maintenance purposes. Interruptions will be kept as brief as possible.

The GACRC will notify all users at least 10 days in advance that a maintenance window will be in effect. The notification will describe the nature and extent (partial or full) of the interruptions of cluster and or storage services. In case a maintenance window has to be extended due to unavoidable technical reasons, adequate communications will be made to all users.

The impact of the outages will vary, and the GACRC will do its best to preserve pending and running jobs, which is often very doable. Nevertheless, users will need to plan their job submissions around the maintenance windows.
===Unplanned Maintenance and System Outage===

From time to time, hardware, software, and/or environmental factors may cause a system or subsystem to malfunction, causing disruption to service. Also, there may be circumstances or events related to possible security or intrusions which will cause GACRC staff to take systems offline while the nature of the apparent breach is analyzed and appropriate action is taken.

Whenever possible, account holders will be notified by e-mail of these outages in advance, but that may not always be possible. Account holders will be notified by e-mail if the disruption should last more than 30 minutes.

GACRC staff will strive to preserve the work and/or prevent disruption of jobs in process during such outages. However, there may be circumstances which cause disruption of jobs and loss of data. Users are encouraged to implement methods in their code which minimize the effect of unplanned interruption of a job’s execution, such as checkpoints.

==Adding Department or Grant-Sponsored Resources to the GACRC==

Researchers may benefit by adding resources sponsored by grants or departments to the GACRC. In many cases, the cost of doing so will be less than the researcher’s acquisition and maintenance of the resources within their own laboratory or group.
===Usage Model===

When a department or research project sponsors the addition of compute power, storage capacity, and/or software to the GACRC’s compliment of high performance computing resources, the project will have access to the resource capacities that they have sponsored, throughout the duration of the research project, or as agreed upon in a separate service level agreement. When the project could benefit from resources beyond those that the project sponsored, if those resources are available through the GACRC, they will be allocated to the project. When the resources sponsored by a project are not being used by the project, they will become available to other projects.

The project will benefit from the security, environmental, and system administration provided by the GACRC.
===Usage Policy Enforcement===

The GACRC strives to enforce this usage model through the use of resource management software. From time to time the software may not perform in accordance with the policy. Such events, when detected, should be reported to the GACRC system administrator or manager such that corrective action can be taken to prevent such events in the future.
===Funding Model===

During the grant design and writing process, GACRC staff, in collaboration with the Office of the Vice President for Research Office of Sponsored Programs, is available to assist in estimating the level of computing, storage, network bandwidth, software, and services required to meet the objectives of the proposed research project. GACRC staff will provide the cost of acquiring, installing, and maintaining the proposed resources (in compliance with the architectures of the GACRC as well as established best-practices) over the life of the grant. If the grant is awarded, the GACRC will acquire and implement the resources sponsored by the project using funds allocated for such purposes.

Instructional Accounts

2016-01-14T14:32:50Z

Derda:

Generic instructional user accounts are available on the Linux cluster for courses that teach high-performance computing methods. Besides the number of accounts needed, we also need to know the course name (e.g., GENE8940) and what type of resources are required (e.g., special queue, large memory, specific applications or data sets, or none). Typical requests are for 10 - 30 user accounts, with a common data directory where the instructor provides data files for the class.

All users of instructional accounts must attend a training session led by one of our instructors, before being given access to GACRC resources. The training session usually takes about an hour, so please plan for that in your class scheduling.

Requests for accounts should be made via the [http://help.gacrc.uga.edu/ GACRC Support Form] at the beginning of the semester in which the course is taught (i.e., please don't wait until the last minute).

Instructional Accounts

2016-01-14T14:32:15Z

Derda:

Generic instructional user accounts are available on the Linux cluster for courses that teach high-performance computing methods. Besides the number of accounts needed, we also need to know the course name (e.g., GENE8940) and what type of resources are required (e.g., special queue, large memory, specific applications or data sets, or none). Typical requests are for 10 - 30 user accounts, with a common data directory where the instructor provides data files for the class.

All users of instructional accounts must attend a training session led by one of our instructors, before being giving access to GACRC resources. The training session usually takes about an hour, so please plan for that in your class scheduling.

Requests for accounts should be made via the [http://help.gacrc.uga.edu/ GACRC Support Form] at the beginning of the semester in which the course is taught (i.e., please don't wait until the last minute).

User Accounts

2015-07-02T17:48:15Z

Derda:

All persons using GACRC resources must meet the requirements described on the GACRC [http://gacrc.uga.edu/about/policies Policies] page.

GACRC user accounts are managed as part of a "computing lab" group which is registered by a UGA principal investigator (PI) i.e. faculty member. Users are typically post-docs and graduate students working under the direction of the PI. The account creation process is twofold:

====Computing Lab Registration====

A UGA faculty member may request a computing lab with us by providing the following information on our [http://help.gacrc.uga.edu/labAcct.php Lab Registration Form]:

*Your Name
*Email Address (primary account where GACRC's notifications will be sent)
*Phone Number
*Your MyID
*Department Name
*Department Number (3-digit)
*Please let us know if you need a user account yourself e.g., "user account = No"

By applying for a group account, the Principal Investigator is agreeing that all users will abide by the [http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup UGA Policies on the Use of Computers].

====User Accounts====

The PI of a computing lab must request user accounts for members of his/her computing lab. Please use our [http://help.gacrc.uga.edu/userAcct.php User Accounts Form] and provide the following information:

*Your Name
*Your Email Address
*Your Phone Number
*User's Name
*User's MyID
*User's Email Address (primary account where GACRC's notifications will be sent)
*A phone number where the user can be reached

'''NOTE: New users are required to attend an introductory training session and information about that will be sent once an account is requested.
'''

User Accounts

2015-07-02T17:47:44Z

Derda:

All persons using GACRC resources must meet the requirements described on the GACRC [http://gacrc.uga.edu/about/policies Policies] page.

GACRC user accounts are managed as part of a "computing lab" group which is registered by a UGA principal investigator (PI) i.e. faculty member. Users are typically post-docs and graduate students working under the direction of the PI. The account creation process is twofold:

====Computing Lab Registration====

A UGA faculty member may request a computing lab with us by providing the following information on our [http://help.gacrc.uga.edu/labAcct.php Lab Registration Form]:

*Your Name
*Email Address (primary account where GACRC's notifications will be sent)
*Phone Number
*Your MyID
*Department Name
*Department Number (3-digit)
*Please let us know if you need a user account yourself e.g., "user account = No"

By applying for a group account, the Principal Investigator is agreeing that all users will abide by the [http://eits.uga.edu/access_and_security/infosec/pols_regs/policies/aup UGA Policies on the Use of Computers].

====User Accounts====

The PI of a computing lab must request user accounts for members of his/her computing lab. Please use our [http://help.gacrc.uga.edu/userAcct.php User Accounts Form] and provide the following information:

*Your Name
*Your Email Address
*Your Phone Number
*User's Name
*User's MyID
*User's Email Address (primary account where GACRC's notifications will be sent)
*A phone number where the user can be reached

**NOTE: New users are required to attend an introductory training session and information about that will be sent once an account is requested.

Georgia Advanced Computing Resource Center

2015-01-22T14:24:50Z

Derda:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"
*[https://blog.gacrc.uga.edu/ Blog] – announcements
*[https://forums.gacrc.uga.edu/ Forums] – user discussion area







<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Online </span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Running Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>

Consulting

2014-12-09T13:58:48Z

Derda: /* GACRC IT Staff */

===Perl and UNIX-shell scripting===

Automating and simplifying tasks with command-scripts is a routine procedure in the Unix/Linux environment. We regularly help users develop scripts to suit their specific needs.

===Code Development===

We provide assistance with code compilation and debugging in serial and parallel environments.

===Computational Physics===

Computational physics methods are sometimes useful and applicable to problems in other fields. We provide some
consultation for computational physics algorithm selection and implementation.

===Bioinformatics===

From the installation of programs and data (e.g., genetic databases), to helping users get started, bioinformatics support is a key support area.

===General Usage===

GACRC staff are available to provide consultation in order to facilitate and optimize the use of GACRC’s resources. We are available to meet with small groups of users, and to provide guest lectures for courses that use GACRC resources.

==People==

===Administrators===

*'''Guy Cormier, Ph.D.''' <br />Director, UGA Georgia Advanced Computing Resource Center<br />Associate Research Scientist, Institute of Bioinformatics

===GACRC Consultants===

*'''Greg Derda, Ph.D.''' <br /> Operations Manager
*'''Yecheng Huang''' <br /> Bioinformatics consultant
*'''Shan-Ho Tsai, Ph.D.''' <br /> Computational Physics and High Performance Computing consultant

===GACRC IT Staff===

*'''Paul Brunk''' <br /> Principal Unix Systems Administrator
*'''Help Wanted''' <br /> Principal Unix Systems Administrator

==Office Hours==
GACRC staff are located in 4098C Stegeman Coliseum. Our office is open Monday though Friday from 9am until 5pm. You are welcome to stop by and ask questions, or you can schedule an appointment using the [http://help.gacrc.uga.edu/ GACRC Support Form].

==Directions==
As you face Stegeman Coliseum from Carlton St., there is a ramp that leads from the street up to the Coliseum, to the left of the new glass wall. From the top of that ramp, you will see a small stairwell leading down to a single glass door. Come in through that door and climb the steps to the 4th floor door, which opens into the GACRC workspace.

Consulting

2014-12-09T13:58:37Z

Derda: /* GACRC IT Staff */

===Perl and UNIX-shell scripting===

Automating and simplifying tasks with command-scripts is a routine procedure in the Unix/Linux environment. We regularly help users develop scripts to suit their specific needs.

===Code Development===

We provide assistance with code compilation and debugging in serial and parallel environments.

===Computational Physics===

Computational physics methods are sometimes useful and applicable to problems in other fields. We provide some
consultation for computational physics algorithm selection and implementation.

===Bioinformatics===

From the installation of programs and data (e.g., genetic databases), to helping users get started, bioinformatics support is a key support area.

===General Usage===

GACRC staff are available to provide consultation in order to facilitate and optimize the use of GACRC’s resources. We are available to meet with small groups of users, and to provide guest lectures for courses that use GACRC resources.

==People==

===Administrators===

*'''Guy Cormier, Ph.D.''' <br />Director, UGA Georgia Advanced Computing Resource Center<br />Associate Research Scientist, Institute of Bioinformatics

===GACRC Consultants===

*'''Greg Derda, Ph.D.''' <br /> Operations Manager
*'''Yecheng Huang''' <br /> Bioinformatics consultant
*'''Shan-Ho Tsai, Ph.D.''' <br /> Computational Physics and High Performance Computing consultant

===GACRC IT Staff===

*'''Paul Brunk''' <br /> Principal Unix Systems Administrator
*'''Help Wanter''' <br /> Principal Unix Systems Administrator

==Office Hours==
GACRC staff are located in 4098C Stegeman Coliseum. Our office is open Monday though Friday from 9am until 5pm. You are welcome to stop by and ask questions, or you can schedule an appointment using the [http://help.gacrc.uga.edu/ GACRC Support Form].

==Directions==
As you face Stegeman Coliseum from Carlton St., there is a ramp that leads from the street up to the Coliseum, to the left of the new glass wall. From the top of that ramp, you will see a small stairwell leading down to a single glass door. Come in through that door and climb the steps to the 4th floor door, which opens into the GACRC workspace.

Georgia Advanced Computing Resource Center

2014-11-11T14:38:35Z

Derda:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"
*[https://blog.gacrc.uga.edu/ Blog] – announcements
*[https://forums.gacrc.uga.edu/ Forums] – user discussion area







<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Online </span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Running Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]
* [[Archive Storage]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>

Frequently Asked Questions

2014-08-12T12:42:48Z

Derda: /* How do I forward X Window applications running on GACRC machines to my Windows desktop? */

===How do I find out which compute platforms and software are available at GACRC?===

A list of GACRC systems, including a brief description of the compute platforms, is available at the [[Systems]] page.

Applications and software available on GACRC machines are grouped by category.
For an alphabetical list of all software installed please refer to the [[Software]] page. You can sort by Category or by software Name.

===How do I apply for accounts on GACRC machines?===

User accounts are created as part of a "lab group" which has been registered by a Principal Investigator (PI), i.e. a UGA faculty. Once the group is registered, the PI will receive an email stating that he/she can request individual accounts for members of his/her group. For more information, please see http://gacrc.uga.edu/accounts

===How do I connect to a GACRC machine?===

Users can access GACRC machines using secure shell (ssh) from their local machines either on-campus or off-campus. To connect via ssh, you must have an ssh software on your local machine and a connection to the UGA campus network. ssh software is included in recent releases of Unix based operating systems (including Linux and Mac OSX). An SSH Secure Shell Utility is available for Windows on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page (you will need to enter your UGA myID and password to download the software). Note that connection via telnet is not allowed .

For more detailed information on how to connect to a specific GACRC machine, please see the [[Connecting]] page.

===How do I copy files to/from a GACRC machine?===

Users can transfer files between their local machines and GACRC machines using FTP with explicit SSL encryption, a secure copy (scp), or an SSH secure File Transfer software (SSH file transfer). To transfer files using scp (or SSH file transfer) you must have scp (or SSH) on your local machine and a connection to the UGA campus network. An scp software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure File Transfer Utility is available for Windows on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page (you will need to enter your UGA myID and password to download the software).

Users are encouraged to use FTP with explicit SSL encryption for file transfer to zcluster. Two file transfer software that support FTP with explicit SSL encryption are the open source software [http://filezilla-project.org/ FileZilla] (available for Windows, Mac OS X, and Linux) and [http://winscp.net/ WinSCP] (available for Windows machines).

For more detailed information on how to copy files to/from a specific GACRC machine, please see the [[Transferring Files]] page.

===How do I print files that are on a GACRC machine?===

Files on any GACRC machine cannot be directly printed on your local printer. Please transfer the files from the GACRC machine to your local machine (see question above) and print them from your local machine.

===How do I run jobs on a GACRC machine?===

Jobs should not be run on the login node of GACRC machines. The Linux cluster (zcluster) uses a variant of the Sun Grid Engine queueing system, which should be used for both interactive and batch jobs. For more information, please see the [[Running Jobs on zcluster]] page.

===How do I change my default shell on a given machine?===

When you login to an GACRC machine, the environment on your terminal and the commands that you type at the prompt are defined/interpreted by a program called shell. Examples of shell are bash, csh, ksh, tcsh. The syntax for setting environment variables and some of the functionality of your keyboard depend on the shell that you are running. For example, with bash and tcsh it is straightforward to use up arrows to recover previous commands. All users have a default shell defined at account creation time. Users can change their default shell by typing '''chsh''' at the command prompt. Note that this command needs to the run on the login node and on the interactive nodes (if you want the shell changed on them as well).

===Can I use text files (programs, scripts, etc) created on a Windows machine on the GACRC Unix/Linux machines?===

Text (ascii) files created on Windows machines might have Windows newlines that are not interpreted correctly by a Unix/Linux system. However, you can convert a Windows text file to the Unix/Linux format with the dos2unix command available on the GACRC altix and rcluster. The syntax is

<pre class="gcommand">
dos2unix filename
</pre>

where ''filename'' is the name of the ascii file (such as program.c, program.f, run.sh, input.txt, etc) created on a Windows machine.

===Can I use text files (programs, scripts, etc) created on a Mac on the GACRC Unix/Linux machines?===

Text (ascii) files created on computers running Mac OS might have Mac newlines that are not interpreted correctly by a Unix/Linux system. However, you can convert a Mac text file to the Unix/Linux format with the mac2unix command available on the GACRC altix and rcluster. The syntax is:

<pre class="gcommand">
mac2unix filename
</pre>

where ''filename'' is the name of the ascii file (such as program.c, program.f, run.sh, input.txt, etc) created on a machine with Mac OS.

===How do I forward X Window applications running on GACRC machines to my Windows desktop?===

A number of software installed on GACRC machines have X Window (GUI) front ends. Examples of such applications are Mathematica, Maple, some text editors and debuggers, etc. In order to export such X Window applications to your Windows desktop, your desktop needs to have an X Window client (or server) running on it. A free X Window server for Microsoft Windows (XP/2008/Windows7) is [http://sourceforge.net/projects/xming/ Xming]. You can download it from [http://sourceforge.net/projects/xming/ Sourceforge] and make a default installation. You will need to install the Xming server and the Xming-fonts package. Some applications also require having Xming-mesa installed. During the installation of Xming, you might want to select the option to create a desktop icon for Xming. When the installation of these two packages is complete, double click on the Xming icon to start the X Window server (a capital X will appear on your task bar). Now you need to configure your SSH client to allow tunneling of X11 connections. For example, if you use the SSH client provided at the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page you need to open the SSH client, then click on

Edit -> Settings -> Tunneling (under Profile Settings)

and check the box for "Tunnel X11 connections", and then save the settings. Once that is done, you can SSH into your GACRC account (zcluster, stats) and run X Window applications. The application should appear on your local Windows desktop. Each time you logout and log back into your Windows desktop, you might need to start the Xming Server manually before using SSH to connect to your GACRC accounts.

===How do I forward X Window applications running on GACRC machines to my Mac?===
For Mac Mountain Lion (v10.8 so far), user has to manually one-time install XQuartz to enable X11 feature according to [http://support.apple.com/kb/HT5293 Mac]. It is free and available at [http://xquartz.macosforge.org/landing/ XQuartz].

For other version lower than 10.8 (non Mountain), X11 is built in with OS, no need to install extra component.

Then connect to zcluster as:

<pre class="gcommand">
ssh -X username@zcluster.rcc.uga.edu
</pre>

===How do I acknowledge the GACRC in my publication?===

A sample acknowledgment statement is provided at http://gacrc.uga.edu/about/acknowledgment-statement

Connecting

2014-08-12T12:42:07Z

Derda:

[[Category:zcluster]]
[[Category:SAS Server]]

===Connecting to zcluster===

Access to the Linux cluster (zcluster) is through secure shell (ssh) to zcluster.rcc.uga.edu (note that connection via telnet is not safe and it is therefore disabled). If you have a local system that runs an X Window client or server, you can run X applications remotely.

To connect via ssh, you must have an ssh software and a connection to the UGA campus network. An ssh software is included in recent releases of Unix based operating systems (including Linux and Mac OS X). An SSH Secure Shell Utility is available for Windows on the [https://eits.uga.edu/hardware_and_software/software UGA Sitesoft] page (you will need to enter your UGA myID and password to download the software).

Open a connection to host: zcluster.rcc.uga.edu

'''Examples of usage on Unix/Linux/MacOSX'''

To open a connection to zcluster.rcc.uga.edu, first open a terminal and at the command prompt type

<pre class="gcommand">
ssh username@zcluster.rcc.uga.edu
</pre>

or

<pre class="gcommand">
ssh -X username@zcluster.rcc.uga.edu
</pre>

where '''''username''''' is your username on zcluster. The -X option might be necessary to allow X windows applications running on zcluster to be forwarded to your local machine. If your username on your local system and on the zcluster are the same, you can simply type

<pre class="gcommand">
ssh zcluster.rcc.uga.edu
</pre>

In order to enable X forwarding from zcluster to a local '''Mac OS X machine''', the ssh command given above (i.e. ssh -X username@zcluster.rcc.uga.edu) has to be issued from an '''X11 terminal''' (and not from a regular terminal).

===To run X Window applications remotely on your local machine===

If you have a local machine that runs an X Window client or server, you can forward X Window applications running on zcluster to your local machine. Recent releases of Unix based operating systems (including Linux and Mac OS X) come with an X windows client. For Windows OS, a free X windows server is Xming. More information on how to install Xming is available on the [[Frequently Asked Questions | GACRC FAQ]] page.

Your ssh may be configured to allow you to run X Window application on your local machine automatically. If it does not, try including the -X flag when connecting to the zcluster (ssh -X username@zcluster.rcc.uga.edu).

Now you can issue commands to start up X applications on zcluster.rcc.uga.edu, such as xterm.

If you plan to run resource consuming X Window applications, such as Matlab, graphics programs, etc, please use the queueing system to get a command prompt on an interactive node. Information on how to run interactive jobs on zcluster is at the [[Running Jobs on zcluster]] page.

===Logging in===

When your ssh connection is established, you will be prompted for your password. The first time you login to zcluster, please change your initial password with the '''passwd''' command. This command can be used afterwards as well, to change your password.

===Logging Out===

To leave the system, be sure that all your open files are closed and all stopped jobs have been terminated then enter:

exit

Systems

2014-08-12T12:40:03Z

Derda:

[[Category:Zcluster]]
[[Category:SAS Server]]

'''Linux cluster (zcluster)'''

The Linux cluster is comprised of compute nodes with 4-, 6-, 8-, and 12-core processors from both Intel and AMD. Subsets of nodes have "large memory" (e.g., 128, 256, or 512 GB of RAM), while others have InfiniBand connectivity or GPU capabilities. Total CPU compute power is 25.9 Tflops.

The cluster is currently comprised of the following resources:

*230 compute nodes (2600 compute cores), 32 with InfiniBand connectivity.
*Four 8-core, 192GB high-memory compute nodes
*Ten 12-core, 256GB high-memory compute nodes
*Two 32-core, 512GB high-memory compute nodes
*Six 32-core, 64GB high-memory compute nodes
*One NVIDIA Tesla S1070 with four GPU cards (4 x 240 = 960 GPU cores) for programs written to use this architecture.
*One NVIDIA Tesla (Fermi) C2075 GPU processor (448 GPU cores)
*Nine NVIDIA Tesla (Fermi) M2070 GPU cards (9 x 448 = 4032 GPU cores). These cards are installed on 2 hosts each of which has dual 6-core Intel Xeon CPUs and 48GB of RAM; there are 6 GPU cards on one host and 3 on the other.
*32 NVIDIA Tesla (Kepler) K20X GPU cards (32 x 2688 = 86016 GPU cores). These cards are installed on 4 hosts each of which has dual 6-core Intel Xeon CPUs and 96GB of RAM; there are 8 GPU cards per host.

[[Connecting]]

[[Transferring Files]]

[[Disk Storage]]

[[Code Compilation on zcluster]]

[[Running Jobs on zcluster]]

Systems

2014-08-12T12:39:33Z

Derda:

Consulting

2014-08-04T16:32:35Z

Derda: /* GACRC IT Staff */

===Perl and UNIX-shell scripting===

Automating and simplifying tasks with command-scripts is a routine procedure in the Unix/Linux environment. We regularly help users develop scripts to suit their specific needs.

===Code Development===

We provide assistance with code compilation and debugging in serial and parallel environments.

===Computational Physics===

Computational physics methods are sometimes useful and applicable to problems in other fields. We provide some
consultation for computational physics algorithm selection and implementation.

===Bioinformatics===

From the installation of programs and data (e.g., genetic databases), to helping users get started, bioinformatics support is a key support area.

===General Usage===

GACRC staff are available to provide consultation in order to facilitate and optimize the use of GACRC’s resources. We are available to meet with small groups of users, and to provide guest lectures for courses that use GACRC resources.

==People==

===Administrators===

*'''Guy Cormier, Ph.D.''' <br />Director, UGA Georgia Advanced Computing Resource Center<br />Associate Research Scientist, Institute of Bioinformatics

===GACRC Consultants===

*'''Greg Derda, Ph.D.''' <br /> Operations Manager
*'''Yecheng Huang''' <br /> Bioinformatics consultant
*'''Shan-Ho Tsai, Ph.D.''' <br /> Computational Physics and High Performance Computing consultant

===GACRC IT Staff===

*'''Paul Brunk''' <br /> Principal Unix Systems Administrator
*'''Todd Sirmans''' <br /> Principal Unix Systems Administrator

==Office Hours==
GACRC staff are located in 4098C Stegeman Coliseum. Our office is open Monday though Friday from 9am until 5pm. You are welcome to stop by and ask questions, or you can schedule an appointment using the [http://help.gacrc.uga.edu/ GACRC Support Form].

==Directions==
As you face Stegeman Coliseum from Carlton St., there is a ramp that leads from the street up to the Coliseum, to the left of the new glass wall. From the top of that ramp, you will see a small stairwell leading down to a single glass door. Come in through that door and climb the steps to the 4th floor door, which opens into the GACRC workspace.

Georgia Advanced Computing Resource Center

2014-04-17T18:38:29Z

Derda:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"
*[https://blog.gacrc.uga.edu/ Blog] – announcements
*[https://forums.gacrc.uga.edu/ Forums] – user discussion area







<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Online </span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]
* [[Hadoop Cluster Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Running Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]
* [[Archive Storage]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>

Walts rule

2014-04-17T18:38:05Z

Derda: Created page with "From one of our favorite users, Dr. Walt (Bubba) Lorenz: Important README: Remember that before any updates or changes are made on the zcluster anyone who makes said changes..."

From one of our favorite users, Dr. Walt (Bubba) Lorenz:

Important README:

Remember that before any updates or changes are made on the zcluster anyone who makes said changes/updates should always check first to make sure that Walt's jobs (wlorenz or qbcg2) cannot possibly be negatively impacted- this will henceforth be known as "Walt's Rule." Also, the Corollary to Walt's Rule states that, and I quote, "Any changes made without first assessing Walt's job status will inevitably result in his jobs crashing and lead to the elicitation of numerous harassing phone calls and inquiries from Walt to the GACRC staff." Neither of these should not be confused with Walt's Rule of Roots, which states: "The square root of the cumulative number of days that Walt has been refused sudoer privileges since first asking divided by the number of jobs he has running is proportional to the number of phone calls he makes each day to the GACRC helpline."

These should probably be entered on the GACRC Wiki when someone has some free time (Greg).

Best,

Walt

Georgia Advanced Computing Resource Center

2014-04-17T18:36:57Z

Derda:

__NOTOC__
Welcome to the Georgia Advanced Computing Resource Center wiki. The information provided here is a supplement to the GACRC webpage. The GACRC online information resources include:

*[http://gacrc.uga.edu/ Web Site] – general overview
*[https://wiki.gacrc.uga.edu/ Wiki] – software docs and how-to’s - "You Are Here"
*[https://blog.gacrc.uga.edu/ Blog] – announcements
*[https://forums.gacrc.uga.edu/ Forums] – user discussion area







<div style="width=100%; margin:0; background:#00CC33; font-size:120%; font-weight:bold; border:1px solid #00CC33; text-align:left; color:white; padding:0.2em 0.4em;"> Current Status: <span style="color:black"> Online </span></div>

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Getting Started </div>
Welcome to the Georgia Advanced Computing Resource Center at the University of Georgia. If you're new to the GACRC, start with these links to get acquainted with our resources.
* [[User Accounts]]
* [[Connecting]]
* [[Transferring Files]]
* [[Password | Changing your Password]]
* [[Frequently Asked Questions]]
* [[Command List]]
* [[Getting Help]]
* [[Policies]]
* [[Consulting]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Services </div>
Services and other resources.
* [[Instructional Accounts]]
* [[Hadoop Cluster Accounts]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> System Information </div>
Hardware information and operational procedures are described below.
* [[Systems]]
* [[Disk Storage]]
* [[Running Jobs]]
* [[Job Submission Queues]]
* [[Sample Scripts | Sample Job Scripts]]
* [[Archive Storage]]

<div style="width=100%; margin:0; background:#333333; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee; padding:0.2em 0.4em;"> Software and Libraries </div>
Documentation for software applications, programming tools, and usage.
* [[Software]]
* [[Bioinformatics Databases]]
* [[OpenMP]]
* [[MPI | Message Passing Interface (MPI)]]
* [[Compilers]]
* [[GPU|GPU and CUDA Programming]]
* [[Galaxy]]

<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Knowledge Base]]</div>
<br />
<div style="width=100%; margin:0; background:#eeeeee; font-size:120%; font-weight:bold; border:1px solid #f9f9f9; text-align:left; color:#eeeeee padding:0.2em 0.4em;">
[[GACRC Advisory Committee]]</div>

[[walts_rule]]

Disk Storage

2014-02-21T19:35:27Z

Derda:

[[Category:Zcluster]][[Category:Storage]]

== Storage Overview ==
Network attached storage systems at the GACRC are tiered in three collections of systems based on speed and capacity. Our fastest network attached storage system is a Panasas ActiveStor 12 which exports 156TB of data, mounted on every node at /panfs and is divided into two categories: home and scratch.

== Home Directories ==

'''Home directories are for highly static datasets. This volume can not support a high degree of variability with the features (like snapshots) it provides. Please make sure that any data stored in your home directory is limited to applications and inputs that you use frequently.'''

All users have a default 300GB home quota (i.e., maximum limit) on their home directory; however, justifiable requests for quotas up to 2TB can be made by making a formal request through the [http://help.gacrc.uga.edu/ GACRC Website contact form]. '''Storage in the home directory to avoid archive storage fees is not a justifiable request.''' Requests for home quotas greater than 2TB must be submitted by the PI of a lab group, and approved by the GACRC advisory committee (via the IT Manager). Users may create lab directories for data that is shared by a lab group, but those directories count against the quota of the creating user. An example of this, for the “abclab” users, would be: /home/abclab/labdata. Home directories are snapshotted.

=== Snapshots ===
Lab home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be copied from to restore files that may have been accidentally deleted or overwritten.

Home directories have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them.

Any directory on the /home filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /home directories may retrieve files from these snapshots by using the "cd" command and copying files from the appropriate snapshot to any location they would like.

'''Note: ANY user, from any HOME directory can access the snapshots *from that directory* to restore files'''

For example:

<pre>
[cecombs@sites test]$ pwd
/home/rccstaff/cecombs/test
[cecombs@sites ~]$ cd test
[cecombs@sites test]$ ls
Main.java
[cecombs@sites test]$ rm -rf Main.java
[cecombs@sites test]$ cd .snapshot
[cecombs@sites .snapshot]$ ls
2013.04.16.00.00.01.daily 2013.04.17.00.00.01.daily 2013.04.18.00.00.01.daily
[cecombs@sites .snapshot]$ cd 2013.04.18.00.00.01.daily/
[cecombs@sites 2013.04.18.00.00.01.daily]$ ls
Main.java
[cecombs@sites 2013.04.18.00.00.01.daily]$ cp Main.java /home/rccstaff/cecombs/test
[cecombs@sites 2013.04.18.00.00.01.daily]$ cd /home/rccstaff/cecombs/test
[cecombs@sites test]$ ls
Main.java
</pre>

== Scratch ==

Scratch Directories are for dynamic data. Scratch space is specifically designed to handle datasets that grow and shrink on-demand. The GACRC does not and can not snapshot scratch directories because the amount of data which changes periodically is too great and snapshots would only serve to slow the file systems down.

=== eScratch ===
eScratch directories are for ephemeral datasets; most commonly, the output of large calculations that need to be stored in a temporary place for a short period of time. Any user can make escratch directories for their work. Ephemeral scratch directories on GACRC clusters reside on a Panasas ActiveStor 12 storage cluster.

==== Making an eScratch Directory ====
Researchers who need to use scratch space can type '''(''on the login node zcluster.rcc ONLY'')'''

<pre class="gcommand">
make_escratch
</pre>

a sub-directory will be created, and the user will be told the path to the sub-directory, e.g. /panfs/pstor.storage/escratch1/jsmith_Oct_22. The life span of the directory will be one week longer than the longest duration queue, which is currently 30 days (i.e., life span = 37 days). '''At that time, the directory and its contents will be systematically deleted. ''' Users can create one escratch directory per day if needed. The total space a user can use on escratch (all escratch directories '''combined''') is 4TB. The escratch directories are not backed up.

If a user needs to retain their self-created escratch directory for more than the 37 days which they were allocated, they may contact the GACRC staff for an extension through the [http://help.gacrc.uga.edu/ support form]. We will grant almost all requests, but escratch directories do use precious HPC storage space and we must ask that a new request be submitted for every 37 day period that they are needed to ensure that the space is freed as soon as possible.

=== lscratch ===

lscratch stands for local scratch and is available on every node in the zcluster.

==== lscratch information ====
All /lscratch filesystems on every node have these properties:
* lscratch is by far the fastest possible filesystem at the GACRC, however the lscratch directory is only available to the node that a job gets scheduled to.
* lscratch filesystem resides on the local hard drive of the node.
* Represents the remainder of unused disk after the OS is installed.
* Multiple different sizes for /lscratch; nodes have different sized disks.
* Not accessible from other nodes
* Every user has a directory on every node, /lscratch/<username>

==== lscratch Guidelines ====
This is a list of guidelines for /lscratch usage:
* Do not count on any lscratch sizes above 10G unless you know the size of the local hard drive and target that node specifically (e.g.: qsub -l h=compute-15-36)
* You will be responsible for migrating your data from the node after your job finishes. The job itself can transfer the data.
* Make sure that your output goes to: /lscratch/<username> (e.g: /lscratch/cecombs)

== Quotas ==

To see how much space you are consuming on the home and scratch file systems, please use the command

<pre class="gcommand">
quota_rep
</pre>

== Overflow/Archival Storage ==

Some labs also have a subscription archival storage space, which is mounted on the zcluster login node and on the [[Transferring Files | copy nodes]] as /oflow (note that /oflow is not mounted on the compute nodes). The archival storage system is for long-term storage of large, static datasets.

This filesystem is snapshotted. The snapshots are available only from the mount point under the hidden ".zfs" directory (e.g.: /oflow/jlmlab/.zfs). Overflow devices are snapshotted once every hour, day, week and month. 24 hourlies, 7 dailies, 4 weeklies and 4 monthlies are kept.

Please contact the GACRC staff to request Overflow storage.