Disk Storage - Revision history

Shtsai: /* Scratch file system */

2025-04-17T19:03:32Z

Scratch file system

← Older revision		Revision as of 15:03, 17 April 2025
Line 45:		Line 45:

	<blockquote style="background-color: lightyellow; border: solid thin grey;">		<blockquote style="background-color: lightyellow; border: solid thin grey;">
	Any file that is not accessed or modified by a compute job in a time period of at least 30 days will be automatically deleted off the /scratch file system. Measures circumventing this policy will be monitored and actively discouraged.		Any file that is not accessed/read or modified by a compute job in a time period of at least 30 days will be automatically deleted off the /scratch file system. Measures circumventing this policy will be monitored and actively discouraged.

	There is no storage size quota for /scratch usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their /scratch directories or reduce temporarily the 30 day limit to a lower limit.		There is no storage size quota for /scratch usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their /scratch directories or reduce temporarily the 30 day limit to a lower limit.

Gcormier: /* Current Storage Systems */

2023-03-27T23:15:58Z

Current Storage Systems

← Older revision		Revision as of 19:15, 27 March 2023
Line 269:		Line 269:
	== Current Storage Systems ==		== Current Storage Systems ==

	(1) ~~Seagate~~ (~~Xyratex) ClusterStor1500 Lustre appliance (480TB~~) - $~~SCRATCH~~ on Sapelo2		(1) ZFS storage chain (300TB) - $HOME on Sapelo2

	(2) DDN SFA14KX Lustre appliance (1.~~26PB~~) - $SCRATCH & $WORK on Sapelo2		(2) DDN SFA14KX Lustre appliance (2.5PB) - $SCRATCH & $WORK on Sapelo2

	(3) ~~Penguin IceBreakers ZFS storage chains~~ (~~84TB usable capacity~~) - $~~HOME on~~ Sapelo2		(3) Panasas ActiveStor 100H (1PB) - $PROJECT research groups' long-term space - only for active projects requiring Sapelo2 access

	(4) ~~Penguin IceBreakers~~ ZFS storage ~~chains~~ (~~374TB usable capacity~~) - $PROJECT research groups' long-term space - only for active projects requiring Sapelo2 access		(4) ZFS storage chain (1.2PB) - $PROJECT research groups' long-term space - only for active projects requiring Sapelo2 access

	(5) ~~Panasas ActiveStor 100H~~ (~~1PB~~) - $PROJECT ~~research groups' long-term space - only for active projects requiring Sapelo2 access~~		(4) ZFS storage chains (2.4PB) - backup environment for $HOME and $PROJECT.

	~~(6) ZFS storage chains (720TB) - backup environment for $HOME and $PROJECT.~~

Ben: Update /home snapshot info

2022-09-22T15:36:32Z

Update /home snapshot info

← Older revision		Revision as of 11:36, 22 September 2022
Line 160:		Line 160:
	Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.		Home directories are snapshotted. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

	Home directories on ~~sapelo2~~ have snapshots taken once a day and maintained for 4 days, giving the user the ability to retrieve old files for up to 4 days after they have deleted them. ~~Weekly and monthly snapshots are also made and a few recent snapshots are stored.~~		Home directories on Sapelo2 have snapshots taken once a day and are maintained on Sapelo2 for 14 days, giving the user the ability to retrieve old files for up to 14 days after they have deleted them.

	'''Note: Users can access the snapshots of their own home directories and restore their files.'''		'''Note: Users can access the previous 14 days of snapshots of their own home directories and restore their files.'''

	If you would like to recover a file that you have deleted from your home directory, you can check if the file is available in any of the snapshots and, if so, copy the file back. This can be done on an xfer ~~node~~ or on a Sapelo2 compute node.		If you would like to recover a file that you have deleted from your home directory within the last 14 days, you can check if the file is available in any of the snapshots and, if so, copy the file back. This can be done on a transfer node (xfer.gacrc.uga.edu) or on a Sapelo2 compute node.

	Here is an example for user jsmith, on an xfer node:		Here is an example for user jsmith, on an xfer node:
Line 172:		Line 172:
	/home/jsmith		/home/jsmith

	[jsmith@xfer1 ]$ cd /home/.zfs/snapshot		[jsmith@xfer1 ]$ ls /home/.zfs/snapshot/
			zrepl_20220907_063420_000 zrepl_20220920_220422_000 zrepl_20220921_200422_000
			zrepl_20220908_070420_000 zrepl_20220920_230422_000 zrepl_20220921_210422_000
			zrepl_20220909_073421_000 zrepl_20220921_000422_000 zrepl_20220921_220422_000
			zrepl_20220910_073421_000 zrepl_20220921_010422_000 zrepl_20220921_230422_000
			zrepl_20220911_073421_000 zrepl_20220921_020422_000 zrepl_20220922_000422_000
			zrepl_20220912_073421_000 zrepl_20220921_030422_000 zrepl_20220922_010421_000
			zrepl_20220913_073421_000 zrepl_20220921_040422_000 zrepl_20220922_020422_000
			zrepl_20220914_073421_000 zrepl_20220921_050422_000 zrepl_20220922_030422_000
			zrepl_20220915_073421_000 zrepl_20220921_060422_000 zrepl_20220922_040422_000
			zrepl_20220916_073421_000 zrepl_20220921_070422_000 zrepl_20220922_050421_000
			zrepl_20220917_073421_000 zrepl_20220921_080422_000 zrepl_20220922_060422_000
			zrepl_20220918_080421_000 zrepl_20220921_090421_000 zrepl_20220922_070422_000
			zrepl_20220919_080422_000 zrepl_20220921_100422_000 zrepl_20220922_080422_000
			zrepl_20220920_083422_000 zrepl_20220921_113421_000 zrepl_20220922_090422_000
			zrepl_20220920_143422_000 zrepl_20220921_123422_000 zrepl_20220922_100422_000
			zrepl_20220920_153422_000 zrepl_20220921_133423_000 zrepl_20220922_110422_000
			zrepl_20220920_163422_000 zrepl_20220921_150422_000 zrepl_20220922_120422_000
			zrepl_20220920_173422_000 zrepl_20220921_160422_000 zrepl_20220922_130422_000
			zrepl_20220920_190421_000 zrepl_20220921_170422_000 zrepl_20220922_140422_000
			zrepl_20220920_200422_000 zrepl_20220921_180421_000 zrepl_20220922_143422_000
			zrepl_20220920_210422_000 zrepl_20220921_190422_000 zrepl_20220922_150422_000

	[jsmith@xfer1 ~~snapshot~~]$ ls		[jsmith@xfer1 ]$ cd /home/.zfs/snapshot/zrepl_20220907_063420_000/jsmith
	~~autosnap_2020-03-01_00:00:21_monthly autosnap_2020-03-22_01:00:49_hourly~~
	~~autosnap_2020-03-19_00:00:21_daily autosnap_2020-03-22_02:00:49_hourly~~
	~~autosnap_2020-03-20_00:00:21_daily autosnap_2020-03-22_03:00:49_hourly~~
	~~autosnap_2020-03-21_00:00:21_daily autosnap_2020-03-22_04:00:49_hourly~~
	~~autosnap_2020-03-22_00:00:21_daily autosnap_2020-03-22_05:00:49_hourly~~

	[jsmith@xfer1 ~~snapshot~~]$ ~~cd autosnap_2020~~-03-~~21_00:00:21_daily~~/jsmith		[jsmith@xfer1 ]$ cp my-to-restore-file /home/jsmith
			</pre>

	[~~jsmith@xfer1 jsmith]$ cp my-to-restore-file~~ /~~home~~/~~jsmith~~/~~test~~		Weekly and monthly snapshots are also made going as far back as 6 months, but GACRC staff must retrieve these snapshots for you upon [https://uga.teamdynamix.com/TDClient/2060/Portal/Requests/ServiceDet?ID=25844 request].
	</~~pre>~~

Shtsai at 17:28, 9 May 2022

2022-05-09T17:28:22Z

← Older revision		Revision as of 13:28, 9 May 2022
Line 15:		Line 15:
	When you login into a system (e.g. sapelo2 or xfer nodes), you will land on your home directory. Home directories are "auto mounted" on the login nodes and xfer nodes when you login. Your home directory on the xfer nodes is the same as your home directory on sapelo2. Sapelo2 compute nodes will also mount a user's home directory when a job starts (be that interactive or batch). Users of the teaching cluster have a separate home directory, which is not the same as on Sapelo2.		When you login into a system (e.g. sapelo2 or xfer nodes), you will land on your home directory. Home directories are "auto mounted" on the login nodes and xfer nodes when you login. Your home directory on the xfer nodes is the same as your home directory on sapelo2. Sapelo2 compute nodes will also mount a user's home directory when a job starts (be that interactive or batch). Users of the teaching cluster have a separate home directory, which is not the same as on Sapelo2.

	Home directories have a per user quota and have snapshots. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten. A user's snapshot is stored within his/her home file system, thus snapshots consume a user's home directory quota. If files are created and deleted with frequency, the snapshots will grow and might end up using a large fraction (or all) the space available within a user's home file system.		Home directories have a per user quota and have snapshots. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten.

	The recommended data workflow is to have files in the home directory change as little as possible. These should be databases, applications that you use frequently but do not need to modify that often and other things that you, primarily, read from. Think of snapshots as the memory of the files that were stored there - no matter if you add, change or delete the files, the total sum of that activity will build up over time and may exceed your quota.		<!-- A user's snapshot is stored within his/her home file system, thus snapshots consume a user's home directory quota. If files are created and deleted with frequency, the snapshots will grow and might end up using a large fraction (or all) the space available within a user's home file system.
			-->
			The recommended data workflow is to have files in the home directory change as little as possible. These should be databases, applications that you use frequently but do not need to modify that often and other things that you, primarily, read from. <!-- Think of snapshots as the memory of the files that were stored there - no matter if you add, change or delete the files, the total sum of that activity will build up over time and may exceed your quota. -->

	Summary of the home directory characteristics for a sample user 'jsmith' in 'abclab':		Summary of the home directory characteristics for a sample user 'jsmith' in 'abclab':

Moses at 18:11, 28 April 2022

2022-04-28T18:11:19Z

← Older revision		Revision as of 14:11, 28 April 2022
Line 85:		Line 85:
	The data in lscratch is not backed up and it needs to be deleted when job on the compute node is finished.		The data in lscratch is not backed up and it needs to be deleted when job on the compute node is finished.

	Jobs that do not need to write large output files, but that need to access the files often (for example, to write small amounts of data into disk), can benefit from using /lscratch. Jobs that use /lscratch should request the amount of space in /lscratch. For information on how to request lscratch space for jobs, please refer to [https://wiki.gacrc.uga.edu/wiki/Running_Jobs_on_Sapelo2#~~How_to_run_a_job_from_the_compute_node~~.~~27s_local_disk_.28.2Flscratch.29~~ How to run a job from lscratch]		Jobs that do not need to write large output files, but that need to access the files often (for example, to write small amounts of data into disk), can benefit from using /lscratch. Jobs that use /lscratch should request the amount of space in /lscratch. For information on how to request lscratch space for jobs, please refer to [https://wiki.gacrc.uga.edu/wiki/Running_Jobs_on_Sapelo2#How_to_run_a_job_using_the_local_scratch_.2Flscratch_on_a_compute_node How to run a job from lscratch]

	Summary of the lscratch directory characteristics for a sample user 'jsmith' in 'abclab':		Summary of the lscratch directory characteristics for a sample user 'jsmith' in 'abclab':

Shtsai at 17:06, 4 March 2022

2022-03-04T17:06:53Z

← Older revision		Revision as of 13:06, 4 March 2022
Line 43:		Line 43:

	<blockquote style="background-color: lightyellow; border: solid thin grey;">		<blockquote style="background-color: lightyellow; border: solid thin grey;">
	Any file that is not accessed or modified by a compute job in a time period ~~no longer than~~ 30 days will be automatically deleted off the /scratch file system. Measures circumventing this policy will be monitored and actively discouraged.		Any file that is not accessed or modified by a compute job in a time period of at least 30 days will be automatically deleted off the /scratch file system. Measures circumventing this policy will be monitored and actively discouraged.

	There is no storage size quota for /scratch usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their /scratch directories or reduce temporarily the 30 day limit to a lower limit.		There is no storage size quota for /scratch usage. Space is only limited by the physical size of the scratch space being used. If usage across the entire file system is more than 80% of total capacity, the GACRC will take additional measures to reduce usage to a more suitable level. Amongst possible actions, request/force users to clean up their /scratch directories or reduce temporarily the 30 day limit to a lower limit.

Ben at 16:11, 23 September 2021

2021-09-23T16:11:26Z

← Older revision		Revision as of 12:11, 23 September 2021
Line 22:		Line 22:
	<pre>		<pre>
	sapelo2		sapelo2
	home dir quota = ~~100GB~~		home dir quota = 200GB
	home dir path = /home/jsmith		home dir path = /home/jsmith
	snapshots = yes		snapshots = yes

Ben at 16:10, 23 September 2021

2021-09-23T16:10:46Z

← Older revision		Revision as of 12:10, 23 September 2021
Line 13:		Line 13:
	=== Home file system ===		=== Home file system ===

	When you login into a system (e.g. sapelo2 or xfer nodes), you will land on your home directory. Home directories are "auto mounted" on the login nodes and xfer nodes when you login. Your home directory on the xfer nodes is the same as your home directory on sapelo2. Sapelo2 ~~interactive ("qlogin") nodes will mount a user's home directory when the qlogin happens and~~ compute nodes will mount a user's home directory when a job ~~submitted by this user is dispatched to those compute nodes~~. Users of the teaching cluster have a separate home directory, which is not the same as on Sapelo2.		When you login into a system (e.g. sapelo2 or xfer nodes), you will land on your home directory. Home directories are "auto mounted" on the login nodes and xfer nodes when you login. Your home directory on the xfer nodes is the same as your home directory on sapelo2. Sapelo2 compute nodes will also mount a user's home directory when a job starts (be that interactive or batch). Users of the teaching cluster have a separate home directory, which is not the same as on Sapelo2.

	Home directories have a per user quota and have snapshots. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten. A user's snapshot is stored within his/her home file system, thus snapshots consume a user's home directory quota. If files are created and deleted with frequency, the snapshots will grow and might end up using a large fraction (or all) the space available within a user's home file system.		Home directories have a per user quota and have snapshots. Snapshots are like backups in that they are read-only moment-in-time captures of files and directories which can be used to restore files that may have been accidentally deleted or overwritten. A user's snapshot is stored within his/her home file system, thus snapshots consume a user's home directory quota. If files are created and deleted with frequency, the snapshots will grow and might end up using a large fraction (or all) the space available within a user's home file system.
Line 138:		Line 138:
	sapelo2		sapelo2

	home= ~~100GB~~		home= 200GB
	scratch= Currently no quota		scratch= Currently no quota
	work= (to be added)		work= (to be added)
Line 149:		Line 149:
	== Auto Mounting Filesystems ==		== Auto Mounting Filesystems ==

	Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo2 home directories and the project filesystems. Sapelo2 ~~interactive ("qlogin")~~ nodes will mount a user's home directory when ~~the qlogin happens~~.		Some filesystems are "auto mounted" when they are first accessed on a server. For the xfer nodes, this includes Sapelo2 home directories and the project filesystems. Sapelo2 compute nodes will mount a user's home directory when a job starts.

Shtsai at 15:42, 18 August 2021

2021-08-18T15:42:25Z

@@ Line 187: / Line 187: @@
 '''Project file systems'''
-The project file systems are also snapshotted. Each /project filesystem contains a completely invisible directory named ".snapshot". This directory cannot be listed with ls or viewed by any program at all. Only the "cd" command can be used to enter this directory. Users of /project directories may retrieve files from these snapshots by changing into their snapshot directory /project/abclab/.snapshot and then cd into an appropriate snapshot directory and copying files from the that snapshot to any location they would like.
+The project file systems are also snapshotted and the method to recover a file from a snapshot depends on whether your project directory is located on the Panasas storage device or on a ZFS storage device (SN13). One of the two methods below should work for you.
 '''Note: ANY user in a lab can access the snapshots of his/her group project file system and restore files he/she has there.'''
 Here is an example for user jsmith who is in the abclab group, on an xfer node:
@@ Line 208: / Line 212: @@
 [jsmith@xfer1 2019.03.03.06.00.03.Daily]$ cp my-to-restore-file /home/jsmith/test
 </pre>

Keeko: /* Scratch file system */

2020-11-03T18:47:12Z

Scratch file system

← Older revision		Revision as of 14:47, 3 November 2020
Line 32:		Line 32:

	The scratch file system resides on a high-speed storage device and it should be used to store temporary files needed for current jobs. Files that are not needed for current jobs should not be left on the scratch file system. This file system is mounted on the login nodes, xfer nodes, and compute nodes.		The scratch file system resides on a high-speed storage device and it should be used to store temporary files needed for current jobs. Files that are not needed for current jobs should not be left on the scratch file system. This file system is mounted on the login nodes, xfer nodes, and compute nodes.

	~~The scratch file system is mounted on the login nodes, xfer nodes, and compute nodes.~~

	The recommended data workflow will have jobs write output files, including intermediate data, such as checkpoint files, and final results into the scratch file system. Final results, intermediate files, and other data should then be transferred out of and immediately deleted from the scratch file system, if these are not needed for other jobs that are being submitted soon.		The recommended data workflow will have jobs write output files, including intermediate data, such as checkpoint files, and final results into the scratch file system. Final results, intermediate files, and other data should then be transferred out of and immediately deleted from the scratch file system, if these are not needed for other jobs that are being submitted soon.