Array Jobs - Revision history

Chelsea: /* Numbered Input Files */

2025-07-07T18:27:10Z

Numbered Input Files

← Older revision		Revision as of 14:27, 7 July 2025
Line 19:		Line 19:
	#SBATCH --array=1-5		#SBATCH --array=1-5

	ml R/4.3.1-~~foss~~-~~2022a~~		ml R/4.3.2-gfbf-2023a

	Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}		Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
Line 41:		Line 41:

	<pre class="gcommand">		<pre class="gcommand">
	ml R/4.3.1-~~foss~~-~~2022a~~		ml R/4.3.2-gfbf-2023a

	Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}		Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
Line 61:		Line 61:
	#SBATCH --array=1,3,5		#SBATCH --array=1,3,5

	ml R/4.3.1-~~foss~~-~~2022a~~		ml R/4.3.2-gfbf-2023a

	Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}		Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
Line 102:		Line 102:
	#SBATCH --array=1-3		#SBATCH --array=1-3

	ml R/4.3.1-~~foss~~-~~2022a~~		ml R/4.3.2-gfbf-2023a

	file=$(awk "NR==${SLURM_ARRAY_TASK_ID}" input.lst)		file=$(awk "NR==${SLURM_ARRAY_TASK_ID}" input.lst)

Shtsai at 01:27, 6 September 2023

2023-09-06T01:27:13Z

← Older revision		Revision as of 21:27, 5 September 2023
Line 19:		Line 19:
	#SBATCH --array=1-5		#SBATCH --array=1-5

	ml R/4.0.0-foss-~~2019b~~		ml R/4.3.1-foss-2022a

	Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}		Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
Line 41:		Line 41:

	<pre class="gcommand">		<pre class="gcommand">
	ml R/4.0.0-foss-~~2019b~~		ml R/4.3.1-foss-2022a

	Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}		Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
Line 61:		Line 61:
	#SBATCH --array=1,3,5		#SBATCH --array=1,3,5

	ml R/4.0.0-foss-~~2019b~~		ml R/4.3.1-foss-2022a

	Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}		Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
Line 102:		Line 102:
	#SBATCH --array=1-3		#SBATCH --array=1-3

	ml R/4.0.0-foss-~~2019b~~		ml R/4.3.1-foss-2022a

	file=$(awk "NR==${SLURM_ARRAY_TASK_ID}" input.lst)		file=$(awk "NR==${SLURM_ARRAY_TASK_ID}" input.lst)
Line 113:		Line 113:
	One way that you can quickly create a file containing a list of your input files would be to redirect the output of <code>ls</code>. For example, say you had 100+ .csv's in your current directory that you wanted to spread across an array job. You could create your input list file with the command <code>ls *.csv > input.lst</code>.		One way that you can quickly create a file containing a list of your input files would be to redirect the output of <code>ls</code>. For example, say you had 100+ .csv's in your current directory that you wanted to spread across an array job. You could create your input list file with the command <code>ls *.csv > input.lst</code>.

			==Email Notification for Array Jobs==

			By default, email notifications set for an array job will generate one email message for the array job. If you would like to receive an email message for individual array job elements (up to a certain limit), please add ARRAY_TASKS to the --mail-type option.

			To get one email notification per array job, when it starts and ends, use:
			<pre class="gscript">
			#SBATCH --mail-user=MYID@uga.edu
			#SBATCH --mail-type=BEGIN,END,FAIL
			</pre>

			To get one email notification per array job element (up to a limit), when it starts and ends, use:
			<pre class="gscript">
			#SBATCH --mail-user=MYID@uga.edu
			#SBATCH --mail-type=BEGIN,END,FAIL,ARRAY_TASKS
			</pre>
			where in MYID@uga.edu needs to be replaced by your email address.
	==Further Reading==		==Further Reading==

	For more information on Slurm array jobs, please see Slurm's [https://slurm.schedmd.com/job_array.html documentation].		For more information on Slurm array jobs, please see Slurm's [https://slurm.schedmd.com/job_array.html documentation].

Ben at 21:00, 17 June 2021

2021-06-17T21:00:46Z

@@ Line 46: / Line 46: @@
 </pre>
-with ${SLURM_ARRAY_TASK_ID} being replaced by one of the numbers in the range defined in the --array Slurm header.  This ensures each array job element is working one unique input file, but as you can see, this does require naming your files a certain way to leverage the SLURM_ARRAY_TASK_ID environment variable, which may not always be desirable.
+with ${SLURM_ARRAY_TASK_ID} being replaced by one of the numbers in the range defined in the --array Slurm header.  This ensures each array job element is working one unique input file.
 ===Non-Numbered Input Files===
@@ Line 77: / Line 109: @@
 </pre>
-In the above submission script, ${SLURM_ARRAY_TASK_ID} is being replaced by one of the integers in the range defined by #SBATCH --array, for each array job element.  The <code>awk</code> built-in variable NR is getting the content of the line number given of input.lst.  This ensures each array job element gets a single unique input file.  One way that you can quickly create a file containing a list of your input files would be to redirect the output of <code>ls</code>.  For example, say you had 100+ .csv's in your current directory that you wanted to spread across an array job.  You could create your input list file with the command <code>ls *.csv > input.lst</code>.
+In the above submission script, ${SLURM_ARRAY_TASK_ID} is being replaced by one of the integers in the range defined by #SBATCH --array, for each array job element.  The <code>awk</code> built-in variable NR is getting the content of the line number given of input.lst.  This ensures each array job element gets a single unique input file.  Please note that this method would require you not start your array index range at 0, as <code>awk</code> will start counting line numbers from 1.
 ==Further Reading==
 For more information on Slurm array jobs, please see Slurm's [https://slurm.schedmd.com/job_array.html documentation].

Ben at 15:26, 17 June 2021

2021-06-17T15:26:45Z

@@ Line 3: / Line 3: @@
 An array job is a collection of jobs (called array job "elements") initiated from a single submission script. Array jobs work well for problems that are [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarassingly parallel], meaning a problem can be easily split up into concurrently running tasks that are not dependent on one another.  Imagine you have 10 input files that you want to perform the same action(s) against.  Rather than looping through the input one at a time, or rather than writing 10 almost identical submission scripts, you could write and submit one array job submission script.
-==Example Submission Script==
+==Example Submission Scripts==
 Writing an array job submission script is hardly different from any other type Slurm submission script.  The two key things to remember are the Slurm array header (#SBATCH --array), and the SLURM_ARRAY_TASK_ID environment variable.  Below is an array job submission script in which there are 5 input files to be ran as arguments for myScript.R, assuming the input files were named myinput-1, myinput-2, myinput-3, etc...
@@ Line 44: / Line 46: @@
 </pre>
-with ${SLURM_ARRAY_TASK_ID} being replaced by one of the numbers in the range defined in the --array Slurm header.
+with ${SLURM_ARRAY_TASK_ID} being replaced by one of the numbers in the range defined in the --array Slurm header.  This ensures each array job element is working one unique input file, but as you can see, this does require naming your files a certain way to leverage the SLURM_ARRAY_TASK_ID environment variable, which may not always be desirable.

Ben at 16:12, 16 June 2021

2021-06-16T16:12:32Z

← Older revision		Revision as of 12:12, 16 June 2021
Line 44:		Line 44:
	</pre>		</pre>

	with $SLURM_ARRAY_TASK_ID being replaced by one of the numbers in the range defined in the --array Slurm header.		with ${SLURM_ARRAY_TASK_ID} being replaced by one of the numbers in the range defined in the --array Slurm header.

Ben: Created page with "==Introduction== An array job is a collection of jobs (called array job "elements") initiated from a single submission script. Array jobs work well for problems that are [htt..."

2021-06-16T16:11:24Z

Created page with "==Introduction== An array job is a collection of jobs (called array job "elements") initiated from a single submission script. Array jobs work well for problems that are [htt..."

New page

==Introduction==

An array job is a collection of jobs (called array job "elements") initiated from a single submission script. Array jobs work well for problems that are [https://en.wikipedia.org/wiki/Embarrassingly_parallel embarassingly parallel], meaning a problem can be easily split up into concurrently running tasks that are not dependent on one another. Imagine you have 10 input files that you want to perform the same action(s) against. Rather than looping through the input one at a time, or rather than writing 10 almost identical submission scripts, you could write and submit one array job submission script.

==Example Submission Script==

Writing an array job submission script is hardly different from any other type Slurm submission script. The two key things to remember are the Slurm array header (#SBATCH --array), and the SLURM_ARRAY_TASK_ID environment variable. Below is an array job submission script in which there are 5 input files to be ran as arguments for myScript.R, assuming the input files were named myinput-1, myinput-2, myinput-3, etc...

<pre class="gscript">
#!/bin/bash

#SBATCH --job-name=array-test
#SBATCH --partition=batch
#SBATCH --ntasks=1
#SBATCH --mem=20gb
#SBATCH --time=1:00:00
#SBATCH --array=1-5

ml R/4.0.0-foss-2019b

Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
</pre>

Submitting the above script would create five array job elements as shown below:

<pre class="gcommand">
bc06026@b1-24 arraytest$ sbatch sub.sh
Submitted batch job 3341751
bc06026@b1-24 arraytest$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3341751_1 batch array-te bc06026 R 0:08 1 c4-9
3341751_2 batch array-te bc06026 R 0:08 1 c4-21
3341751_3 batch array-te bc06026 R 0:08 1 c4-21
3341751_4 batch array-te bc06026 R 0:08 1 c4-21
3341751_5 batch array-te bc06026 R 0:08 1 c4-11
</pre>

As you can see in the <code>squeue --me</code> output, by submitting this one submission script, we have 5 jobs running concurrently. Each one of these jobs is allocated the resources requested in the submission script and is running the commands:

<pre class="gcommand">
ml R/4.0.0-foss-2019b

Rscript myScript.R myinput-${SLURM_ARRAY_TASK_ID}
</pre>

with $SLURM_ARRAY_TASK_ID being replaced by one of the numbers in the range defined in the --array Slurm header.