Migrating from Torque to Slurm - Revision history

Shtsai: /* Training material */

2026-04-05T03:10:47Z

Training material

← Older revision		Revision as of 23:10, 4 April 2026
Line 360:		Line 360:


			<!--
	This page was adapted from https://arc-ts.umich.edu/migrating-from-torque-to-slurm/ and https://hpcc.usc.edu/support/documentation/pbs-to-slurm/		This page was adapted from https://arc-ts.umich.edu/migrating-from-torque-to-slurm/ and https://hpcc.usc.edu/support/documentation/pbs-to-slurm/
			-->

Shtsai: /* Job Environment and Environment Variables */

2026-04-05T03:07:39Z

Job Environment and Environment Variables

← Older revision		Revision as of 23:07, 4 April 2026
Line 279:		Line 279:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Frequently used environment variables
	\|-		\|-
	! style="width: 15%" scope="col" \| Info		! style="width: 15%" scope="col" \| Info
Line 287:		Line 288:

	\|-		\|-
	\| Version \|\| <code>$PBS_VERSION</code> \|\| – \|\| <small>Can extract from</small> <code>sbatch --version</code>		! scope="row" \| Version
			\| <code>$PBS_VERSION</code> \|\| – \|\| <small>Can extract from</small> <code>sbatch --version</code>
	\|-		\|-
	\| Job name \|\| <code>$PBS_JOBNAME</code> \|\| <code>$SLURM_JOB_NAME</code> \|\|		! scope="row" \| Job name
			\| <code>$PBS_JOBNAME</code> \|\| <code>$SLURM_JOB_NAME</code> \|\|
	\|-		\|-
	\| Job ID \|\| <code>$PBS_JOBID</code> \|\| <code>$SLURM_JOB_ID</code> \|\|		! scope="row" \| Job ID
			\| <code>$PBS_JOBID</code> \|\| <code>$SLURM_JOB_ID</code> \|\|
	\|-		\|-
	\| Batch or interactive \|\| <code>$PBS_ENVIRONMENT</code> \|\| – \|\|		! scope="row" \| Batch or interactive
			\| <code>$PBS_ENVIRONMENT</code> \|\| – \|\|
	\|-		\|-
	\| Submit directory \|\| <code>$PBS_O_WORKDIR</code> \|\| <code>$SLURM_SUBMIT_DIR</code> \|\| <small>Slurm jobs start from the submit directory by default.</small>		! scope="row" \| Submit directory
			\| <code>$PBS_O_WORKDIR</code> \|\| <code>$SLURM_SUBMIT_DIR</code> \|\| <small>Slurm jobs start from the submit directory by default.</small>
	\|-		\|-
	\| Submit host \|\| <code>$PBS_O_HOST</code> \|\| <code>$SLURM_SUBMIT_HOST</code> \|\|		! scope="row" \| Submit host
			\| <code>$PBS_O_HOST</code> \|\| <code>$SLURM_SUBMIT_HOST</code> \|\|
	\|-		\|-
	\| Node file \|\| <code>$PBS_NODEFILE</code> \|\| \|\| <small>A filename and path that lists the nodes a job has been allocated.</small>		! scope="row" \| Node file
			\| <code>$PBS_NODEFILE</code> \|\| \|\| <small>A filename and path that lists the nodes a job has been allocated.</small>
	\|-		\|-
	\| Node list \|\| <code>cat $PBS_NODEFILE</code> \|\| <code>$SLURM_JOB_NODELIST</code> \|\| <small>The Slurm variable has a different format to the Torque/PBS one. To get a list of nodes use: </small><br> <code>scontrol show hostnames $SLURM_JOB_NODELIST</code>		! scope="row" \| Node list
			\| <code>cat $PBS_NODEFILE</code> \|\| <code>$SLURM_JOB_NODELIST</code> \|\| <small>The Slurm variable has a different format to the Torque/PBS one. To get a list of nodes use: </small><br> <code>scontrol show hostnames $SLURM_JOB_NODELIST</code>
	\|-		\|-
	\| Job array index \|\| <code>$PBS_ARRAYID</code> <br> <code>$PBS_ARRAY_INDEX</code> \|\| <code>$SLURM_ARRAY_TASK_ID</code> \|\| <small>Only set when submitting a job array (with -a or –array)</small>		! scope="row" \| Job array index
			\| <code>$PBS_ARRAYID</code> <br> <code>$PBS_ARRAY_INDEX</code> \|\| <code>$SLURM_ARRAY_TASK_ID</code> \|\| <small>Only set when submitting a job array (with -a or –array)</small>
	\|-		\|-
	\| Walltime \|\| <code>$PBS_WALLTIME</code> \|\| – \|\|		! scope="row" \| Walltime
			\| <code>$PBS_WALLTIME</code> \|\| – \|\|
	\|-		\|-
	\| Queue name \|\| <code>$PBS_QUEUE</code> \|\| <code>$SLURM_JOB_PARTITION</code> \|\|		! scope="row" \| Queue name
			\| <code>$PBS_QUEUE</code> \|\| <code>$SLURM_JOB_PARTITION</code> \|\|
	\|-		\|-
	\| Number of nodes allocated \|\| <code>$PBS_NUM_NODES</code> \|\| <code>$SLURM_JOB_NUM_NODES</code> <br> <code>$SLURM_NNODES</code> \|\|		! scope="row" \| Number of nodes allocated
			\| <code>$PBS_NUM_NODES</code> \|\| <code>$SLURM_JOB_NUM_NODES</code> <br> <code>$SLURM_NNODES</code> \|\|
	\|-		\|-
	\| Number of processes \|\| <code>$PBS_NP</code> \|\| <code>$SLURM_NTASKS</code> \|\|		! scope="row" \| Number of processes
			\| <code>$PBS_NP</code> \|\| <code>$SLURM_NTASKS</code> \|\|
	\|-		\|-
	\| Number of processes per node \|\| <code>$PBS_NUM_PPN</code> \|\| <code>$SLURM_TASKS_PER_NODE</code> \|\|		! scope="row" \| Number of processes per node
			\| <code>$PBS_NUM_PPN</code> \|\| <code>$SLURM_TASKS_PER_NODE</code> \|\|
	\|-		\|-
	\| List of allocated GPUs \|\| <code>$PBS_GPUFILE</code> \|\| – \|\|		! scope="row" \| List of allocated GPUs
			\| <code>$PBS_GPUFILE</code> \|\| – \|\|
	\|-		\|-
	\| Requested tasks per node \|\| – \|\| <code>$SLURM_NTASKS_PER_NODE</code> \|\|		! scope="row" \| Requested tasks per node
			\| – \|\| <code>$SLURM_NTASKS_PER_NODE</code> \|\|
	\|-		\|-
	\| Requested CPUs per task \|\| – \|\| <code>$SLURM_CPUS_PER_TASK</code> \|\|		! scope="row" \| Requested CPUs per task
			\| – \|\| <code>$SLURM_CPUS_PER_TASK</code> \|\|
	\|-		\|-
	\| Scheduling priority \|\| – \|\| <code>$SLURM_PRIO_PROCESS</code> \|\|		! scope="row" \| Scheduling priority
			\| – \|\| <code>$SLURM_PRIO_PROCESS</code> \|\|
	\|-		\|-
	\| Job user \|\| – \|\| <code>$SLURM_JOB_USER</code> \|\|		! scope="row" \| Job user
			\| – \|\| <code>$SLURM_JOB_USER</code> \|\|
	<!-- \|-		<!-- \|-
	\| Hostname \|\| <code>$HOSTNAME</code> \|\| <code>$HOSTNAME == $SLURM_SUBMIT_HOST</code> \|\| <small>Unless a shell is invoked on an allocated resource, the HOSTNAME variable is propagated (copied) from the submit machine environments will be the same on all allocated nodes.</small>		! scope="row" \| Hostname
			\| <code>$HOSTNAME</code> \|\| <code>$HOSTNAME == $SLURM_SUBMIT_HOST</code> \|\| <small>Unless a shell is invoked on an allocated resource, the HOSTNAME variable is propagated (copied) from the submit machine environments will be the same on all allocated nodes.</small>
	-->		-->
	\|}		\|}

Shtsai: /* How to Monitor Jobs */

2026-04-05T03:04:58Z

How to Monitor Jobs

← Older revision		Revision as of 23:04, 4 April 2026
Line 220:		Line 220:

	{\| width="40%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="40%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Valid job states
	\|-		\|-
	! style="width: 5%" scope="col" \| Code		! style="width: 5%" scope="col" \| Code
Line 227:		Line 228:

	\|-		\|-
	\| CA \|\| Canceled \|\| Job was canceled		! scope="row" \| CA
			\| Canceled \|\| Job was canceled
	\|-		\|-
	\| CD \|\| Completed \|\| Job completed		! scope="row" \| CD
			\| Completed \|\| Job completed
	\|-		\|-
	\| CF \|\| Configuring \|\| Job resources being configured		! scope="row" \| CF
			\| Configuring \|\| Job resources being configured
	\|-		\|-
	\| CG \|\| Completing \|\| Job is completing		! scope="row" \| CG
			\| Completing \|\| Job is completing
	\|-		\|-
	\| F \|\| Failed \|\| Job terminated with non-zero exit code		! scope="row" \| F
			\| Failed \|\| Job terminated with non-zero exit code
	\|-		\|-
	\| NF \|\| Node Fail \|\| Job terminated due to failure of node(s)		! scope="row" \| NF
			\| Node Fail \|\| Job terminated due to failure of node(s)
	\|-		\|-
	\| PD \|\| Pending \|\| Job is waiting for compute node(s)		! scope="row" \| PD
			\| Pending \|\| Job is waiting for compute node(s)
	\|-		\|-
	\| R \|\| Running \|\| Job is running on compute node(s)		! scope="row" \| R
			\| Running \|\| Job is running on compute node(s)
	\|-		\|-
	\|-TO \|\| Timeout \|\| Job terminated upon reaching its time limit		! scope="row" \|-TO
			\| Timeout \|\| Job terminated upon reaching its time limit
	\|}		\|}

Shtsai: /* How to Monitor Jobs */

2026-04-05T03:03:28Z

How to Monitor Jobs

← Older revision		Revision as of 23:03, 4 April 2026
Line 181:		Line 181:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Commands to monitor jobs
	\|-		\|-
	! style="width: 20%" scope="col" \| Info		! style="width: 20%" scope="col" \| Info
Line 188:		Line 189:

	\|-		\|-
	\| Job status (all) \|\| <code>qstat</code> <br> <code>showq</code> \|\| <code>squeue</code>		! scope="row" \| Job status (all)
			\| <code>qstat</code> <br> <code>showq</code> \|\| <code>squeue</code>
	\|-		\|-
	\| Job status (by job) \|\| <code>qstat <job ID></code> \|\| <code>squeue -j <job ID></code>		! scope="row" \| Job status (by job)
			\| <code>qstat <job ID></code> \|\| <code>squeue -j <job ID></code>
	\|-		\|-
	\| Job status (by user) \|\| <code>qstat -u <user></code> \|\| <code>squeue -u <user></code>		! scope="row" \| Job status (by user)
			\| <code>qstat -u <user></code> \|\| <code>squeue -u <user></code>
	\|-		\|-
	\| Job status (only own jobs) \|\| <code>qstat_me</code> \|\| <code>squeue --me</code> <br> <code>squeue --me -l</code>		! scope="row" \| Job status (only own jobs)
			\| <code>qstat_me</code> \|\| <code>squeue --me</code> <br> <code>squeue --me -l</code>
	\|-		\|-
	\| Job status (detailed) \|\| <code>qstat -f <job ID></code> <br> <code>checkjob <job ID> </code>\|\| <code>scontrol show job -dd <job ID></code>		! scope="row" \| Job status (detailed)
			\| <code>qstat -f <job ID></code> <br> <code>checkjob <job ID> </code>\|\| <code>scontrol show job -dd <job ID></code>
	\|-		\|-
	\| Show expected start time \|\| <code>showstart <job ID> </code>\|\| <code>squeue -j <job ID> --start</code>		! scope="row" \| Show expected start time
			\| <code>showstart <job ID> </code>\|\| <code>squeue -j <job ID> --start</code>
	\|-		\|-
	\| Monitor or review a job’s resource usage \|\| <code>qstat -f <job ID> </code>\|\| <code>sacct -j <job ID> --format JobID,jobname,NTasks,nodelist,CPUTime,ReqMem,Elapsed </code>		! scope="row" \| Monitor or review a job’s resource usage
			\| <code>qstat -f <job ID> </code>\|\| <code>sacct -j <job ID> --format JobID,jobname,NTasks,nodelist,CPUTime,ReqMem,Elapsed </code>
	\|-		\|-
	\| View job batch script \|\| \|\| <code>scontrol write batch_script <job ID> [filename]</code>		! scope="row" \| View job batch script
			\| \|\| <code>scontrol write batch_script <job ID> [filename]</code>

	\|}		\|}

Shtsai: /* How to View Resources on the Cluster */

2026-04-05T03:02:26Z

How to View Resources on the Cluster

← Older revision		Revision as of 23:02, 4 April 2026
Line 157:		Line 157:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Commands to view resources on the cluster
	\|-		\|-
	! style="width: 20%" scope="col" \| Info		! style="width: 20%" scope="col" \| Info
Line 164:		Line 165:

	\|-		\|-
	\| Queue list / info \|\| <code>qstat -q [queue]</code> \|\| <code>scontrol show partition [queue]</code>		! scope="row" \| Queue list / info
			\| <code>qstat -q [queue]</code> \|\| <code>scontrol show partition [queue]</code>
	\|-		\|-
	\| Node list \|\| <code>pbsnodes -a</code><br> <code>mdiag -n -v </code>\|\| <code>scontrol show nodes</code>		! scope="row" \| Node list
			\| <code>pbsnodes -a</code><br> <code>mdiag -n -v </code>\|\| <code>scontrol show nodes</code>
	\|-		\|-
	\| Node details \|\| <code>pbsnodes <node></code> \|\| <code>scontrol show node <node></code>		! scope="row" \| Node details
			\| <code>pbsnodes <node></code> \|\| <code>scontrol show node <node></code>
	\|-		\|-
	\| Cluster status \|\| <code>qstat -B </code>\|\| <code>sinfo</code>		! scope="row" \| Cluster status
			\| <code>qstat -B </code>\|\| <code>sinfo</code>
	\|}		\|}

Shtsai: /* How to Submit and Manage Jobs */

2026-04-05T03:01:37Z

How to Submit and Manage Jobs

← Older revision		Revision as of 23:01, 4 April 2026
Line 126:		Line 126:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Commands to submit and manage jobs
	\|-		\|-
	! style="width: 20%" scope="col" \| Info		! style="width: 20%" scope="col" \| Info
Line 133:		Line 134:

	\|-		\|-
	\| Submit a job \|\| <code>qsub <job script></code> \|\| <code>sbatch <job script></code>		! scope="row" \| Submit a job
			\| <code>qsub <job script></code> \|\| <code>sbatch <job script></code>
	\|-		\|-
	\| Delete a job \|\| <code>qdel <job ID></code> \|\| <code>scancel <job ID></code>		! scope="row" \| Delete a job
			\| <code>qdel <job ID></code> \|\| <code>scancel <job ID></code>
	\|-		\|-
	\| Hold a job \|\| <code>qhold <job ID></code> \|\| <code>scontrol hold <job ID></code>		! scope="row" \| Hold a job
			\| <code>qhold <job ID></code> \|\| <code>scontrol hold <job ID></code>
	\|-		\|-
	\| Release a job \|\| <code>qrls <job ID></code> \|\| <code>scontrol release <job ID></code>		! scope="row" \| Release a job
			\| <code>qrls <job ID></code> \|\| <code>scontrol release <job ID></code>
	\|-		\|-
	\| Start an interactive job \|\| <code>qsub -I <options></code> \|\| <code>salloc <options></code> <br> <code>srun --pty <options></code>		! scope="row" \| Start an interactive job
			\| <code>qsub -I <options></code> \|\| <code>salloc <options></code> <br> <code>srun --pty <options></code>
	\|-		\|-
	\| Start an interactive job with X forwarding \|\| <code>qsub -I -X <options></code> \|\| <code>srun --x11 <options> </code>		! scope="row" \| Start an interactive job with X forwarding
			\| <code>qsub -I -X <options></code> \|\| <code>srun --x11 <options> </code>

	\|}		\|}

Shtsai: /* Job Submission Options */

2026-04-05T02:58:58Z

Job Submission Options

← Older revision		Revision as of 22:58, 4 April 2026
Line 107:		Line 107:
	\| <code>-W depend=after:jobID[:jobID...]</code> <br> <code>-W depend=afterok:jobID[:jobID...]</code> <br> <code>-W depend=afternotok:jobID[:jobID...]</code> <br> <code>-W depend=afterany:jobID[:jobID...]</code> \|\| <code>--dependency=after:jobID[:jobID...]</code> <br> <code>--dependency=afterok:jobID[:jobID...]</code> <br> <code>--dependency=afternotok:jobID[:jobID...]</code> <br> <code>--dependency=afterany:jobID[:jobID...]</code>		\| <code>-W depend=after:jobID[:jobID...]</code> <br> <code>-W depend=afterok:jobID[:jobID...]</code> <br> <code>-W depend=afternotok:jobID[:jobID...]</code> <br> <code>-W depend=afterany:jobID[:jobID...]</code> \|\| <code>--dependency=after:jobID[:jobID...]</code> <br> <code>--dependency=afterok:jobID[:jobID...]</code> <br> <code>--dependency=afternotok:jobID[:jobID...]</code> <br> <code>--dependency=afterany:jobID[:jobID...]</code>
	\|-		\|-
	\|! scope="row" Request event notification		! scope="row" \| Request event notification
	\| <code>-m <events> </code>\|\| <code>--mail-type=<events></code> <br> <small>Note: multiple mail-type requests may be specified in a comma separated list: </small><br><code> --mail-type=BEGIN,END,NONE,FAIL,REQUEUE</code>		\| <code>-m <events> </code>\|\| <code>--mail-type=<events></code> <br> <small>Note: multiple mail-type requests may be specified in a comma separated list: </small><br><code> --mail-type=BEGIN,END,NONE,FAIL,REQUEUE</code>
	\|-		\|-

Shtsai: /* Job Submission Options */

2026-04-05T02:58:15Z

Job Submission Options

← Older revision		Revision as of 22:58, 4 April 2026
Line 41:		Line 41:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Job submission options
	\|-		\|-
	! scope="col" \| Option		! scope="col" \| Option
Line 48:		Line 49:

	\|-		\|-
	\| Script directive \|\| <code>#PBS</code> \|\| <code>#SBATCH</code>		! scope="row" \| Script directive
			\| <code>#PBS</code> \|\| <code>#SBATCH</code>
	\|-		\|-
	\| Job name \|\| <code>-N <name> </code> \|\| <code>--job-name=<name></code> <br> <code>-J <name></code>		! scope="row" \| Job name
			\| <code>-N <name> </code> \|\| <code>--job-name=<name></code> <br> <code>-J <name></code>
	\|-		\|-
	\| Queue \|\| <code>-q <queue> </code>\|\| <code>--partition=<queue></code>		! scope="row" \| Queue
			\| <code>-q <queue> </code>\|\| <code>--partition=<queue></code>
	\|-		\|-
	\| Wall time limit \|\| <code>-l walltime=<hh:mm:ss> </code>\|\| <code>--time=<hh:mm:ss></code>		! scope="row" \| Wall time limit
			\| <code>-l walltime=<hh:mm:ss> </code>\|\| <code>--time=<hh:mm:ss></code>
	\|-		\|-
	\| Node count \|\| <code>-l nodes=<count></code> \|\| <code>--nodes=<count></code> <br><code> -N <count></code>		! scope="row" \| Node count
			\| <code>-l nodes=<count></code>
			\| <code>--nodes=<count></code> <br><code> -N <count></code>
	\|-		\|-
	\| Process count per node \|\| <code>-l ppn=<count></code> \|\| <code>--ntasks-per-node=<count></code>		! scope="row" \| Process count per node
			\| <code>-l ppn=<count></code> \|\| <code>--ntasks-per-node=<count></code>
	\|-		\|-
	\| core count (per process) \|\| \|\| <code>--cpus-per-task=<cores></code>		! scope="row" \| core count (per process)
			\| \|\| <code>--cpus-per-task=<cores></code>
	\|-		\|-
	\| Memory limit \|\| <code>-l mem=<limit></code> \|\| <code>--mem=<limit> </code> <small>(Memory per node in mega bytes – MB)</small>		! scope="row" \| Memory limit
			\| <code>-l mem=<limit></code> \|\| <code>--mem=<limit> </code> <small>(Memory per node in mega bytes – MB)</small>
	\|-		\|-
	\| Minimum memory per processor \|\| <code>-l pmem=<limit> </code>\|\| <code>--mem-per-cpu=<memory></code>		! scope="row" \| Minimum memory per processor
			\| <code>-l pmem=<limit> </code>\|\| <code>--mem-per-cpu=<memory></code>
	\|-		\|-
	\| Request GPUs \|\| <code>-l gpus=<count></code> \|\| <code>--gres=gpu:<count></code>		! scope="row" \| Request GPUs
			\| <code>-l gpus=<count></code> \|\| <code>--gres=gpu:<count></code>
	\|-		\|-
	\| Request specific nodes \|\| <code>-l nodes=<node>[,node2[,...]]></code> \|\| <code>-w, --nodelist=<node>[,node2[,...]]></code> <br><code> -F, --nodefile=<node file></code>		! scope="row" \| Request specific nodes
			\| <code>-l nodes=<node>[,node2[,...]]></code> \|\| <code>-w, --nodelist=<node>[,node2[,...]]></code> <br><code> -F, --nodefile=<node file></code>
	\|-		\|-
	\| Request node feature \|\| <code>-l nodes=<count>:ppn=<count>:<feature></code> \|\| <code>--constraint=<feature></code>		! scope="row" \| Request node feature
			\| <code>-l nodes=<count>:ppn=<count>:<feature></code> \|\| <code>--constraint=<feature></code>
	\|-		\|-
	\| Job array \|\| <code>-t <array indices></code> \|\| <code>--array <indexes></code> <br> <code>-a <indexes></code> <br> <small>Where <indexes> is replaced by a range (0-15), a list (0, 6, 16-32), or a step function (0-15:4)</small>		! scope="row" \| Job array
			\| <code>-t <array indices></code> \|\| <code>--array <indexes></code> <br> <code>-a <indexes></code> <br> <small>Where <indexes> is replaced by a range (0-15), a list (0, 6, 16-32), or a step function (0-15:4)</small>
	\|-		\|-
	\| Standard output file \|\|<code>-o <file path></code> \|\| <code>--output=<file path></code> <small>(path must exist)</small>		! scope="row" \| Standard output file
			\|<code>-o <file path></code> \|\| <code>--output=<file path></code> <small>(path must exist)</small>
	\|-		\|-
	\| Standard error file \|\| <code>-e <file path></code> \|\| <code>--error=<file path></code> <small>(path must exist)</small>		! scope="row" \| Standard error file
			\| <code>-e <file path></code> \|\| <code>--error=<file path></code> <small>(path must exist)</small>
	\|-		\|-
	\| Combine stdout/stderr to stdout \|\| <code>-j oe</code> \|\| <code>--output=<combined out and err file path></code>		! scope="row" \| Combine stdout/stderr to stdout
			\| <code>-j oe</code> \|\| <code>--output=<combined out and err file path></code>
	\|-		\|-
	\| Copy environment \|\| <code>-V </code>\|\| <code>--export=ALL (default)</code> <br> <code>--export=NONE </code> <small>to not export environment</small>		! scope="row" \| Copy environment
			\| <code>-V </code>\|\| <code>--export=ALL (default)</code> <br> <code>--export=NONE </code> <small>to not export environment</small>
	\|-		\|-
	\| Copy environment variable \|\| <code>-v <variable[=value][,variable2=value2[,...]]></code> \|\| <code>--export=<variable[=value][,variable2=value2[,...]]></code>		! scope="row" \| Copy environment variable
			\| <code>-v <variable[=value][,variable2=value2[,...]]></code> \|\| <code>--export=<variable[=value][,variable2=value2[,...]]></code>
	\|-		\|-
	\| Job dependency \|\| <code>-W depend=after:jobID[:jobID...]</code> <br> <code>-W depend=afterok:jobID[:jobID...]</code> <br> <code>-W depend=afternotok:jobID[:jobID...]</code> <br> <code>-W depend=afterany:jobID[:jobID...]</code> \|\| <code>--dependency=after:jobID[:jobID...]</code> <br> <code>--dependency=afterok:jobID[:jobID...]</code> <br> <code>--dependency=afternotok:jobID[:jobID...]</code> <br> <code>--dependency=afterany:jobID[:jobID...]</code>		! scope="row" \| Job dependency
			\| <code>-W depend=after:jobID[:jobID...]</code> <br> <code>-W depend=afterok:jobID[:jobID...]</code> <br> <code>-W depend=afternotok:jobID[:jobID...]</code> <br> <code>-W depend=afterany:jobID[:jobID...]</code> \|\| <code>--dependency=after:jobID[:jobID...]</code> <br> <code>--dependency=afterok:jobID[:jobID...]</code> <br> <code>--dependency=afternotok:jobID[:jobID...]</code> <br> <code>--dependency=afterany:jobID[:jobID...]</code>
	\|-		\|-
	\| Request event notification \|\| <code>-m <events> </code>\|\| <code>--mail-type=<events></code> <br> <small>Note: multiple mail-type requests may be specified in a comma separated list: </small><br><code> --mail-type=BEGIN,END,NONE,FAIL,REQUEUE</code>		\|! scope="row" Request event notification
			\| <code>-m <events> </code>\|\| <code>--mail-type=<events></code> <br> <small>Note: multiple mail-type requests may be specified in a comma separated list: </small><br><code> --mail-type=BEGIN,END,NONE,FAIL,REQUEUE</code>
	\|-		\|-
	\| Email address \|\| <code>-M <email address></code> \|\| <code>--mail-user=<email address></code>		! scope="row" \| Email address
			\| <code>-M <email address></code> \|\| <code>--mail-user=<email address></code>
	\|-		\|-
	\| Defer job until the specified time \|\| <code>-a <date/time></code> \|\| <code>--begin=<date/time></code>		! scope="row" \| Defer job until the specified time
			\| <code>-a <date/time></code> \|\| <code>--begin=<date/time></code>
	\|-		\|-
	\| Node exclusive job \|\| <code>qsub -n</code> \|\| <code>--exclusive</code>		! scope="row" \| Node exclusive job
			\| <code>qsub -n</code> \|\| <code>--exclusive</code>
	\|}		\|}


	==Common Job Commands==		==Common Job Commands==

Shtsai: /* How to Submit Jobs */

2026-04-05T02:55:13Z

How to Submit Jobs

← Older revision		Revision as of 22:55, 4 April 2026
Line 19:		Line 19:

	{\| width="60%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="60%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Job submission commands
	\|-		\|-
	! scope="col" \| Info		! scope="col" \| Info
Line 26:		Line 27:

	\|-		\|-
	\| Submit a batch job to the queue \|\| <code>qsub <job script></code> \|\| <code>sbatch <job script></code>		! scope="row" \| Submit a batch job to the queue
			\| <code>qsub <job script></code> \|\| <code>sbatch <job script></code>
	\|-		\|-
	\| Start an interactive job \|\| <code>qsub -I <options></code> \|\| <code> salloc <options> </code>		! scope="row" \| Start an interactive job
			\| <code>qsub -I <options></code> \|\| <code> salloc <options> </code>
	\|}		\|}
	where <job script> needs to be replaced by the name of your job submission script (e.g. sub.sh). Note that the syntax in the job submission script for Slurm is different from the Torque syntax.		where <job script> needs to be replaced by the name of your job submission script (e.g. sub.sh). Note that the syntax in the job submission script for Slurm is different from the Torque syntax.

Shtsai at 02:54, 5 April 2026

2026-04-05T02:54:21Z

← Older revision		Revision as of 22:54, 4 April 2026
Line 3:		Line 3:
	for job scheduling and resource management on Sapelo2. Slurm replaced the Torque (PBS) resource manager and the Moab scheduling system that Sapelo2 used before then.		for job scheduling and resource management on Sapelo2. Slurm replaced the Torque (PBS) resource manager and the Moab scheduling system that Sapelo2 used before then.

	===How is Slurm different from Torque?===		==How is Slurm different from Torque?==

	Slurm is different from Torque in a number of ways, including the commands used to submit and monitor jobs, the syntax used to request resources, and the way environment variables behave.		Slurm is different from Torque in a number of ways, including the commands used to submit and monitor jobs, the syntax used to request resources, and the way environment variables behave.
Line 14:		Line 14:
	*In Slurm, environmental variables of the submitting process are passed to the job by default <!--, except that by default Slurm does not source the files <code>~/.bashrc</code> when requesting resources via <code>sbatch</code>. -->		*In Slurm, environmental variables of the submitting process are passed to the job by default <!--, except that by default Slurm does not source the files <code>~/.bashrc</code> when requesting resources via <code>sbatch</code>. -->

	===How to Submit Jobs===		==How to Submit Jobs==

	To submit jobs in Slurm, replace <code>qsub</code> with one of the commands from the table below.		To submit jobs in Slurm, replace <code>qsub</code> with one of the commands from the table below.
Line 32:		Line 32:
	where <job script> needs to be replaced by the name of your job submission script (e.g. sub.sh). Note that the syntax in the job submission script for Slurm is different from the Torque syntax.		where <job script> needs to be replaced by the name of your job submission script (e.g. sub.sh). Note that the syntax in the job submission script for Slurm is different from the Torque syntax.

	===Job Submission Options===		==Job Submission Options==

	As with Torque, job options and resource requests in Slurm can be set in the job submission script or as options to the job submission command. However, the syntax used to request resources is different and the table below summarizes some of the options that are frequently used.		As with Torque, job options and resource requests in Slurm can be set in the job submission script or as options to the job submission command. However, the syntax used to request resources is different and the table below summarizes some of the options that are frequently used.
Line 93:		Line 93:


	===Common Job Commands===		==Common Job Commands==


	====How to Submit and Manage Jobs====		===How to Submit and Manage Jobs===

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
Line 120:		Line 120:
	\|}		\|}

	====How to View Resources on the Cluster====		===How to View Resources on the Cluster===

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
Line 139:		Line 139:
	\|}		\|}

	====How to Monitor Jobs====		===How to Monitor Jobs===

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
Line 198:		Line 198:
	\|}		\|}

	===Job Environment and Environment Variables===		==Job Environment and Environment Variables==

	In Slurm, environment variables will get passed to your job by default.		In Slurm, environment variables will get passed to your job by default.
Line 271:		Line 271:
	\|}		\|}

	===Slurm Documentation===		==Slurm Documentation==

	Extensive documentation on Slurm is available at https://slurm.schedmd.com/documentation.html		Extensive documentation on Slurm is available at https://slurm.schedmd.com/documentation.html


	===Training material===		==Training material==

	To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the GACRC's Kaltura channel at https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required). Training sessions and slides are available at https://wiki.gacrc.uga.edu/wiki/Training		To help users familiarize with Slurm and the test cluster environment, we have prepared some training videos that are available from the GACRC's Kaltura channel at https://kaltura.uga.edu/channel/GACRC/176125031 (login with MyID and password is required). Training sessions and slides are available at https://wiki.gacrc.uga.edu/wiki/Training

← Older revision		Revision as of 23:07, 4 April 2026
Line 279:		Line 279:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Frequently used environment variables
	\|-		\|-
	! style="width: 15%" scope="col" \| Info		! style="width: 15%" scope="col" \| Info
Line 287:		Line 288:

	\|-		\|-
	\| Version \|\| <code>$PBS_VERSION</code> \|\| – \|\| <small>Can extract from</small> <code>sbatch --version</code>		! scope="row" \| Version
			\| <code>$PBS_VERSION</code> \|\| – \|\| <small>Can extract from</small> <code>sbatch --version</code>
	\|-		\|-
	\| Job name \|\| <code>$PBS_JOBNAME</code> \|\| <code>$SLURM_JOB_NAME</code> \|\|		! scope="row" \| Job name
			\| <code>$PBS_JOBNAME</code> \|\| <code>$SLURM_JOB_NAME</code> \|\|
	\|-		\|-
	\| Job ID \|\| <code>$PBS_JOBID</code> \|\| <code>$SLURM_JOB_ID</code> \|\|		! scope="row" \| Job ID
			\| <code>$PBS_JOBID</code> \|\| <code>$SLURM_JOB_ID</code> \|\|
	\|-		\|-
	\| Batch or interactive \|\| <code>$PBS_ENVIRONMENT</code> \|\| – \|\|		! scope="row" \| Batch or interactive
			\| <code>$PBS_ENVIRONMENT</code> \|\| – \|\|
	\|-		\|-
	\| Submit directory \|\| <code>$PBS_O_WORKDIR</code> \|\| <code>$SLURM_SUBMIT_DIR</code> \|\| <small>Slurm jobs start from the submit directory by default.</small>		! scope="row" \| Submit directory
			\| <code>$PBS_O_WORKDIR</code> \|\| <code>$SLURM_SUBMIT_DIR</code> \|\| <small>Slurm jobs start from the submit directory by default.</small>
	\|-		\|-
	\| Submit host \|\| <code>$PBS_O_HOST</code> \|\| <code>$SLURM_SUBMIT_HOST</code> \|\|		! scope="row" \| Submit host
			\| <code>$PBS_O_HOST</code> \|\| <code>$SLURM_SUBMIT_HOST</code> \|\|
	\|-		\|-
	\| Node file \|\| <code>$PBS_NODEFILE</code> \|\| \|\| <small>A filename and path that lists the nodes a job has been allocated.</small>		! scope="row" \| Node file
			\| <code>$PBS_NODEFILE</code> \|\| \|\| <small>A filename and path that lists the nodes a job has been allocated.</small>
	\|-		\|-
	\| Node list \|\| <code>cat $PBS_NODEFILE</code> \|\| <code>$SLURM_JOB_NODELIST</code> \|\| <small>The Slurm variable has a different format to the Torque/PBS one. To get a list of nodes use: </small><br> <code>scontrol show hostnames $SLURM_JOB_NODELIST</code>		! scope="row" \| Node list
			\| <code>cat $PBS_NODEFILE</code> \|\| <code>$SLURM_JOB_NODELIST</code> \|\| <small>The Slurm variable has a different format to the Torque/PBS one. To get a list of nodes use: </small><br> <code>scontrol show hostnames $SLURM_JOB_NODELIST</code>
	\|-		\|-
	\| Job array index \|\| <code>$PBS_ARRAYID</code> <br> <code>$PBS_ARRAY_INDEX</code> \|\| <code>$SLURM_ARRAY_TASK_ID</code> \|\| <small>Only set when submitting a job array (with -a or –array)</small>		! scope="row" \| Job array index
			\| <code>$PBS_ARRAYID</code> <br> <code>$PBS_ARRAY_INDEX</code> \|\| <code>$SLURM_ARRAY_TASK_ID</code> \|\| <small>Only set when submitting a job array (with -a or –array)</small>
	\|-		\|-
	\| Walltime \|\| <code>$PBS_WALLTIME</code> \|\| – \|\|		! scope="row" \| Walltime
			\| <code>$PBS_WALLTIME</code> \|\| – \|\|
	\|-		\|-
	\| Queue name \|\| <code>$PBS_QUEUE</code> \|\| <code>$SLURM_JOB_PARTITION</code> \|\|		! scope="row" \| Queue name
			\| <code>$PBS_QUEUE</code> \|\| <code>$SLURM_JOB_PARTITION</code> \|\|
	\|-		\|-
	\| Number of nodes allocated \|\| <code>$PBS_NUM_NODES</code> \|\| <code>$SLURM_JOB_NUM_NODES</code> <br> <code>$SLURM_NNODES</code> \|\|		! scope="row" \| Number of nodes allocated
			\| <code>$PBS_NUM_NODES</code> \|\| <code>$SLURM_JOB_NUM_NODES</code> <br> <code>$SLURM_NNODES</code> \|\|
	\|-		\|-
	\| Number of processes \|\| <code>$PBS_NP</code> \|\| <code>$SLURM_NTASKS</code> \|\|		! scope="row" \| Number of processes
			\| <code>$PBS_NP</code> \|\| <code>$SLURM_NTASKS</code> \|\|
	\|-		\|-
	\| Number of processes per node \|\| <code>$PBS_NUM_PPN</code> \|\| <code>$SLURM_TASKS_PER_NODE</code> \|\|		! scope="row" \| Number of processes per node
			\| <code>$PBS_NUM_PPN</code> \|\| <code>$SLURM_TASKS_PER_NODE</code> \|\|
	\|-		\|-
	\| List of allocated GPUs \|\| <code>$PBS_GPUFILE</code> \|\| – \|\|		! scope="row" \| List of allocated GPUs
			\| <code>$PBS_GPUFILE</code> \|\| – \|\|
	\|-		\|-
	\| Requested tasks per node \|\| – \|\| <code>$SLURM_NTASKS_PER_NODE</code> \|\|		! scope="row" \| Requested tasks per node
			\| – \|\| <code>$SLURM_NTASKS_PER_NODE</code> \|\|
	\|-		\|-
	\| Requested CPUs per task \|\| – \|\| <code>$SLURM_CPUS_PER_TASK</code> \|\|		! scope="row" \| Requested CPUs per task
			\| – \|\| <code>$SLURM_CPUS_PER_TASK</code> \|\|
	\|-		\|-
	\| Scheduling priority \|\| – \|\| <code>$SLURM_PRIO_PROCESS</code> \|\|		! scope="row" \| Scheduling priority
			\| – \|\| <code>$SLURM_PRIO_PROCESS</code> \|\|
	\|-		\|-
	\| Job user \|\| – \|\| <code>$SLURM_JOB_USER</code> \|\|		! scope="row" \| Job user
			\| – \|\| <code>$SLURM_JOB_USER</code> \|\|
	<!-- \|-		<!-- \|-
	\| Hostname \|\| <code>$HOSTNAME</code> \|\| <code>$HOSTNAME == $SLURM_SUBMIT_HOST</code> \|\| <small>Unless a shell is invoked on an allocated resource, the HOSTNAME variable is propagated (copied) from the submit machine environments will be the same on all allocated nodes.</small>		! scope="row" \| Hostname
			\| <code>$HOSTNAME</code> \|\| <code>$HOSTNAME == $SLURM_SUBMIT_HOST</code> \|\| <small>Unless a shell is invoked on an allocated resource, the HOSTNAME variable is propagated (copied) from the submit machine environments will be the same on all allocated nodes.</small>
	-->		-->
	\|}		\|}

← Older revision		Revision as of 23:04, 4 April 2026
Line 220:		Line 220:

	{\| width="40%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="40%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Valid job states
	\|-		\|-
	! style="width: 5%" scope="col" \| Code		! style="width: 5%" scope="col" \| Code
Line 227:		Line 228:

	\|-		\|-
	\| CA \|\| Canceled \|\| Job was canceled		! scope="row" \| CA
			\| Canceled \|\| Job was canceled
	\|-		\|-
	\| CD \|\| Completed \|\| Job completed		! scope="row" \| CD
			\| Completed \|\| Job completed
	\|-		\|-
	\| CF \|\| Configuring \|\| Job resources being configured		! scope="row" \| CF
			\| Configuring \|\| Job resources being configured
	\|-		\|-
	\| CG \|\| Completing \|\| Job is completing		! scope="row" \| CG
			\| Completing \|\| Job is completing
	\|-		\|-
	\| F \|\| Failed \|\| Job terminated with non-zero exit code		! scope="row" \| F
			\| Failed \|\| Job terminated with non-zero exit code
	\|-		\|-
	\| NF \|\| Node Fail \|\| Job terminated due to failure of node(s)		! scope="row" \| NF
			\| Node Fail \|\| Job terminated due to failure of node(s)
	\|-		\|-
	\| PD \|\| Pending \|\| Job is waiting for compute node(s)		! scope="row" \| PD
			\| Pending \|\| Job is waiting for compute node(s)
	\|-		\|-
	\| R \|\| Running \|\| Job is running on compute node(s)		! scope="row" \| R
			\| Running \|\| Job is running on compute node(s)
	\|-		\|-
	\|-TO \|\| Timeout \|\| Job terminated upon reaching its time limit		! scope="row" \|-TO
			\| Timeout \|\| Job terminated upon reaching its time limit
	\|}		\|}

← Older revision		Revision as of 23:03, 4 April 2026
Line 181:		Line 181:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Commands to monitor jobs
	\|-		\|-
	! style="width: 20%" scope="col" \| Info		! style="width: 20%" scope="col" \| Info
Line 188:		Line 189:

	\|-		\|-
	\| Job status (all) \|\| <code>qstat</code> <br> <code>showq</code> \|\| <code>squeue</code>		! scope="row" \| Job status (all)
			\| <code>qstat</code> <br> <code>showq</code> \|\| <code>squeue</code>
	\|-		\|-
	\| Job status (by job) \|\| <code>qstat <job ID></code> \|\| <code>squeue -j <job ID></code>		! scope="row" \| Job status (by job)
			\| <code>qstat <job ID></code> \|\| <code>squeue -j <job ID></code>
	\|-		\|-
	\| Job status (by user) \|\| <code>qstat -u <user></code> \|\| <code>squeue -u <user></code>		! scope="row" \| Job status (by user)
			\| <code>qstat -u <user></code> \|\| <code>squeue -u <user></code>
	\|-		\|-
	\| Job status (only own jobs) \|\| <code>qstat_me</code> \|\| <code>squeue --me</code> <br> <code>squeue --me -l</code>		! scope="row" \| Job status (only own jobs)
			\| <code>qstat_me</code> \|\| <code>squeue --me</code> <br> <code>squeue --me -l</code>
	\|-		\|-
	\| Job status (detailed) \|\| <code>qstat -f <job ID></code> <br> <code>checkjob <job ID> </code>\|\| <code>scontrol show job -dd <job ID></code>		! scope="row" \| Job status (detailed)
			\| <code>qstat -f <job ID></code> <br> <code>checkjob <job ID> </code>\|\| <code>scontrol show job -dd <job ID></code>
	\|-		\|-
	\| Show expected start time \|\| <code>showstart <job ID> </code>\|\| <code>squeue -j <job ID> --start</code>		! scope="row" \| Show expected start time
			\| <code>showstart <job ID> </code>\|\| <code>squeue -j <job ID> --start</code>
	\|-		\|-
	\| Monitor or review a job’s resource usage \|\| <code>qstat -f <job ID> </code>\|\| <code>sacct -j <job ID> --format JobID,jobname,NTasks,nodelist,CPUTime,ReqMem,Elapsed </code>		! scope="row" \| Monitor or review a job’s resource usage
			\| <code>qstat -f <job ID> </code>\|\| <code>sacct -j <job ID> --format JobID,jobname,NTasks,nodelist,CPUTime,ReqMem,Elapsed </code>
	\|-		\|-
	\| View job batch script \|\| \|\| <code>scontrol write batch_script <job ID> [filename]</code>		! scope="row" \| View job batch script
			\| \|\| <code>scontrol write batch_script <job ID> [filename]</code>

	\|}		\|}

← Older revision		Revision as of 22:58, 4 April 2026
Line 41:		Line 41:

	{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"		{\| width="100%" border="1" cellspacing="0" cellpadding="2" align="top" class="wikitable unsortable"
			\|+ Job submission options
	\|-		\|-
	! scope="col" \| Option		! scope="col" \| Option
Line 48:		Line 49:

	\|-		\|-
	\| Script directive \|\| <code>#PBS</code> \|\| <code>#SBATCH</code>		! scope="row" \| Script directive
			\| <code>#PBS</code> \|\| <code>#SBATCH</code>
	\|-		\|-
	\| Job name \|\| <code>-N <name> </code> \|\| <code>--job-name=<name></code> <br> <code>-J <name></code>		! scope="row" \| Job name
			\| <code>-N <name> </code> \|\| <code>--job-name=<name></code> <br> <code>-J <name></code>
	\|-		\|-
	\| Queue \|\| <code>-q <queue> </code>\|\| <code>--partition=<queue></code>		! scope="row" \| Queue
			\| <code>-q <queue> </code>\|\| <code>--partition=<queue></code>
	\|-		\|-
	\| Wall time limit \|\| <code>-l walltime=<hh:mm:ss> </code>\|\| <code>--time=<hh:mm:ss></code>		! scope="row" \| Wall time limit
			\| <code>-l walltime=<hh:mm:ss> </code>\|\| <code>--time=<hh:mm:ss></code>
	\|-		\|-
	\| Node count \|\| <code>-l nodes=<count></code> \|\| <code>--nodes=<count></code> <br><code> -N <count></code>		! scope="row" \| Node count
			\| <code>-l nodes=<count></code>
			\| <code>--nodes=<count></code> <br><code> -N <count></code>
	\|-		\|-
	\| Process count per node \|\| <code>-l ppn=<count></code> \|\| <code>--ntasks-per-node=<count></code>		! scope="row" \| Process count per node
			\| <code>-l ppn=<count></code> \|\| <code>--ntasks-per-node=<count></code>
	\|-		\|-
	\| core count (per process) \|\| \|\| <code>--cpus-per-task=<cores></code>		! scope="row" \| core count (per process)
			\| \|\| <code>--cpus-per-task=<cores></code>
	\|-		\|-
	\| Memory limit \|\| <code>-l mem=<limit></code> \|\| <code>--mem=<limit> </code> <small>(Memory per node in mega bytes – MB)</small>		! scope="row" \| Memory limit
			\| <code>-l mem=<limit></code> \|\| <code>--mem=<limit> </code> <small>(Memory per node in mega bytes – MB)</small>
	\|-		\|-
	\| Minimum memory per processor \|\| <code>-l pmem=<limit> </code>\|\| <code>--mem-per-cpu=<memory></code>		! scope="row" \| Minimum memory per processor
			\| <code>-l pmem=<limit> </code>\|\| <code>--mem-per-cpu=<memory></code>
	\|-		\|-
	\| Request GPUs \|\| <code>-l gpus=<count></code> \|\| <code>--gres=gpu:<count></code>		! scope="row" \| Request GPUs
			\| <code>-l gpus=<count></code> \|\| <code>--gres=gpu:<count></code>
	\|-		\|-
	\| Request specific nodes \|\| <code>-l nodes=<node>[,node2[,...]]></code> \|\| <code>-w, --nodelist=<node>[,node2[,...]]></code> <br><code> -F, --nodefile=<node file></code>		! scope="row" \| Request specific nodes
			\| <code>-l nodes=<node>[,node2[,...]]></code> \|\| <code>-w, --nodelist=<node>[,node2[,...]]></code> <br><code> -F, --nodefile=<node file></code>
	\|-		\|-
	\| Request node feature \|\| <code>-l nodes=<count>:ppn=<count>:<feature></code> \|\| <code>--constraint=<feature></code>		! scope="row" \| Request node feature
			\| <code>-l nodes=<count>:ppn=<count>:<feature></code> \|\| <code>--constraint=<feature></code>
	\|-		\|-
	\| Job array \|\| <code>-t <array indices></code> \|\| <code>--array <indexes></code> <br> <code>-a <indexes></code> <br> <small>Where <indexes> is replaced by a range (0-15), a list (0, 6, 16-32), or a step function (0-15:4)</small>		! scope="row" \| Job array
			\| <code>-t <array indices></code> \|\| <code>--array <indexes></code> <br> <code>-a <indexes></code> <br> <small>Where <indexes> is replaced by a range (0-15), a list (0, 6, 16-32), or a step function (0-15:4)</small>
	\|-		\|-
	\| Standard output file \|\|<code>-o <file path></code> \|\| <code>--output=<file path></code> <small>(path must exist)</small>		! scope="row" \| Standard output file
			\|<code>-o <file path></code> \|\| <code>--output=<file path></code> <small>(path must exist)</small>
	\|-		\|-
	\| Standard error file \|\| <code>-e <file path></code> \|\| <code>--error=<file path></code> <small>(path must exist)</small>		! scope="row" \| Standard error file
			\| <code>-e <file path></code> \|\| <code>--error=<file path></code> <small>(path must exist)</small>
	\|-		\|-
	\| Combine stdout/stderr to stdout \|\| <code>-j oe</code> \|\| <code>--output=<combined out and err file path></code>		! scope="row" \| Combine stdout/stderr to stdout
			\| <code>-j oe</code> \|\| <code>--output=<combined out and err file path></code>
	\|-		\|-
	\| Copy environment \|\| <code>-V </code>\|\| <code>--export=ALL (default)</code> <br> <code>--export=NONE </code> <small>to not export environment</small>		! scope="row" \| Copy environment
			\| <code>-V </code>\|\| <code>--export=ALL (default)</code> <br> <code>--export=NONE </code> <small>to not export environment</small>
	\|-		\|-
	\| Copy environment variable \|\| <code>-v <variable[=value][,variable2=value2[,...]]></code> \|\| <code>--export=<variable[=value][,variable2=value2[,...]]></code>		! scope="row" \| Copy environment variable
			\| <code>-v <variable[=value][,variable2=value2[,...]]></code> \|\| <code>--export=<variable[=value][,variable2=value2[,...]]></code>
	\|-		\|-
	\| Job dependency \|\| <code>-W depend=after:jobID[:jobID...]</code> <br> <code>-W depend=afterok:jobID[:jobID...]</code> <br> <code>-W depend=afternotok:jobID[:jobID...]</code> <br> <code>-W depend=afterany:jobID[:jobID...]</code> \|\| <code>--dependency=after:jobID[:jobID...]</code> <br> <code>--dependency=afterok:jobID[:jobID...]</code> <br> <code>--dependency=afternotok:jobID[:jobID...]</code> <br> <code>--dependency=afterany:jobID[:jobID...]</code>		! scope="row" \| Job dependency
			\| <code>-W depend=after:jobID[:jobID...]</code> <br> <code>-W depend=afterok:jobID[:jobID...]</code> <br> <code>-W depend=afternotok:jobID[:jobID...]</code> <br> <code>-W depend=afterany:jobID[:jobID...]</code> \|\| <code>--dependency=after:jobID[:jobID...]</code> <br> <code>--dependency=afterok:jobID[:jobID...]</code> <br> <code>--dependency=afternotok:jobID[:jobID...]</code> <br> <code>--dependency=afterany:jobID[:jobID...]</code>
	\|-		\|-
	\| Request event notification \|\| <code>-m <events> </code>\|\| <code>--mail-type=<events></code> <br> <small>Note: multiple mail-type requests may be specified in a comma separated list: </small><br><code> --mail-type=BEGIN,END,NONE,FAIL,REQUEUE</code>		\|! scope="row" Request event notification
			\| <code>-m <events> </code>\|\| <code>--mail-type=<events></code> <br> <small>Note: multiple mail-type requests may be specified in a comma separated list: </small><br><code> --mail-type=BEGIN,END,NONE,FAIL,REQUEUE</code>
	\|-		\|-
	\| Email address \|\| <code>-M <email address></code> \|\| <code>--mail-user=<email address></code>		! scope="row" \| Email address
			\| <code>-M <email address></code> \|\| <code>--mail-user=<email address></code>
	\|-		\|-
	\| Defer job until the specified time \|\| <code>-a <date/time></code> \|\| <code>--begin=<date/time></code>		! scope="row" \| Defer job until the specified time
			\| <code>-a <date/time></code> \|\| <code>--begin=<date/time></code>
	\|-		\|-
	\| Node exclusive job \|\| <code>qsub -n</code> \|\| <code>--exclusive</code>		! scope="row" \| Node exclusive job
			\| <code>qsub -n</code> \|\| <code>--exclusive</code>
	\|}		\|}


	==Common Job Commands==		==Common Job Commands==