Monitoring Jobs on the teaching cluster
How to list all user jobs in the queues
To list all running and pending jobs (by all users), use the command
squeue
Sample squeue output:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
274 batch slurmenv ab12345 PD 0:00 1 (Dependency)
278 interactive bash shtsai R 0:04 1 rb1-11
277 batch mpitest shtsai R 2:05 2 rb1-[9-10]
276 batch slurmenv shtsai R 0:08 1 rb1-6
273 batch slurmenv ab12345 R 0:44 1 rb1-7
Output explained:
The column entitled JOBID gives the job id of each job.
The column entitled PARTITION gives the partition or queue where the job is running. A job in the interactive partition is an interactive session (e.g. one that was launched with the interact command).
The column entitled NAME gives the name of the job (specified in the job submission script with the --job-name or -J option.
The column entitled USER gives the username (MyID) of the user running the job.
The column entitled ST gives the status of a job, which could be
- R : job is running
- PD : job is pending, waiting for resources to become available or for a job dependency to be satisfied
- CG : job is being cleared (completed, was deleted, crashed) and is no longer running.
The column entitled TIME gives the walltime of the job (this is not the CPU time of the job).
The column entitled NODES specifies how many nodes are being used by the job.
The column entitled NODELIST(REASON) lists the hostnames of the nodes used by running jobs. For pending jobs, this column lists the reason the job is pending. For example Dependency means the job is waiting for a job dependency to be satisfied.
The command
squeue -l
adds an extra column in the output for the TIME_LIMIT of the jobs.
How to list only my jobs in the queues
To list all your running and pending jobs, use the command
squeue -u MyID
where MyID needs to be replaced by your own cluster username (UGA MyID).
Sample output (as ran by user shtsai):
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
278 interactive bash shtsai R 0:04 1 rb1-11
277 batch mpitest shtsai R 2:05 2 rb1-[9-10]
276 batch slurmenv shtsai R 0:08 1 rb1-6
How to determine on which node(s) jobs are running
To see which nodes were assigned to each or your jobs, use the command
squeue
The column entitled NODELIST lists the hostnames of the nodes allocated to the jobs.
How to see detailed information about a given job
To see a long listing with detailed information for a running job (STATE = R), including the initial working directory, the number of cores and memory requested, the job submission and start time, use the command
scontrol show job JOBID
where JOBID should be replaced by the JOBID for the job you wish to check (the JOBID is given in the first column of the squeue output).
Sample output of a long job listing scontrol show job 11279:
JobId=11279 JobName=mpitest UserId=ab12345(10012) GroupId=abclab(1001) MCS_label=N/A Priority=1 Nice=0 Account=gacrc QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:02:24 TimeLimit=02:00:00 TimeMin=N/A SubmitTime=2018-08-14T12:19:23 EligibleTime=2018-08-14T12:19:23 StartTime=2018-08-14T12:19:23 EndTime=2018-08-14T14:19:23 Deadline=N/A PreemptTime=None SuspendTime=None SecsPreSuspend=0 LastSchedEval=2018-08-14T12:19:23 Partition=batch AllocNode:Sid=c2-4:872 ReqNodeList=(null) ExcNodeList=(null) NodeList=rb1-[7-8] BatchHost=rb1-7 NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=24,mem=14400M,node=2,billing=24 Socks/Node=* NtasksPerN:B:S:C=12:0:*:* CoreSpec=* MinCPUsNode=12 MinMemoryCPU=600M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 Gres=(null) Reservation=(null) OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/home/ab12345/tests/slurm/mpi/sub.sh WorkDir=/home/ab12345/tests/slurm/mpi StdErr=/home/ab12345/tests/slurm/mpi/mpitest.o11279 StdIn=/dev/null StdOut=/home/shtsai/tests/slurm/mpi/mpitest.o11279 Power=
The long listing of information is not available any more for jobs that are no longer in a running state.
Note that the long listing above does not include the resources utilized by the job.
For your own jobs that are still running or that already completed, you can check how much memory was used, how much CPU time was used, the working directory, etc, please use the command
sacct_zh
Sample output of an sacct_zh command:
JobID JobName User Partition NodeList AllocNodes NTasks NCPUS ReqMem MaxVMSize State CPUTime Elapsed Timelimit ExitCode WorkDir ------------ ---------- --------- ---------- --------------- ---------- -------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------- -------------------- 275 slurmenv shtsai batch rb1-5 1 2 10Gn COMPLETED 00:00:00 00:00:00 02:00:00 0:0 /home/shtsai/tests/+ 275.batch batch rb1-5 1 1 2 10Gn 197240K COMPLETED 00:00:00 00:00:00 0:0 275.extern extern rb1-5 1 1 2 10Gn 169800K COMPLETED 00:00:00 00:00:00 0:0 276 slurmenv shtsai batch rb1-6 1 1 10Gn CANCELLED+ 00:03:19 00:03:19 02:00:00 0:0 /home/shtsai/tests/+ 276.batch batch rb1-6 1 1 1 10Gn 221140K CANCELLED 00:03:20 00:03:20 0:15 276.extern extern rb1-6 1 1 1 10Gn 169800K COMPLETED 00:03:19 00:03:19 0:0 277 mpitest shtsai batch rb1-[9-10] 2 24 600Mc COMPLETED 04:01:12 00:10:03 02:00:00 0:0 /home/shtsai/tests/+ 277.batch batch rb1-9 1 1 12 600Mc 221268K COMPLETED 02:00:36 00:10:03 0:0 277.extern extern rb1-[9-10] 2 2 24 600Mc 169800K COMPLETED 04:01:12 00:10:03 0:0 277.0 orted rb1-10 1 1 1 600Mc 265640K COMPLETED 00:00:01 00:00:01 0:0 278 bash shtsai interq rb1-4 1 1 2Gn RUNNING 00:13:37 00:13:37 12:00:00 0:0 /home/shtsai/tests/+ 278.extern extern rb1-4 1 1 1 2Gn RUNNING 00:13:37 00:13:37 0:0 278.0 bash rb1-4 1 1 1 2Gn RUNNING 00:13:37 00:13:37 0:0
Note that each job will have several entries in this output, and these correspond to the different steps of the job. In general, non-MPI jobs that are still running will have two entries listed (jobid and jobid.extern), non-MPI jobs that completed or that were cancelled will have 3 entries (jobid, jobid.batch, jobid.extern), and MPI jobs will have an extra entry (jobid.0).
To see all fields reported by the accounting command for a given job, use the command
sacct -l -j JOBID
where JOBID should be replaced by the JOBID for the job you wish to check (the JOBID is given in the first column of the squeue output).
Monitoring Array Jobs
To view an array job, use the command
squeue -l
Sample output of the squeue -l command for an array job that has JOBID=81 and that has 10 elements:
Fri Aug 11 16:38:03 2023
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
81_0 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_1 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_2 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_3 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_4 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_5 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_6 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_7 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_8 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7
81_9 batch arrayjob shtsai RUNNING 0:03 10:00:00 1 rb1-7