Running Jobs on zcluster

From Research Computing Center Wiki
Jump to: navigation, search


Using the Queueing System

The zcluster login node (zcluster.rcc.uga.edu) should be used for text editing, and job submissions. No jobs should be run directly on the login node. Processes that use too much CPU or RAM on the login node may be terminated by GACRC staff, or automatically, in order to keep the cluster running properly. Jobs should be run using the zcluster queueing system, which is a variant of the Sun Grid Engine (SGE).

The queueing system should be used to run both interactive and batch jobs.


Back to Top

Batch Queues on the zcluster

The zcluster Intel compute nodes have 2 processors each. Some of the nodes have two 4-core and some have two 6-core processors, which means these nodes behave as though they had 8 and 12 CPUs each, respectively. Because the zcluster is comprised of both AMD Opterons and Intel 4-core and 6-core Xeons, code compiled on one type of node might not be optimized to run on the other type. For more information, please refer to Code Compilation on zcluster. The zcluster also has some large memory nodes, NVidia GPU cards, and nodes connected with the Infiniband interconnect. For more information on the hardware available on zcluster, please refer to the Systems page.

The queue names specify the type of hardware to which the job will be submitted to and the maximum runtime of the job. The commonly used queue names are listed here:

Queue Name Description Notes
rcc-30d Up to 75 CPU cores, maximum run time of 30 days, sends job to 8-core and 12-core nodes (Intel Xeon processors). Open to all users.
rcc-mc-30d Up to 75 CPU cores, maximum run time of 30 days, sends jobs to 32-core AMD nodes with 64 GB. Permission needed.
rcc-m128-30d Up to 8 CPU cores, maximum run time of 30 days, sends job to Intel Xeon nodes with 128GB, 192GB, or 256GB RAM. Permission needed.
rcc-m512-30d Up to 8 CPU cores, maximum run time of 30 days, sends job to Intel Xeon nodes with 512GB of RAM. Permission needed.
rcc-kgpu-30d For jobs that use the NVIDIA Tesla (Kepler) K20X GPUs, maximum run time of 30 days. Permission needed.
rcc-mgpu-30d For jobs that use the NVIDIA Tesla (Fermi) M2070 GPUs, maximum run time of 30 days. Permission needed.
rcc-sgpu-30d For jobs that use the NVIDIA Tesla S1070 GPUs, maximum run time of 30 days. Permission needed.

The batch queue can be used for serial jobs (that is, jobs that require only one CPU core) and for parallel jobs. To submit a job to the resource, first determine your CPU core requirement (for multi-thread jobs, the number of cores should be equal to the maximum number of threads used). This will determine the job submission command options.

For a list of all valid queue names, please use the command qconf -sql from a zcluster shell prompt.

We recommend that users checkpoint their codes whenever possible to avoid losing valuable compute time if the system goes down before a job is completed. A long job that can be checkpointed can be run as a sequence of shorter jobs, which can be automatically submitted to the queue as described below in the Runchaining Jobs section. If you cannot fit your job within the established processor and runtime limits, please let us know.


Back to Top

SGE Usage Information

These are the common SGE commands:

  • qsub Submit a job to the queue
  • qdel Cancel a queued or running job
  • qhold Place a queued job on hold
  • qstat Check the status of queued and running jobs
  • qstat -g c List all valid queue names

Back to Top

Submitting a Batch Job to the Queue

The preferred way to submit a batch job to the queue is to use the qsub command to submit a job submission shell script. To submit a serial job, the syntax of the qsub command is:

qsub -q queuename shellscriptname

where

  • queuename is the name of the batch queue
  • shellscriptname is the name of the job submission shell script file

By default the standard output and the standard error of the job will be written into files called shellscriptname.ojobid and shellscriptname.ejobid, respectively, where jobid is the job id number. These files will be written in the directory from which the job was submitted.

Serial Jobs:

In the example below, the executable name will be called myprog and it requires input parameters to be piped in. The input parameters are in a file called myin and the output data will be stored in a file called myout. The working_directory is the path to your working directory (e.g., it could be /home/labname/username/subdir, which can be referenced as ${HOME}/subdir ). The second line in the script below, namely cd working_directory or cd ${HOME}/subdirectory can be replaced by cd `pwd` if the job will be submitted from within the working directory.

Sample submission script to run a serial job (e.g. script name sub.sh)

#!/bin/bash
cd working_directory
time ./myprog < myin > myout

Note that the 'time' command included in the sample script above is optional (this command measures the amount of time it takes to run an executable). Entering a standard input file (myin) and standard output file (myout) is only necessary if your executable requires input from standard input and if it outputs data to standard output.

NOTE:Do NOT put the job into the background with a '&' in the shell script. This will confuse the queueing system.

To submit a serial job with script sub.sh to the rcc-30d batch queue and have the standard output and error go to the files sub.sh.ojobid and sub.sh.ejobid, respectively, in the directory from which the job was submitted, use

qsub -q rcc-30d sub.sh

Back to Top

Parallel job

#!/bin/bash
cd working_directory
time ./myprog -number-of-thread 4

Note that the '-number-of-thread' in the sample script above is an example of thread number parameter. Each program has own name of such parameter.

To submit a parallel job using 4 threads with script sub.sh to the rcc-30d batch queue and have the standard output and error go to the files sub.sh.ojobid and sub.sh.ejobid, respectively, in the directory from which the job was submitted, use

qsub -q rcc-30d -pe thread 4 sub.sh

Parallel Jobs using MPI libraries

Several MPI libraries are installed on zcluster. Here are sample scripts to use some version of MPI llibraries. For other versions, you need to change the path to mpirun appropriately. For a list of all MPI libraries installed on zcluster, please see Code Compilation on zcluster.

In the examples below, the executable name will be called myprog. The second line in the scripts below, namely cd working_directory or cd ${HOME}/subdirectory can be replaced by cd `pwd` if the job will be submitted from within the working directory.

NOTE: MPICH jobs executed with mpirun have to use the -machinefile option as shown in the example below, otherwise your mpi job will not use the processors assigned to it by the queueing system. In contract, when using MPICH2, OpenMPI and MVAPICH2 it is not necessary to use the -machinefile option.

To run a parallel MPI job using the default MPICH2 (PGI) compilers. For example, if you compiled the code with mpicc or mpif90, etc, without full path (e.g. script name submpich2.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/mpich2/1.4.1p1/pgi123/lib:${LD_LIBRARY_PATH}
mpirun -np $NSLOTS ./myprog

To run a parallel MPI job using MPICH2 1.4.1p1 and e.g. GNU 4.4.4 compilers (e.g. script name submpich2.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/mpich2/1.4.1p1/gcc_4.4.4/lib:${LD_LIBRARY_PATH}
/usr/local/mpich2/1.4.1p1/gcc_4.4.4/bin/mpirun -np $NSLOTS ./myprog

Note that with MPICH2 it is not necessary to include the -machinefile option when submitting the job to a batch queue. When using other MPICH2 compilations, such as for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script.

To run a parallel MPI job using MPICH2 3.0.4 and e.g. GNU 4.5.3 compilers (e.g. script name submpich2.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/gcc/4.5.3/lib64:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH=/usr/local/mpich2/3.0.4/gcc453/lib:${LD_LIBRARY_PATH}
/usr/local/mpich2/3.0.4/gcc453/bin/mpirun -np $NSLOTS ./myprog

Note that with MPICH2 it is not necessary to include the -machinefile option when submitting the job to a batch queue. When using other MPICH2 compilations, such as for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script.

To run a parallel MPI job using MPICH2 3.2 and e.g. GNU 5.3.0 compilers (e.g. script name submpich2.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/gcc/5.3.0/lib64:${LD_LIBRARY_PATH}
/usr/local/mpich2/3.2/gcc530/bin/mpirun -np $NSLOTS ./myprog

Note that with MPICH2 it is not necessary to include the -machinefile option when submitting the job to a batch queue. When using other MPICH2 compilations, such as for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script.

To run a parallel MPI job using OpenMPI 1.4.4 and e.g. GNU 4.1.2 compilers (e.g. script name subompi.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/openmpi/1.4.4/gcc412/lib:${LD_LIBRARY_PATH}
/usr/local/openmpi/1.4.4/gcc412/bin/mpirun --mca btl "tcp,self" -np $NSLOTS ./myprog

To run a parallel MPI job using OpenMPI 1.6.2 and e.g. GNU 4.7.1 compilers (e.g. script name subompi.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/gcc/4.7.1/lib64:/usr/local/openmpi/1.6.2/gcc471/lib:${LD_LIBRARY_PATH}
/usr/local/openmpi/1.6.2/gcc471/bin/mpirun --mca btl "tcp,self" -np $NSLOTS ./myprog

Note that with OpenMPI you can use the mpirun command and there is no need to include the -machinefile option. When using other OpenMPI compilations, such as the one for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script. To use OpenMPI over Infiniband, remove the mpirun option

––mca btl "tcp,self"

from the script.

To run a parallel MPI job using MVAPICH2/GNU over Infiniband (e.g. script name submvapich2.sh)

#!/bin/bash
cd working_directory
export LD_LIBRARY_PATH=/usr/local/mvapich2/1.8/gcc444/lib:${LD_LIBRARY_PATH}
/usr/local/mvapich2/1.8/gcc444/bin/mpirun -np $NSLOTS ./myprog

Note that with MVAPICH2 it is not necessary to include the -machinefile option when submitting the job to the batch queue. When using other MVAPICH2 compilations, such as the one for PGI compilers, users need to adjust the path to the libraries and to mpirun appropriately in the script.

To run a parallel MPI job using MVAPICH2 v. 2.0.1 using Intel 14.0 over Infiniband (e.g. script name submvapich2.sh)

#!/bin/bash
cd working_directory
export MV2_SMP_USE_CMA=0
export LD_LIBRARY_PATH=/usr/local/mvapich2/2.0.1/intel140/lib:${LD_LIBRARY_PATH}
/usr/local/mvapich2/2.0.1/intel140/bin/mpirun -np $NSLOTS ./myprog

To run a parallel MPI job using legacy MPICH1/PGI (e.g. script name submpi.sh):

#!/bin/bash
cd working_directory
echo "Got $NSLOTS processors."
echo "Machines:"
cat $TMPDIR/machines
/usr/local/pgi/linux86-64/2012/mpi/mpich/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines ./myprog

Note that lines 3, 4, and 5 in this script are optional.

NOTE:Do NOT put the job into the background with a '&' in the shell script. This will confuse the queueing system.

Example of MPI job submission commands

1.To submit a parallel MPI job that uses MPICH and 4 CPU cores, with script submpi.sh to the biof-30d batch queue and have the standard output and error go to the files submpi.sh.ojobid and submpi.sh.ejobid, respectively, use

qsub -q biof-30d -pe mpi 4 submpi.sh

2.To submit a parallel MPI job that uses MPICH2 and 4 CPU cores, with script submpich2.sh to the rcc-30d batch queue

qsub -q rcc-30d -pe mpi 4 submpich2.sh

3.To submit a parallel MPI job that uses OpenMPI and 12 CPU cores, with script subompi.sh to the rcc-30d batch queue

qsub -q rcc-30d -pe ompi 12 subompi.sh

4.To submit a parallel MPI job that uses MVAPICH2 over Infiniband and 32 CPU cores, with script submvapich2.sh to the rcc-ib-30d batch queue

qsub -q rcc-ib-30d -pe mpi 32 submvapich2.sh

Back to Top

Parallel Jobs using shared-memory threads

In the example below, the executable name will be called myprog and it requires input parameters to be piped in. The input parameters are in a file called myin and the output data will be stored in a file called myout. The working_directory is the path to your working directory (e.g., it could be /home/labname/username/subdir, which can be referenced as ${HOME}/subdir )

Sample script to run a parallel shared-memory job using OpenMP (e.g. script name sub.sh)

#!/bin/bash
cd working_directory
export OMP_NUM_THREADS=4
./myprog < myin > myout

NOTE:

  • Do NOT put the job into the background with a '&' in the shell script. This will confuse the queueing system.
  • Some applications use a command option to specify the number of threads to use. This is application dependent, please refer to the documentation of each application.

The file myin in the examples above is only necessary if your program requires standard input data and the file myout is only necessary if you want the standard output data (if any) to be stored in a separate file instead of the standard output file of the batch job (e.g sub.sh.ojobid in the example below for the serial job). If your program does not require one or both of these files, you have to remove the corresponding piping symbols ( < and/or > ) in the last line of the scripts above.

To submit a parallel job using 4 threads with script sub.sh to the rcc-30d batch queue and have the standard output and error go to the files sub.sh.ojobid and sub.sh.ejobid, respectively, in the directory from which the job was submitted, use

qsub -q rcc-30d -pe thread 4 sub.sh

NOTE:

  • The number of threads specified in the job submission script (e.g. sub.sh) has to match the number following the qsub option -pe thread
  • Although the rcc-30d queue targets nodes that have 8 and 12 cores, the maximum number of threads that you can request per job in this queue is 10.

Back to Top

Parallel Jobs using hybrid MPI/shared-memory threads

In the example below, the executable name will be called myprog and it does not require input parameters to be piped in. The working_directory is the path to your working directory (e.g., it could be /home/labname/username/subdir, which can be referenced as ${HOME}/subdir ).

Sample script to run a hybrid MPI/OpenMP job, using MPICH2 3.0.4/GNU 4.4.7 (e.g. script name subhybrid.sh)

#!/bin/bash
cd working_directory
export OMP_NUM_THREADS=2
/usr/local/mpich2/3.0.4/gcc447/bin/mpirun -np ${NSLOTS} ./myprog

where we assumed that each MPI process will run with 2 threads.

NOTE:

  • Do NOT put the job into the background with a '&' in the shell script. This will confuse the queueing system.
  • Some applications use a command option to specify the number of threads to use. This is application dependent, please refer to the documentation of each application.


To submit a parallel job to the rcc-30d batch queue, using 10 MPI processes and 2 threads per MPI process, use

qsub -q rcc-30d -pe mpi 10 -l hybrid_slots=2 subhybrid.sh

With the sample script subhybrid.sh above, the number that follows -pe mpi will be the number of MPI processes and the number specified by -l hybrid_slots= is the number of threads per MPI process.


Back to Top

Additional Notes

  • You can include the qsub option -j y to have the standard error of the job merged into the standard output file for the job.
  • If you wish to submit a binary as a batch job, without using a script (not recommended in general, but in some cases there might be a good reason for the user to prefer this approach), you will need to add the qsub option -b y
  • To have an email notification at the end of the job, include the qsub option -m e
  • To have an email notification when the job starts, include the qsub option -m b

Notes about the job submission script

  • You can use the pound sign # to start comment lines in your script, but # cannot be immediately followed by a dollar sign $. The combination #$ is reserved to indicate a qsub option defined in the script.
  • The job submission script does not need to have execution permission.

Back to Top

Memory Considerations and Large Memory Jobs

The queueing system provides an option to request the amount of memory that a job needs. This qsub option is

-l mem_total=number

where number is the amount of memory requested.

Sample command to submit a serial batch job with a script called sub.sh to the rcc-30d queue and request 20GB of RAM:

qsub -q rcc-30d -l mem_total=20g sub.sh       

Note: the -l here is a lower case L.

The rcc-30d queue sends jobs to nodes that have 16GB of RAM and nodes that have 48GB of RAM. If your job needs more than about 12GB of RAM, we recommend that you include the option

-l mem_total=20g 

to ensure that it is assigned to a node with 48GB of RAM (as the RAM is shared by all jobs running on the node).

If the memory requirement for your job cannot be met by the nodes targetted by the rcc-30d queue, then you are welcome to request permission to use the large memory queues.


Back to Top

Running GPU jobs

The GPU cards are set to exclusive-process compute mode, so each GPU card will only be used by one job at a time. If interested in running jobs that use the GPU cards, please submit a support request to ask the GACRC staff to add your name to the permission list of this queue.

Example of job submission shell scripts (sub.sh) and job submission commands

In the examples below, the executable name will be called myprog and it requires input parameters to be piped in. The input parameters are in a file called myin and the output data will be stored in a file called myout. The working_directory is the path to your working directory (e.g., it could be ${HOME}/subdir or a scratch directory, etc). You can also replace the second line by cd `pwd` if you are submitting the job from the working directory. We will also assume that the program uses CUDA 5.0.35 (if you are using a different version of CUDA, please change the PATH and LD_LIBRARY_PATH variables appropriately). Note that the path to the CUDA binaries and libraries can be set in the user's dotfiles (.bashrc and .profile for bash and .cshrc for csh), then they do not have to be included in the job submission scripts.

1. To run a serial job

#!/bin/bash
cd working_directory
export PATH=/usr/local/cuda/5.0.35/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/cuda/5.0.35/lib64:${LD_LIBRARY_PATH}
./myprog < myin > myout

Entering a standard input file (myin) and standard output file myout is only necessary if your executable requires input from standard input and if it outputs data to standard output.

To request 1 CPU core and 1 GPU card, submit the job to the queue with (for example, using the rcc-kgpu-30d queue):

qsub -q rcc-kgpu-30d -l cuda=1 sub.sh

Note that in addition to specifying the correct GPU queue, it is necessary to explictly request a cuda instance with the -l cuda=1 (i.e. minus lower case L, followed by a space and cuda= followed by the number one) option. The -l cuda=... option specifies the number of GPU cards needed by your program. So if your program needs 1 CPU core and 2 GPU cards, submit the job to the queue with

qsub -q rcc-kgpu-30d -l cuda=2 sub.sh

2. To run a parallel MPI job (MPICH2/GNU) using one GPU card per CPU core (bash shell)

The queueing system does not allow a job to use GPU cards from different nodes. So at the moment the maximum number of GPU cards a single job can use is 6 for the rcc-mgpu-30d queue (with targets the M2070 cards) and 8 for the rcc-kgpu-30d queue (which targets the K20X cards). If all your MPI processes need to be on the same node as the GPU cards, please use the qsub "-pe mpi1n" option (instead of the "-pe mpi" option).

#!/bin/bash
cd working_directory
export PATH=/usr/local/cuda/5.0.35/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/cuda/5.0.35/lib64:${LD_LIBRARY_PATH}
/usr/local/mpich2/1.4.1p1/gcc412/bin/mpirun -np $NSLOTS ./myprog < myin > myout

Submit the job to the queue with (for example, using the rcc-kgpu-30d queue):

a. If using 2 CPUs and 2 GPUs:

qsub -q rcc-kgpu-30d -pe mpi1n 2 -l cuda=2 sub.sh

b. If using 6 CPUs and 6 GPUs:

qsub -q rcc-kgpu-30d -pe mpi1n 6 -l cuda=6 sub.sh

Note that in this example each CPU core will be able to use 1 GPU card.

c. If using 6 CPUs (with MPI) and 1 GPU:

qsub -q rcc-kgpu-30d -pe mpi 6 -l cuda=1 sub.sh

In this example the MPI master process will be on the same node where the GPU card is located, but the other MPI processes might be allocated on other nodes.

3. To run a parallel job with CPU threads

Sample job submission command if using 4 CPU threads and 1 GPU:

qsub -q rcc-kgpu-30d -pe thread 4 -l cuda=1 sub.sh

Back to Top

Checking the Status of Jobs

While jobs are pending or running

Use the qstat command to check the status of running or pending jobs.

qstat [-u username] [-j jobid]

where username is the user whose jobs you want to check and jobid is the JOBID of a specific job. The -j option gives more detailed information about the job(s) (such as working directory, script name, submit time, job start time, cpu and memory usage).

Example

  • qstat shows your jobs in the pool (running and pending)
  • qstat -u "*" shows all the jobs in the pool
  • qstat -u johndoe shows all jobs for user johndoe
  • qstat -j 10407 gives detailed information about the job with JOBID 10407

Back to Top

After a job is done

To check information about jobs that have already ended, use the qacct command. Note that qacct does not return any information for a job that is still running or pending.

qacct [-o username] [-j jobid]

Example

  • qacct -j 1004 shows info about job with JOBID 1004, which completed successfully or exited the queue for other reasons

Sample output of a qstat command:

job-ID prior name user state submit/start at queue slots ja-task-ID
9920 0.50250 sub.sh johndoe r 02/22/2012 15:45:10 rcc-30d@compute-12-27.local 1

The output explained

The first five columns show the job ID number (JOBID), priority, name (which is the name of the job submission script file, unless a job name is explicity specified), the username that submitted the job and the state of the job, respectively. Typical values for the state are (1). r, which indicates the job is running, (2). qw, which means the job is pending (queued, waiting), and (3) Eqw, which indicates an error state, that is, the job cannot run because it has encountered an error. Jobs in the Eqw state require attention, you can use the qstat -j JOBID and the qacct -j JOBID commands (replace JOBID by the ID number of your job) to look for error messages that might help you identify the problem and fix it, before resubmitting the job (and deleting the job in Eqw state). For jobs in the qw state, the 6th column shows the date and time when the job was submitted; for jobs that are running, the date and time correspond to when the job started running. The 7th column shows the queue name and the hostname on which the job is running. The last column shows the number of cores requested by the job. Note that for multi-node jobs, the hostname in the 7th column is the node where the master process is running.


Sample output lines in a qacct -j jobid command:

qname        rcc-30d
hostname     compute-15-13.local
group        abclab
owner        johndoe
jobname      sub.sh
jobnumber    1004
qsub_time    Sat Dec  1 16:19:25 2012
start_time   Sat Dec  1 16:19:29 2012
end_time     Sun Dec  2 11:46:23 2012
slots        1
failed       0
exit_status  0
ru_wallclock 70014
cpu          69994.589
mem          92808.195
maxvmem      1.828G

Note that the ru_wallclock variable returns the wall clock time, cpu gives the amount of cpu time used, and maxvmem corresponds to the maximum amount of virtual memory used. The mem variable gives the amount of memory integrated over time, so in general it is not a relevant variable to consider.

When a job is killed by the user or if it exits the queue because it exceeded the maximum runtime of the queue, one of the lines returned by the qacct command is "failed 100 : assumedly after job". In general, the standard output and error of the job will have no indication that the job was killed by user, or that it exceeded any queue limits.


Back to Top

Files Created at Job Start

By default, the standard output and the standard error of the job will be written into files called shellscriptname.ojobid and shellscriptname.ejobid, respectively, where jobid is the job id number. If you submit your job with the -j y option, then the standard error will be written into the standard output file.


Canceling/Removing a Job

Use the qdel command to cancel/remove a job from the job pool:

qdel [-u username] jobid [jobid]

Example

  • qdel 10408 cancels your job with JOBID 10408
  • qdel 10408 10409 cancels your jobs with JOBIDs 10408 and 10409
  • qdel -u your_user_name cancels all jobs you have in the queue

Occasionally, if the node does not respond to qdel, then your job may not be able to be removed. Please contact us with the job number through our support form GACRC Contact Form.


Back to Top

Receiving an Email when Job Terminates

When you submit a batch job with qsub with the option -m e, you will receive an email at the end of the job. You can also use the option -m b to receive an email when the job starts running. Use the -M option to specify your email address.

qsub -q queuename -m e -M myemail.uga.edu script_name

Back to Top

Runchaining Jobs

We have found that a common need is to be able to run the same job over and over. For instance when you need to do a large number of iterations, you run so many and write in a data set the information needed to restart the job where it left off. When the job is restarted it reads the restart information and continues where the previous execution left off.

To have one job automatically submit the next one once it finishes, you can add the following lines at the end of your job submission script:

qsub -q queuename next_script_name
exit

Example

In the examples below we assume that the executable myprog does not require any standard input. The working directory is assumed to be ${HOME}/subdirectory.

Serial job using bash:

#!/bin/bash
cd ${HOME}/suddirectory
time ./myprog
qsub -q rcc-30d sub2.sh
exit

Parallel job using bash (running MPICH/PGI with 4 MPI processes):

#!/bin/bash
cd ${HOME}/subdirectory
mpirun -np $NSLOTS -machinefile $TMPDIR/machines ./myprog
qsub -q rcc-30d -pe mpi 4 sub2.sh
exit

First the script sub1.sh is submitted to the queue. Once it finishes running, it automatically submits script sub2.sh to the queue. This script can in turn submit sub3.sh to the queue when it completes, and so on. For this procedure, the user can prepare a sequence of scripts, which will then be submitted one at a time to the queue and run in sequence. Alternatively, the script sub1.sh can resubmit itself back to the queue once it finishes running. This would create an "infinite loop", a situation that is not recommended. To break the infinite loop, the user can set some termination rules for the job resubmission process.

Example of a termination rule

One way to break out of an infinite job resubmission loop is to have the code generate a file when the program finally "converges" (or when it completes a predetermined number of steps, for example). Let us call this file finalresults.txt. The job submission script sub.sh checks whether the file finalresults.txt exists. If it does not, then the script sub.sh is submitted to the queue again, otherwise the script simply exits and the resubmission chain is terminated.

A simple script sub.sh that accomplishes this is the following:

Serial job using csh (tcsh)

#!/bin/csh
cd ${HOME}/subdirectory
time ./myprog
if ( ! -e finalresults.txt ) then
qsub -q rcc-30d sub.sh
endif
exit

Serial job using ksh (bash):

#!/bin/ksh
cd ${HOME}/subdirectory
time ./myprog
if [ ! -f finalresults.txt ]
then
qsub -q rcc-30d sub.sh
fi
exit

Back to Top

Running an Interactive Job

Interactive submissions are jobs that require user interaction to run. Examples would be any jobs that prompt the user for keystrokes, button pushes or input. Interactive submissions are also a good way to prototype a submission before actually submitting the job to the cluster queues.

Interactive jobs have a maximum lifetime of 12 hours and they will be automatically killed by gridengine after accumulating 12 hours of runtime

For better management of resources, users will not ssh into the interactive nodes to run jobs. Instead, interactive jobs have to be run using the SGE interactive queue, which will open a session in one of the interactive nodes. To start an interactive session, on a zcluster terminal prompt type

qlogin

If an idle core is available on the interactive nodes, this command will log you into one of the interactive nodes and it will return a shell prompt from that node. At that prompt you can run your interactive application, including ones that require an X windows front end (graphical front end). More information on running X-windows applications is given below.

Example of an interactive session:

zcluster> qlogin
Your job 9242 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 9242 has been successfully scheduled.
compute-14-7.local>

In this example the interactive session was established on a node called compute-14-7.local.

To end an interactive session, type exit at the interactive session shell prompt. Please end your interactive session as soon as you finish running your interactive application. Please do not leave interactive sessions idle, as this will tie up resources that can be used by other interactive jobs.

If you lose connection to the shell terminal where your interactive session is running, your qlogin session might still be running (and the interactive process running in it might also continue to run), so when that happens, it is important for you to use qstat to see if you have any left over qlogin sessions. If so, you can see on which node your previous qlogin session was running (this information is provided by qstat) and connect back to the same node with qlogin -l h=hostname in order to check on any processes that you had started. If there are no processes left on that previous qlogin session, please use qdel to delete it. Note that once you delete a qlogin session, any interactive processes running in that session will be terminated automatically.

Running an interactive parallel job

In order to run a parallel (MPI or threaded) job interactively, users have to give a qlogin option to request the number of cores that will be used. The number of cores should match the maximum number of threads or MPI processes that will be used. For example, to run a shared-memory application with 4 threads or an MPI program using 4 MPI processes, type

qlogin -pe parallel 4

At the interactive session shell prompt, you can change directory as if you were logged in directory to the machine and then you can run your shared-memory application. To run an interactive MPI process, after changing to your working directory, you will need to create a file (called e.g. mlist) with the name of the host to be used with the mpirun -machinefile option. This can be done by typing the command

hostname

to check the name of the interactive node and then creating a file with this name (e.g. compute-14-7.local) repeated multiple times in a column (the number of times that it is repeated should be equal to the number of CPU cores you want to use for the MPI program). That is, the contents of mlist will be for example

compute-14-7.local
compute-14-7.local
compute-14-7.local
compute-14-7.local

Alternatively, you can use the following commands to create an mlist (e.g. with 4 entries)

echo `hostname` > mlist
echo `hostname` >> mlist
echo `hostname` >> mlist
echo `hostname` >> mlist

Note that in the first line there is a single > before mlist and in the subsequent lines there are two such symbols (i.e. >> before mlist. Put this file (mlist) in your working directory and then run the MPI program a.out as follows (e.g. using 4 CPU cores):

compute-14-7>  mpirun -np 4 -machinefile mlist ./a.out

If your code links to MPICH2 or OpenMPI, use the corresponding full path to the mpirun command in the lines above. For example to use MPICH2 with gcc 4.4.4:

compute-14-7>  /usr/local/mpich2/1.4.1p1/gcc444/bin/mpirun -n 4 -f mlist ./a.out

To end an interactive parallel session, type exit at the interactive session shell prompt. Please end your interactive session as soon as you finish running your interactive application. Please do not leave interactive sessions idle, as this will tie up resources that can be used by other interactive jobs.


Back to Top

Running an X-windows application

In order to run X-windows applications on zcluster and have the X windows display on your local machine, your local machine needs to have an X client. Most recent Unix based operating systems, including Linux and Mac OSX, include an X client and you can use the ssh -X option to enable X forwarding. On a Mac OSX this ssh command has to be issued in an X11 terminal (not a regular terminal) in order for X-forwarding to work.

ssh -X username@zcluster.rcc.uga.edu

For Apple OSX Mountain Lion (10.8) and future versions, X11 is not included in the OS. Please refer to Frequently Asked Questions for detailed installation procedure.

A free X client for Microsoft Windows operating systems is described on the Frequently Asked Questions page. For more information on how to connect to zcluster, please see Connecting to zcluster.

X-windows utilities or applications that do not consume much cputime, such as graphical front-end to text editors (e.g. xemacs), pdf readers (e.g. evince) to display user guide of software, etc, can be run directly on zcluster (i.e., the login node). Other X-windows applications should be run on an interactive node, using the SGE interactive queue as described in the section above. Example of an interactive session to run Matlab with the graphical front end:

zcluster> qlogin
Your job 9242 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 9242 has been successfully scheduled.
compute-14-7.local> matlab

Back to Top