MATLAB-Sapelo2: Difference between revisions
(41 intermediate revisions by 2 users not shown) | |||
Line 10: | Line 10: | ||
=== Version === | === Version === | ||
R2023a, R2023b | |||
=== Author / Distributor === | === Author / Distributor === | ||
Line 18: | Line 18: | ||
=== Description === | === Description === | ||
MATLAB is a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis, and numeric computation. | |||
=== Running Program === | === Running Program === | ||
Line 26: | Line 26: | ||
For more information on Environment Modules on Sapelo2 please see the [[Lmod]] page. | For more information on Environment Modules on Sapelo2 please see the [[Lmod]] page. | ||
* Version | * Version R2023a is installed in /apps/gb/MATLAB/R2023a. In order to use this version of MATLAB, please first load the matlab/R2023a module with | ||
<pre class="gcommand"> | <pre class="gcommand"> | ||
ml matlab/ | ml matlab/R2023a | ||
</pre> | </pre> | ||
====Running MATLAB interactively==== | |||
Please do not run MATLAB interactively on the Sapelo2 login node, instead please run it using the [[Running_Jobs_on_Sapelo2#How_to_open_an_interactive_session | interactive partition]] (without GUI) or using [[OnDemand]] (with GUI). | |||
The best way to run MATLAB interactively with a graphical front-end (GUI) is to run the MATLAB interactive application in the [[OnDemand]] interface to Sapelo2. | |||
To run MATLAB interactively without a GUI, please first start an interactive job with the '''interact''' command. | |||
For example: | For example: | ||
1. To run with the graphical front-end on a regular node: | <!--- | ||
1. To run with the graphical front-end on a regular compute node: | |||
<pre class="gcommand"> | <pre class="gcommand"> | ||
xqlogin | xqlogin | ||
Line 63: | Line 51: | ||
matlab & | matlab & | ||
</pre> | </pre> | ||
Note that in order to have the | Note that in order to have the MATLAB GUI display on your local machine, you will need to have an X client running on your local machine and configure your ssh session to tunnel X11. '''See below for an option on [https://wiki.gacrc.uga.edu/wiki/MatLab-Sapelo2#Using_a_MATLAB_client_installed_on_your_local_machine_to_run_jobs_on_the_cluster run MATLAB directly on your local machine and have it submit jobs to the cluster]. ''' | ||
---> | |||
1. To run without the graphical front-end on a regular compute node: | |||
<pre class="gcommand"> | <pre class="gcommand"> | ||
interact | |||
ml matlab/ | ml matlab/R2023a | ||
matlab -nodisplay | matlab -nodisplay | ||
</pre> | </pre> | ||
2. To run without the graphical front-end on a node in a different partition, e.g. in abc_p, or to request more resources (cores or memory), use for example | |||
<pre class="gcommand"> | <pre class="gcommand"> | ||
interact -p abc_p -c 4 --mem 20gb | |||
ml matlab/R2023a | |||
ml matlab/ | |||
matlab -nodisplay | matlab -nodisplay | ||
</pre> | </pre> | ||
For more information on how to run interactive jobs, please see [[Running_Jobs_on_Sapelo2#How_to_open_an_interactive_session | interactive partition]] | |||
====Running | ====Running MATLAB as a batch job==== | ||
MATLAB can also be run as a batch job, for example in the batch partition. To do this, first create a MATLAB M-file with the MATLAB commands. Then use a job submission file to submit this job to the batch partition. | |||
Sample | Sample MATLAB M-file (matrixinv.m): | ||
<pre class="gscript"> | <pre class="gscript"> | ||
n = 500; | n = 500; | ||
Line 115: | Line 104: | ||
#SBATCH --output=%x.%j.out | #SBATCH --output=%x.%j.out | ||
#SBATCH --error=%x.%j.err | #SBATCH --error=%x.%j.err | ||
cd $SLURM_SUBMIT_DIR | cd $SLURM_SUBMIT_DIR | ||
ml matlab/ | ml matlab/R2023a | ||
matlab -nodisplay < matrixinv.m | matlab -nodisplay < matrixinv.m | ||
Line 137: | Line 124: | ||
#SBATCH --output=%x.%j.out | #SBATCH --output=%x.%j.out | ||
#SBATCH --error=%x.%j.err | #SBATCH --error=%x.%j.err | ||
cd $SLURM_SUBMIT_DIR | cd $SLURM_SUBMIT_DIR | ||
ml matlab/ | ml matlab/R2023a | ||
echo functionname | matlab -nodisplay -nosplash | echo functionname | matlab -nodisplay -nosplash | ||
Line 151: | Line 136: | ||
sbatch sub.sh | sbatch sub.sh | ||
</pre> | </pre> | ||
====Parallel Computing - Using multiple CPU cores on a single compute node ==== | ====Parallel Computing - Using multiple CPU cores on a single compute node ==== | ||
The Parallel Computing toolbox allows a user to use many CPU cores. If you want to use cores on the same node for the job you can use the <code>defaultProfile</code>. Here is a simple example using the '''parfor''' loop with 24 | The Parallel Computing toolbox allows a user to use many CPU cores. If you want to use cores on the same node for the job you can use the <code>defaultProfile</code>. Here is a simple example using the '''parfor''' loop with 24 MATLAB workers. | ||
Sample code psine.m | Sample code psine.m | ||
<pre class="gscript"> | <pre class="gscript"> | ||
p=parcluster('local'); | |||
p=parcluster( | |||
p.NumWorkers=25; | p.NumWorkers=25; | ||
ppool=parpool(p,24); | ppool=parpool(p,24); | ||
Line 183: | Line 166: | ||
#SBATCH --output=%x.%j.out | #SBATCH --output=%x.%j.out | ||
#SBATCH --error=%x.%j.err | #SBATCH --error=%x.%j.err | ||
cd $SLURM_SUBMIT_DIR | cd $SLURM_SUBMIT_DIR | ||
ml matlab/ | ml matlab/R2023a | ||
matlab -nodisplay < psine.m | matlab -nodisplay < psine.m | ||
</pre> | </pre> | ||
Note that the number that follows '''--cpus-per-tasks''' needs to match the number of | Note that the number that follows '''--cpus-per-tasks''' needs to match the number of MATLAB workers defined with <code>NumWorkers</code> in the MATLAB code . | ||
Sample job submission command | Sample job submission command | ||
Line 217: | Line 197: | ||
====Parallel Computing - Using cores from one or more compute nodes ==== | ====Parallel Computing - Using cores from one or more compute nodes ==== | ||
In order to use the Parallel Computing toolbox to run | In order to use the Parallel Computing toolbox to run MATLAB using either cores on a single node, or to use cores on multiple nodes, you need to configure MATLAB and create a new cluster profile. To do this, please login to Sapelo2, start an interactive session with <code>interact</code>, load the matlab module you want to use and start matlab. For example, to configure this for matlab/R2023a: | ||
<pre class="gcommand"> | <pre class="gcommand"> | ||
interact | |||
ml matlab/R2021a | ml matlab/R2021a | ||
Line 226: | Line 206: | ||
</pre> | </pre> | ||
In | In MATLAB, call the <code>configCluster</code> function: | ||
<pre class="gscript"> | |||
< | |||
>> configCluster | >> configCluster | ||
</ | </pre> | ||
This function only needs to be called '''once''' per version of | This function only needs to be called '''once''' per version of MATLAB. Please be aware that running <code>configCluster</code> more than once per version will reset your cluster profile back to default settings and erase any saved modifications to the profile. If calling the <code>configCluster</code> function returns the error '''Unrecognized function or variable 'configCluster' ''', then run the following command in an interactive MATLAB session: | ||
< | <pre class="gscript"> | ||
>>rehash toolboxcache | >>rehash toolboxcache | ||
</ | </pre> | ||
Sample | Sample MATLAB code that can be run in a Slurm batch partition and use more than one node (psine.m) | ||
<pre class="gscript"> | <pre class="gscript"> | ||
c = parcluster; | c = parcluster; | ||
Line 252: | Line 231: | ||
p.delete | p.delete | ||
</pre> | </pre> | ||
Note that the resources you want this parallel job to use (e.g. partition name, the walltime limit and the memory per CPU | Note that the resources you want this parallel job to use (e.g. partition name, the walltime limit and the memory per CPU need to be specified in this MATLAB code. This parallel job will automatically request as many cores as needed for the parpool defined in the code, and the cores can be allocated on more than one node, if needed. For more details on how to request resources in the MATLAB code, please see the [https://wiki.gacrc.uga.edu/wiki/MatLab-Sapelo2#Configuring_Jobs_from_within_MATLAB_on_the_cluster_or_on_your_local_machine Configuring Jobs from within MATLAB] session below. | ||
Sample job submission script sub.sh: | Sample job submission script sub.sh: | ||
Line 266: | Line 245: | ||
#SBATCH --output=%x.%j.out | #SBATCH --output=%x.%j.out | ||
#SBATCH --error=%x.%j.err | #SBATCH --error=%x.%j.err | ||
cd $SLURM_SUBMIT_DIR | cd $SLURM_SUBMIT_DIR | ||
ml matlab/ | ml matlab/R2023a | ||
matlab -nodisplay < psine.m | matlab -nodisplay < psine.m | ||
</pre> | </pre> | ||
Note that this job only needs to request one core. When this job runs, | Note that this job only needs to request one core. When this job runs, MATLAB will submit another parallel job using the resources specified in the psine.m code. It is important that this script (sub.sh) uses the '''--mem-per-cpu''' option to request memory per cpu and '''not''' the total memory with '''--mem'''. | ||
Sample job submission command | Sample job submission command | ||
Line 281: | Line 258: | ||
</pre> | </pre> | ||
====Using a | ====Using a MATLAB client installed on your local machine to run jobs on the cluster==== | ||
In order to use a MATLAB client installed on your local machine and have it offload work onto the cluster (Sapelo2), you will need to first install some cluster integration files on your local machine. Please note that you need to have the same version of MATLAB client installed on your local machine, as the MATLAB version you will use on the cluster. | |||
The Sapelo2 MATLAB support package can be found on Sapelo2, at the following location: | |||
'''For R2023a''' | |||
Windows: | Windows: /apps/gb/MATLAB/UGA.nonshared.R2023a.zip | ||
Linux/macOS: /apps/gb/MATLAB/UGA.nonshared.R2023a.tar.gz | |||
Download the appropriate archive file to your local machine and start MATLAB on your local machine. The archive file should be untarred/unzipped in the location returned by calling | Download the appropriate archive file to your local machine and start MATLAB on your local machine. The archive file should be untarred/unzipped in the location returned by calling | ||
<pre class="gscript"> | |||
< | |||
>> userpath | >> userpath | ||
</ | </pre> | ||
Configure MATLAB to run parallel jobs on your cluster by calling <code>configCluster</code>, which only needs to be called once per version of MATLAB. | Configure MATLAB to run parallel jobs on your cluster by calling <code>configCluster</code>, which only needs to be called once per version of MATLAB. | ||
<pre class="gscript"> | |||
< | |||
>> configCluster | >> configCluster | ||
</ | </pre> | ||
Submission to the remote cluster (Sapelo2) requires SSH credentials and you will need to configure key-based SSH. You will be prompted for your ssh username and your identity file (private key). | Submission to the remote cluster (Sapelo2) requires SSH credentials and you will need to configure key-based SSH. For information on how to set key-based ssh, please see https://www.ssh.com/academy/ssh/keygen. If your local machine is a Mac, please generate the ssh key with <code>ssh-keygen -t rsa -m PEM</code>. Once you have the key-based ssh set up, you can submit jobs to the cluster from within the MATLAB client on your local machine. You will be prompted for your ssh username and your identity file (private key). The username and location of the private key will be stored in MATLAB for future sessions. | ||
Jobs will now default to the cluster rather than submit to the local machine. | Jobs will now default to the cluster rather than submit to the local machine. | ||
NOTE: If you would like to submit to the local machine then run the following command: | NOTE: If you would like to submit to the local machine then run the following command: | ||
<pre class="gscript"> | |||
< | |||
>> % Get a handle to the local resources | >> % Get a handle to the local resources | ||
>> c = parcluster('local'); | >> c = parcluster('local'); | ||
</ | </pre> | ||
====Configuring Jobs from within | ====Configuring Jobs from within MATLAB on the cluster or on your local machine==== | ||
Prior to having | Prior to having MATLAB submit a job to the cluster, we can specify various parameters to pass to our jobs, such as partition, e-mail, walltime, etc. The WallTime, MemUsage, and QueueName (partition name) fields are mandatory in order to submit a job. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
Line 350: | Line 327: | ||
</pre> | </pre> | ||
Save changes after modifying AdditionalProperties for the above changes to persist between | Save changes after modifying AdditionalProperties for the above changes to persist between MATLAB sessions. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> c.saveProfile | >> c.saveProfile | ||
Line 357: | Line 333: | ||
To see the values of the current configuration options, display AdditionalProperties. | To see the values of the current configuration options, display AdditionalProperties. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> % To view current properties | >> % To view current properties | ||
Line 364: | Line 339: | ||
Unset a value when no longer needed. | Unset a value when no longer needed. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> % Turn off email notifications | >> % Turn off email notifications | ||
Line 371: | Line 345: | ||
</pre> | </pre> | ||
==== | ====Submitting Independent Batch Jobs from within locally installed MATLAB==== | ||
Use the <code>batch</code> command to submit asynchronous jobs to the cluster. Users can either run a single function or a Matlab script as a batch job. You must make sure the script is in the MATLAB path. If running MATLAB locally, the command <code>userpath</code> displays the location you can save your .m scripts. See the MATLAB documentation for <code>userpath</code> if you would like to add directories to that path. Since your local file system is different from the worker file system(Sapelo2), if you are submitting jobs to Sapelo 2 from local MATLAB you also have to set the 'AutoAddClientPath' option to false. The <code>batch</code> command will return a job object which is used to access the output of the submitted job. See the MATLAB documentation for more help on <code>batch</code>. | |||
Shown below is an example submitting a batch job from local MATLAB which runs the script testscript.m. Notice that the .m extension is left off. | |||
<pre class="gscript"> | <pre class="gscript"> | ||
Line 378: | Line 354: | ||
>> c = parcluster; | >> c = parcluster; | ||
>> % | >> % Submit job to query where MATLAB is running on the cluster | ||
>> | >> j = batch(testscript,'AutoAddClientPath',false); | ||
>> % Query job for state | |||
>> j.State | |||
>> % If state is finished, fetch the results | |||
>> load(j) | |||
>> % Delete the job after results are no longer needed | |||
>> % | >> j.delete | ||
>> | |||
</pre> | </pre> | ||
<pre class="gscript"> | You can also run functions with multiple arguments using the <code>batch</code> command. For a function to be available it must be in your MATLAB path. Built in functions will automatically be included, but your own functions you will have to save in your userpath. When running functions, you must specify that you are using the worker established with the <code>parcluster</code> command, so use <code>c.batch</code> instead of <code>batch</code>. The example below shows a job which prints the working directory. When retrieving the data, use the command <code>fetchOutputs</code> instead of <code>load</code>.<pre class="gscript"> | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
>> c = parcluster; | >> c = parcluster; | ||
>> % Submit job to query where MATLAB is running on the cluster | >> % Submit job to query where MATLAB is running on the cluster | ||
>> j = c.batch(@pwd, 1 | >> j = c.batch(@pwd, 1,'AutoAddClientPath',false); | ||
>> % Query job for state | >> % Query job for state | ||
>> j.State | >> j.State | ||
>> j.fetchOutputs{:} | >> j.fetchOutputs{:} | ||
ans = | |||
'/home/keekov' | |||
</pre> | </pre> | ||
To retrieve a list of currently running or completed jobs, call <code>parcluster</code> to retrieve the cluster object. The cluster object stores an array of jobs that were run, are running, or are queued to run. This allows us to fetch the results of completed jobs. Retrieve and view the list of jobs as shown below. | To retrieve a list of currently running or completed jobs, call <code>parcluster</code> to retrieve the cluster object. The cluster object stores an array of jobs that were run, are running, or are queued to run. This allows us to fetch the results of completed jobs. Retrieve and view the list of jobs as shown below. The <code>Jobs</code> command also allows you to see the job id and status of previously submitted jobs. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> c = parcluster; | >> c = parcluster; | ||
Line 422: | Line 390: | ||
</pre> | </pre> | ||
Data that has been written to files on the cluster needs be retrieved directly from the file system (e.g. via ftp). | |||
To | To fetch the outputs of a previously completed job use the <code>fetchOutputs</code> or <code>load</code>command. This job used the function method and so data is retrieved with <code>fetchOutputs</code> instead of <code>load.</code> | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> % Get a handle to the job with ID 2 | >> % Get a handle to the job with ID 2 | ||
>> j2 = c.Jobs(2); | >> j2 = c.Jobs(2); | ||
>> % Fetch results for job with ID 2 | >> % Fetch results for job with ID 2 | ||
>> j2.fetchOutputs{:} | >> j2.fetchOutputs{:} | ||
</pre> | </pre> | ||
====Submitting Parallel Batch Jobs from within | ====Submitting Parallel Batch Jobs from within locally installed MATLAB==== | ||
Users can also submit parallel workflows with the batch command. Let’s use the following example for a parallel job, which is saved as parallel_example.m. | Users can also submit parallel workflows with the <code>batch</code> command. Let’s use the following example for a parallel job, which is saved as ''parallel_example.m''. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
Line 457: | Line 421: | ||
</pre> | </pre> | ||
This time when we use the batch command, to run a parallel job, | This time when we use the <code>c.batch</code> command, to run a parallel job, we will also specify a MATLAB Pool. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
Line 481: | Line 444: | ||
NOTE: For some applications, there will be a diminishing return when allocating too many workers, as the overhead may exceed computation time. | NOTE: For some applications, there will be a diminishing return when allocating too many workers, as the overhead may exceed computation time. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
Line 500: | Line 462: | ||
Once we have a handle to the cluster, we will call the <code>findJob</code> method to search for the job with the specified job ID. | Once we have a handle to the cluster, we will call the <code>findJob</code> method to search for the job with the specified job ID. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> % Get a handle to the cluster | >> % Get a handle to the cluster | ||
Line 521: | Line 482: | ||
The job now runs in 4.73 seconds using eight workers. Run code with different number of workers to determine the ideal number to use. | The job now runs in 4.73 seconds using eight workers. Run code with different number of workers to determine the ideal number to use. | ||
====Debugging a | ==== Parallel Interactive Jobs using MATLAB on the cluster ==== | ||
To run an interactive pool job on the cluster, continue to use parpool as you’ve done before. Start an interactive session with qlogin (or with srun), load the matlab module and start MATLAB with <code>matlab -nodislay</code>. In Matlab, run | |||
<pre class="gscript"> | |||
>> % Get a handle to the cluster | |||
>> c = parcluster; | |||
>> % Open a pool of 64 workers on the cluster | |||
>> p = c.parpool(64); | |||
</pre> | |||
Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster. | |||
<pre class="gscript"> | |||
>> % Run a parfor over 1000 iterations | |||
>> parfor idx = 1:1000 | |||
a(idx) = … | |||
end | |||
</pre> | |||
Once we are done with the pool, delete it. | |||
<pre class="gscript"> | |||
>> % Delete the pool | |||
>> p.delete | |||
</pre> | |||
====Debugging a MATLAB job==== | |||
If a serial job produces an error, call the <code>getDebugLog</code> method to view the error log file. When submitting independent jobs, with multiple tasks, specify the task number. | If a serial job produces an error, call the <code>getDebugLog</code> method to view the error log file. When submitting independent jobs, with multiple tasks, specify the task number. | ||
Line 529: | Line 514: | ||
For Pool jobs, only specify the job object. | For Pool jobs, only specify the job object. | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> c.getDebugLog(j) | >> c.getDebugLog(j) | ||
</pre> | </pre> | ||
When troubleshooting a job, the cluster admin may request the scheduler ID of the job. This can be derived by calling schedID | When troubleshooting a job, the cluster admin may request the scheduler ID of the job. This can be derived by calling <code>schedID</code> | ||
<pre class="gscript"> | <pre class="gscript"> | ||
>> schedID(j) | >> schedID(j) | ||
Line 544: | Line 527: | ||
=== Documentation === | === Documentation === | ||
Some documentation and sample files are available in /apps/gb/MATLAB/ | MATLAB documentation is available at https://www.mathworks.com/help/matlab/ | ||
Some documentation and sample files are available on Sapelo2, in /apps/gb/MATLAB/R2023a/help | |||
To learn more about the MATLAB Parallel Computing Toolbox, check out these resources: | To learn more about the MATLAB Parallel Computing Toolbox, check out these resources: | ||
*[http://www.mathworks.com/ | *[http://www.mathworks.com/help/parallel-computing/examples.html Parallel Computing Coding Examples] | ||
*[http://www.mathworks.com/help/distcomp/index.html Parallel Computing Documentation] | *[http://www.mathworks.com/help/distcomp/index.html Parallel Computing Documentation] | ||
*[http://www.mathworks.com/products/parallel-computing/index.html Parallel Computing Overview] | *[http://www.mathworks.com/products/parallel-computing/index.html Parallel Computing Overview] | ||
Line 557: | Line 542: | ||
=== Installation === | === Installation === | ||
'''Version | '''Version 2023a''' | ||
Installed in /apps/gb/MATLAB/R2023a. | |||
Available toolboxes: Almost all toolboxes for which UGA has a license. For details, see the directories in /apps/gb/MATLAB/R2023a/toolbox | |||
=== System === | === System === | ||
64-bit Linux | 64-bit Linux |
Latest revision as of 11:32, 13 May 2024
Category
Other, Programming, Graphics
Program On
Sapelo2
Version
R2023a, R2023b
Author / Distributor
The MathWorks (see http://www.mathworks.com)
Description
MATLAB is a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis, and numeric computation.
Running Program
Also refer to Running Jobs on Sapelo2
For more information on Environment Modules on Sapelo2 please see the Lmod page.
- Version R2023a is installed in /apps/gb/MATLAB/R2023a. In order to use this version of MATLAB, please first load the matlab/R2023a module with
ml matlab/R2023a
Running MATLAB interactively
Please do not run MATLAB interactively on the Sapelo2 login node, instead please run it using the interactive partition (without GUI) or using OnDemand (with GUI).
The best way to run MATLAB interactively with a graphical front-end (GUI) is to run the MATLAB interactive application in the OnDemand interface to Sapelo2.
To run MATLAB interactively without a GUI, please first start an interactive job with the interact command.
For example:
1. To run without the graphical front-end on a regular compute node:
interact ml matlab/R2023a matlab -nodisplay
2. To run without the graphical front-end on a node in a different partition, e.g. in abc_p, or to request more resources (cores or memory), use for example
interact -p abc_p -c 4 --mem 20gb ml matlab/R2023a matlab -nodisplay
For more information on how to run interactive jobs, please see interactive partition
Running MATLAB as a batch job
MATLAB can also be run as a batch job, for example in the batch partition. To do this, first create a MATLAB M-file with the MATLAB commands. Then use a job submission file to submit this job to the batch partition.
Sample MATLAB M-file (matrixinv.m):
n = 500; Q = orth(randn(n,n)); d = logspace(0,-10,n); A = Q*diag(d)*Q'; x = randn(n,1); b = A*x; tic, z = A\b, toc err = norm(z-x) res = norm(A*z-b)
Sample job submission script file (sub.sh) to run a serial (single-core) matlab program:
#!/bin/bash #SBATCH --job-name=myjobname #SBATCH --partition=batch #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=5gb #SBATCH --time=48:00:00 #SBATCH --output=%x.%j.out #SBATCH --error=%x.%j.err cd $SLURM_SUBMIT_DIR ml matlab/R2023a matlab -nodisplay < matrixinv.m
The parameters of the job, such as the maximum wall clock time, maximum memory, email address, the number of cores per task, and the job name need to be modified appropriately.
If you are using functions, you might have to use a sample script like this:
#!/bin/bash #SBATCH --job-name=myjobname #SBATCH --partition=batch #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=5gb #SBATCH --time=48:00:00 #SBATCH --output=%x.%j.out #SBATCH --error=%x.%j.err cd $SLURM_SUBMIT_DIR ml matlab/R2023a echo functionname | matlab -nodisplay -nosplash
To submit either of the two sample files sub.sh to the queue:
sbatch sub.sh
Parallel Computing - Using multiple CPU cores on a single compute node
The Parallel Computing toolbox allows a user to use many CPU cores. If you want to use cores on the same node for the job you can use the defaultProfile
. Here is a simple example using the parfor loop with 24 MATLAB workers.
Sample code psine.m
p=parcluster('local'); p.NumWorkers=25; ppool=parpool(p,24); parfor i=1:1024 A(i) = sin(i*2*pi/1024); end p = gcp; delete(p)
Sample job submission script sub.sh
#!/bin/bash #SBATCH --job-name=myjobname #SBATCH --partition=batch #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=25 #SBATCH --mem=50gb #SBATCH --time=48:00:00 #SBATCH --output=%x.%j.out #SBATCH --error=%x.%j.err cd $SLURM_SUBMIT_DIR ml matlab/R2023a matlab -nodisplay < psine.m
Note that the number that follows --cpus-per-tasks needs to match the number of MATLAB workers defined with NumWorkers
in the MATLAB code .
Sample job submission command
sbatch sub.sh
Parallel Computing - Using cores from one or more compute nodes
In order to use the Parallel Computing toolbox to run MATLAB using either cores on a single node, or to use cores on multiple nodes, you need to configure MATLAB and create a new cluster profile. To do this, please login to Sapelo2, start an interactive session with interact
, load the matlab module you want to use and start matlab. For example, to configure this for matlab/R2023a:
interact ml matlab/R2021a matlab -nodisplay
In MATLAB, call the configCluster
function:
>> configCluster
This function only needs to be called once per version of MATLAB. Please be aware that running configCluster
more than once per version will reset your cluster profile back to default settings and erase any saved modifications to the profile. If calling the configCluster
function returns the error Unrecognized function or variable 'configCluster' , then run the following command in an interactive MATLAB session:
>>rehash toolboxcache
Sample MATLAB code that can be run in a Slurm batch partition and use more than one node (psine.m)
c = parcluster; c.AdditionalProperties.QueueName = 'batch'; c.AdditionalProperties.WallTime = '24:00:00'; c.AdditionalProperties.MemUsage = '5G'; c.saveProfile p = c.parpool(5); parfor i=1:1024 A(i) = sin(i*2*pi/1024); end p.delete
Note that the resources you want this parallel job to use (e.g. partition name, the walltime limit and the memory per CPU need to be specified in this MATLAB code. This parallel job will automatically request as many cores as needed for the parpool defined in the code, and the cores can be allocated on more than one node, if needed. For more details on how to request resources in the MATLAB code, please see the Configuring Jobs from within MATLAB session below.
Sample job submission script sub.sh:
#!/bin/bash #SBATCH --job-name=myjobname #SBATCH --partition=batch #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem-per-cpu=5gb #SBATCH --time=48:00:00 #SBATCH --output=%x.%j.out #SBATCH --error=%x.%j.err cd $SLURM_SUBMIT_DIR ml matlab/R2023a matlab -nodisplay < psine.m
Note that this job only needs to request one core. When this job runs, MATLAB will submit another parallel job using the resources specified in the psine.m code. It is important that this script (sub.sh) uses the --mem-per-cpu option to request memory per cpu and not the total memory with --mem.
Sample job submission command
sbatch sub.sh
Using a MATLAB client installed on your local machine to run jobs on the cluster
In order to use a MATLAB client installed on your local machine and have it offload work onto the cluster (Sapelo2), you will need to first install some cluster integration files on your local machine. Please note that you need to have the same version of MATLAB client installed on your local machine, as the MATLAB version you will use on the cluster.
The Sapelo2 MATLAB support package can be found on Sapelo2, at the following location:
For R2023a
Windows: /apps/gb/MATLAB/UGA.nonshared.R2023a.zip
Linux/macOS: /apps/gb/MATLAB/UGA.nonshared.R2023a.tar.gz
Download the appropriate archive file to your local machine and start MATLAB on your local machine. The archive file should be untarred/unzipped in the location returned by calling
>> userpath
Configure MATLAB to run parallel jobs on your cluster by calling configCluster
, which only needs to be called once per version of MATLAB.
>> configCluster
Submission to the remote cluster (Sapelo2) requires SSH credentials and you will need to configure key-based SSH. For information on how to set key-based ssh, please see https://www.ssh.com/academy/ssh/keygen. If your local machine is a Mac, please generate the ssh key with ssh-keygen -t rsa -m PEM
. Once you have the key-based ssh set up, you can submit jobs to the cluster from within the MATLAB client on your local machine. You will be prompted for your ssh username and your identity file (private key). The username and location of the private key will be stored in MATLAB for future sessions.
Jobs will now default to the cluster rather than submit to the local machine.
NOTE: If you would like to submit to the local machine then run the following command:
>> % Get a handle to the local resources >> c = parcluster('local');
Configuring Jobs from within MATLAB on the cluster or on your local machine
Prior to having MATLAB submit a job to the cluster, we can specify various parameters to pass to our jobs, such as partition, e-mail, walltime, etc. The WallTime, MemUsage, and QueueName (partition name) fields are mandatory in order to submit a job.
>> % Get a handle to the cluster >> c = parcluster; [REQUIRED] >> % Specify memory to use for MATLAB jobs, per core >> c.AdditionalProperties.MemUsage = '5G'; >> % Specify a queue to use for MATLAB jobs >> c.AdditionalProperties.QueueName = 'partition-name'; >> % Specify the walltime (e.g. 5 hours) >> c.AdditionalProperties.WallTime = '05:00:00'; [OPTIONAL] >> % Specify an account to use for MATLAB jobs >> c.AdditionalProperties.AccountName = 'account-name'; >> % Specify e-mail address to receive notifications about your job >> c.AdditionalProperties.EmailAddress = 'user-id@uga.edu'; >> % Specify constraint for you job >> c.AdditionalProperties.Constraint = 'Intel'; >> % Specify number of GPUs >> c.AdditionalProperties.GpusPerNode = 1; >> % Specify GPU type >> c.AdditionalProperties.GpuType = 'K40';
Save changes after modifying AdditionalProperties for the above changes to persist between MATLAB sessions.
>> c.saveProfile
To see the values of the current configuration options, display AdditionalProperties.
>> % To view current properties >> c.AdditionalProperties
Unset a value when no longer needed.
>> % Turn off email notifications >> c.AdditionalProperties.EmailAddress = ''; >> c.saveProfile
Submitting Independent Batch Jobs from within locally installed MATLAB
Use the batch
command to submit asynchronous jobs to the cluster. Users can either run a single function or a Matlab script as a batch job. You must make sure the script is in the MATLAB path. If running MATLAB locally, the command userpath
displays the location you can save your .m scripts. See the MATLAB documentation for userpath
if you would like to add directories to that path. Since your local file system is different from the worker file system(Sapelo2), if you are submitting jobs to Sapelo 2 from local MATLAB you also have to set the 'AutoAddClientPath' option to false. The batch
command will return a job object which is used to access the output of the submitted job. See the MATLAB documentation for more help on batch
.
Shown below is an example submitting a batch job from local MATLAB which runs the script testscript.m. Notice that the .m extension is left off.
>> % Get a handle to the cluster >> c = parcluster; >> % Submit job to query where MATLAB is running on the cluster >> j = batch(testscript,'AutoAddClientPath',false); >> % Query job for state >> j.State >> % If state is finished, fetch the results >> load(j) >> % Delete the job after results are no longer needed >> j.delete
You can also run functions with multiple arguments using the batch
command. For a function to be available it must be in your MATLAB path. Built in functions will automatically be included, but your own functions you will have to save in your userpath. When running functions, you must specify that you are using the worker established with the parcluster
command, so use c.batch
instead of batch
. The example below shows a job which prints the working directory. When retrieving the data, use the command fetchOutputs
instead of load
.
>> % Get a handle to the cluster >> c = parcluster; >> % Submit job to query where MATLAB is running on the cluster >> j = c.batch(@pwd, 1,'AutoAddClientPath',false); >> % Query job for state >> j.State >> j.fetchOutputs{:} ans = '/home/keekov'
To retrieve a list of currently running or completed jobs, call parcluster
to retrieve the cluster object. The cluster object stores an array of jobs that were run, are running, or are queued to run. This allows us to fetch the results of completed jobs. Retrieve and view the list of jobs as shown below. The Jobs
command also allows you to see the job id and status of previously submitted jobs.
>> c = parcluster; >> jobs = c.Jobs;
Data that has been written to files on the cluster needs be retrieved directly from the file system (e.g. via ftp).
To fetch the outputs of a previously completed job use the fetchOutputs
or load
command. This job used the function method and so data is retrieved with fetchOutputs
instead of load.
>> % Get a handle to the job with ID 2 >> j2 = c.Jobs(2); >> % Fetch results for job with ID 2 >> j2.fetchOutputs{:}
Submitting Parallel Batch Jobs from within locally installed MATLAB
Users can also submit parallel workflows with the batch
command. Let’s use the following example for a parallel job, which is saved as parallel_example.m.
function t = parallel_example(iter) if nargin==0, iter = 8; end disp('Start sim') t0 = tic; parfor idx = 1:iter A(idx) = idx; pause(2) end t = toc(t0); disp('Sim Completed')
This time when we use the c.batch
command, to run a parallel job, we will also specify a MATLAB Pool.
>> % Get a handle to the cluster >> c = parcluster; >> % Submit a batch pool job using 4 workers for 16 simulations >> j = c.batch(@parallel_example, 1, {16}, 'Pool',4, … 'CurrentFolder','.', 'AutoAddClientPath',false); >> % View current job status >> j.State >> % Fetch the results after a finished state is retrieved >> j.fetchOutputs{:} ans = 8.8872
The job ran in 8.89 seconds using four workers. Note that these jobs will always request N+1 CPU cores, since one worker is required to manage the batch job and pool of workers. For example, a job that needs eight workers will consume nine CPU cores.
We will run the same simulation but increase the Pool size. This time, to retrieve the results later, we will keep track of the job ID.
NOTE: For some applications, there will be a diminishing return when allocating too many workers, as the overhead may exceed computation time.
>> % Get a handle to the cluster >> c = parcluster; >> % Submit a batch pool job using 8 workers for 16 simulations >> j = c.batch(@parallel_example, 1, {16}, 'Pool', 8, … 'CurrentFolder','.', 'AutoAddClientPath',false); >> % Get the job ID >> id = j.ID id = 4 >> % Clear j from workspace (as though we quit MATLAB) >> clear j
Once we have a handle to the cluster, we will call the findJob
method to search for the job with the specified job ID.
>> % Get a handle to the cluster >> c = parcluster; >> % Find the old job >> j = c.findJob('ID', 4); >> % Retrieve the state of the job >> j.State ans = finished >> % Fetch the results >> j.fetchOutputs{:}; ans = 4.7270
The job now runs in 4.73 seconds using eight workers. Run code with different number of workers to determine the ideal number to use.
Parallel Interactive Jobs using MATLAB on the cluster
To run an interactive pool job on the cluster, continue to use parpool as you’ve done before. Start an interactive session with qlogin (or with srun), load the matlab module and start MATLAB with matlab -nodislay
. In Matlab, run
>> % Get a handle to the cluster >> c = parcluster; >> % Open a pool of 64 workers on the cluster >> p = c.parpool(64);
Rather than running local on the local machine, the pool can now run across multiple nodes on the cluster.
>> % Run a parfor over 1000 iterations >> parfor idx = 1:1000 a(idx) = … end
Once we are done with the pool, delete it.
>> % Delete the pool >> p.delete
Debugging a MATLAB job
If a serial job produces an error, call the getDebugLog
method to view the error log file. When submitting independent jobs, with multiple tasks, specify the task number.
>> c.getDebugLog(j.Tasks(3))
For Pool jobs, only specify the job object.
>> c.getDebugLog(j)
When troubleshooting a job, the cluster admin may request the scheduler ID of the job. This can be derived by calling schedID
>> schedID(j) ans = 25539
Documentation
MATLAB documentation is available at https://www.mathworks.com/help/matlab/
Some documentation and sample files are available on Sapelo2, in /apps/gb/MATLAB/R2023a/help
To learn more about the MATLAB Parallel Computing Toolbox, check out these resources:
- Parallel Computing Coding Examples
- Parallel Computing Documentation
- Parallel Computing Overview
- Parallel Computing Tutorials
- Parallel Computing Videos
- Parallel Computing Webinars
Installation
Version 2023a
Installed in /apps/gb/MATLAB/R2023a.
Available toolboxes: Almost all toolboxes for which UGA has a license. For details, see the directories in /apps/gb/MATLAB/R2023a/toolbox
System
64-bit Linux