Meraculous-Sapelo2 - Revision history

Moses at 18:57, 14 December 2022

2022-12-14T18:57:29Z

← Older revision		Revision as of 14:57, 14 December 2022
Line 99:		Line 99:
	'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays		'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays

	'''cluster_ram_request''' specifies the maximum amount of memory (GB) to be allocated for task arrays; If elements of the task array are multi-threaded, Meraculous will automatically divide this number by the number of allocated CPU cores. In the example given above, at some pipeline steps, each CPU core can get 16 GB memory, that is (the number of cluster_ram_request (128)) / (the number of cluster_slots_per_task (8)) = 16.		'''cluster_ram_request''' specifies the maximum amount of memory (GB) to be allocated for each element in task arrays; If elements of the task array are multi-threaded, Meraculous will automatically divide this number by the number of allocated CPU cores. In the example given above, at some pipeline steps, each CPU core can get 16 GB memory, that is (the number of cluster_ram_request (128)) / (the number of cluster_slots_per_task (8)) = 16.

	'''cluster_walltime''' specifies the wall-clock time limit for cluster tasks. It must be specified as hh:mm:ss (168 hours is the upper limit for Sapelo2 batch partition).		'''cluster_walltime''' specifies the wall-clock time limit for cluster tasks. It must be specified as hh:mm:ss (168 hours is the upper limit for Sapelo2 batch partition).

Moses at 18:56, 14 December 2022

2022-12-14T18:56:15Z

← Older revision		Revision as of 14:56, 14 December 2022
Line 95:		Line 95:
	where '''use_cluster 1''' specifies to use a cluster for job submissions.		where '''use_cluster 1''' specifies to use a cluster for job submissions.

	'''cluster_num_nodes''' is specifies the number of available cluster compute nodes. This number can be approximate (mentioned in Meraculous user manual). In test, we ~~found~~ some pipeline steps (e.g., gapCloser) can use more nodes (e.g. 15) than the number you specified here (e.g. 10).		'''cluster_num_nodes''' is specifies the number of available cluster compute nodes. This number can be approximate (mentioned in Meraculous user manual). In test, we observed some pipeline steps (e.g., gapCloser) can use more nodes (e.g. 15) than the number you specified here (e.g. 10).

	'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays		'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays

Moses at 18:55, 14 December 2022

2022-12-14T18:55:31Z

← Older revision		Revision as of 14:55, 14 December 2022
Line 95:		Line 95:
	where '''use_cluster 1''' specifies to use a cluster for job submissions.		where '''use_cluster 1''' specifies to use a cluster for job submissions.

	'''cluster_num_nodes''' is specifies the number of available cluster compute nodes. This number can be approximate. In test, we found some pipeline steps (e.g., gapCloser) can use more nodes (e.g. 15) than the number you specified here (e.g. 10).		'''cluster_num_nodes''' is specifies the number of available cluster compute nodes. This number can be approximate (mentioned in Meraculous user manual). In test, we found some pipeline steps (e.g., gapCloser) can use more nodes (e.g. 15) than the number you specified here (e.g. 10).

	'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays		'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays

Moses at 18:53, 14 December 2022

2022-12-14T18:53:16Z

← Older revision		Revision as of 14:53, 14 December 2022
Line 95:		Line 95:
	where '''use_cluster 1''' specifies to use a cluster for job submissions.		where '''use_cluster 1''' specifies to use a cluster for job submissions.

	'''cluster_num_nodes''' is specifies the number of available cluster compute nodes (can be an approximate ~~number~~, e.g., 10).		'''cluster_num_nodes''' is specifies the number of available cluster compute nodes. This number can be approximate. In test, we found some pipeline steps (e.g., gapCloser) can use more nodes (e.g. 15) than the number you specified here (e.g. 10).

	'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays		'''cluster_slots_per_task''' specifies the maximum number of CPU cores to be allocated for multi-threaded elements in task arrays

	'''cluster_ram_request''' specifies the maximum amount of memory (GB) to be allocated for task arrays; If elements of the task array are multi-threaded ~~elements~~, this number ~~will be divided automatically~~ by ~~Meraculous for each~~ allocated CPU ~~core~~. In the example given above, at some pipeline steps, each CPU core can get 16 GB memory, that is (the number of cluster_ram_request 128) / (the number of cluster_slots_per_task 8) = 16.		'''cluster_ram_request''' specifies the maximum amount of memory (GB) to be allocated for task arrays; If elements of the task array are multi-threaded, Meraculous will automatically divide this number by the number of allocated CPU cores. In the example given above, at some pipeline steps, each CPU core can get 16 GB memory, that is (the number of cluster_ram_request (128)) / (the number of cluster_slots_per_task (8)) = 16.

	'''cluster_walltime''' specifies the wall-clock time limit for cluster tasks. It must be specified as hh:mm:ss (168 hours is the upper limit for Sapelo2 batch partition).		'''cluster_walltime''' specifies the wall-clock time limit for cluster tasks. It must be specified as hh:mm:ss (168 hours is the upper limit for Sapelo2 batch partition).

Moses at 18:39, 14 December 2022

2022-12-14T18:39:20Z

@@ Line 49: / Line 49: @@
   use_cluster 0
 . Create a job submission script, called sub.sh in the example here, with the sample content:
   #!/bin/bash
@@ Line 90: / Line 92: @@
   cluster_walltime 168:00:00
   cluster_queue batch
 . Create a job submission script, called sub.sh in the example here, with the sample content:

Moses at 17:56, 14 December 2022

2022-12-14T17:56:29Z

← Older revision		Revision as of 13:56, 14 December 2022
Line 83:		Line 83:
	num_prefix_blocks 4		num_prefix_blocks 4
	min_depth_cutoff 3		min_depth_cutoff 3
	~~bubble_depth_threshold 1~~

	use_cluster 1		use_cluster 1

Moses: /* Running Program */

2022-12-14T17:54:33Z

Running Program

← Older revision		Revision as of 13:54, 14 December 2022
Line 26:		Line 26:
	*Version 2.2.6, installed as Conda virtual environment in /apps/eb/Meraculous/2.2.6		*Version 2.2.6, installed as Conda virtual environment in /apps/eb/Meraculous/2.2.6

	To use this version of ~~magma~~, please first load the module with		To use this version of Meraculous, please first load the module with
	<pre class="gscript">		<pre class="gscript">
	module load Meraculous/2.2.6		module load Meraculous/2.2.6

Moses at 17:51, 14 December 2022

2022-12-14T17:51:58Z

← Older revision		Revision as of 13:51, 14 December 2022
Line 83:		Line 83:
	num_prefix_blocks 4		num_prefix_blocks 4
	min_depth_cutoff 3		min_depth_cutoff 3
			bubble_depth_threshold 1

	use_cluster 0		use_cluster 1
			cluster_num_nodes 10
			cluster_slots_per_task 8
			cluster_ram_request 128
			cluster_walltime 168:00:00
			cluster_queue batch

	2. Create a job submission script, called sub.sh in the example here, with the sample content:		2. Create a job submission script, called sub.sh in the example here, with the sample content:
	#!/bin/bash		#!/bin/bash
	#SBATCH --job-name=~~meraculoue_standalone~~		#SBATCH --job-name=meraculoue_cluster
	#SBATCH --partition=batch		#SBATCH --partition=batch
	#SBATCH --nodes=1		#SBATCH --nodes=1
	#SBATCH --ntasks=1		#SBATCH --ntasks=1
	#SBATCH --cpus-per-task=16		#SBATCH --ntasks-per-node=1
			#SBATCH --cpus-per-task=4
	#SBATCH --mem=128gb		#SBATCH --mem=128gb
	#SBATCH --time=~~7-00~~:00:00		#SBATCH --time=168:00:00
	#SBATCH --output=log.%j.out		#SBATCH --output=log.%j.out
	#SBATCH --error=log.%j.err		#SBATCH --error=log.%j.err
Line 101:		Line 109:
	ml Meraculous/2.2.6		ml Meraculous/2.2.6

	run_meraculous.sh -c meraculous.~~standalone~~.config -dir output -cleanup_level 1		export SLURM_ROOT=/opt/apps/slurm/21.08.8

			run_meraculous.sh -c meraculous.cluster.config -dir output -cleanup_level 1

	The parameters of the job, such as the maximum wall clock time (--time), maximum memory (--mem), CPU cores (--cpus-per-task), and the job name (--job-name) need to be modified appropriately. In this example, the standard output and error of the run_meraculous.sh command will be saved into two files called log.JobID.out and log.JobID.err, respectively, where jobID is the jobid number.		The parameters of the job, such as the maximum wall clock time (--time), maximum memory (--mem), CPU cores (--cpus-per-task), and the job name (--job-name) need to be modified appropriately. In this example, the standard output and error of the run_meraculous.sh command will be saved into two files called log.JobID.out and log.JobID.err, respectively, where jobID is the jobid number.

Line 111:		Line 122:

	=== Documentation ===		=== Documentation ===
	~~Tutorials and user~~ guide ~~are~~ available at ~~http:~~//~~magma~~.~~maths~~.~~usyd.edu.au~~/~~magma~~/~~documentation~~/		User guide Manual.pdf is available at /apps/eb/Meraculous/2.2.6/share/doc/meraculous/Manual.pdf . Please feel free to download it to your local computer to browse.

	=== Installation ===		=== Installation ===
	*~~Binaries for Intel and AMD processors with AVX support downloaded from http~~://~~magma~~.~~maths.usyd.edu.au~~/~~magma~~/		*https://anaconda.org/bioconda/meraculous

	=== System ===		=== System ===
	64-bit Linux		64-bit Linux

Moses at 17:42, 14 December 2022

2022-12-14T17:42:33Z

@@ Line 49: / Line 49: @@
   use_cluster 0
-. Create a job submission script, called sub.sh in the example here, with the sample content
+. Create a job submission script, called sub.sh in the example here, with the sample content:
   #!/bin/bash
   #SBATCH --job-name=meraculoue_standalone
@@ Line 66: / Line 66: @@
   run_meraculous.sh -c meraculous.standalone.config -dir output -cleanup_level 1
-The parameters of the job, such as the maximum wall clock time (--time), maximum memory (--mem), CPU cores (--cpus-per-task), and the job name (--job-name) need to be modified appropriately. In this example, the standard output and error of the run_meraculous.sh command will be saved into two files called log.JobID.out and log.JobID}.err, respectively, where jobID is the jobid number.
+The parameters of the job, such as the maximum wall clock time (--time), maximum memory (--mem), CPU cores (--cpus-per-task), and the job name (--job-name) need to be modified appropriately. In this example, the standard output and error of the run_meraculous.sh command will be saved into two files called log.JobID.out and log.JobID.err, respectively, where jobID is the jobid number.

Moses at 17:39, 14 December 2022

2022-12-14T17:39:29Z

← Older revision		Revision as of 13:39, 14 December 2022
Line 66:		Line 66:

	run_meraculous.sh -c meraculous.standalone.config -dir output -cleanup_level 1		run_meraculous.sh -c meraculous.standalone.config -dir output -cleanup_level 1
	~~where the~~ parameters of the job, such as the maximum wall clock time (--time), maximum memory (--mem), CPU cores (--cpus-per-task), and the job name (--job-name) need to be modified appropriately. In this example, the standard output and error of the run_meraculous.sh command will be saved into two files called log.JobID.out and log.JobID}.err, respectively, where jobID is the jobid number.		The parameters of the job, such as the maximum wall clock time (--time), maximum memory (--mem), CPU cores (--cpus-per-task), and the job name (--job-name) need to be modified appropriately. In this example, the standard output and error of the run_meraculous.sh command will be saved into two files called log.JobID.out and log.JobID}.err, respectively, where jobID is the jobid number.