Trinity-HpcGridRunner - Revision history

Shtsai at 19:28, 11 May 2026

2026-05-11T19:28:26Z

← Older revision		Revision as of 15:28, 11 May 2026
Line 6:		Line 6:

	* Note: normal Trinity jobs should be run utilizing /lscratch, however, this method running with HpcGridRunner does not benefit from /lscratch so this example will not have any /lscratch components in it. For more information on running normal Trinity jobs with /lscratch, please see [[Trinity-Sapelo2#Utilizing /lscratch in Trinity Job Submission Script\|Utilizing /lscratch in Trinity Job Submission Script]].		* Note: normal Trinity jobs should be run utilizing /lscratch, however, this method running with HpcGridRunner does not benefit from /lscratch so this example will not have any /lscratch components in it. For more information on running normal Trinity jobs with /lscratch, please see [[Trinity-Sapelo2#Utilizing /lscratch in Trinity Job Submission Script\|Utilizing /lscratch in Trinity Job Submission Script]].
	* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity ~~v2.15.1 Software Module on Sapelo2\|here~~]].		* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Running Program\|Running Trinity Jobs]].

	Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity command (see how to format --grid exec below)		Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity command (see how to format --grid exec below)

Shtsai at 19:27, 11 May 2026

2026-05-11T19:27:06Z

← Older revision		Revision as of 15:27, 11 May 2026
Line 5:		Line 5:
	Step 1: Create normal Trinity script		Step 1: Create normal Trinity script

	* Note: normal Trinity jobs should be run utilizing /lscratch, however, this method running with HpcGridRunner does not benefit from /lscratch so this example will not have any /lscratch components in it. For more information on running normal Trinity jobs with /lscratch, please see [[Trinity-Sapelo2#Utilizing /lscratch in Trinity Job Submission Script\|~~here~~]].		* Note: normal Trinity jobs should be run utilizing /lscratch, however, this method running with HpcGridRunner does not benefit from /lscratch so this example will not have any /lscratch components in it. For more information on running normal Trinity jobs with /lscratch, please see [[Trinity-Sapelo2#Utilizing /lscratch in Trinity Job Submission Script\|Utilizing /lscratch in Trinity Job Submission Script]].
	* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity v2.15.1 Software Module on Sapelo2\|here]].		* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity v2.15.1 Software Module on Sapelo2\|here]].

Shtsai at 18:59, 11 May 2026

2026-05-11T18:59:35Z

← Older revision		Revision as of 14:59, 11 May 2026
Line 1:		Line 1:

	===Description===		==Description==
	Instruction on how to modify a Trinity script to run in conjunction with HpcGridRunner		Instruction on how to modify a Trinity script to run in conjunction with HpcGridRunner
	===Running Program===		==Running Program==
	Step 1: Create normal Trinity script		Step 1: Create normal Trinity script

Line 60:		Line 60:


	===Other Resources===		==Other Resources==

	https://github.com/trinityrnaseq/trinityrnaseq/wiki/Running-Trinity#optional-adapting-trinity-to-a-computing-grid-for-massively-parallel-processing-of-embarrassingly-parallel-steps		https://github.com/trinityrnaseq/trinityrnaseq/wiki/Running-Trinity#optional-adapting-trinity-to-a-computing-grid-for-massively-parallel-processing-of-embarrassingly-parallel-steps

Chelsea: /* Running Program */

2026-04-23T16:25:55Z

Running Program

← Older revision		Revision as of 12:25, 23 April 2026
Line 42:		Line 42:

	# template for a grid submission:		# template for a grid submission:
	cmd=sbatch -p batch --mem 80gb -n 1 --cpus-per-task 24 -t 01:00:00		cmd=sbatch -p batch --mem 80gb -n 1 --cpus-per-task 8 -t 01:00:00

	##########################################################################################		##########################################################################################
Line 51:		Line 51:
	max_nodes=5		max_nodes=5

	# number of commands that are batched into a single grid submission job.		# number of commands that are batched into a single grid submission job. These run sequentially, not concurrently, so you only need to request enough cpus-per-task for the threads of a single command
	cmds_per_node=3		cmds_per_node=3
	</pre>		</pre>

Chelsea: /* Running Program */

2026-04-21T13:47:35Z

Running Program

← Older revision		Revision as of 09:47, 21 April 2026
Line 42:		Line 42:

	# template for a grid submission:		# template for a grid submission:
	cmd=sbatch -p batch --mem 80gb -n 1 -t 01:00:00		cmd=sbatch -p batch --mem 80gb -n 1 --cpus-per-task 24 -t 01:00:00

	##########################################################################################		##########################################################################################

Chelsea: /* Running Program */

2024-09-16T11:49:34Z

Running Program

← Older revision		Revision as of 07:49, 16 September 2024
Line 10:		Line 10:
	Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity command (see how to format --grid exec below)		Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity command (see how to format --grid exec below)

	* Your grid_exec flag should look exactly like the one ~~above~~, where the only part you'll change is the '''location to your config.conf''' file after --grid_conf. Note the location of the quotation marks as they are necessary.		* Your grid_exec flag should look exactly like the one below, where the only part you'll change is the '''location to your config.conf''' file after --grid_conf. Note the location of the quotation marks as they are necessary.
	<pre class="gscript">		<pre class="gscript">
	#!/bin/bash		#!/bin/bash

Chelsea: /* Running Program */

2024-09-16T11:49:13Z

Running Program

← Older revision		Revision as of 07:49, 16 September 2024
Line 8:		Line 8:
	* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity v2.15.1 Software Module on Sapelo2\|here]].		* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity v2.15.1 Software Module on Sapelo2\|here]].

	Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity ~~comma~~<pre class="gscript">		Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity command (see how to format --grid exec below)

			* Your grid_exec flag should look exactly like the one above, where the only part you'll change is the '''location to your config.conf''' file after --grid_conf. Note the location of the quotation marks as they are necessary.
			<pre class="gscript">
	#!/bin/bash		#!/bin/bash
	#SBATCH --job-name=Trinity_HpcGridRunner		#SBATCH --job-name=Trinity_HpcGridRunner
Line 34:		Line 37:

	</pre>		</pre>

	* Your grid_exec flag should look exactly like the one above, where the only part you'll change is the '''location to your config.conf''' file after --grid_conf. Note the location of the quotation marks as they are necessary.
	Step 3: create config.conf file<pre class="gscript">		Step 3: create config.conf file<pre class="gscript">
	# grid type:		# grid type:

Chelsea at 19:35, 1 August 2024

2024-08-01T19:35:52Z

← Older revision		Revision as of 15:35, 1 August 2024
Line 59:		Line 59:


	Running ~~Program~~		===Other Resources===

			https://github.com/trinityrnaseq/trinityrnaseq/wiki/Running-Trinity#optional-adapting-trinity-to-a-computing-grid-for-massively-parallel-processing-of-embarrassingly-parallel-steps

Chelsea at 19:34, 1 August 2024

2024-08-01T19:34:52Z

← Older revision		Revision as of 15:34, 1 August 2024
Line 25:		Line 25:

	Trinity --seqType <string> --max_memory 100G \		Trinity --seqType <string> --max_memory 100G \
	--CPU 1 \		--CPU 8 \
	--left reads.left.fq.gz \		--left reads.left.fq.gz \
	--right reads.right.fq.gz \		--right reads.right.fq.gz \
Line 37:		Line 37:
	* Your grid_exec flag should look exactly like the one above, where the only part you'll change is the '''location to your config.conf''' file after --grid_conf. Note the location of the quotation marks as they are necessary.		* Your grid_exec flag should look exactly like the one above, where the only part you'll change is the '''location to your config.conf''' file after --grid_conf. Note the location of the quotation marks as they are necessary.
	Step 3: create config.conf file<pre class="gscript">		Step 3: create config.conf file<pre class="gscript">
	To be updated after maintenance		# grid type:
			grid=SLURM

			# template for a grid submission:
			cmd=sbatch -p batch --mem 80gb -n 1 -t 01:00:00

			##########################################################################################
			# settings below configure the Trinity job submission system, not tied to the grid itself.
			##########################################################################################

			# number of grid submissions to be maintained at steady state by the Trinity submission system
			max_nodes=5

			# number of commands that are batched into a single grid submission job.
			cmds_per_node=3
	</pre>		</pre>

	* The main things to change in your config.conf file are the max_nodes and the number of ~~jobs~~, which together determine how the grid will run on the cluster.		* The main things to change in your config.conf file are the max_nodes (how many individual grid jobs will be allowed to run at the same time) and the cmds_per_node (number of commands per grid job), which together determine how the grid will run on the cluster. You can also alter the cmd, which is the command that will be used to submit the individual grid jobs.
			* The number of grid jobs that will be submitted depends on how many recursive Trinity commands you have and what variables you choose for max_nodes and cmds_per_node.
			** Increasing the number of cmds_per_node and max_nodes will lower the total time it takes for all of the recursive Trinity commands to finish running, however it is important to note that though you may put a large number for max_nodes, the amount actually used will be determined by SLURM and ultimately depends on what resources are currently available and how many jobs you already have running. Similarly, setting a large number for the variable cmds_per_node can make the overall job complete more quickly, but then those individual jobs (submitted by the cmd variable) may need more memory.


			Running Program

Chelsea: /* Running Program */

2024-07-31T18:08:08Z

Running Program

← Older revision		Revision as of 14:08, 31 July 2024
Line 8:		Line 8:
	* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity v2.15.1 Software Module on Sapelo2\|here]].		* For more information on creating a Trinity job (without utilizing /lscratch) please see [[Trinity-Sapelo2#Trinity v2.15.1 Software Module on Sapelo2\|here]].

	Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity ~~command~~<pre class="gscript">		Step 2: add a line to load HpcGridRunner module AND add the --grid_exec flag in your Trinity comma<pre class="gscript">
	#!/bin/bash		#!/bin/bash
	#SBATCH --job-name=Trinity_HpcGridRunner		#SBATCH --job-name=Trinity_HpcGridRunner