EGAPx-Sapelo2 - Revision history

Shtsai at 00:32, 5 April 2026

2026-04-05T00:32:16Z

← Older revision		Revision as of 20:32, 4 April 2026
Line 1:		Line 1:
	[[Category:Sapelo2]][[Category:Software]][[Category:Bioinformatics]]		[[Category:Sapelo2]][[Category:Software]][[Category:Bioinformatics]]

	=== Category ===		== Category ==
	Bioinformatics		Bioinformatics

	=== Program On ===		== Program On ==
	N/A		N/A

	=== Author / Distributor ===		== Author / Distributor ==
	Please see https://github.com/ncbi/egapx		Please see https://github.com/ncbi/egapx

	=== Description ===		== Description ==
	From https://github.com/ncbi/egapx: "EGAPx is the publicly accessible version of the updated NCBI Eukaryotic Genome Annotation Pipeline. "		From https://github.com/ncbi/egapx: "EGAPx is the publicly accessible version of the updated NCBI Eukaryotic Genome Annotation Pipeline. "

	=== Running Program ===		== Running Program ==
	Also refer to [[Running Jobs on Sapelo2]]		Also refer to [[Running Jobs on Sapelo2]]

Line 115:		Line 115:
	(Please see https://github.com/ncbi/egapx?tab=readme-ov-file#input-data-format for more information on the correct formatting for input data, and https://github.com/ncbi/egapx?tab=readme-ov-file#input-example for more examples.)		(Please see https://github.com/ncbi/egapx?tab=readme-ov-file#input-data-format for more information on the correct formatting for input data, and https://github.com/ncbi/egapx?tab=readme-ov-file#input-example for more examples.)

	=== Documentation ===		== Documentation ==

	Details and references are at: https://github.com/ncbi/egapx		Details and references are at: https://github.com/ncbi/egapx

Jordan: /* Running Program */

2026-01-27T19:20:12Z

Running Program

← Older revision		Revision as of 15:20, 27 January 2026
Line 27:		Line 27:
	<pre class="gcommand">		<pre class="gcommand">
	ml Nextflow/24.10.2		ml Nextflow/24.10.2
	ml Python/3.11.3-GCCcore-12.3.0		ml Python/3.12.3-GCCcore-13.3.0
	ml PyYAML/6.0-GCCcore-12.3.0		ml PyYAML/6.0.2-GCCcore-13.3.0
	</pre>		</pre>
	~~(Note~~: ~~while you can almost certainly use other versions of Python and PyYAML, as recent~~ as EGAPx v0.5.0, the developers have been highly inconsistent in their use of version 1 <i>vs</i>. version 2 of Nextflow's domain-specific language, or "DSL," and Nextflow v25 does not tolerate these inconsistencies; <b>therefore, you <i>have to</i> use Nextflow/24.10.2 in order for the pipeline to run.</b>)		Notes:
			* As recently as EGAPx v0.5.0, the developers have been highly inconsistent in their use of version 1 <i>vs</i>. version 2 of Nextflow's domain-specific language, or "DSL," and Nextflow v25 does not tolerate these inconsistencies; <b>therefore, you <i>have to</i> use Nextflow/24.10.2 in order for the pipeline to run.</b>
			* As of EGAPx v0.5.0, it appears that the software depends on having at least v13.3.0 of GCC/GCCcore available, so the versions of Python and PyYAML listed above are strongly recommended.
	</br>		</br>
	</br>		</br>

Jordan: Updated to reflect that Nextflow v24 is required because the EGAPx developers have not fully updated their scripts to version 2 of Nextflow's domain-specific language (DSL2) and Nextflow v25 doesn't tolerate DSL1 artifacts.

2026-01-09T18:36:53Z

Updated to reflect that Nextflow v24 is required because the EGAPx developers have not fully updated their scripts to version 2 of Nextflow's domain-specific language (DSL2) and Nextflow v25 doesn't tolerate DSL1 artifacts.

← Older revision		Revision as of 14:36, 9 January 2026
Line 26:		Line 26:
	Since the dependencies for EGAPx are already installed, you don't need to create a new virtual environment; simply load the relevant software modules:		Since the dependencies for EGAPx are already installed, you don't need to create a new virtual environment; simply load the relevant software modules:
	<pre class="gcommand">		<pre class="gcommand">
	ml Nextflow/23.10.1		ml Nextflow/24.10.2
	ml Python/3.11.3-GCCcore-12.3.0		ml Python/3.11.3-GCCcore-12.3.0
	ml PyYAML/6.0-GCCcore-12.3.0		ml PyYAML/6.0-GCCcore-12.3.0
	</pre>		</pre>
			(Note: while you can almost certainly use other versions of Python and PyYAML, as recent as EGAPx v0.5.0, the developers have been highly inconsistent in their use of version 1 <i>vs</i>. version 2 of Nextflow's domain-specific language, or "DSL," and Nextflow v25 does not tolerate these inconsistencies; <b>therefore, you <i>have to</i> use Nextflow/24.10.2 in order for the pipeline to run.</b>)
			</br>
			</br>
	Now you need to run one of the test examples provided by the developers in order to acquire the necessary config files:		Now you need to run one of the test examples provided by the developers in order to acquire the necessary config files:
	<pre class="gcommand">		<pre class="gcommand">
Line 87:		Line 89:
	<nowiki>#</nowiki>SBATCH --error=%x.%j.err<br>		<nowiki>#</nowiki>SBATCH --error=%x.%j.err<br>

	ml purge		</br>ml purge
	ml Nextflow/23.10.1		</br>ml Nextflow/24.10.2
	ml Python/3.11.3-GCCcore-12.3.0		</br>ml Python/3.11.3-GCCcore-12.3.0
	ml PyYAML/6.0-GCCcore-12.3.0		</br>ml PyYAML/6.0-GCCcore-12.3.0

	python3 <u>PATH/to/</u>egapx/ui/egapx.py <u>PATH/to/input_and_database_options.yaml</u> -e slurm -o <u>PATH/to/Output_Directory</u>		</br>python3 <u>PATH/to/</u>egapx/ui/egapx.py <u>PATH/to/input_and_database_options.yaml</u> -e slurm -o <u>PATH/to/Output_Directory</u>

	</div>		</div>

Jordan: /* Running Program */

2026-01-06T20:29:53Z

Running Program

← Older revision		Revision as of 16:29, 6 January 2026
Line 16:		Line 16:
	Also refer to [[Running Jobs on Sapelo2]]		Also refer to [[Running Jobs on Sapelo2]]

	As of 05/20/~~2025~~, due to its current implementation of Nextflow, installing EGAPx centrally would potentially run the risk of overwriting important files; it currently also depends on the NCBI C++ toolkit, which has documented security vulnerabilities. As such, this page exists to outline the steps needed to run EGAPx from the /home directory.		As of 01/06/2026, due to its current implementation of Nextflow, installing EGAPx centrally would potentially run the risk of overwriting important files; it currently also depends on the NCBI C++ toolkit, which has documented security vulnerabilities. As such, this page exists to outline the steps needed to run EGAPx from the /home directory.

	First, <u><b>start an interactive job</b></u> and, in your /home directory, clone the repository and then cd to the new directory that this creates:		First, <u><b>start an interactive job</b></u> and, in your /home directory, clone the repository and then cd to the new directory that this creates:

Jordan: Updated the reasons why we can't install this centrally on Sapelo2.

2026-01-06T20:27:31Z

Updated the reasons why we can't install this centrally on Sapelo2.

← Older revision		Revision as of 16:27, 6 January 2026
Line 6:		Line 6:
	=== Program On ===		=== Program On ===
	N/A		N/A

	~~=== Version ===~~
	~~0.3.2-alpha~~

	=== Author / Distributor ===		=== Author / Distributor ===
Line 19:		Line 16:
	Also refer to [[Running Jobs on Sapelo2]]		Also refer to [[Running Jobs on Sapelo2]]

	As of 05/20/2025, due to its current implementation of Nextflow, installing EGAPx centrally would potentially run the risk of overwriting important files ~~(this might change in~~ the ~~future when the software is out of alpha/beta)~~. As such, this page exists to outline the steps needed to run EGAPx from the /home directory.		As of 05/20/2025, due to its current implementation of Nextflow, installing EGAPx centrally would potentially run the risk of overwriting important files; it currently also depends on the NCBI C++ toolkit, which has documented security vulnerabilities. As such, this page exists to outline the steps needed to run EGAPx from the /home directory.

	First, <u><b>start an interactive job</b></u> and, in your /home directory, clone the repository and then cd to the new directory that this creates:		First, <u><b>start an interactive job</b></u> and, in your /home directory, clone the repository and then cd to the new directory that this creates:

Jordan: Initial creation of this page

2025-05-20T19:14:42Z

Initial creation of this page

New page

[[Category:Sapelo2]][[Category:Software]][[Category:Bioinformatics]]

=== Category ===
Bioinformatics

=== Program On ===
N/A

=== Version ===
0.3.2-alpha

=== Author / Distributor ===
Please see https://github.com/ncbi/egapx

=== Description ===
From https://github.com/ncbi/egapx: "EGAPx is the publicly accessible version of the updated NCBI Eukaryotic Genome Annotation Pipeline. "

=== Running Program ===
Also refer to [[Running Jobs on Sapelo2]]

As of 05/20/2025, due to its current implementation of Nextflow, installing EGAPx centrally would potentially run the risk of overwriting important files (this might change in the future when the software is out of alpha/beta). As such, this page exists to outline the steps needed to run EGAPx from the /home directory.

First, start an interactive job and, in your /home directory, clone the repository and then cd to the new directory that this creates:
<pre class="gcommand">
git clone https://github.com/ncbi/egapx.git
cd egapx
</pre>

Since the dependencies for EGAPx are already installed, you don't need to create a new virtual environment; simply load the relevant software modules:
<pre class="gcommand">
ml Nextflow/23.10.1
ml Python/3.11.3-GCCcore-12.3.0
ml PyYAML/6.0-GCCcore-12.3.0
</pre>

Now you need to run one of the test examples provided by the developers in order to acquire the necessary config files:
<pre class="gcommand">
python3 ui/egapx.py ./examples/input_D_farinae_small.yaml -o example_out
</pre>

Note that it isn't strictly necessary to include "python3" at the start of the command, so long as the 'egapx.py' script in the 'ui/' directory has executable permissions (which it does by default). Running the execution script the first time will put all of the config files into a new subdirectory in your working directory ('egapx/') called 'egapx_config'. Of these, the one to use is 'slurm.config', which you will need to open with your preferred text editor and make the following changes:

The line
<pre class="gscript">
cacheDir = "/data/$USER/singularity"
</pre>
should be changed to
<pre class="gscript">
cacheDir = "$PWD/singularity"
</pre>

while the section
<pre class="gscript">
env {
SINGULARITY_CACHEDIR="/data/$USER/singularity"
SINGULARITY_TMPDIR="/data/$USER/tmp"
}
</pre>

should be changed to:
<pre class="gscript">
env {
SINGULARITY_CACHEDIR="$PWD/singularity"
SINGULARITY_TMPDIR="/$PWD/tmp"
}
</pre>

Finally, in the section called "process," there are only three things that need to be un-commented (remove the "//" at the start of the line) and changed as follows:
<pre class="gscript">
queue = 'batch'
queueSize = 20
</pre>
and
<pre class="gscript">
clusterOptions = ' --ntasks=1 '
</pre>

After this, you can create a job submission script that looks like this:

<div class="gscript2">
<nowiki>#</nowiki>!/bin/bash 
<nowiki>#</nowiki>SBATCH --partition=batch 
<nowiki>#</nowiki>SBATCH --job-name=ExampleJobName 
<nowiki>#</nowiki>SBATCH --ntasks=1 
<nowiki>#</nowiki>SBATCH --cpus-per-task=1 
<nowiki>#</nowiki>SBATCH --mem=4G 
<nowiki>#</nowiki>SBATCH --time=48:00:00 
<nowiki>#</nowiki>SBATCH --output=%x.%j.out 
<nowiki>#</nowiki>SBATCH --error=%x.%j.err 

ml purge
ml Nextflow/23.10.1
ml Python/3.11.3-GCCcore-12.3.0
ml PyYAML/6.0-GCCcore-12.3.0

python3 PATH/to/egapx/ui/egapx.py PATH/to/input_and_database_options.yaml -e slurm -o PATH/to/Output_Directory

</div>
In the real submission script, all the above underlined values need to be replaced by the actual values, particularly the relevant paths. Please note that, while it is perfectly fine to use EGAPx in your /home directory, the input and output directories (i.e., where files will be read from and written to) should NOT be located in your /home directory; please use somewhere else for input (if using local data) and output files, such your /scratch directory.
 
 
Example input YAML file:
<pre class="gcommand">
genome: https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/020/809/275/GCF_020809275.1_ASM2080927v1/GCF_020809275.1_ASM2080927v1_genomic.fna.gz
taxid: 6954
reads:
- https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR8506572.1
- https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR8506572.2
- https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR9005248.1
- https://ftp.ncbi.nlm.nih.gov/genomes/TOOLS/EGAP/data/Dermatophagoides_farinae_small/SRR9005248.2
</pre>

(Please see https://github.com/ncbi/egapx?tab=readme-ov-file#input-data-format for more information on the correct formatting for input data, and https://github.com/ncbi/egapx?tab=readme-ov-file#input-example for more examples.)

=== Documentation ===

Details and references are at: https://github.com/ncbi/egapx

Full help options
<pre class="gcommand">
$ ui/egapx.py -h
usage: egapx.py [-h] [-o OUTPUT] [-e EXECUTOR] [-c CONFIG_DIR] [-w WORKDIR] [-r REPORT] [-n] [-st] [-so] [-lo] [-ot ORTHO_TAXID] [-dl] [-lc LOCAL_CACHE] [-q] [-v] [-V] [-fn FUNC_NAME] [filename]

Main script for EGAPx

options:
-h, --help show this help message and exit
-e EXECUTOR, --executor EXECUTOR
Nextflow executor, one of docker, singularity, aws, or local (for NCBI internal use only). Uses corresponding Nextflow config file
-c CONFIG_DIR, --config-dir CONFIG_DIR
Directory for executor config files, default is ./egapx_config. Can be also set as env EGAPX_CONFIG_DIR
-w WORKDIR, --workdir WORKDIR
Working directory for cloud executor
-r REPORT, --report REPORT
Report file prefix for report (.report.html) and timeline (.timeline.html) files, default is in output directory
-n, --dry-run
-st, --stub-run
-so, --summary-only Print result statistics only if available, do not compute result
-lo, --logs-only Collect execution logs if available, put them in output directory, do not compute result
-ot ORTHO_TAXID, --ortho-taxid ORTHO_TAXID
Taxid of reference data for orthology tasks
-lc LOCAL_CACHE, --local-cache LOCAL_CACHE
Where to store the downloaded files
-q, --quiet
-v, --verbose
-V, --version Report software version
-fn FUNC_NAME, --func_name FUNC_NAME
func_name

run:
filename YAML file with input: section with at least genome: and reads: parameters
-o OUTPUT, --output OUTPUT
Output path

download:
-dl, --download-only Download external files to local storage, so that future runs can be isolated
</pre>

[[#top|Back to Top]]