Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fmalmeida/ngs-preprocess
A pipeline for preprocessing NGS data from Illumina, Nanopore and PacBio technologies
https://github.com/fmalmeida/ngs-preprocess
bax2bam bioinformatics illumina nanopack nextflow ngs ngs-preprocess pacbio pacbio-ccs pipeline porechop reproducible-research reproducible-science trimgalore workflow
Last synced: 3 months ago
JSON representation
A pipeline for preprocessing NGS data from Illumina, Nanopore and PacBio technologies
- Host: GitHub
- URL: https://github.com/fmalmeida/ngs-preprocess
- Owner: fmalmeida
- License: gpl-3.0
- Created: 2019-09-19T19:01:22.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2024-06-30T13:46:44.000Z (4 months ago)
- Last Synced: 2024-07-03T01:45:22.691Z (4 months ago)
- Topics: bax2bam, bioinformatics, illumina, nanopack, nextflow, ngs, ngs-preprocess, pacbio, pacbio-ccs, pipeline, porechop, reproducible-research, reproducible-science, trimgalore, workflow
- Language: Nextflow
- Homepage: https://ngs-preprocess.readthedocs.io/
- Size: 5.27 MB
- Stars: 30
- Watchers: 4
- Forks: 4
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
- Awesome-Bioinformatics - ngs-preprocess - A pipeline for preprocessing short and long sequencing reads, built with Nextflow. [ [web](https://ngs-preprocess.readthedocs.io/en/latest/?badge=latest) ] (Next Generation Sequencing / Pipelines)
README
[](https://doi.org/10.12688/f1000research.139488.1)
[](https://github.com/fmalmeida/ngs-preprocess/releases)
[](https://ngs-preprocess.readthedocs.io/en/latest/?badge=latest)
[](https://hub.docker.com/r/fmalmeida/ngs-preprocess)
[](https://www.nextflow.io/)
[](https://docs.conda.io/en/latest/)
[](https://www.docker.com/)
[](https://sylabs.io/docs/)
[](https://twitter.com/fmarquesalmeida)
[](https://github.com/fmalmeida/ngs-preprocess/blob/master/LICENSE)
[](https://doi.org/10.5281/zenodo.3451405)
ngs-preprocess pipeline
A pipeline for preprocessing short and long sequencing reads
See the documentation »
Report Bug
·
Request Feature
## About
ngs-preprocess is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. It is an easy to use pipeline that uses state-of-the-art software for quality check and pre-processing ngs reads of Illumina, Pacbio and Oxford Nanopore Technologies.
It wraps up the following software:
| Step | tools |
| :--- | :---- |
| SRA NBCI fetch | [Entrez-direct](https://anaconda.org/bioconda/entrez-direct) & [sra-tools](https://github.com/ncbi/sra-tools) |
| Illumina pre-processing | [Fastp](https://github.com/OpenGene/fastp) |
| Nanopore pre-processing | [Porechop](https://github.com/rrwick/Porechop), [Porechop ABI](https://github.com/bonsai-team/Porechop_ABI), [pycoQC](https://github.com/tleonardi/pycoQC), [NanoPack](https://github.com/wdecoster/nanopack) |
| Pacbio pre-processing | [bam2fastx](https://github.com/PacificBiosciences/pbtk#bam2fastx), [bax2bam](https://anaconda.org/bioconda/bax2bam), [lima](https://github.com/PacificBiosciences/barcoding), [pacbio ccs](https://ccs.how/) |
## Further reading
This pipeline has two complementary pipelines (also written in nextflow) for [genome assembly](https://github.com/fmalmeida/mpgap) and [prokaryotic genome annotation](https://github.com/fmalmeida/bacannot) that can give the user a complete workflow for bacterial genomics analyses.
## Quickstart
1. Install Nextflow:
```bash
curl -s https://get.nextflow.io | bash
```
2. Give it a try:
```bash
nextflow run fmalmeida/ngs-preprocess --help
```
3. Download required tools
* for docker
```bash
# for docker
docker pull fmalmeida/ngs-preprocess:v2.7
# run
nextflow run fmalmeida/ngs-preprocess -profile docker [options]
```
* for singularity
```bash
# for singularity
# remember to properly set NXF_SINGULARITY_LIBRARYDIR
# read more at https://www.nextflow.io/docs/latest/singularity.html#singularity-docker-hub
export NXF_SINGULARITY_LIBRARYDIR=MY_SINGULARITY_IMAGES # your singularity storage dir
export NXF_SINGULARITY_CACHEDIR=MY_SINGULARITY_CACHE # your singularity cache dir
singularity pull \
--dir $NXF_SINGULARITY_LIBRARYDIR \
fmalmeida-ngs-preprocess-v2.7.img docker://fmalmeida/ngs-preprocess:v2.7
# run
nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
```
* for conda
```bash
# for conda
# it is better to create envs with mamba for faster solving
wget https://github.com/fmalmeida/ngs-preprocess/raw/master/environment.yml
conda env create -f environment.yml # advice: use mamba
# must be executed from the base environment
# This tells nextflow to load the available ngs-preprocess environment when required
nextflow run fmalmeida/ngs-preprocess -profile conda [options]
```
4. Start running your analysis
```bash
nextflow run fmalmeida/ngs-preprocess -profile
```
:fire: Please read the documentation below on [selecting between conda, docker or singularity](https://github.com/fmalmeida/ngs-preprocess/tree/master#selecting-between-profiles) profiles, since the tools will be made available differently depending on the profile desired.
## Documentation
### Selecting between profiles
Nextflow profiles are a set of "sensible defaults" for the resource requirements of each of the steps in the workflow, that can be enabled with the command line flag `-profile`. You can learn more about nextflow profiles at:
+ https://nf-co.re/usage/configuration#basic-configuration-profiles
+ https://www.nextflow.io/docs/latest/config.html#config-profiles
The pipeline have "standard profiles" set to run the workflows with either conda, docker or singularity using the [local executor](https://www.nextflow.io/docs/latest/executor.html), which is nextflow's default and basically runs the pipeline processes in the computer where Nextflow is launched. If you need to run the pipeline using another executor such as sge, lsf, slurm, etc. you can take a look at [nextflow's manual page](https://www.nextflow.io/docs/latest/executor.html) to proper configure one in a new custom profile set in your personal copy of [ngs-preprocess config file](https://github.com/fmalmeida/ngs-preprocess/blob/master/nextflow.config) and take advantage that nextflow allows multiple profiles to be used at once, e.g. `-profile conda,sge`.
By default, if no profile is chosen, the pipeline will try to load tools from the local machine $PATH. Available pre-set profiles for this pipeline are: `docker/conda/singularity`, you can choose between them as follows:
* conda
```bash
# must be executed from the base environment
# This tells nextflow to load the available ngs-preprocess environment when required
nextflow run fmalmeida/ngs-preprocess -profile conda [options]
```
* docker
```bash
nextflow run fmalmeida/ngs-preprocess -profile docker [options]
```
* singularity
```bash
nextflow run fmalmeida/ngs-preprocess -profile singularity [options]
```
:book: Please use conda as last resource since the packages will not be "frozen and pre-installed", problems may arise.
### Usage
For understading pipeline usage and configuration, users must read the complete online documentation »
### Using a configuration file
All the parameters showed above can be, and are advised to be, set through the configuration file. When a configuration file is set the pipeline is run by simply executing:
```bash
nextflow run fmalmeida/ngs-preprocess -c ./configuration-file
```
Your configuration file is what will tell to the pipeline the type of data you have, and which processes to execute. Therefore, it needs to be correctly set up.
Create a configuration file in your working directory:
```bash
nextflow run fmalmeida/ngs-preprocess [ --get_config ]
```
### Interactive graphical configuration and execution
#### Via NF tower launchpad (good for cloud env execution)
Nextflow has an awesome feature called [NF tower](https://tower.nf). It allows that users quickly customise and set-up the execution and configuration of cloud enviroments to execute any nextflow pipeline from nf-core, github (this one included), bitbucket, etc. By having a compliant JSON schema for pipeline configuration it means that the configuration of parameters in NF tower will be easier because the system will render an input form.
Checkout more about this feature at: https://seqera.io/blog/orgs-and-launchpad/
#### Via nf-core launch (good for local execution)
Users can trigger a graphical and interactive pipeline configuration and execution by using [nf-core launch](https://nf-co.re/launch) utility. nf-core launch will start an interactive form in your web browser or command line so you can configure the pipeline step by step and start the execution of the pipeline in the end.
```bash
# Install nf-core
pip install nf-core
# Launch the pipeline
nf-core launch fmalmeida/ngs-preprocess
```
It will result in the following:
# Citation
In order to cite this pipeline, please refer to:
> Almeida FMd, Campos TAd and Pappas Jr GJ. Scalable and versatile container-based pipelines for de novo genome assembly and bacterial annotation. F1000Research 2023, 12:1205 (https://doi.org/10.12688/f1000research.139488.1)
Additionally, archived versions of the pipeline are also found in [Zenodo](https://doi.org/10.5281/zenodo.3451405).
This pipeline uses code and infrastructure developed and maintained by the [nf-core](https://nf-co.re) community, reused here under the [GPLv3](https://github.com/fmalmeida/ngs-preprocess/blob/master/LICENSE).
> The nf-core framework for community-curated bioinformatics pipelines.
>
> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.
>
> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.
In addition, users are encouraged to cite the programs used in this pipeline whenever they are used. Links to resources of tools and data used in this pipeline are as follows:
* [Entrez-direct](https://anaconda.org/bioconda/entrez-direct)
* [sra-tools](https://github.com/ncbi/sra-tools)
* [Fastp](https://github.com/OpenGene/fastp)
* [Porechop](https://github.com/rrwick/Porechop)
* [Porechop ABI](https://github.com/bonsai-team/Porechop_ABI)
* [pycoQC](https://github.com/a-slide/pycoQC)
* [bax2bam](https://anaconda.org/bioconda/bax2bam)
* [bam2fastq](https://github.com/PacificBiosciences/pbtk#bam2fastx)
* [lima](https://github.com/PacificBiosciences/barcoding)
* [pacbio ccs](https://ccs.how/)
* [NanoPack](https://github.com/wdecoster/nanopack).