{"id":24527243,"url":"https://github.com/biobakery/kneaddata","last_synced_at":"2025-04-05T02:09:09.906Z","repository":{"id":38817494,"uuid":"253871273","full_name":"biobakery/kneaddata","owner":"biobakery","description":"Quality control tool on metagenomic and metatranscriptomic sequencing data, especially data from microbiome experiments.","archived":false,"fork":false,"pushed_at":"2025-03-20T16:47:29.000Z","size":1001,"stargazers_count":116,"open_issues_count":13,"forks_count":31,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-29T01:09:49.435Z","etag":null,"topics":["biobakery","bowtie2","kneaddata","public","quality-control","tools"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/biobakery.png","metadata":{"files":{"readme":"readme.md","changelog":"history.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-07T17:53:23.000Z","updated_at":"2025-03-27T01:13:54.000Z","dependencies_parsed_at":"2025-02-12T10:13:27.789Z","dependency_job_id":"31ded67b-b568-4dce-87bb-ab4a343a6dbc","html_url":"https://github.com/biobakery/kneaddata","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Fkneaddata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Fkneaddata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Fkneaddata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/biobakery%2Fkneaddata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/biobakery","download_url":"https://codeload.github.com/biobakery/kneaddata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276164,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["biobakery","bowtie2","kneaddata","public","quality-control","tools"],"created_at":"2025-01-22T06:17:43.126Z","updated_at":"2025-04-05T02:09:09.881Z","avatar_url":"https://github.com/biobakery.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ***ATTENTION***\r\n\r\nBefore opening a new issue here, please check the appropriate help channel on the [KneadData bioBakery Support Forum](https://forum.biobakery.org/c/infrastructure-and-utilities/kneaddata/8) and consider opening or commenting on a thread there.\r\n\r\nFor additional information, visit the [KneadData Tutorial](https://github.com/biobakery/biobakery/wiki/kneaddata).\r\n\r\n----\r\n# KneadData User Manual #\r\n\r\nKneadData is a tool designed to perform quality control on metagenomic and\r\nmetatranscriptomic sequencing data, especially data from microbiome experiments.\r\nIn these experiments, samples are typically taken from a host in hopes of\r\nlearning something about the microbial community on the host. However,\r\nsequencing data from such experiments will often contain a high ratio of host to\r\nbacterial reads. This tool aims to perform principled  *in silico* separation of\r\nbacterial reads from these \"contaminant\" reads, be they from the host, from\r\nbacterial 16S sequences, or other user-defined sources. Additionally, KneadData\r\ncan be used for other filtering tasks. For example, if one is trying to clean\r\ndata derived from a human sequencing experiment, KneadData can be used to\r\nseparate the human and the non-human reads.\r\n\r\n**If you use the KneadData software, please cite our manuscript: TBA**\r\n\r\n## Contents ##\r\n- [Requirements](#requirements)\r\n- [Installation](#installation)\r\n- [Create a Custom Database](#create-a-custom-database)\r\n- [How to Run](#how-to-run)\r\n    - [Single End Run](#single-end-run)\r\n    - [Paired End Run](#paired-end-run)\r\n    - [Demo Run](#demo-run)\r\n    - [Sequencer Source for trimming Adapter Contents](#sequencer-source-for-trimming-adapter-contents)\r\n    - [Trim Overrepresented/Repetitive sequences](#trim-overrepresentedrepetitive-sequences)\r\n    - [Additional Arguments](#additional-arguments)\r\n- [Complete Option List](#complete-option-list)\r\n\r\n## Requirements ##\r\n\r\n1.  [Trimmomatic](http://www.usadellab.org/cms/?page=trimmomatic) (version == 0.33) (automatically installed)\r\n2.  [Bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml) (version \u003e= 2.2) (automatically installed)\r\n3.  [Python](http://www.python.org/) (version \u003e= 2.7)\r\n4.  [Java Runtime Environment](http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html)\r\n5.  [TRF](https://tandem.bu.edu/trf/trf.html) (optional)\r\n6.  [Fastqc](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) (optional)\r\n7.  [SAMTools](https://github.com/samtools/samtools) (only required if input file is in BAM format)\r\n8.  Memory (\u003e= 4 Gb if using Bowtie2, \u003e= 8 Gb if using BMTagger)\r\n9.  Operating system (Linux or Mac)\r\n\r\nOptionally, [BMTagger](ftp://ftp.ncbi.nlm.nih.gov/pub/agarwala/bmtagger/) can be used instead of Bowtie2.\r\n\r\nThe executables for the required software packages should be installed in your $PATH. Alternatively, you can provide the location of the Bowtie2 install ($BOWTIE2_DIR) with the following KneadData option “--bowtie2 $BOWTIE2_DIR”.\r\n\r\n![kneaddata_workflow](https://huttenhower.sph.harvard.edu/wp-content/uploads/2021/11/kneaddata_workflow.drawio.png)\r\n\r\n## Installation ##\r\n\r\nBefore installing KneadData, please install the Java Runtime Environment (JRE). First [download](http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html) the JRE for your platform. Then follow the instructions for your platform: [Linux 64-bit](http://docs.oracle.com/javase/8/docs/technotes/guides/install/linux_jre.html#CFHIEGAA) or [Mac OS](http://docs.oracle.com/javase/8/docs/technotes/guides/install/mac_jre.html#jre_8u40_osx). At the end of the installation, add the location of the java executable to your $PATH.\r\n\r\n## Download KneadData ###\r\n\r\nYou can download the latest KneadData release or the development version. The source contains example files. If installing with pip, it is optional to first download the KneadData source.\r\n\r\nOption 1: Latest Release (Recommended)\r\n\r\n* Download [kneaddata.tar.gz](https://pypi.python.org/pypi/kneaddata) and unpack the latest release of KneadData.\r\n\r\nOption 2: Development Version\r\n\r\n* Create a clone of the repository:\r\n\r\n    `` $ git clone https://github.com/biobakery/kneaddata.git ``\r\n\r\n    Note: Creating a clone of the repository requires [Git](https://git-scm.com/) to be installed.\r\n\r\n## Install KneadData ###\r\n\r\n#### Install with pip ####\r\n\r\n* `` $ pip install kneaddata ``\r\n* This command will automatically install Trimmomatic and Bowtie2. To bypass the install of dependencies, add the option \"--install-option='--bypass-dependencies-install'\".\r\n* If you do not have write permissions to '/usr/lib/', then add the option \"--user\" to the install command. This will install the python package into subdirectories of '$HOME/.local'. Please note when using the \"--user\" install option on some platforms, you might need to add '$HOME/.local/bin/' to your $PATH as it might not be included by default. You will know if it needs to be added if you see the following message ``kneaddata: command not found`` when trying to run KneadData after installing with the \"--user\" option.\r\n\r\n#### Install from source ####\r\n\r\n1. Follow the instructions to download KneadData\r\n2. Move to the KneadData source directory: ``$ cd kneaddata``\r\n3. Install KneadData\r\n    * ``$ python setup.py install``\r\n    * This command will automatically install Trimmomatic and Bowtie2. To bypass the install of dependencies, add the option \"--bypass-dependencies-install\".\r\n    * If you do not have write permissions to '/usr/lib/', then add the option \"--user\" to the install command. This will install the python package into subdirectories of '$HOME/.local'. Please note when using the \"--user\" install option on some platforms, you might need to add '$HOME/.local/bin/' to your $PATH as it might not be included by default. You will know if it needs to be added if you see the following message ``kneaddata: command not found`` when trying to run KneadData after installing with the \"--user\" option.\r\n\r\n### Download the database ###\r\n\r\nIt is recommended that you download the human ([Homo_sapiens_hg39_T2T_Bowtie2_v0.1.tar.gz](https://huttenhower.sph.harvard.edu/kneadData_databases/Homo_sapiens_hg39_T2T_Bowtie2_v0.1.tar.gz) - [source](https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_009914755.1/) ) reference database (approx. size = 3.6 GB). However, this step is not required if you are using your own custom reference database or if you will not be running with a reference database.\r\n\r\n* `` $ kneaddata_database --download human_genome bowtie2 $DIR ``\r\n* When running this command, $DIR should be replaced with the full path to the directory you have selected to store the database.\r\n\r\nIf you are running with bmtagger instead of bowtie2, then download the bmtagger database instead of the bowtie2 database with the following command.\r\n\r\n* `` $ kneaddata_database --download human_genome bmtagger $DIR ``\r\n* When running this command, $DIR should be replaced with the full path to the directory you have selected to store the database.\r\n\r\nThe human transcriptome (hg38) reference database is also available for download (approx. size = 254 MB).\r\n\r\n* `` $ kneaddata_database --download human_transcriptome bowtie2 $DIR ``\r\n\r\nThe SILVA Ribosomal RNA reference database is also available for download (approx. size = 11 GB).\r\n\r\n* `` $ kneaddata_database --download ribosomal_RNA bowtie2 $DIR ``\r\n\r\nThe mouse (C57BL) reference database is also available for download (approx. size = 3 GB).\r\n\r\n* `` $ kneaddata_database --download mouse_C57BL bowtie2 $DIR ``\r\n\r\nThe dog reference database is also available for download (approx. size = 1.4 GB). This database is based on the genomic DNA sequences for the [Canis familiaris (domestic dog) assembly version ROS_Cfam_1.0.](https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/014/441/545/GCF_014441545.1_ROS_Cfam_1.0/GCF_014441545.1_ROS_Cfam_1.0_genomic.fna.gz) This file includes the nucleotide sequences of the assembled chromosomes and unplaced scaffolds.  \r\n\r\n* `` $ kneaddata_database --download dog_genome bowtie2 $DIR ``\r\n\r\nThe cat reference database is available for download (approx. size = 3.7 GB). This database is based on the genomic DNA sequences for the [Felis catus (domestic cat)](https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_018350175.1/) This link includes the nucleotide sequences of the assembled chromosomes and unplaced scaffolds.  \r\n\r\n* `` $ kneaddata_database --download cat_genome bowtie2 $DIR ``\r\n\r\n\r\n## Create a Custom Database ##\r\n\r\nA reference database can be downloaded to use when running KneadData. Alternatively, you can create your own custom reference database.\r\n\r\n### Select Reference Sequences ###\r\n\r\nFirst you must select reference sequences for the contamination you are trying to\r\nremove. Say you wish to filter reads from a particular \"host.\" Broadly\r\ndefined, the host can be an organism, or a set of organisms, or just a set of\r\nsequences. Then, you simply must generate a reference database for KneadData from a\r\n[FASTA](http://en.wikipedia.org/wiki/FASTA_format) file containing these\r\nsequences. Usually, researchers want to remove reads from the human genome, the\r\nhuman transcriptome, or ribosomal RNA. You can access some of these FASTA files\r\nusing the resources below:\r\n\r\n- Ribosomal RNA: [Silva](http://www.arb-silva.de/) provides a comprehensive\r\n  database for ribosomal RNA sequences spanning all three domains of life\r\n  (*Bacteria*, *Archaea*, and *Eukarya*).\r\n\r\n- Human Genome \u0026 Transcriptome: Information about the newest assembly of human\r\n  genomic data can be found at the [NCBI project\r\n  page](http://www.ncbi.nlm.nih.gov/projects/genome/assembly/grc/human/). USCS\r\n  provides a convenient [website](http://hgdownload.cse.ucsc.edu/downloads.html#human) to download\r\n  this data.\r\n\r\n### Generating KneadData Databases ###\r\n\r\nKneadData requires that your reference sequences (FASTA files) be indexed to\r\nform KneadData databases beforehand. This only needs to be done once per\r\nreference sequence.\r\n\r\nFor certain common databases, we provide indexed files. If you use these, you\r\ncan skip the manual build steps below. Alternatively if you would like to bypass\r\nthe reference alignment portion of the workflow, a database does not need to be\r\nprovided when running KneadData.\r\n\r\nTo download the indexed human reference database, run the following command:\r\n\r\n* `` $ kneaddata_database --download human bowtie2 $DIR ``\r\n* When running this command, $DIR should be replaced with the full path to the directory you have selected to store the database.\r\n\r\n### Creating a Bowtie2 Database #####\r\n\r\nSimply run the `bowtie2-build` indexer included with Bowtie2 as follows:\r\n\r\n``$ bowtie2-build \u003creference\u003e \u003cdb-name\u003e``\r\n\r\nWhere `\u003creference\u003e` is the reference FASTA file, and `\u003cdb-name\u003e` is the name you\r\nwish to call your Bowtie2 database. For more details, refer\r\nto the [bowtie2-build-documentation](http://bowtie-bio.sourceforge.net/bowtie2/manual.shtml#the-bowtie2-build-indexer)\r\n \r\n \r\n##### **Note: Creating SILVA ribosomal_RNA Database**\r\nCreating the **SILVA ribosomal_RNA** database requires one additional step. Run the following python program before `bowtie2-build` command which converts the \"U\"s to \"T\"s in the fasta sequences.  \r\nScript link: [modify_RNA_to_DNA.py](https://github.com/biobakery/kneaddata/blob/master/kneaddata/db_preprocessing/modify_RNA_to_DNA.py)\r\n``$ python -u modify_RNA_to_DNA.py input.fasta  output.fa``\r\n\r\n\r\n### Creating a BMTagger Database #####\r\n\r\nKneadData includes `kneaddata_build_database`, an executable that\r\nwill automatically generate these databases for BMTagger. Simply run\r\n\r\n``$ kneaddata_build_database reference.fasta``\r\n\r\nBy default, this will generate the reference databases, whose names are prefixed\r\nwith `reference.fasta`.\r\n\r\nA note on PATH: The above command will fail if the tools in the BMTagger suite\r\n(specifically, bmtool and srprism) and the NCBI BLAST executables are not in\r\nyour PATH. If this is the case, you can specify a path to these tools using the\r\n`-b`, `-s`, and `-m` options. Run\r\n\r\n``$ kneaddata_build_database --help``\r\n\r\nfor more details.\r\n\r\n#### Example Custom Database Build #####\r\n\r\nSay you want to remove human reads from your metagenomic sequencing data.\r\nYou downloaded the human genome in a file called `Homo_sapiens.fasta`. \r\n\r\nThen, you can generate the KneadData database by executing:\r\n\r\n``$ bowtie2-build Homo_sapiens.fasta -o Homo_sapiens_db``\r\n\r\nfor Bowtie2, or\r\n\r\n``$ kneaddata_build_database Homo_sapiens.fasta -o Homo_sapiens_db``\r\n\r\nAll of the required KneadData database files will have file names prefixed by\r\n`Homo_sapiens_db` and have various file extensions.\r\n\r\n### **Note**: For creating SILVA ribosomal_RNA database\r\nRun the following python program before `bowtie2-build` command which converts the \"U\"s to \"T\"s in the fasta sequences for creating SILVA ribosomal_RNA database.  \r\nScript link: [modify_RNA_to_DNA.py](https://github.com/biobakery/kneaddata/blob/master/kneaddata/db_preprocessing/modify_RNA_to_DNA.py)\r\n``$ python -u modify_RNA_to_DNA.py input.fasta  output.fa``\r\n\r\n## How to Run ###\r\n\r\nAfter downloading or generating your database file, you can start to remove contaminant reads.\r\nAs input, KneadData requires FASTQ files. It supports both single end and paired\r\nend reads. KneadData uses either Bowtie2 (default) or BMTagger to identify the\r\ncontaminant reads.\r\n\r\n## Single End Run ####\r\n\r\nTo run KneadData in single end mode, run\r\n\r\n` $ kneaddata --unpaired seq.fastq --reference-db $DATABASE --output kneaddata_output `\r\n\r\nThis will create files in the folder `kneaddata_output` named\r\n\r\n+ `seq_kneaddata_$DATABASE_bowtie2_contam.fastq`: FASTQ file containing reads that were\r\n  identified as contaminants from the database (named $DATABASE).\r\n+ `seq_kneaddata.fastq`: This file includes reads that were not in the reference database.\r\n+ `seq_kneaddata.trimmed.fastq`: This file has trimmed reads.\r\n+ `seq_kneaddata.log`\r\n\r\nTo run KneadData in single end mode with BMTagger, run\r\n\r\n` $ kneaddata --unpaired seq.fastq --reference-db $DATABASE --run-bmtagger`\r\n\r\nBy default, this will create the same four files as running with bowtie2. The only differences are the contaminants file will have \"bmtagger\" in the name instead of \"bowtie2\" and the included $DATABASE name would differ.\r\n\r\nIf you wanted to use BMTagger and the BMTagger executable was located at\r\n`$HOME/bmtagger/bmtagger.sh` which is not in your $PATH you would add the option \"--bmtagger $HOME/bmtagger/bmtagger.sh\" to the command. \r\n\r\nIf you wanted to select the basenames of the output files, you would add the option \"--output-prefix $NAME\", replacing $NAME with the name you would like used.\r\n\r\n## Paired End Run ####\r\n\r\nTo run KneadData in paired end mode with Bowtie2, run\r\n\r\n` $ kneaddata --input1 seq1.fastq --input2 seq2.fastq -db $DATABASE --output kneaddata_output`\r\n\r\nTo run KneadData in paired end mode with BMTagger, run\r\n\r\n` $ kneaddata --input seq1.fastq --input seq2.fastq -db $DATABASE --run-bmtagger --output kneaddata_output `\r\n\r\n+ `seq1.fastq`: Your input FASTQ file, first mate\r\n+ `seq2.fastq`: Your input FASTQ file, second mate\r\n+ `$DATABASE`: Prefix for the KneadData database.\r\n+ `kneaddata_output`: The folder to write the output files.\r\n\r\nThe outputs depend on what happens during the quality filtering and trimming\r\npart of the pipeline.\r\n\r\nWhen performing quality filtering and trimming for paired end files, three things\r\ncan happen:\r\n\r\n1. Both reads in the pair pass.\r\n2. The read in the first mate passes, and the one in the second does not pass.\r\n3. The read in the second mate passes, and the one in the first does not pass.\r\n\r\nThe number of outputs are a function of the read quality.\r\n\r\nKneadData + Bowtie2 (or BMTagger) Outputs: There can be up to 8 outputs per reference\r\ndatabase, plus up to 5 aggregate outputs.\r\n\r\nInstead of single end reads, say you have paired end reads and you want to\r\nseparate the reads that came from bacterial mRNA, bacterial rRNA, and human RNA.\r\nYou have two databases, one prefixed `bact_rrna_db` and the other prefixed\r\n`human_rna_db`, and your sequence files are `seq1.fastq` and `seq2.fastq`. To\r\nrun with Bowtie2, execute\r\n\r\n`$ kneaddata --input1 seq1.fastq --input2 seq2.fastq -db bact_rrna_db -db human_rna_db --output seq_out `\r\n\r\nThis will output files in the folder `seq_out` named:\r\n\r\nFiles for just the `bact_rrna_db` database:\r\n\r\n+ `seq_kneaddata_paired_bact_rrna_db_bowtie2_contam_1.fastq`: Reads from the first mate in\r\n  situation (1) above that were identified as belonging to the `bact_rrna_db`\r\n  database.\r\n+ `seq_kneaddata_paired_bact_rrna_db_bowtie2_contam_2.fastq`: Reads from the second mate in\r\n  situation (1) above that were identified as belonging to the `bact_rrna_db`\r\n  database.\r\n+ `seq_kneaddata_paired_bact_rrna_db_bowtie2_clean_1.fastq`: Reads from the first mate in\r\n  situation (1) above that were identified as NOT belonging to the\r\n  `bact_rrna_db` database.\r\n+ `seq_kneaddata_paired_bact_rrna_db_bowtie2_clean_2.fastq`: Reads from the second mate in\r\n  situation (1) above that were identified as NOT belonging to the\r\n  `bact_rrna_db` database.\r\n\r\nDepending on the input FASTQ, one or more of the following may be output:\r\n\r\n+ `seq_kneaddata_unmatched_1_bact_rrna_db_bowtie2_contam.fastq`: Reads from the first mate in\r\n  situation (2) above that were identified as belonging to the `bact_rrna_db`\r\n  database.\r\n+ `seq_kneaddata_unmatched_1_bact_rrna_db_bowtie2_clean.fastq`: Reads from the first mate in\r\n  situation (2) above that were identified as NOT belonging to the\r\n  `bact_rrna_db` database.\r\n+ `seq_kneaddata_unmatched_2_bact_rrna_db_bowtie2_contam.fastq`: Reads from the second mate in\r\n  situation (3) above that were identified as belonging to the `bact_rrna_db`\r\n  database.\r\n+ `seq_kneaddata_unmatched_2_bact_rrna_db_bowtie2_clean.fastq`: Reads from the second mate in\r\n  situation (3) above that were identified as NOT belonging to the\r\n  `bact_rrna_db` database.\r\n\r\nFiles for just the `human_rna_db` database:\r\n\r\n+ `seq_kneaddata_paired_human_rna_db_bowtie2_contam_1.fastq`: Reads from the first mate in\r\n  situation (1) above that were identified as belonging to the `human_rna_db`\r\n  database.\r\n+ `seq_kneaddata_paired_human_rna_db_bowtie2_contam_2.fastq`: Reads from the second mate in\r\n  situation (1) above that were identified as belonging to the `human_rna_db`\r\n  database.\r\n+ `seq_kneaddata_paired_human_rna_db_bowtie2_clean_1.fastq`: Reads from the first mate in\r\n  situation (1) above that were identified as NOT belonging to the\r\n  `human_rna_db` database.\r\n+ `seq_kneaddata_paired_human_rna_db_bowtie2_clean_2.fastq`: Reads from the second mate in\r\n  situation (1) above that were identified as NOT belonging to the\r\n  `human_rna_db` database.\r\n\r\nDepending on the input FASTQ, one or more of the following may be output:\r\n\r\n+ `seq_kneaddata_unmatched_1_human_rna_db_bowtie2_contam.fastq`: Reads from the first mate in\r\n  situation (2) above that were identified as belonging to the `human_rna_db`\r\n  database.\r\n+ `seq_kneaddata_unmatched_1_human_rna_db_bowtie2_clean.fastq`: Reads from the first mate in\r\n  situation (2) above that were identified as NOT belonging to the\r\n  `human_rna_db` database.\r\n+ `seq_kneaddata_unmatched_2_human_rna_db_bowtie2_contam.fastq`: Reads from the second mate in\r\n  situation (2) above that were identified as belonging to the `human_rna_db`\r\n  database.\r\n+ `seq_kneaddata_unmatched_2_human_rna_db_bowtie2_clean.fastq`: Reads from the second mate in\r\n  situation (2) above that were identified as NOT belonging to the\r\n  `human_rna_db` database.\r\n\r\nNote, the files named \"*_clean.fastq\" will only be written if running with the option \"--store-temp-output\".\r\n\r\nAggregated files:\r\n\r\n+ `seq_kneaddata.log`: Log file containing statistics about the run.\r\n+ `seq_kneaddata_paired_1.fastq`: Reads from the first mate in situation (1) identified as\r\n  NOT belonging to any of the reference databases.\r\n+ `seq_kneaddata_paired_2.fastq`: Reads from the second mate in situation (1) identified as\r\n  NOT belonging to any of the reference databases.\r\n+ `seq_kneaddata_unmatched_1.fastq`: Reads from the first mate in situation (2) identified as\r\n  NOT belonging to any of the reference databases.\r\n+ `seq_kneaddata_unmatched_2.fastq`: Reads from the second mate in situation (3) identified as\r\n  NOT belonging to any of the reference databases.\r\n\r\n## Demo Run ####\r\n\r\nThe examples folder contains a demo input file. This file is a single read, fastq format.\r\n\r\n`` $ kneaddata --unpaired examples/demo.fastq --reference-db examples/demo_db --output kneaddata_demo_output ``\r\n\r\nThis will create four output files:\r\n\r\n1. `` kneaddata_demo_output/demo_kneaddata.fastq ``\r\n2. `` kneaddata_demo_output/demo_kneaddata_demo_db_bowtie2_contam.fastq ``\r\n3. `` kneaddata_demo_output/demo_kneaddata.log ``\r\n3. `` kneaddata_demo_output/demo_kneaddata.trimmed.fastq ``\r\n\r\n## Sequencer Source for trimming Adapter Contents ####\r\nKneaddata will use **\"NexteraPE\"** adapters provided by trimomatic to trim the adapter contents `by default`.\r\n\r\nThe other available options are: `[\"NexteraPE\", \"TruSeq2\", \"TruSeq3\",\"none\"]`. Based on the source of the sequencer and the FASTQC report, it is **highly reccommended**\r\nto choose the correct sequencer source to ensure the removal of adapter contents by Kneaddata. \r\n\r\n###### Example: Trimmming adapter sequence using **TruSeq3** sequencer adapters in the workflow: \r\n```\r\nkneaddata --unpaired demo.fastq -db demo_db -o kneaddata_output --sequencer-source TruSeq3 --fastqc FastQC\r\n```\r\n###### Example: Skipping adapter trimming in the workflow:\r\n```\r\nkneaddata --unpaired demo.fastq -db demo_db -o kneaddata_output --sequencer-source none --fastqc FastQC\r\n```\r\n\r\n## Trim Overrepresented/Repetitive sequences ####\r\nIt is highly recommeded to use **--run-trim-repetitive** flag for **Shotgun sequences (Metatranscriptomics-MTX, Metagenomics-MGX)** to trim the overrepresented sequences if shown in FASTQC reports.\r\n\r\nHowever, Kneaddata will **not** trim the overrepresented sequences **by default** as **Amplicon sequences** usually have a large number of repetitive reads resulting in depletion of the read count.\r\n\r\n###### Example: Trimming overrepresented sequences using the Fastqc reports:\r\n```\r\nkneaddata --unpaired demo.fastq -db demo_db -o kneaddata_output --run-trim-repetitive --fastqc FastQC\r\n```\r\n###### Example: Trimming overrepresented sequences and TruSeq3 adapters:\r\n```\r\nkneaddata --unpaired demo.fastq -db demo_db -o kneaddata_output --run-trim-repetitive --sequencer-source TruSeq3 --fastqc FastQC\r\n```\r\n\r\n## Additional Arguments ####\r\n\r\nIf you want to specify additional arguments for Bowtie2 using the\r\n`--bowtie2-options` flag, you will need to use the equals sign along with quotes. Add additional flags for each option.\r\n\r\nFor example:\r\n\r\n`$ kneaddata --unpaired demo.fastq --output kneaddata_output --reference-db database_folder --bowtie2-options=\"--very-fast\" --bowtie2-options=\"-p 2\"`\r\n\r\nA similar approach is used to specify additional arguments for Trimmomatic:\r\n\r\n`$ kneaddata --unpaired demo.fastq --output kneaddata_output --reference-db database_folder --trimmomatic-options=\"LEADING:3\" --trimmomatic-options=\"TRAILING:3\"`\r\n\r\n*NOTE*: Manually specifying additional arguments will completely override the defaults.\r\n\r\nAlso more than one database can be provided for each run. The database argument can contain the folder that includes the database or the prefix of the database files. \r\n\r\nFor example:\r\n\r\n`$ kneaddata --unpaired demo.fastq --output kneaddata_output --reference-db database_folder --reference-db database_folder2/demo`\r\n\r\n## Contributions ##\r\nThanks go to these wonderful people:\r\n- weichi weichi.syu@atgenomix.com\r\n  - TRF parallel run bug fix    \r\n\r\n## Complete Option List ##\r\n\r\nAll options can be accessed with `$ kneaddata --help`.\r\n\r\n```\r\nusage: kneaddata [-h] [--version] [-v] [-i1 INPUT1] [-i2 INPUT2]\r\n                 [-un UNPAIRED]  -o OUTPUT_DIR\r\n                 [-db REFERENCE_DB] [--bypass-trim] [--run-trim-repetitive]\r\n                 [--output-prefix OUTPUT_PREFIX] [-t \u003c1\u003e] [-p \u003c1\u003e]\r\n                 [-q {phred33,phred64}] [--run-bmtagger]\r\n                 [--run-fastqc-start] [--run-fastqc-end] [--store-temp-output]\r\n                 [--cat-final-output]\r\n                 [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--log LOG]\r\n                 [--trimmomatic TRIMMOMATIC_PATH] [--max-memory MAX_MEMORY]\r\n                 [--trimmomatic-options TRIMMOMATIC_OPTIONS]\r\n                 [--bowtie2 BOWTIE2_PATH] [--bowtie2-options BOWTIE2_OPTIONS]\r\n                 [--bmtagger BMTAGGER_PATH] [--trf TRF_PATH] [--match MATCH]\r\n                 [--mismatch MISMATCH] [--delta DELTA] [--pm PM] [--pi PI]\r\n                 [--minscore MINSCORE] [--maxperiod MAXPERIOD]\r\n                 [--fastqc FASTQC_PATH]\r\n\r\nKneadData\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -v, --verbose         additional output is printed\r\n\r\nglobal options:\r\n  --version             show program's version number and exit\r\n  -i INPUT, --input INPUT\r\n                        input FASTQ file (add a second argument instance to run with paired input files)\r\n  -o OUTPUT_DIR, --output OUTPUT_DIR\r\n                        directory to write output files\r\n  -db REFERENCE_DB, --reference-db REFERENCE_DB\r\n                        location of reference database (additional arguments add databases)\r\n  --run-trim-repetitive Option to trim repetitive/overrepresented sequences generated by FASTQC reports \r\n  --bypass-trim         bypass the trim step\r\n  --output-prefix OUTPUT_PREFIX\r\n                        prefix for all output files\r\n                        [ DEFAULT : $SAMPLE_kneaddata ]\r\n  -t \u003c1\u003e, --threads \u003c1\u003e\r\n                        number of threads\r\n                        [ Default : 1 ]\r\n  -p \u003c1\u003e, --processes \u003c1\u003e\r\n                        number of processes\r\n                        [ Default : 1 ]\r\n  -q {phred33,phred64}, --quality-scores {phred33,phred64}\r\n                        quality scores\r\n                        [ DEFAULT : phred33 ]\r\n  --run-bmtagger        run BMTagger instead of Bowtie2 to identify contaminant reads\r\n  --bypass-trf          option to bypass the removal of tandem repeats\r\n  --run-fastqc-start    run fastqc at the beginning of the workflow\r\n  --run-fastqc-end      run fastqc at the end of the workflow\r\n  --store-temp-output   store temp output files\r\n                        [ DEFAULT : temp output files are removed ]\r\n  --cat-final-output    concatenate all final output files\r\n                        [ DEFAULT : final output is not concatenated ]\r\n  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}\r\n                        level of log messages\r\n                        [ DEFAULT : DEBUG ]\r\n  --log LOG             log file\r\n                        [ DEFAULT : $OUTPUT_DIR/$SAMPLE_kneaddata.log ]\r\n\r\ntrimmomatic arguments:\r\n  --trimmomatic TRIMMOMATIC_PATH\r\n                        path to trimmomatic\r\n                        [ DEFAULT : $PATH ]\r\n  --max-memory MAX_MEMORY\r\n                        max amount of memory\r\n                        [ DEFAULT : 500m ]\r\n  --trimmomatic-options TRIMMOMATIC_OPTIONS\r\n                        options for trimmomatic\r\n                        [ DEFAULT : SLIDINGWINDOW:4:20 MINLEN:50 ]\r\n                        MINLEN is set to 50 percent of total input read length. The user can alternatively specify a length (in bases) for MINLEN.\r\n  --sequencer-source    options for sequencer-source\r\n                        [ DEFAULT: NexteraPE]\r\n                        Available sequencers: [\"NexteraPE\",\"TruSeq2\",\"TruSeq3\"]\r\n\r\nbowtie2 arguments:\r\n  --bowtie2 BOWTIE2_PATH\r\n                        path to bowtie2\r\n                        [ DEFAULT : $PATH ]\r\n  --bowtie2-options BOWTIE2_OPTIONS\r\n                        options for bowtie2\r\n                        [ DEFAULT : --very-sensitive ]\r\n\r\nbmtagger arguments:\r\n  --bmtagger BMTAGGER_PATH\r\n                        path to BMTagger\r\n                        [ DEFAULT : $PATH ]\r\n\r\ntrf arguments:\r\n  --bypass-trf          bypass the TRF step\r\n  --trf TRF_PATH        path to TRF\r\n                        [ DEFAULT : $PATH ]\r\n  --match MATCH         matching weight\r\n                        [ DEFAULT : 2 ]\r\n  --mismatch MISMATCH   mismatching penalty\r\n                        [ DEFAULT : 7 ]\r\n  --delta DELTA         indel penalty\r\n                        [ DEFAULT : 7 ]\r\n  --pm PM               match probability\r\n                        [ DEFAULT : 80 ]\r\n  --pi PI               indel probability\r\n                        [ DEFAULT : 10 ]\r\n  --minscore MINSCORE   minimum alignment score to report\r\n                        [ DEFAULT : 50 ]\r\n  --maxperiod MAXPERIOD\r\n                        maximum period size to report\r\n                        [ DEFAULT : 500 ]\r\n\r\nfastqc arguments:\r\n  --fastqc FASTQC_PATH  path to fastqc\r\n                        [ DEFAULT : $PATH ]\r\n```\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiobakery%2Fkneaddata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbiobakery%2Fkneaddata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiobakery%2Fkneaddata/lists"}