Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/agordon/fastx_toolkit

FASTA/FASTQ pre-processing programs
https://github.com/agordon/fastx_toolkit

Last synced: 3 months ago
JSON representation

FASTA/FASTQ pre-processing programs

Awesome Lists containing this project

README 

        
FASTX-Toolkit

=============



*******************************************************************

*                                                                 *

* FASTX TOOLKIT is unmaintained software.                         *

* No new features have been added since 2010.                     *

*                                                                 *

* There are many better alternatives for low-level FASTQ/FASTA    *

* manipulation. Use at your own risk.                             *

*                                                                 *

*******************************************************************



Short Summary

===============



The FASTX-Toolkit is a collection of command line tools for Short-Reads 

FASTA/FASTQ files preprocessing.



More Details

==============



Next-Generation sequencing machines usually produce FASTA or FASTQ files, 

containing multiple short-reads sequences (possibly with quality information).



The main processing of such FASTA/FASTQ files is mapping (aka aligning)

the sequences to reference genomes or other databases using specialized

programs. 



Example of such mapping programs are:

Blat (http://www.kentinformatics.com/index.asp), 

SHRiMP (http://compbio.cs.toronto.edu/shrimp),

LastZ (http://www.bx.psu.edu/miller_lab),

MAQ (http://maq.sourceforge.net/)

And many many others.



However, 

It is sometimes more productive to preprocess the FASTA/FASTQ files before 

mapping the sequences to the genome - manipulating the sequences to 

produce better mapping results.



The FASTX-Toolkit tools perform some of these preprocessing tasks.



Available Tools

===============



FASTQ-to-FASTA - Converts a FASTQ file to a FASTA file..



FASTQ-Statistics - scans a FASTQ file, and produces some statistics about the 

	quality and the sequences in the file.

	

FASTQ-Quality-BoxPlot, and

FASTQ-Nucleotides-Distribution - Generates charts based on the statistics 

	generated by FASTQ-Statistics. These charts can be used to quickly

	see the quality of the sequenced library.

	

FASTQ-Quality-Converter - Converts from ASCII to numeric quality scores.



FASTQ-Quality-Filter - removes low-quality sequences from FASTQ files.



FASTX-Artifacts-Filter - removes some sequencing artifacts from FASTA/Q files.



FASTX-Barcode-Splitter - A common practice is to sequence multiple biological

	samples in the same library (marking each sample using a dedicated 

	barcode). The resulting FASTA/Q file contains intermixed sequences 

	from those samples. This tool separates FASTA/Q files into several 

	individual files, based on the barcodes.

	

FASTX-Clipper - Adapters (aka Linkers) are added to the library (before 

	sequencing), and should be removed from the resulting FASTA/Q file.

	This tool removes (clips) adapters.

	

FASTA-Clipping-Histogram - After clipping a FASTA file, this tool generates a

	chart showing the length of the clipped sequences.

	

FASTX-Reverse-Complement - Produces a reverse-complement of FASTA/Q file.

	If a FASTQ file is given, the quality scores are also reversed.

	

FASTX-Trimmer - Extract sub-seqeunces from FASTA/Q file. Two examples are:

	Removing barcodes from the 5'-end of all sequences in a FASTQ file;

	Cutting 7 nucleotides from the 3'-end of all sequences in a FASTA file.



Galaxy

======



Galaxy (https://usegalaxy.org) is web-based framework for computational biology.



While the programs in the FASTX-Toolkit are command-line based, the package 

include the necessary files to integrate the tools into a Galaxy server,

Allowing users to execute this tools from their web-browser.



If you run your own local mirror of a Galaxy server, you can integrate the

FASTX-Toolkit into your Galaxy server.



Software Requirements

=====================



1. GCC is required to compile most tools.



2. FASTA-Clipping-Histogram tool requires Perl, the "PerlIO::gzip",

   "GD::Graph::bars" modules.

   

   Installing the perl modules can be accomplised by running:



   $ sudo cpan 'PerlIO::gzip'

   $ sudo cpan 'GD::Graph::bars'

   

3. FASTX-Barcode-Splitter requires the GNU Sed program.

   

4. FASTQ-Quality-Boxplot and FASTQ-Nucleotides-Distribution requires the

   'gnuplot' program.



Installation

============



When downloading the git repository from github, use the following:



   $ git clone https://github.com/agordon/fastx_toolkit

   $ cd fastx_toolkit

   $ ./reconf

   $ ./configure

   $ make



When downloading a released version archive:



   $ wget https://github.com/agordon/fastx_toolkit/releases/download/0.0.14/fastx_toolkit-0.0.14.tar.bz2

   $ tar -xjvf fastx_toolkit-0.0.14.tar.bz2

   $ cd fastx_toolkit-0.0.14

   $ ./configure

   $ make



The available releases are here:

   https://github.com/agordon/fastx_toolkit/releases



To install the tools, run (as root):



  $ sudo make install



This will install the tools into /usr/local/bin.

To install the tools to a different location, change the 'configure' step to:



  $ ./configure --prefix=/DESTINATION/DIRECTORY



The libgtextutils package is required to build fastx-toolkit,

see https://github.com/agordon/libgtextutils/ .



Command Line Usage

==================



Most tools support "-h" argument to show a short help screen.

Better documentation is not available at this moment.

Some more details and examples are available in the  section

of the XML tool files (in the 'galaxy' subdirectory).

  

 

Galaxy Installation

===================



Galaxy Installation should be done manually, and requires technical

understading of the Galaxy framework.



1. build and install the command line tools (as described above).



2. Make backup of your galaxy installation (better safe than sorry).



3. Run the 'install_galaxy_files.sh' script, 

   and specify the galaxy root directory.

   This script copies the files from the 'galaxy' sub-directory into

   your galaxy mirror directory.

   

4. Manually add the content of ./galaxy/fastx_toolkit_conf.xml file,

   into your Galaxy's tool_conf.xml

   

5. Edit [YOUR-GALAXY]/tool-data/fastx_clipper_sequences.txt file,

   And add your custom adapters/linkers.

   

6. Modify the "fastx_barcode_splitter_galaxy_wrapper.sh" as explained

   Below (see section "Special configuration for Barcode-Splitter").



7. Restart Galaxy.



Always make backup of your galaxy server files before trying to install 

the FASTX-Toolkit. 



Galaxy Testing

==============



The following tools support Galaxy's functional testing:

(Run from Galaxy's main directory)

  $ sh run_functional_tests.sh -id cshl_fastq_qual_conv

  $ sh run_functional_tests.sh -id cshl_fastq_to_fasta

  $ sh run_functional_tests.sh -id cshl_fastq_qual_stat

  $ sh run_functional_tests.sh -id cshl_fastx_trimmer

  $ sh run_functional_tests.sh -id cshl_fastx_reverse_complement

  $ sh run_functional_tests.sh -id cshl_fastx_artifacts_filter

  $ sh run_functional_tests.sh -id cshl_fasta_collapser

  $ sh run_functional_tests.sh -id cshl_fastx_clipper

 



Special configuration for Barcode-Splitter

==========================================



When running the barcode-splitter tool from the command line you specify a 

prefix direcotry - the output files will be written to that directory (similar

to GNU's split program usage).



Running the barcode-splittter inside galaxy requires a special hack beacuse

(I don't know how to|Galaxy can't) create a variable number of output datasets.

The number of required output files is determined by the tool only AFTER reading 

the barcodes description file.



The Galaxy-version of Barcode-Splitter works like this:

1. A FASTA/FASTQ file, and a Barcode description file are fed to the tool.

2. The tool produces a single output dataset (inside galaxy). This output

   is an HTML file, containing links to the split FASTA files.

3. Users can use the links to get the split FASTA files.

   (Since Galaxy's 'upload data' tool accepts URLs, this is not a real problem).

   

4. As the galaxy administrator, you'll have to edit 

   'fastx_barcode_splitter_galaxy_wrapper.sh' script and change BASEPATH and 

   PUBLICURL to point to a publicly accesibly path on your server.

   

Example:



fastx_barcode_splitter_galaxy_wrapper.sh contains:



   BASEPATH="/media/sdb1/galaxy/barcode_splits/"

   PUBLICURL="http://tango.cshl.edu/barcode_splits/"



When a user runs the barcode splitter tool, the FASTA files will be generated in 

"/media/sdb1/galaxy/barcode_splits/".  

The URL "http://tango.cshl.edu/barcode_splits" is set (in an apache server) to

serve files from "/media/sdb1/galaxy/barcode_splits/", with the following 

configuration:



    Alias /barcode_splits "/media/sdb1/galaxy/barcode_splits/"

    

        AllowOverride None

        Order allow,deny

        Allow from all

    



Licenses

========



FASTX-Toolkit is distributed under the Affero GPL version 3 or later (AGPLv3),



EXCEPT



All files under the 'galaxy' sub-directory are distributed under the

same license as Galaxy itself (which is an MIT-style license).



While IANAL, these licenses basically mean that:

1. You're free to use FASTX-toolkit,



2. You're free to integrate FASTX-toolkit in your Galaxy mirror server 

   (or any other server).

   

3. You're free to modify the files under 'galaxy',

   without making your modifications public.

   

4. If you modify the FASTX-toolkit tools, and make those modifications 

   publicly available (either as downloadable tools, part of another product),

   or as a web-based server - you must make the modified source code freely 

   available (free as in speech).

   

See the COPYING file for the full Affero GPL.

See the GALAXY-LICENSE file for galaxy's license.



Please remember: 

  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY

APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT

HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY

OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,

THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR

PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM

IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF

ALL NECESSARY SERVICING, REPAIR OR CORRECTION.



=============

Please send all comments, suggestions, bug reports (or better yet - bug fixes)

to [email protected] .