{"id":26573244,"url":"https://github.com/linsalrob/fastq-pair","last_synced_at":"2025-06-30T07:35:30.650Z","repository":{"id":54638960,"uuid":"98881309","full_name":"linsalrob/fastq-pair","owner":"linsalrob","description":"Match up paired end fastq files quickly and efficiently.","archived":false,"fork":false,"pushed_at":"2024-06-02T07:11:54.000Z","size":249,"stargazers_count":146,"open_issues_count":7,"forks_count":33,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-08T21:32:10.510Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://edwards.flinders.edu.au/sorting-and-paring-fastq-files/","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linsalrob.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.md","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-31T11:24:48.000Z","updated_at":"2025-04-04T16:52:15.000Z","dependencies_parsed_at":"2022-08-13T22:30:44.567Z","dependency_job_id":"cacbde98-1338-4688-bd53-2e4c45e45011","html_url":"https://github.com/linsalrob/fastq-pair","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/linsalrob/fastq-pair","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Ffastq-pair","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Ffastq-pair/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Ffastq-pair/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Ffastq-pair/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linsalrob","download_url":"https://codeload.github.com/linsalrob/fastq-pair/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2Ffastq-pair/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262731858,"owners_count":23355438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-23T00:39:53.554Z","updated_at":"2025-06-30T07:35:30.588Z","avatar_url":"https://github.com/linsalrob.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Edwards Lab](https://img.shields.io/badge/Bioinformatics-EdwardsLab-03A9F4)](https://edwards.sdsu.edu/research)\n[![DOI](https://zenodo.org/badge/98881309.svg)](https://zenodo.org/badge/latestdoi/98881309)\n[![Build Status](https://travis-ci.org/linsalrob/fastq-pair.svg?branch=master)](https://travis-ci.org/linsalrob/fastq-pair)\n\n# FASTQ PAIR\n\nRewrite paired end fastq files to make sure that all reads have a mate and to separate out singletons.\n\nThis code does one thing: it takes two fastq files, and generates four fastq files. That's right, for free it doubles\nthe number of fastq files that you have!!\n\nUsually when you get paired end read files you have two files with a /1 sequence in one and a /2 sequence in the other\n(or a /f and /r or just two reads with the same ID).  However, often when working with files from a third party source\n(e.g. [the SRA](http://edwards.sdsu.edu/research/sra/)) there are different numbers of reads in each file (because some\nreads fail QC). Spades, bowtie2 and other tools break because they demand paired end files have the same number of reads.\n\nThis program solves that problem.\n\nIt rewrites the files with the sequences in order, with matching files for the two files provided on the command line,\nand then any single reads that are not matched are place in two separate files, one for each original file.\n\nThis code is designed to be fast and memory efficient, and works with large fastq files. It does not store the whole\nfile in memory, but rather just stores the locations of each of the indices in the first file provided in memory.\n\n### Speed and efficiency considerations\n\nThe most efficient way to use this code is to provide the smallest file first (though it doesn't matter which way you\nprovide the files), and then to manipulate the `-t` parameter on the command line. The code implementation is based\non a [hash table](https://en.wikipedia.org/wiki/Hash_table) and the size of that table is the biggest way to make this\ncode run faster. If you set the hash table size too low, then the data structure quickly fills up and the performance\ndegrades to what we call _O_(n). On the other hand if you  set the table size too big, then you waste a lot of memory,\nand it takes longer to initialize the data structures safely.\n\nThe optimal table size is basically somewhere around the number of sequences in your fastq files. You can quickly find\nout how many sequences there are in your fastq file:\n\n```\nwc -l fastq_filename\n```\nThe number of sequences will be the number printed here, divided by 4.\n\n_Note_: If you get an error that looks like \n```\n\"We cannot allocate the memory for a table size of -436581356. Please try a smaller value for -t\"\n```\n\nyou are probably suffering from an integer overflow, so try reducing the value you are providing to the `-t` option.\nSee [issue 12](https://github.com/linsalrob/fastq-pair/issues/12) for more details.\n\nIf you are not sure, you can run this code with the `-p` parameter. Before it prints out the matched pairs of sequences,\nit will print out the number of sequences in each \"bucket\" in the table. If this number is more than about a dozen you\nneed to increase the value you provide to `-t`. If most of the entries are zero, then you should decrease the size of\n`-t`.\n\nAs an aside, this code is also _really_ slow if _none_ of your sequences are paired. You should most likely use this\nafter taking a peek at your files and making sure there are at least _some_ paired sequences in your files!\n\n## Installing fastq_pair\n\nWe recommend installing fastq-pair using [bioconda](https://bioconda.github.io/recipes/fastq-pair/README.html)\n\n```\nmamba install -c bioconda fastq-pair\n```\n\nor in its own environment:\n\n```\nmamba create --name fastq-pair -c bioconda fastq-pair\n```\n\n### Installing from source\n\nTo install the code, grab the github repository, then make a build directory:\n```$xslt\nmkdir build \u0026\u0026 cd build\ncmake3 ..\nmake \u0026\u0026 sudo make install\n```\nThere are more instructions on the [installation](INSTALLATION.md) page.\n\n## Running fastq_pair\n\n`fastq_pair` takes two primary arguments. The name of the two fastq files that you want to pair up.\n\n```$xslt\nfastq_pair file1.fastq file2.fastq\n```\n\nYou can also change the size of the hash table using the `-t` parameter:\n\n```$xslt\nfastq_pair -t 50021 file1.fastq file2.fastq\n```\n\nYou can also print out the number of elements in each bucket using the `-p` parameter:\n\n```$xslt\nfastq_pair -p -t 100 file1.fastq file2.fastq\n```\n\n\n## Testing fastq_pair\n\nIn the [test](test/) directory there are two fastq files that you can use to test `fastq_pair`. There are 250 sequences\nin the [left](test/left.fastq) file and 75 sequences in the [right](test/right.fastq) file. Only 50 sequences are common\nbetween the two files.\n\nYou can test the code with:\n\n```$xslt\nfastq_pair -t 1000 test/left.fastq test/right.fastq\n```\n\nThis will make four files in the [test/](test) directory:\n- left.fastq.paired.fq\n- left.fastq.single.fq\n- right.fastq.paired.fq\n- right.fastq.single.fq\n\nThe _paired_ files have 50 sequences each, and the two _single_ files have 200 and 25 sequences (left and right respectively).\n\n### A note about gzipped fastq files\n\nUnfortunately `fastq_pair` doesn't work with gzipped files at the moment, because it relies heavily on random access of\nthe file stream. That is complex with gzipped files, especially when the uncompressed file exceeds available memory\n(which is exactly the situation that `fastq_pair` was designed to handle).\n\nTherefore, at this time, `fastq_pair` does not support gzipped files. You need to uncompress the files before using\n`fastq_pair`.\n\nIf you really need to use gzipped files, and can accept slightly worse performance, then\n[we have some alternative](https://edwards.sdsu.edu/research/sorting-and-paring-fastq-files/) approaches\nwritten in Python that you can try.\n\n### Testing for gzipped files ([issue #6](https://github.com/linsalrob/fastq-pair/issues/6))\n\nWe take a peek at the first couple of bytes in the file to see if the file is gzip compressed. Per the standard, the\nfiles should start 0x1F and 0x8B as the first two bytes. There is a small tester for the gzip program, called `test_gzip.c`,\nthat takes a single argument and reports whether it is gzipped or not. You can compile that tester with the command:\n\n```\ngcc -std=gnu99  -o testgz ./test_gzip.c  is_gzipped.c\n```\n\nWe now test both files and exit (hopefully gracefully) if either is gzip compressed. The easiest solution is to\nuncompress your files, and we recommend and love [pigz](https://zlib.net/pigz/) because it is awesome!\n\n## Citing fastq_pair\n\nPlease see the [CITATION](CITATION.md) file for the current citation for fastq-pair\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Ffastq-pair","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinsalrob%2Ffastq-pair","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Ffastq-pair/lists"}