{"id":13639122,"url":"https://github.com/shenwei356/kmcp","last_synced_at":"2025-12-30T00:11:30.425Z","repository":{"id":37719863,"uuid":"308636966","full_name":"shenwei356/kmcp","owner":"shenwei356","description":"Accurate metagenomic profiling \u0026\u0026 Fast large-scale sequence/genome searching","archived":false,"fork":false,"pushed_at":"2023-09-22T04:09:54.000Z","size":86047,"stargazers_count":176,"open_issues_count":10,"forks_count":13,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-08-03T01:13:50.796Z","etag":null,"topics":["bigsi","cobs","fracminhash","kmer","metagenomics","scaled-minhash","searching","sketch","sketching","syncmers","taxonomic-classification","taxonomic-profiling","virome"],"latest_commit_sha":null,"homepage":"https://bioinf.shenwei.me/kmcp","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shenwei356.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-10-30T13:21:43.000Z","updated_at":"2024-07-11T03:47:26.000Z","dependencies_parsed_at":"2023-02-09T11:01:48.461Z","dependency_job_id":"4a66099f-9368-4c32-b3be-79fa6ee5a9a8","html_url":"https://github.com/shenwei356/kmcp","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenwei356%2Fkmcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenwei356%2Fkmcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenwei356%2Fkmcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenwei356%2Fkmcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shenwei356","download_url":"https://codeload.github.com/shenwei356/kmcp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223810278,"owners_count":17206728,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigsi","cobs","fracminhash","kmer","metagenomics","scaled-minhash","searching","sketch","sketching","syncmers","taxonomic-classification","taxonomic-profiling","virome"],"created_at":"2024-08-02T01:00:57.899Z","updated_at":"2025-12-30T00:11:30.389Z","avatar_url":"https://github.com/shenwei356.png","language":"Go","funding_links":[],"categories":["其他_生物医药","Sequence Analysis and Manipulation","Ranked by starred repositories"],"sub_categories":["网络服务_其他"],"readme":"\u003ccenter\u003e\u003cimg src=\"kmcp-logo.jpg\" alt=\"Kmer-based Metagenomic Classification and Profiling\" width=\"800\"/\u003e\u003c/center\u003e\n\n# KMCP: accurate metagenomic profiling of both prokaryotic and viral populations by pseudo-mapping\n\n\n\n## Citation\n\n\u003e **KMCP: accurate metagenomic profiling of both prokaryotic and viral populations by pseudo-mapping**.\u003cbr\u003e\n\u003e Wei Shen, Hongyan Xiang, Tianquan Huang, Hui Tang, Mingli Peng, Dachuan Cai, Peng Hu, Hong Ren.\u003cbr\u003e\n\u003e Bioinformatics, btac845, [https://doi.org/10.1093/bioinformatics/btac845](https://doi.org/10.1093/bioinformatics/btac845)\n\n## Table of contents\n\n* [Documents](#documents)\n* [What can we do?](#what-can-we-do-)\n    + [1. Accurate metagenomic profiling](#1-accurate-metagenomic-profiling)\n    + [2. Fast sequence search against large scales of genomic datasets](#2-fast-sequence-search-against-large-scales-of-genomic-datasets)\n    + [3. Fast genome similarity estimation](#3-fast-genome-similarity-estimation)\n* [Features](#features)\n* [Installation](#installation)\n* [Commands](#commands)\n* [Quickstart](#quickstart)\n* [KMCP vs COBS](#kmcp-vs-cobs)\n* [Support](#support)\n* [License](#license)\n* [Acknowledgments](#acknowledgments)\n\n## Documents\n\nhttps://bioinf.shenwei.me/kmcp\n\n- [Installation](https://bioinf.shenwei.me/kmcp/download)\n- [Databases](https://bioinf.shenwei.me/kmcp/database)\n- Tutorials\n    - [Taxonomic profiling](https://bioinf.shenwei.me/kmcp/tutorial/profiling)\n    - [Detecting specific pathogens](https://bioinf.shenwei.me/kmcp/tutorial/detecting-pathogens)\n    - [Detecting contaminated sequences](https://bioinf.shenwei.me/kmcp/tutorial/detecting-contaminated-seqs)\n    - [Sequence and genome searching](https://bioinf.shenwei.me/kmcp/tutorial/searching)\n- [Usage](https://bioinf.shenwei.me/kmcp/usage)\n- [Benchmarks](https://bioinf.shenwei.me/kmcp/benchmark)\n- [FAQs](https://bioinf.shenwei.me/kmcp/faq)\n\n\n## What can we do?\n\n### 1. Accurate metagenomic profiling\n\nKMCP utilizes genome coverage information by splitting the reference genomes\ninto chunks and stores k-mers in a modified and optimized [COBS](https://github.com/bingmann/cobs) index for\nfast alignment-free sequence searching. KMCP **combines k-mer similarity and\ngenome coverage information to reduce the false positive rate** of k-mer-based\ntaxonomic classification and profiling methods.\n\nThe read mapping process in KMCP is referred to as **pseudo-mapping**,\nwhich is similar to but different from the lightweight algorithm in Sailfish (Patro et al., 2014),\npseudoalignment in Kallisto (Bray et al., 2016), quasi-mapping in RapMap (Srivastava et al., 2016),\nand lightweight mapping in Salmon (Patro et al., 2017). All of these methods seek to elide the\ncomputation of base-to-base alignment using distinct strategies (Srivastava et al., 2016).\n*In KMCP, each reference genome is pre-split into chunks of equal size, and the k-mers of a query,\nas a whole, are compared to each genome chunk to find all possible ones sharing a predefined\nproportion of k-mers with the query*. Like quasi-mapping in RapMap, KMCP tracks the target\nand position for each query. However, the read position in KMCP is approximate and in a\npredefined resolution (the number of genome chunks).\n\nBenchmarking results based on simulated and real data demonstrate that **KMCP,\ndespite a longer running time than some other methods,\nnot only allows the accurate taxonomic profiling of prokaryotic and viral populations\nbut also provides more confident pathogen detection in clinical samples of low depth**.\n\n***Genome collections with custom taxonomy***,\ne.g., [GTDB](https://gtdb.ecogenomic.org/) uses its own taxonomy and\n[MGV](https://doi.org/10.1038/s41564-021-00928-6) uses [ICTV taxonomy](https://talk.ictvonline.org/),\nare also supported by generating NCBI-style taxdump files with [taxonkit create-taxdump](https://bioinf.shenwei.me/taxonkit/usage/#create-taxdump).\nYou can even [merge the GTDB taxonomy (for prokaryotic genomes from GTDB) and NCBI taxonomy (for genomes from NCBI)](https://bioinf.shenwei.me/kmcp/database/#merging-gtdb-and-ncbi-taxonomy).\n\n### 2. Fast sequence search against large scales of genomic datasets\n\nKMCP can be used for fast sequence search against large scales of genomic datasets\nas [BIGSI](https://github.com/Phelimb/BIGSI) and [COBS](https://github.com/bingmann/cobs) do.\nWe reimplemented and modified the Compact Bit-Sliced Signature index (COBS) algorithm,\nbringing a smaller index size and [much faster searching speed (2x for genome search and 10x for short reads) faster than COBS](https://bioinf.shenwei.me/kmcp/benchmark/searching/#result)\n (check the [tutorial](https://bioinf.shenwei.me/kmcp/tutorial/searching) and [benchmark](https://bioinf.shenwei.me/kmcp/benchmark/searching)). Also check [the algorithm and data structure differences between KMCP and COBS](#kmcp-vs-cobs).\n \n### 3. Fast genome similarity estimation\n\nKMCP can also be used for fast similarity estimation of assemblies/genomes against known reference genomes.\n\nGenome sketching is a method of utilizing small and approximate summaries of\ngenomic data for fast searching and comparison.\n[Mash](https://github.com/marbl/Mash) and [Sourmash](https://github.com/sourmash-bio/sourmash)\nprovide fast genome distance estimation using MinHash (Mash) or FracMinHash (Sourmash).\nKMCP supports multiple k-mer sketches \n([Minimizer](https://academic.oup.com/bioinformatics/article/20/18/3363/202143), \n[FracMinHash](https://www.biorxiv.org/content/10.1101/2022.01.11.475838v2)\n(previously named [Scaled MinHash](https://f1000research.com/articles/8-1006)), and\n[Closed Syncmers](https://peerj.com/articles/10805/)) for genome similarity estimation.\nAnd [KMCP is 5x-7x faster than Mash/Sourmash](https://bioinf.shenwei.me/kmcp/benchmark/searching/#result)\n (check the [tutorial](https://bioinf.shenwei.me/kmcp/tutorial/searching) and [benchmark](https://bioinf.shenwei.me/kmcp/benchmark/searching)).\n\n\n## Features\n\n- **Easy to install**\n    - [Statically linked executable binaries for multiple platforms](https://bioinf.shenwei.me/kmcp/download) (Linux/Windows/macOS, AMD64/ARM64).\n    - No dependencies, no configurations.\n    - `conda install -c bioconda kmcp`\n- **Easy to use**\n    - Supporting [shell autocompletion](https://bioinf.shenwei.me/kmcp/usage/#autocompletion).\n    - Detailed [usage](https://bioinf.shenwei.me/kmcp/usage), [database](https://bioinf.shenwei.me/kmcp/database),\n      [tutorials](https://bioinf.shenwei.me/kmcp/tutorial), and [FAQs](https://bioinf.shenwei.me/kmcp/faq).\n- **Building database is easy and fast**\n    - [~25 min for 47894 genomes from GTDB-r202](https://bioinf.shenwei.me/kmcp/benchmark/searching/#kmcp-vs-cobs) on a sever with 40 CPU threads and solid disk drive.\n- **Fast searching speed (for sequence/genome search)**\n    - The index structure is modified from COBS, while [KMCP is 2x-10x faster](https://bioinf.shenwei.me/kmcp/benchmark/searching/#result) in sequence searching.\n    - Automatically scales to exploit all available CPU cores.\n    - Searching time is linearly related to the number of reference genomes (chunks).\n- **Scalable searching**. *Searching results against multiple databases can be fast merged*.\n    This brings many benefits:\n    - *There's no need to re-built the database with newly added reference genomes*. \n    - The searching step can be parallelized with a computer cluster in which each computation node searches against a small database.\n    - Computers with limited main memory can also utilize an extensive collection of reference genomes by building and searching against small databases..\n- **Accurate taxonomic profiling**\n    - Some k-mer based taxonomic profilers suffer from high false positive rates,\n      while [KMCP adopts multiple strategies](https://bioinf.shenwei.me/kmcp/tutorial/profiling/#methods)\n      to **improve specificity and keeps high sensitivity at the same time**.\n    - In addition to archaea and bacteria, KMCP performed well on **viruses/phages**.\n    - KMCP also provides **confident infectious pathogen detection**.\n    - [Preset six modes for multiple scenarios](https://bioinf.shenwei.me/kmcp/tutorial/profiling/#profiling-modes).\n    - [Supports CAMI and MetaPhlAn profiling format](https://bioinf.shenwei.me/kmcp/tutorial/profiling/#profiling-result-formats).\n- **Flexible support of taxonomy database**\n    - Taxonomy data, in the format of NCBI taxdump files, are only needed in the profiling step.\n      Therefore, it is easy to utilize an updated version of taxonomy data.\n    - GTDB, ICTV and custom taxonomy database are supported by creating taxdump files with [taxonkit create-taxdump](https://bioinf.shenwei.me/taxonkit/usage/#create-taxdump).\n    - [You can even merge the GTDB taxonomy (for prokaryotic genomes from GTDB) and NCBI taxonomy (for genomes from NCBI)](https://bioinf.shenwei.me/kmcp/database/#merging-gtdb-and-ncbi-taxonomy).\n    - Profiling without taxonomy data is also supported by setting `--level strain` in `kmcp profile`.\n\n\u003chr/\u003e\n    \n\u003cimg src=\"kmcp.jpg\" alt=\"\" width=\"800\"/\u003e\n\n\u003chr/\u003e\n\n\n## Installation\n\n[![Latest Version](https://img.shields.io/github/release/shenwei356/kmcp.svg?style=flat?maxAge=86400)](https://github.com/shenwei356/kmcp/releases)\n[![Github Releases](https://img.shields.io/github/downloads/shenwei356/kmcp/latest/total.svg?maxAge=3600)](http://bioinf.shenwei.me/kmcp/download/)\n[![Cross-platform](https://img.shields.io/badge/platform-any-ec2eb4.svg?style=flat)](http://bioinf.shenwei.me/kmcp/download/)\n[![Anaconda Cloud](https://anaconda.org/bioconda/kmcp/badges/version.svg)](https://anaconda.org/bioconda/kmcp)\n\nDownload [executable binaries](https://github.com/shenwei356/kmcp/releases),\nor install using conda:\n\n    conda install -c bioconda kmcp\n    \nSIMD extensions including `AVX512`, `AVX2`, `SSE2` are sequentially detected and used\nin two packages for better searching performance.\n\n- [pand](https://github.com/shenwei356/pand),\n  for accelerating searching on databases constructed with multiple hash functions.\n- [pospop](https://github.com/shenwei356/pospop/tree/short-input3),\n  for batch counting matched k-mers in bloom filters.\n\nARM architecture is supported, but `kmcp search` would be slower.\n\n## Commands\n\n|Subcommand                                                                |Function                                                        |\n|:-------------------------------------------------------------------------|:---------------------------------------------------------------|\n|[**compute**](https://bioinf.shenwei.me/kmcp/usage/#compute)              |Generate k-mers (sketch) from FASTA/Q sequences                 |\n|[**index**](https://bioinf.shenwei.me/kmcp/usage/#index)                  |Construct a database from k-mer files                           |\n|[**search**](https://bioinf.shenwei.me/kmcp/usage/#search)                |Search sequences against a database                             |\n|[**merge**](https://bioinf.shenwei.me/kmcp/usage/#merge)                  |Merge search results from multiple databases                    |\n|[**profile**](https://bioinf.shenwei.me/kmcp/usage/#profile)              |Generate the taxonomic profile from search results              |\n|[utils split-genomes](https://bioinf.shenwei.me/kmcp/usage/#split-genomes)|Split genomes into chunks                                       |\n|[utils unik-info](https://bioinf.shenwei.me/kmcp/usage/#unik-info)        |Print information of .unik files                                |\n|[utils index-info](https://bioinf.shenwei.me/kmcp/usage/#index-info)      |Print information of index files                                |\n|[utils index-density](https://bioinf.shenwei.me/kmcp/usage/#index-density)|Plot the element density of bloom filters for an index file     |\n|[utils ref-info](https://bioinf.shenwei.me/kmcp/usage/#ref-info)          |Print information of reference chunks in a database             |\n|[utils cov2simi](https://bioinf.shenwei.me/kmcp/usage/#icov2simi)         |Convert k-mer coverage to sequence similarity                   |\n|[utils query-fpr](https://bioinf.shenwei.me/kmcp/usage/#query-fpr)        |Compute the false positive rate of a query                      |\n|[utils filter](https://bioinf.shenwei.me/kmcp/usage/#filter)              |Filter search results and find species/assembly-specific queries|\n|[utils merge-regions](https://bioinf.shenwei.me/kmcp/usage/#merge-regions)|Merge species/assembly-specific regions                         |\n\n## Quickstart\n\n![](kmcp-workflow.jpg)\n\n    # compute k-mers\n    kmcp compute -k 21 --split-number 10 --split-overlap 150 \\\n        --in-dir genomes/ --out-dir genomes-k21-n10\n\n    # index k-mers\n    kmcp index --false-positive-rate 0.1 --num-hash 1 \\\n        --in-dir genomes-k21-n10/ --out-dir genomes.kmcp\n    \n    # delete temporary files\n    # rm -rf genomes-k21-n10/\n    \n    # search    \n    kmcp search --db-dir genomes.kmcp/ test.fa.gz --out-file search.kmcp@db1.kmcp.tsv.gz\n\n    # merge search results against multiple databases\n    kmcp merge -o search.kmcp.tsv.gz search.kmcp@*.kmcp.tsv.gz\n\n    # profile and binning\n    kmcp profile search.kmcp.tsv.gz \\\n        --taxid-map        taxid.map \\\n        --taxdump          taxdump/ \\\n        --out-file         search.tsv.gz.k.profile \\\n        --metaphlan-report search.tsv.gz.m.profile \\\n        --cami-report      search.tsv.gz.c.profile \\\n        --binning-result   search.tsv.gz.binning.gz\n\nNext:\n\n- [Demo of taxonomic profiling](https://github.com/shenwei356/kmcp/tree/main/demo-profiling)\n- [Tutorial of taxonomic profiling](https://bioinf.shenwei.me/kmcp/tutorial/profiling)\n\n## KMCP vs COBS\n\nWe reimplemented and modified the Compact Bit-Sliced Signature index ([COBS](https://github.com/bingmann/cobs)) algorithm,\nbringing a smaller index size and [much faster searching speed (2x for genome search and 10x for short reads) faster than COBS](https://bioinf.shenwei.me/kmcp/benchmark/searching/#result).\n\n\nThe differences between KMCP and COBS\n\n|Category       |Item                  |COBS                                 |KMCP                                                               |Comment                                                                                           |\n|:--------------|:----------------------|:------------------------------------|:------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------|\n|Algorithm      |K-mer hashing          |xxhash                               |ntHash1                                                            |xxHash is a general-purpose hashing function while ntHash is a recursive hash function for DNA/RNA|\n|               |Bloom filter hashing   |xxhash                               |Using k-mer hash values                                            |Avoid hash computation                                                                            |\n|               |Multiple-hash functions|xxhash with different seeds          |Generating multiple values from a single one                       |Avoid hash computation                                                                            |\n|               |Single-hash function   |Same as multiple-hash functions      |Separated workflow                                                 |Reducing loops                                                                                    |\n|               |AND step               |Serial bitwise AND                   |Vectorised bitwise AND                                             |Bitwise AND for \u003e1 hash functions                                                                 |\n|               |PLUS step              |Serial bit-unpacking                 |Vectorised positional popcount with pospop                         |Counting from bit-packed data                                                                     |\n|Index structure|Size of blocks         |/                                    |Using extra thresholds to split the last block with the most k-mers|Uneven genome size distribution would make bloom filters of the last block extremely huge         |\n|               |Index files            |Concatenated                         |Independent                                                        |                                                                                                  |\n|               |Index loading          |mmap, loading complete index into RAM|mmap, loading complete index into RAM, seek                        |Index loading modes                                                                               |\n|Input/output   |Input files            |FASTA/Q, McCortex, text              |FASTA/Q                                                            |                                                                                                  |\n|               |Output                 |Target and matched k-mers            |Target, matched k-mers, query FPR, etc.                            |                                                                                                  |\n\n\n## Support\n\nPlease [open an issue](https://github.com/shenwei356/kmcp/issues) to report bugs,\npropose new functions, or ask for help.\n\n## License\n\n[MIT License](https://github.com/shenwei356/kmcp/blob/master/LICENSE)\n\n## Acknowledgments\n\n- [Zhi-Luo Deng](https://dawnmy.github.io/CV/) (Helmholtz Centre for Infection Research, Germany)\n  gave a lot of valuable advice on metagenomic profiling and benchmarking.\n- [Robert Clausecker](https://github.com/clausecker/) (Zuse Institute Berlin, Germany)\n  wrote the high-performance vectorized positional popcount package \n  ([pospop](https://github.com/clausecker/pospop)) \n  [during my development of KMCP](https://stackoverflow.com/questions/63248047/),\n  which greatly accelerated the bit-matrix searching.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshenwei356%2Fkmcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshenwei356%2Fkmcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshenwei356%2Fkmcp/lists"}