{"id":19377131,"url":"https://github.com/lgautier/mashing-pumpkins","last_synced_at":"2025-04-23T18:33:41.252Z","repository":{"id":57439874,"uuid":"78150710","full_name":"lgautier/mashing-pumpkins","owner":"lgautier","description":"Minhash and maxhash library in Python, combining flexibility, expressivity, and performance.","archived":false,"fork":false,"pushed_at":"2024-12-14T22:56:06.000Z","size":1466,"stargazers_count":21,"open_issues_count":4,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T18:51:40.762Z","etag":null,"topics":["dice-similarity","jaccard-similarity","locality-sensitive-hashing","minhash","minhash-sketches","murmurhash3","python","xxhash"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lgautier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-05T21:43:21.000Z","updated_at":"2024-12-14T22:16:38.000Z","dependencies_parsed_at":"2022-09-26T16:31:22.731Z","dependency_job_id":null,"html_url":"https://github.com/lgautier/mashing-pumpkins","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgautier%2Fmashing-pumpkins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgautier%2Fmashing-pumpkins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgautier%2Fmashing-pumpkins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgautier%2Fmashing-pumpkins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lgautier","download_url":"https://codeload.github.com/lgautier/mashing-pumpkins/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249664796,"owners_count":21308099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dice-similarity","jaccard-similarity","locality-sensitive-hashing","minhash","minhash-sketches","murmurhash3","python","xxhash"],"created_at":"2024-11-10T08:46:38.254Z","updated_at":"2025-04-23T18:33:41.239Z","avatar_url":"https://github.com/lgautier.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![logo](doc/_static/mashingpumpkins.png)](doc/_static/mashingpumpkins.png)\n# mashing-pumpkins\n\nFlexible-yet-pretty-fast minhashing-related library for Python \u003e= 3.5.\n\n[![Python package](https://github.com/lgautier/mashing-pumpkins/actions/workflows/package.yml/badge.svg)](https://github.com/lgautier/mashing-pumpkins/actions/workflows/package.yml)\n[![codecov](https://codecov.io/gh/lgautier/mashing-pumpkins/branch/master/graph/badge.svg)](https://codecov.io/gh/lgautier/mashing-pumpkins)\n[![Pypi release](https://img.shields.io/pypi/v/mashing-pumpkins.svg)](https://img.shields.io/pypi/v/mashing-pumpkins.svg)\n\nThe package is rather well documented with:\n- Sphinx documentation in [doc/](doc/) (start with [doc/index.rst](doc/index.rst))\n- Docstrings for most of the functions, classes, and methods\n- A showcase notebook in [doc/notebooks](doc/notebooks) (although this needs to be refreshed)\n- A demo command-line tool to build sketches in sourmash's JSON format (`python -m mashingpumpkins.demo.cmdline --help`) (although this is\n  historical and need to be refreshed or deleted)\n\n## Why Minhash sketches ?\n\nBottom-sketches (Minhash sketches) are samples of the elements present in a set.\nThey have been extensively used for text document matching or retrieval, which can\nextend to the context of genomics where strings are DNA or RNA sequences. There the\nset of k-mers present in a genome (called \"k-shingles\" in MinHash-related litterature),\nor in the reads from a sequencing assay, and they have been shown to be useful to\nmeasure similarity between genomes.\n\n## Why this implementation ?\n\n\nThe purpose of this implementation is to provide a library design that is combining flexibility and expressivity with performance\n(speed and memory usage).\n\n### Design\n\nThe design is allowing us to implement with a relatively short code base:\n\n- the use different hash functions (MurmurHash3, XXHash), and with user-specified seeds\n- Minhash and Maxhash sketches\n- \"Count sketches\"\n- Demonstrate quickly the comparative efficiency of alternative hashing strategies for double-stranded genomes (see - https://github.com/marbl/Mash/issues/45#issuecomment-274665746)\n\n### Performance\n\nThe implementation also happens to be pretty fast, making it a reasonable option as a building block for minhash-related research and prototypes.\n\n![perf](doc/_static/perf_benchmark.png)\n\n![perf](doc/_static/perf_benchmark_murmur3.png)\n\nAt the time of writing it is able to build a minhash sketch (k=31, size=1000) for a FASTQ file with ~21M reads (700MB when gzip-compressed)\non a laptop[*] in under 1'30\".\n\n```bash\n$ python -m mashingpumpkins.demo.cmdline --parser=fastqandfurious --ncpu=3 DRR065801.fastq.gz\nProcessing DRR065801.fastq.gz as a FASTQ file...\n    20853697 records in 1m20s (9.43 MB/s)\n```\n\n(*: ASUS ultrabook, dual-core with hyperthreading, running Linux)\n\n## Installation\n\nPython \u003e 3.8 and a C/C++ compiler (C99-aware) are pretty much everything that is needed. At the time of writing the CI\non Travis is testing with gcc and clang on Linux and OSX.\n\nWith pip, installing either latest release or the \"master\" branch can be done with:\n\n```bash\n# latest release\npip install mashing-pumpkins\n\n# master on github\npip install git+https://https://github.com/lgautier/mashing-pumpkins.git\n\n```\n\nThe installation can be tested with:\n\n```bash\npython -m pytest --pyargs mashingpumpkins\n\n# or:\n\npython -m pytest --cov=mashingpumpkins ---cov-report term\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgautier%2Fmashing-pumpkins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flgautier%2Fmashing-pumpkins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgautier%2Fmashing-pumpkins/lists"}