{"id":17043550,"url":"https://github.com/duhaime/minhash","last_synced_at":"2025-04-12T15:11:33.784Z","repository":{"id":30514851,"uuid":"124752750","full_name":"duhaime/minhash","owner":"duhaime","description":"Quickly estimate the similarity between many sets","archived":false,"fork":false,"pushed_at":"2022-12-03T03:34:22.000Z","size":1036,"stargazers_count":51,"open_issues_count":20,"forks_count":11,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T09:45:13.081Z","etag":null,"topics":["locality-sensitive-hashing","lsh","minhash","text-mining"],"latest_commit_sha":null,"homepage":"https://duhaime.github.io/minhash/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/duhaime.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-11T12:10:01.000Z","updated_at":"2024-11-26T08:50:29.000Z","dependencies_parsed_at":"2023-01-14T17:15:25.994Z","dependency_job_id":null,"html_url":"https://github.com/duhaime/minhash","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duhaime%2Fminhash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duhaime%2Fminhash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duhaime%2Fminhash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/duhaime%2Fminhash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/duhaime","download_url":"https://codeload.github.com/duhaime/minhash/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586230,"owners_count":21128997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["locality-sensitive-hashing","lsh","minhash","text-mining"],"created_at":"2024-10-14T09:29:51.038Z","updated_at":"2025-04-12T15:11:33.766Z","avatar_url":"https://github.com/duhaime.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\n                  _           _                     _             _\n      _ __ ___   (_)  _ __   | |__     __ _   ___  | |__         (_)  ___\n     | '_ ` _ \\  | | | '_ \\  | '_ \\   / _` | / __| | '_ \\        | | / __|\n     | | | | | | | | | | | | | | | | | (_| | \\__ \\ | | | |  _    | | \\__ \\\n     |_| |_| |_| |_| |_| |_| |_| |_|  \\__,_| |___/ |_| |_| (_)  _/ | |___/\n                                                               |__/\n```\n\n[![Build Status](https://travis-ci.org/duhaime/minhash.svg?branch=master)](https://travis-ci.org/duhaime/minhash)\n\n[Minhashing](https://en.wikipedia.org/wiki/MinHash) is an efficient similarity estimation technique that is often used to identify near-duplicate documents in large text collections. This package offers a JavaScript implementation of the minhash algorithm and an efficient [Locality Sensitive Hashing Index](https://en.wikipedia.org/wiki/Locality-sensitive_hashing) for finding similar minhashes in Node.js or web applications.\n\n## Installation\n\nTo get started with Minhash.js, you can install the package with npm:\n\n```bash\nnpm install minhash --save\n```\n\nIf you prefer, you can instead load the package directly in a browser:\n\n```html\n\u003cscript src='https://cdn.jsdelivr.net/gh/duhaime/minhash@master/minhash.min.js'\u003e\u003c/script\u003e\n```\n\n#### Minhash Usage\n\nMinhashes are hash representations of the contents within a set. The following example minhashes and then estimates the [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index) between two sets:\n\n```javascript\nimport { Minhash } from 'minhash'; // If using Node.js\n\nvar s1 = ['minhash', 'is', 'a', 'probabilistic', 'data', 'structure', 'for',\n        'estimating', 'the', 'similarity', 'between', 'datasets'];\nvar s2 = ['minhash', 'is', 'a', 'probability', 'data', 'structure', 'for',\n        'estimating', 'the', 'similarity', 'between', 'documents'];\n\n// create a hash for each set of words to compare\nvar m1 = new Minhash();\nvar m2 = new Minhash();\n\n// update each hash\ns1.map(function(w) { m1.update(w) });\ns2.map(function(w) { m2.update(w) });\n\n// estimate the jaccard similarity between two minhashes\nm1.jaccard(m2);\n```\n\n#### LshIndex Usage\n\nWhile one can compare the Jaccard similarity between a minhash and all others in a collection, the complexity of doing so is O(n), as one needs to compare the query set to every other set.\n\nTo estimate the results of the same comparison in sub-linear time, one can instead build a [Locality Sensitive Hash Index](http://infolab.stanford.edu/~ullman/mmds/ch3.pdf), which maps hash sequences from a minhash signature to the list of document identifiers that contain the given hash sequence. Using this indexing technique, one can effectively find sets similar to a query set:\n\n```javascript\nimport { Minhash, LshIndex } from 'minhash'; // If using Node.js\n\nvar s1 = ['minhash', 'is', 'a', 'probabilistic', 'data', 'structure', 'for',\n        'estimating', 'the', 'similarity', 'between', 'datasets'];\nvar s2 = ['minhash', 'is', 'a', 'probability', 'data', 'structure', 'for',\n        'estimating', 'the', 'similarity', 'between', 'documents'];\nvar s3 = ['cats', 'are', 'tall', 'and', 'have', 'been',\n        'known', 'to', 'sing', 'quite', 'loudly'];\n\n// generate a hash for each list of words\nvar m1 = new Minhash();\nvar m2 = new Minhash();\nvar m3 = new Minhash();\n\n// update each hash\ns1.map(function(w) { m1.update(w) });\ns2.map(function(w) { m2.update(w) });\ns3.map(function(w) { m3.update(w) });\n\n// add each document to a Locality Sensitive Hashing index\nvar index = new LshIndex();\nindex.insert('m1', m1);\nindex.insert('m2', m2);\nindex.insert('m3', m3);\n\n// query for documents that appear similar to a query document\nvar matches = index.query(m1);\nconsole.log('Jaccard similarity \u003e= 0.5 to m1:', matches);\n```\n\n### Example\n\nThe [sample application](https://duhaime.github.io/minhash/) uses minhash.js to compute the similarity between several [sample documents](https://github.com/duhaime/minhash/tree/gh-pages/texts):\n\n![app preview](https://raw.githubusercontent.com/duhaime/minhash/master/images/preview.png)\n\nThere is also a sample Node.js script that can be run with `node examples/index.js`.\n\n### Development\n\nTo run the test suite — `npm run test`.\nTo compile and minify minhash.min.js — `npm run build`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduhaime%2Fminhash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fduhaime%2Fminhash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fduhaime%2Fminhash/lists"}