{"id":33308252,"url":"https://github.com/princeton-ddss/lsh","last_synced_at":"2025-11-19T01:01:16.509Z","repository":{"id":320577718,"uuid":"1082613609","full_name":"princeton-ddss/lsh","owner":"princeton-ddss","description":"DuckDB community extension for locality-sensitive hashing (LSH)","archived":false,"fork":false,"pushed_at":"2025-11-18T16:12:53.000Z","size":109,"stargazers_count":5,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-18T18:06:54.982Z","etag":null,"topics":["approximate-nearest-neighbor-search","data-matching","deduplication","duckdb","duckdb-community","duckdb-extension","entity-resolution","fuzzy-matching","locality-sensitive-hashing","lsh","record-linkage"],"latest_commit_sha":null,"homepage":"https://duckdb.org/community_extensions/extensions/lsh","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/princeton-ddss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-24T13:56:59.000Z","updated_at":"2025-11-18T16:22:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"a6b2a393-0506-4b35-a62a-227f9bc7b3ae","html_url":"https://github.com/princeton-ddss/lsh","commit_stats":null,"previous_names":["princeton-ddss/minhash","princeton-ddss/lsh"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/princeton-ddss/lsh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-ddss%2Flsh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-ddss%2Flsh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-ddss%2Flsh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-ddss%2Flsh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/princeton-ddss","download_url":"https://codeload.github.com/princeton-ddss/lsh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/princeton-ddss%2Flsh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285164323,"owners_count":27125519,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-18T02:00:05.759Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","data-matching","deduplication","duckdb","duckdb-community","duckdb-extension","entity-resolution","fuzzy-matching","locality-sensitive-hashing","lsh","record-linkage"],"created_at":"2025-11-19T01:00:43.104Z","updated_at":"2025-11-19T01:01:16.503Z","avatar_url":"https://github.com/princeton-ddss.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Locality-Sensitive Hashing (LSH) DuckDB Extension\n\nDuckDB extension for [locality-sensitive hashing (LSH)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing),\nusing the Rust implementations from the [`zoomerjoin`](https://github.com/beniaminogreen/zoomerjoin) R package.\n(For a conceptual review and a description of that package,\nsee [https://doi.org/10.21105/joss.05693](https://doi.org/10.21105/joss.05693).)\n\n## Installation\n\n`lsh` is a [DuckDB Community Extension](https://github.com/duckdb/community-extensions).\n\nIt can be installed and loaded in DuckDB like so:\n\n```sql\nINSTALL lsh FROM community;\nLOAD lsh;\n```\n\n## Available Functions\n\n### MinHash (for Strings)\n\n- 64-bit: `lsh_min(string, ngram_width, band_count, band_size, seed)`\n- 32-bit: `lsh_min32(string, ngram_width, band_count, band_size, seed)`\n\n```sql\nCREATE TEMPORARY TABLE temp_names (\n    name VARCHAR\n);\n\nINSERT INTO temp_names (name) VALUES\n    ('Alice Johnson'),\n    ('Robert Smith'),\n    (NULL),\n    ('Charlotte Brown'),\n    ('David Martinez'),\n    ('Emily Davis'),\n    ('Michael Wilson'),\n    ('Sophia Taylor'),\n    (NULL),\n    ('James Anderson'),\n    ('Olivia Thomas'),\n    ('Benjamin Lee');\n\nSELECT lsh_min(name, 2, 3, 2, 123) AS hash FROM temp_names;\n```\n\n```\n┌──────────────────────────────────────────────────────────────────┐\n│                               hash                               │\n│                             uint64[]                             │\n├──────────────────────────────────────────────────────────────────┤\n│ [13571929851950895096, 9380027513982184887, 2973452616913389687] │\n│ [8779492002049334510, 6213046290947405081, 13321761559668221936] │\n│ NULL                                                             │\n│ [17147317566672094549, 9868884775472345505, 9544039307031965287] │\n│ [8205471107123956470, 3856457550471365223, 160978381860159594]   │\n│ [5031590273592478399, 2643794611755346220, 10496886524478706543] │\n│ [7351019434982270461, 11969544284460938578, 1096653296545732983] │\n│ [947309311728102588, 6485027977500841069, 11465726828575944543]  │\n│ NULL                                                             │\n│ [6511242524203601686, 5368660891928216176, 4531328875985401258]  │\n│ [6134578107120707744, 8471287122008225606, 13561556383590060017] │\n│ [7926739398273580158, 2501438919389423193, 17085734390799214704] │\n├──────────────────────────────────────────────────────────────────┤\n│                             12 rows                              │\n└──────────────────────────────────────────────────────────────────┘\n```\n\n### Euclidean Hashing (for Points)\n\n- 64-bit: `lsh_euclidean(array, bucket_width, band_count, band_size, seed)`\n- 32-bit: `lsh_euclidean32(array, bucket_width, band_count, band_size, seed)`\n\n```sql\nCREATE OR REPLACE TEMPORARY TABLE temp_vals (\n    val DOUBLE[5],\n);\n\nINSERT INTO temp_vals (val) VALUES\n    (ARRAY[1.1, 2.2, 3.3, 5.8, 3.9]),\n    (NULL),\n    (ARRAY[4.5, 5.5, 2.3, 1.8, 6.3]),\n    (ARRAY[7.3, 1.2, 9.6, 5.5, 7.8]),\n    (NULL),\n    (ARRAY[6.7, 4.5, 1.3, 0.6, 4.6]);\n\nSELECT lsh_euclidean(val, 0.5, 2, 3, 123) AS hash FROM temp_vals;\n```\n\n```\n┌─────────────────────────────────────────────┐\n│                    hash                     │\n│                  uint64[]                   │\n├─────────────────────────────────────────────┤\n│ [4153593470791884295, 13333357882440433242] │\n│ NULL                                        │\n│ [9539244981710099531, 8978554412800410753]  │\n│ [3765229951789618540, 13117769235920535871] │\n│ NULL                                        │\n│ [13333357882440433242, 2643848813970240984] │\n└─────────────────────────────────────────────┘\n```\n\n### Jaccard Similarity (for Strings)\n\n- `lsh_jaccard(string_left, string_right, ngram_width)`\n\n```sql\nCREATE TEMPORARY TABLE temp_names (\n    name_a VARCHAR,\n    name_b VARCHAR\n);\n\nINSERT INTO temp_names (name_a, name_b) VALUES\n    ('Charlotte Brown', 'Charlene Browning'),\n    ('David Martinez', 'Davis Martin'),\n    ('Olivia Thomas', 'Olive Thomason'),\n    ('Alice Johnson', NULL),\n    (NULL, 'Roberta Mills'),\n    ('Emily Davis', 'Laura Bennett'),\n    ('Michael Wilson', 'Mike Wilson'),\n    ('James Anderson', 'Jamie Anders'),\n    ('Sophia Taylor', NULL),\n    ('Benjamin Lee', 'Christopher Grant');\n\nSELECT lsh_jaccard(name_a, name_b, 2) AS similarity FROM temp_names;\n```\n\n```\n┌────────────┐\n│ similarity │\n│   double   │\n├────────────┤\n│        0.5 │\n│        0.6 │\n│     0.5625 │\n│       NULL │\n│       NULL │\n│        0.0 │\n│     0.4375 │\n│        0.5 │\n│       NULL │\n│        0.0 │\n├────────────┤\n│  10 rows   │\n└────────────┘\n```\n\n## Suggested Usage\n\nWe do not recommend creating and storing the full `ARRAY::[band_count]`-type columns,\nas they become large very quickly. Instead, we recommend generating bands on-the-fly\nin join conditions (i.e., when generating comparisons/potential matches). This reduces\nstorage needs and memory consumption. Further, we note that statements generating a set\nof *unique* row pairs based on the output of these functions may be slower than producing\ncomparison pairs *then filtering to matches* within each band (*then* taking the union)\nif the filtering/comparison function(s) are not computationally intensive.\n\nFor example, to identify record pairs satisfying `Jaccard(A.col, B.col) \u003e 0.8` between\ntables `A` and `B` using bigram MinHashing (`band_count = 2, band_size = 3`) to generate\ncomparison pairs, we recommend the following syntax, where each call to `lsh_min()` produces\na single-element array. Holding the seed fixed within join calls and rotating it across\ncalls fixes the hash functions *within* each join but effectively produces additional bands\n*across* each join.\n\n```sql\nSELECT A.ind, B.id\nFROM A\nINNER JOIN B\nON lsh_min(A.col, 2, 1, 3, 1)[1] = lsh_min(A.col, 2, 1, 3, 1)[1]\nWHERE lsh_jaccard(A.col, B.col, 2) \u003e 0.8\n\nUNION\n\nSELECT A.ind, B.id\nFROM A\nINNER JOIN B\nON lsh_min(A.col, 2, 1, 3, 2)[1] = lsh_min(A.col, 2, 1, 3, 2)[1]\nWHERE lsh_jaccard(A.col, B.col, 2) \u003e 0.8\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprinceton-ddss%2Flsh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprinceton-ddss%2Flsh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprinceton-ddss%2Flsh/lists"}