{"id":16780025,"url":"https://github.com/ekzhu/lshensemble","last_synced_at":"2025-07-07T15:36:16.091Z","repository":{"id":40791768,"uuid":"68092131","full_name":"ekzhu/lshensemble","owner":"ekzhu","description":"LSH index for approximate set containment search","archived":false,"fork":false,"pushed_at":"2022-06-27T20:55:14.000Z","size":58,"stargazers_count":57,"open_issues_count":1,"forks_count":12,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-04T02:02:12.731Z","etag":null,"topics":["approximate-nearest-neighbor-search","containment","containment-search","lsh","lsh-ensemble","lsh-forest","nearest-neighbor-search","paper","vldb"],"latest_commit_sha":null,"homepage":"http://www.vldb.org/pvldb/vol9/p1185-zhu.pdf","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ekzhu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-13T08:59:51.000Z","updated_at":"2024-08-12T14:47:25.000Z","dependencies_parsed_at":"2022-09-03T11:30:20.459Z","dependency_job_id":null,"html_url":"https://github.com/ekzhu/lshensemble","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ekzhu/lshensemble","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhu%2Flshensemble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhu%2Flshensemble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhu%2Flshensemble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhu%2Flshensemble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ekzhu","download_url":"https://codeload.github.com/ekzhu/lshensemble/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ekzhu%2Flshensemble/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264103691,"owners_count":23557981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","containment","containment-search","lsh","lsh-ensemble","lsh-forest","nearest-neighbor-search","paper","vldb"],"created_at":"2024-10-13T07:33:44.408Z","updated_at":"2025-07-07T15:36:16.073Z","avatar_url":"https://github.com/ekzhu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LSH Ensemble\n\n[![Build Status](https://travis-ci.org/ekzhu/lshensemble.svg?branch=master)](https://travis-ci.org/ekzhu/lshensemble)\n[![GoDoc](https://godoc.org/github.com/ekzhu/lshensemble?status.svg)](https://godoc.org/github.com/ekzhu/lshensemble)\n[![DOI](https://zenodo.org/badge/68092131.svg)](https://zenodo.org/badge/latestdoi/68092131)\n\n\n[Documentation](https://godoc.org/github.com/ekzhu/lshensemble)\n\nPlease cite this paper if you use this library in your work:\n\u003eErkang Zhu, Fatemeh Nargesian, Ken Q. Pu, Renée J. Miller:\n\u003eLSH Ensemble: Internet-Scale Domain Search. PVLDB 9(12): 1185-1196 (2016)\n\u003e[Link](http://www.vldb.org/pvldb/vol9/p1185-zhu.pdf)\n\n[Presentation slides](http://ekzhu.com/talks/lshensemble-vldb2016.pdf) @ VLDB 2016, New Delhi.\n\n## Datasets\n\nWe used two datasets for evaluation. The datasets are all from public domains\nand can be downloaded directly from the original publisher.\n\n* [Canadian Open Data, tabular domains only (as of June 2015)](https://www.dropbox.com/s/a9qbgloyvhrnrgu/canadian_open_data_tabular_domains_only.tar.gz?dl=0):\nEach file corresponds to a single domain extracted from a column of\na table, which could be a spreadsheet or an CSV file. \nThe filenames follow the `\u003cdata file name\u003e.\u003ccolumn id\u003e` format.\n* [2015 WDC Web Tables, English Relational, 51 compressed files](http://data.dws.informatik.uni-mannheim.de/webtables/2015-07/englishCorpus/compressed):\nSee the data format [here](http://webdatacommons.org/webtables/2015/downloadInstructions.html).\n\nBy using these datasets you agree to use them for academic research purpose\nonly, and we shall not be held responisble for any \ninaccuracy or error that may exist in the \ndatasets, nor we shall be responsible for any consequence of usage of these\ndatasets.\n\n## Quick Start Guide\n\nInstall this library by running:\n\n```\ngo get github.com/ekzhu/lshensemble\n```\n\nImport the library in your `import`:\n\n```go\nimport (\n\t\"github.com/ekzhu/lshensemble\"\n)\n```\n\nFirst you need to obtain the domains, and each domain should have a string ID.\nFor simplicity we represent a domain as `map[string]bool`, whose keys are distinct values.\nAssuming you have obtained the domains from some dataset,\nyou can generate the MinHash signatures from the domains.\n\n```go\ndomains []map[string]bool\nkeys []string // each key corresponds to the domain at the same index\n\n// ... \n// obtaining domains and keys\n// ...\n\n// initializing the domain records to hold the MinHash signatures\ndomainRecords := make([]*lshensemble.DomainRecord, len(domains))\n\n// set the minhash seed\nseed := 42\n\n// set the number of hash functions\nnumHash := 256\n\n// create the domain records with the signatures\nfor i := range domains {\n\tmh := lshensemble.NewMinhash(seed, numHash)\n\tfor v := range domains[i] {\n\t\tmh.Push([]byte(v))\n\t}\n\tdomainRecords[i] := \u0026lshensemble.DomainRecord {\n\t\tKey       : keys[i],\n\t\tSize      : len(domains[i]),\n\t\tSignature : mh.Signature()\n\t}\n}\n```\n\nBefore you can index the domains, you need to sort them in increasing order by\ntheir sizes. `BySize` wrapper allows the domains to tbe sorted using the build-in `sort`\npackage.\n\n```go\nsort.Sort(lshensemble.BySize(domainRecords))\n```\n\nNow you can use `BootstrapLshEnsembleOptimal`/`BootstrapLshEnsembleEquiDepth`\n(or `BootstrapLshEnsemblePlusOptimal`/`BootstrapLshEnsemblePlusEquiDepth`) \nfor better accuracy at higher memory cost\\*) \nto create an LSH Ensemble index. \n`BootstrapLshEnsembleOptimal` uses dynamic programming to create partitions that\nare optimal in the sense that the total number of false positives generated from\nall the partitions are minimized. This method can be\na bit slower due to the dynamic programming overhead, however, it creates\noptimized partitions for any kind of data distribution.\n`BootstrapLshEnsembleEquiDepth` uses simple equi-depth -- same number of domains\nin every partition. This is method is described in the original \n[paper](http://www.vldb.org/pvldb/vol9/p1185-zhu.pdf) as suitable for power-law\ndistributed domain sizes, which is common in real-world domains.\nYou need to\nspecify the number of partitions to use and some other parameters.\nThe LSH parameter K (number of hash functions per band) is dynamically tuned at query-time,\nbut the maximum value should be specified here.\n\n\\* See [explanation](#maxk-explanation) for the reason for the \"Plus\" version.\n\n```go\n// Set the number of partitions\nnumPart := 8\n\n// Set the maximum value for the MinHash LSH parameter K \n// (number of hash functions per band).\nmaxK := 4\n\n// Create index using equi-depth partitioning\n// You can also use BootstrapLshEnsemblePlusEquiDepth for better accuracy\nindex_eqd, err := lshensemble.BootstrapLshEnsembleEquiDepth(numPart, numHash, maxK, \n    len(domainRecords), lshensemble.Recs2Chan(domainRecords))\nif err != nil {\n\tpanic(err)\n}\n\n// Create index using optimal partitioning\n// You can also use BootstrapLshEnsemblePlusOptimal for better accuracy\nindex_opt, err := lshensemble.BootstrapLshEnsembleOptimal(numPart, numHash, maxK,\n    func () \u003c-chan *lshensemble.DomainRecord { \n        return lshensemble.Recs2Chan(domainRecords); \n    })\nif err != nil {\n\tpanic(err)\n}\n```\n\nFor better memory efficiency when the number of domains is large, \nit's wiser to use Golang channels and goroutines\nto pipeline the generation of the signatures, and then use disk-based sorting to sort the domain records. \nThis is why `BootstrapLshEnsembleEquiDepth` accepts a channel of `*DomainRecord` as input.\nFor a small number of domains, you simply use `Recs2Chan` to convert the sorted slice of `*DomainRecord`\ninto a `chan *DomainRecord`.\nTo help serializing the domain records to disk, you can use `SerializeSignature`\nto serialize the signatures.\nYou need to come up with your own serialization schema for the keys and sizes.\n\nLastly, you can use `Query` function, which returns a Golang channel of \nthe keys of the *candidates* domains, which may contain false positives - domains that do not\nmeet the containment threshold.\nTherefore, you can optionally include a post-processing step to remove\nthe false positive domains using the original domain values.\n\n```go\n// pick a domain to use as the query\nquerySig := domainRecords[0].Signature\nquerySize := domainRecords[0].Size\n\n// set the containment threshold\nthreshold := 0.5\n\n// get the keys of the candidate domains (may contain false positives)\n// through a channel with option to cancel early.\ndone := make(chan struct{})\ndefer close(done) // Important!!\nresults := index.Query(querySig, querySize, threshold, done)\n\nfor key := range results {\n\t// ...\n\t// You may want to include a post-processing step here to remove \n\t// false positive domains using the actual domain values.\n\t// ...\n\t// You can call break here to stop processing results.\n}\n```\n\n## Run Canadian Open Data Benchmark\n\nFirst you need to download the [Canadian Open Data domains](https://github.com/ekzhu/lshensemble#datasets)\nand extract the domain files into a directory called `_cod_domains` by running the following command.\n\n```\ntar xzf canadian_open_data_tabular_domains_only.tar.gz\n```\n\nUse Golang's `test` tool to start the benchmark:\n\n```\ngo test -bench=Benchmark_CanadianOpenData -timeout=24h\n```\n\nThe benchmark process is in the following order:\n\n1. Read the domain files into memory\n2. Run Linear Scan to get the ground truth\n3. Run LSH Ensemble to get the query results\n4. Run the accuracy analysis to generate a report on precisions and recalls\n\n## \u003ca name=\"maxk-explanation\"\u003e\u003c/a\u003eExplanation for the Parameter `MaxK` and Bootstrap Options\n\nMinHash LSH has two parameters `K` and `L` (in the \n[paper](http://www.vldb.org/pvldb/vol9/p1185-zhu.pdf)\nI used `r` and `b` respectively). \n`L` is the number of \"bands\" and `K` is the number of hash functions per band. \nThe details about the two parameters can be found in\nChapter 3 of the textbook,\n[Mining of Massive Datasets](http://infolab.stanford.edu/~ullman/mmds/book.pdf).\n\nIn LSH Ensemble, we want to allow the `K` and `L` of the LSH index in every partition to\nvary at query time, in order to optimize them for any given query \n(see Section 5.5 of the paper).\nWe can use achive this by using multiple MinHash LSH, one for each value of `K`.\nThis allows us to vary the parameter `K` and `L` in the following space:\n```\nK * L \u003c= number of hash functions (let this be H)\n1 \u003c= K \u003c= H\n```\nHowever, this means that for every possible value of `K` from 1 to `H`, \nwe need to create a MinHash LSH -- very expensive.\nSo it is not wise to allow `K` to vary from 1 to `H`, \nand that's why we have a `MaxK` parameter, which bounds `K` and saves memory. \nSo the new parameter space is:\n```\nK * L \u003c= H\n1 \u003c= K \u003c= MaxK\n```\nIt is important to note that it is not the case for `L`, \nbecause we can choose how many \"bands\" to use at query time.\n\nNow, if we use [LSH Forest](http://ilpubs.stanford.edu:8090/678/1/2005-14.pdf),\nwe can vary the parameter `K` from 1 to `MaxK` at query time with just one LSH. \nYou can read the paper to understand how this can be done \n(hint: prefix tree). \nThis comes at a price -- the parameter space is more restricted:\n```\nMaxK * L \u003c= H\n1 \u003c= K \u003c= MaxK\n```\nEssentially, we have less freedom in varying `L`, as \n`1 \u003c= L \u003c= min{H / MaxK, H}` base on the above constraints.\n\nIn this library for LSH Ensemble, we provide both implmentations \n(LSH Forest and \"vanilla\" MinHash LSH ).\nSpecifically, \n* `BootstrapLshEnsembleEquiDepth` and `BootstrapLshEnsembleOptimal` \nbuild the index using the LSH Forest implementation, \nwhich use less memory but with a more restricted parameter space for optimization.\n* `BootstrapLshEnsemblePlusEquiDepth` and `BootstrapLshEnsemblePlusOptimal` \nbuild the index using the \"vanilla\" MinHash LSH\nimplementation (one LSH for every `K`), which uses more memory (bounded by `MaxK`)\nbut with no restriction on `L`.\n\nWe found that the optimal `K` for most queries are less than 4. So in practice you\ncan just set `MaxK` to 4.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekzhu%2Flshensemble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fekzhu%2Flshensemble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fekzhu%2Flshensemble/lists"}