{"id":25655563,"url":"https://github.com/joshengels/flinng","last_synced_at":"2025-11-04T09:05:25.274Z","repository":{"id":126653566,"uuid":"372711146","full_name":"JoshEngels/FLINNG","owner":"JoshEngels","description":"A fast high dimensional near neighbor search algorithm based on group testing and locality sensitive hashing","archived":false,"fork":false,"pushed_at":"2023-12-09T16:04:59.000Z","size":122486,"stargazers_count":23,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T13:42:35.768Z","etag":null,"topics":["group-testing","high-dimensional-data","locality-sensitive-hashing","nearest-neighbor-search"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JoshEngels.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-01T05:27:25.000Z","updated_at":"2025-04-16T10:58:43.000Z","dependencies_parsed_at":"2025-02-23T21:32:03.816Z","dependency_job_id":"44db46dc-f0bf-4a8c-8ccf-c6705b622d21","html_url":"https://github.com/JoshEngels/FLINNG","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JoshEngels/FLINNG","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoshEngels%2FFLINNG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoshEngels%2FFLINNG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoshEngels%2FFLINNG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoshEngels%2FFLINNG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JoshEngels","download_url":"https://codeload.github.com/JoshEngels/FLINNG/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JoshEngels%2FFLINNG/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264473885,"owners_count":23613960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["group-testing","high-dimensional-data","locality-sensitive-hashing","nearest-neighbor-search"],"created_at":"2025-02-23T21:31:57.923Z","updated_at":"2025-11-04T09:05:25.243Z","avatar_url":"https://github.com/JoshEngels.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# FLINNG\n\nFilters to Identify Near-Neighbor Groups (FLINNG) is a near neighbor search algorithm outlined in the paper \n[Practical Near Neighbor Search via Group Testing](https://arxiv.org/pdf/2106.11565.pdf). \n\nThis branch (the main branch) contains a moderately cleaned up version of FLINNG. To \naccess the original research code, use the research branch. Only this branch will \nbe actively updated.\n## Features\n\n- If using C++, header-only \n- If using Python, clean and simple bindings\n- Incremental/streaming index construction\n- Parallel index construction and querying\n\nNote that some features of the research branch have yet to be ported over, and there are a few improvements\nthis branch might soon receive:\n- Signed random projection and densified minhash performance optimization (sparsify SRP, improve DOPH)\n- Index dumping to and from disk\n- Addition of random seeds for reproducible experiments\n\n## Installation\n\nTo install Python bindings, run\n```\ngit clone --depth 1 https://github.com/JoshEngels/FLINNG\ncd FLINNG\nmake\nexport PYTHONPATH=$(pwd)/build:$PYTHONPATH\n```\n\nYou will need to have pybind11 installed with conda or pip. This has been tested\non an M1 Mac and on Windows 10 WSL with Ubuntu.\n\nIf you want to be able to use FLINNG without running the export command every time you\nstart a new terminal, add the export command to your .bashrc or another file that \ngets run on terminal startup.\n\nTo use the C++ headers, you just need to clone the repo and copy src/Flinng.h to your project. You \ncan also copy src/LshFunctions.h to hash your data before passing into Flinng.h; see\npybind/pybind.cpp for a direct example of how this works.\n\n\n## Usage/Examples\n\n### Python\nTo use FLINNG we must first create a new index, either a dense or a sparse index. \nA dense index will use the cosine similarity for the similarity search and accept\npoints as dense vectors in R^n (2D numpy array), while a sparse index will use the Jaccard similarity\nfor the similarity search and accept points as sets of positive integers \n(2D numpy array if all sets are the same length, otherwise python list of lists ). \n\nHere are the steps to use the FLINNG Python API:\n\nCreate a dense or sparse index:\n```python\ndense_index = flinng.dense_32_bit(\n                            num_rows, \n                            cells_per_row, \n                            data_dimension, \n                            num_hash_tables, \n                            hashes_per_table)\nsparse_index = flinng.sparse_32_bit(\n                            num_rows, \n                            cells_per_row, \n                            num_hash_tables, \n                            hashes_per_table,\n                            hash_range_pow)\n```\n\nAdd points to the index:\n```python\nindex.add_points(dataset)\n```\n\nPrepare for querying:\n```python\nindex.prepare_for_queries()\n```\n\nQuery:\n```python\nresults = index.query(queries, top_k)\n```\n\ntest/dense_data.py contains a complete example for running FLINNG on a synthetic dense data,\nexpected result ~100% R1@1. \ntest/promethion.py contains a complete example for running FLINNG on real sparse DNA \ndata, expected result ~98% R10@100. Make sure to read the comment at the beginning of promethion.py to see\nhow to download the dataset.\n\n\n### C++\nSimilar to the above, but example usage might look like:\n```C++\nauto flinng = Flinng(num_rows, cells_per_row, num_hashes, hash_range);\nstd::vector\u003cuint64_t\u003e dataHashes = getHashes(data); // You need to implement this yourself or use LshFunctions.h\nflinng.addPoints(hashes);\nflinng.prepareForQueries();\nstd::vector\u003cuint64_t\u003e queryHashes = getHashes(queries); // You need to implement this yourself or use LshFunctions.h\nauto results = flinng.query(queryHashes, topK);\n```\n\n\n## Authors\n\nImplementation by [Josh Engels](https://www.github.com/joshengels). \nFLINNG created in collaboration with [Ben Coleman](https://randorithms.com/about.html)\nand [Anshumali Shrivastava](https://www.cs.rice.edu/~as143/).\n\nPlease feel free to contact josh.adam.engels@gmail.com with any questions.\n\n## Contributing\n\nCurrently, contributions are limited to bug fixes and suggestions. \nFor a bug fix, feel free to submit a PR or send an email. \n\n## Citations\n\nIf you found our work useful, please cite our work as follows:\n\n```\n@inproceedings{NEURIPS2021_5248e511,\n author = {Engels, Joshua and Coleman, Benjamin and Shrivastava, Anshumali},\n booktitle = {Advances in Neural Information Processing Systems},\n editor = {M. Ranzato and A. Beygelzimer and Y. Dauphin and P.S. Liang and J. Wortman Vaughan},\n pages = {9950--9962},\n publisher = {Curran Associates, Inc.},\n title = { Practical Near Neighbor Search via Group Testing},\n url = {https://proceedings.neurips.cc/paper_files/paper/2021/file/5248e5118c84beea359b6ea385393661-Paper.pdf},\n volume = {34},\n year = {2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshengels%2Fflinng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoshengels%2Fflinng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshengels%2Fflinng/lists"}