{"id":20469385,"url":"https://github.com/jorensix/tarsoslsh","last_synced_at":"2025-04-03T02:10:02.041Z","repository":{"id":7646722,"uuid":"9007292","full_name":"JorenSix/TarsosLSH","owner":"JorenSix","description":"A Java library implementing practical nearest neighbour search algorithm for multidimensional vectors that operates in sublinear time. It implements Locality-sensitive Hashing (LSH) and multi index hashing for hamming space.","archived":false,"fork":false,"pushed_at":"2020-07-26T11:35:16.000Z","size":23616,"stargazers_count":201,"open_issues_count":5,"forks_count":85,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-04-01T20:31:49.356Z","etag":null,"topics":["java","lsh","multi-dimensional-hashing","nearest-neighbor-search"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JorenSix.png","metadata":{"files":{"readme":"README.textile","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-03-25T13:57:36.000Z","updated_at":"2025-03-01T06:18:53.000Z","dependencies_parsed_at":"2022-09-10T14:40:09.319Z","dependency_job_id":null,"html_url":"https://github.com/JorenSix/TarsosLSH","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorenSix%2FTarsosLSH","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorenSix%2FTarsosLSH/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorenSix%2FTarsosLSH/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JorenSix%2FTarsosLSH/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JorenSix","download_url":"https://codeload.github.com/JorenSix/TarsosLSH/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246922247,"owners_count":20855345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","lsh","multi-dimensional-hashing","nearest-neighbor-search"],"created_at":"2024-11-15T14:08:53.959Z","updated_at":"2025-04-03T02:10:02.013Z","avatar_url":"https://github.com/JorenSix.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"h1. TarsosLSH - Locality Sensitive Hashing (LSH) in Java\n\nTarsosLSH is a Java library implementing sub-linear nearest neigbour search algorithms. It contains both an approximate and an exact search algorithm. The first, Locality-sensitive Hashing (\u003cabbr title=\"Locality-sensitive Hashing\"\u003eLSH\u003c/abbr\u003e) is a randomized approximate search algorithm for a number of search spaces. The second, Multi-index hashing is an exact nearest neigbour search algorithm which is limited to Hamming space.\n\nLocality-sensitive Hashing (LSH), a practical nearest neighbour search algorithm for multidimensional vectors that operates in sublinear time. It supports several Locality Sensitive Hashing (LSH) families: the Euclidean hash family (L\u003csub\u003e2\u003c/sub\u003e), city block hash family (L\u003csub\u003e1\u003c/sub\u003e) and cosine hash family. The library tries to hit the sweet spot between being capable enough to get real tasks done, and compact enough to serve as a demonstration on how \u003cabbr title=\"Locality-sensitive Hashing\"\u003eLSH\u003c/abbr\u003e works.\n\nMulti-index hashing is an exact, sub-linear nearest neighbour search algorithm in hamming space. It is a simple algorithm that seeks a balance between storage space requirements and query performance.\n\nh2. Quickly Getting Started with TarsosLSH\n\nHead over to the \"TarsosLSH release repository\":http://0110.be/releases/TarsosLSH/ and download the latest \"TarsosLSH library\":http://0110.be/releases/TarsosLSH/TarsosLSH-latest/TarsosLSH-latest.jar. Consult the \"TarsosLSH API documentation\":http://0110.be/releases/TarsosLSH/TarsosLSH-latest/TarsosLSH-latest-Documentation/. If you, for some reason, want to build from source, you need \"Apache Ant\":http://ant.apache.org/ and \"git\":http://git-scm.com/ installed on your system. The following commands fetch the source and build the library and example jars: \n\u003cpre\u003e\u003ccode\u003egit clone https://JorenSix@github.com/JorenSix/TarsosLSH.git\ncd TarsosLSH/build\nant  #Builds the core TarsosLSH library\nant javadoc #build the API documentation\n\u003c/code\u003e\u003c/pre\u003e\nWhen everything runs correctly you should be able to run the command line application, and have the latest version of the TarsosLSH library for inclusion in your projects. Also, the Javadoc documentation for the API should be available in TarsosLSH/doc. Drop me a line (\u003ca href='ma\u0026#105;\u0026#108;to\u0026#58;j\u0026#111;re\u0026#110;\u0026#64;01%310\u0026#46;\u0026#98;\u0026#101;'\u003ejoren _ at _ 0110 _ dot _ be\u003c/a\u003e) if you use TarsosLSH in your project. Always nice to hear how this software is used.\n\nThe fastest way to get something on your screen is executing this on your command line: @java - jar TarsosLSH.jar@ this lets LSH run on a random data set. The full reference of the command line application is included below:\n\n\u003cpre\u003e\nName\n\tTarsosLSH: finds the nearest neighbours in a data set quickly, using LSH.\nSynopsis     \n\tjava - jar TarsosLSH.jar [options]\nDescription\n\tTries to find nearest neighbours for each vector in the\n\tquery file, using Euclidean (L\u003csub\u003e2\u003c/sub\u003e) distance by default.\n\t\n\tBoth dataset.txt and queries.txt have a similar format:\n\tan optional identifier for the vector and a list of N\n\tcoordinates (which should be doubles).\n\t\n\t[Identifier] coord1 coord2 ... coordN\n\t[Identifier] coord1 coord2 ... coordN\n\t\n\tFor an example data set with two elements and 4 dimensions:\n\t\t\n\tHans 12 24 18.5 -45.6\n\tJane 13 19 -12.0 49.8\n\t\n\tOptions are:\n\t\t\n\t-d dataset.txt\t\n\t\tThe dataset with vectors to store in the hash table\n\t-q queries.txt\t\n\t\tA list of vectors to query against the stored dataset\n\t-f cos|l1|l2\n\t\tDefines the hash family to use:\n\t\t\tl1\tCity block hash family (L\u003csub\u003e1\u003c/sub\u003e)\n\t\t\tl2\tEuclidean hash family(L\u003csub\u003e2\u003c/sub\u003e)\n\t\t\tcos\tCosine distance hash family\n\t-r radius\n\t\tDefines the radius in which near neighbours should\n\t\tbe found. Should be a double. By default a reasonable\n\t\tradius is determined automatically.\n\t-h n_hashes\n\t\tAn integer that determines the number of hashes to\n\t\tuse. By default 4, 32 for the cosine hash family.\n\t-t n_tables\n\t\tAn integer that determines the number of hash tables,\n\t\teach with n_hashes, to use. By default 4.\n\t-n n_neighbours\n\t\tNumber of neighbours in the neighbourhood, defaults to 3.\n\t-b\n\t\tBenchmark the settings.\n\t--help\n\t\tPrints this helpful message.\n\t\nExamples\n\tSearch for nearest neighbours using the l2 hash family with a radius of 500\n\tand utilizing 5 hash tables, each with 3 hashes.\n\t\n\tjava -jar TarsosLSH.jar -f l2 -r 500 -h 3 -t 5 -d dataset.txt -q queries.txt\n\u003c/pre\u003e\n \nh2. Source Code Organization \n\nThe source tree is divided in three directories: \n* @src@ contains the source files, the core functionality. \n** @src/be/tarsos/lsh@ Contains the LSH implementation\n** @src/be/tarsos/mih@ Contains an implementation of a multi index hash algorithm\n* @test@ contains unit tests for some of the functionality.\n* @build@ contains ANT build files. Either to build Java documentation and runnable JAR-files.\n\nh2. License\n\nThe TarsosLSH license is distributed under the LGPL license.\n\nh2. Further Reading\n\nThis section includes a links to resources used to implement this library.\n\n* The \"LSH-page\":http://www.mit.edu/~andoni/LSH/ maintained by Alexandr Andoni contains pointers to good resources:\n** \"Locality-Sensitive Hashing Scheme Based on p-Stable Distributions\":http://theory.lcs.mit.edu/~indyk/nips-nn.ps a chapter by Alexandr Andoni, Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab Mirrokni which appeared in the book Nearest Neighbor Methods in Learning and Vision: Theory and Practice, by T. Darrell and P. Indyk and G. Shakhnarovich (eds.), MIT Press, 2006. \n** \"Similarity Search in High Dimensions via Hashing\":http://theory.csail.mit.edu/~indyk/vldb99.ps The original LSH Paper for hamming distance by Gionis, Aristides and Indyk, Piotr and Motwani, Rajeev.\n* \"Locality-Sensitive Hashing for Finding Nearest Neighbors\":http://www.slaney.org/malcolm/yahoo/Slaney2008-LSHTutorial.pdf a good introduction of \u003cabbr title=\"Locality-sensitive Hashing\"\u003eLSH\u003c/abbr\u003e by Malcom Slaney \u0026 Michael Casey\n* \"Finding Similar Items\":http://i.stanford.edu/~ullman/mmds/ch3.pdf, Chapter Three of \"Mining of Massive Datasets\" by Anand Rajaraman and Jeff Ullman is a textbook introducing the LSH concept.\n* \"Szudzik pairing functions\":http://szudzik.com/ElegantPairing.pdf by Matthew Szudzik. Explains how integer hashes can be combined deterministically to form a reversible, unique new hash.\n\nFor more information on Multi-Index Hashing:\n* \"Fast Search in Hamming Space with Multi-Index Hashing\":http://www.cs.toronto.edu/~norouzi/research/papers/multi_index_hashing.pdf, Mohammad Norouzi, Ali Punjani, David Fleet - IEEE Computer Vision and Pattern Recognition (CVPR), 2012. \n* An extended version of \"Fast Exact Search in Hamming Space with Multi-Index Hashing\":http://arxiv.org/abs/1307.2982 \n* \"Compact Discrete Representations for Scalable Similarity Search\":https://tspace.library.utoronto.ca/bitstream/1807/73095/3/Norouzi_Mohammad_201606_PhD_thesis.pdf\n* A github repository with a \"Multi-Index Hashing C++ implementation\":https://github.com/norouzi/mih\n\nh2. Dependencies\n\nTarsosLSH relies on \"MapDB\":http://www.mapdb.org/ for storage which in turn depends on \"eclipse collections\":https://www.eclipse.org/collections/.\n\n\nh2. Change log\n\n\u003cdt\u003eVersion 0.5\u003c/dt\u003e\u003cdd\u003e2013-04-17\u003c/dd\u003e First release which includes several LSH families.\n\n\u003cdt\u003eVersion 0.6\u003c/dt\u003e\u003cdd\u003e2013-06-13\u003c/dd\u003e Added serialization options to store LSH hashes, which makes the library more practical in real-world scenario's. Later database support should be added.\n\n\u003cdt\u003eVersion 0.7\u003c/dt\u003e\u003cdd\u003e2015-03-16\u003c/dd\u003e Maintenance release with changed package names and small fixes to the documentation and build file.\n\n\u003cdt\u003eVersion 0.8\u003c/dt\u003e\u003cdd\u003e2017-03-17\u003c/dd\u003e Adds an implementation of a Hamming space LSH scheme.\n\n\u003cdt\u003eVersion 0.9\u003c/dt\u003e\u003cdd\u003e2017-04-21\u003c/dd\u003e Adds an implementation of a Hamming space multi index hasher algorithm as described in \"Fast Exact Search in Hamming Space with Multi-Index Hashing\":http://arxiv.org/abs/1307.2982 .\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjorensix%2Ftarsoslsh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjorensix%2Ftarsoslsh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjorensix%2Ftarsoslsh/lists"}