{"id":20911419,"url":"https://github.com/brandonrobertz/sparselsh","last_synced_at":"2025-04-06T07:10:18.279Z","repository":{"id":18462667,"uuid":"21657115","full_name":"brandonrobertz/SparseLSH","owner":"brandonrobertz","description":"A Locality Sensitive Hashing (LSH) library with an emphasis on large, highly-dimensional datasets.","archived":false,"fork":false,"pushed_at":"2024-09-04T21:48:17.000Z","size":111,"stargazers_count":146,"open_issues_count":0,"forks_count":27,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-30T06:04:07.426Z","etag":null,"topics":["clustering","data-mining","machine-learning","sparse-matrices","text-mining"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brandonrobertz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-09T15:13:25.000Z","updated_at":"2025-01-20T10:53:52.000Z","dependencies_parsed_at":"2024-11-18T14:23:53.442Z","dependency_job_id":"a0f25368-9d01-4a1f-938d-b02910c0bd14","html_url":"https://github.com/brandonrobertz/SparseLSH","commit_stats":{"total_commits":110,"total_committers":6,"mean_commits":"18.333333333333332","dds":0.5363636363636364,"last_synced_commit":"11f381560a94c8d74af55b3db5e8db1bbddfc212"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brandonrobertz%2FSparseLSH","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brandonrobertz%2FSparseLSH/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brandonrobertz%2FSparseLSH/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brandonrobertz%2FSparseLSH/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brandonrobertz","download_url":"https://codeload.github.com/brandonrobertz/SparseLSH/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247445668,"owners_count":20939958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","data-mining","machine-learning","sparse-matrices","text-mining"],"created_at":"2024-11-18T14:21:35.330Z","updated_at":"2025-04-06T07:10:18.238Z","avatar_url":"https://github.com/brandonrobertz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SparseLSH\n\nA locality sensitive hashing library with an emphasis on large, highly-dimensional datasets.\n\n## Features\n\n- Fast and memory-efficient calculations using sparse matrices.\n- Built-in support for key-value storage backends: pure-Python, Redis (memory-bound), LevelDB, BerkeleyDB\n- Multiple hash indexes support (based on Kay Zhu's lshash)\n- Built-in support for common distance/objective functions for ranking outputs.\n\n## Details\n\nSparseLSH is based on a fork of Kay Zhu's lshash, and is suited for datasets that won't\nfit into main memory or are highly dimensional. Using sparse matrices\nallows for speedups of easily over an order of magnitude compared to using dense, list-based\nor numpy array-based vector math. Sparse matrices also makes it possible to deal with\nthese datasets purely in memory using python dicts or through Redis. When this isn't\nappropriate, you can use one of the disk-based key-value stores, LevelDB and BerkeleyDB.\nSerialization is done using cPickle (for raw C speedups), falling back to python\npickle if it's not available.\n\n## Installation\n\nThe easy way, from [PyPI][sparselsh-pypi]:\n\n    pip install sparselsh\n\nOr you can clone this repo and run the minimal install:\n\n    pip install .\n\nIf you would like to use the LevelDB or Redis storage backends, you can\ninstall those dependencies, too:\n\n    pip install -r .[redis]\n    pip install -r .[leveldb]\n\n## Quickstart\n\nYou can quickly use LSH using the bundled `sparselsh` command line utility. Simply pass the path to a file containing records to be clustered, one per line, and the script will output groups of similar items.\n\n    sparselsh path/to/recordsfile.txt\n\nTo create 4-bit hashes for input data of 7 dimensions:\n\n    from sparselsh import LSH\n    from scipy.sparse import csr_matrix\n\n    X = csr_matrix([\n        [3, 0, 0, 0, 0, 0, -1],\n        [0, 1, 0, 0, 0, 0,  1],\n        [1, 1, 1, 1, 1, 1,  1]\n    ])\n\n    # One label for each input point\n    y = [\"label-one\", \"second\", \"last\"]\n\n    lsh = LSH(\n        4,\n        X.shape[1],\n        num_hashtables=1,\n        storage_config={\"dict\":None}\n    )\n\n    lsh.index(X, extra_data=y)\n\n    # Build a 1-D (single row) sparse matrix\n    X_sim = csr_matrix([[1, 1, 1, 1, 1, 1, 0]])\n    # find the point in X nearest to X_sim\n    points = lsh.query(X_sim, num_results=1)\n    # split up the first result into its parts\n    (point, label), dist = points[0]\n    print(label)  # 'last'\n\nThe query above result in a list of matrix-class tuple \u0026 similarity\nscore tuples. A lower score is better in this case (the score here is 1.0).\nHere's a breakdown of the return value when `query` is called with a\nsingle input row (1-dimensional sparse matrix, the output is different\nwhen passing multiple query points):\n\n    [((\u003c1x7 sparse matrix of type '\u003ctype 'numpy.int64'\u003e' with 7 stored elements in Compressed Sparse Row format\u003e, 'label'), 1.0)]\n\nWe can look at the most similar matched item by accessing the sparse array\nand invoking it's `todense` function:\n\n    #                      ,------- Get first result-score tuple\n    #                     |   ,---- Get data. [1] is distance score\n    #                     |  |  ,-- Get the point. [1] is extra_data\n    #                     |  |  |\n    In [11]: print points[0][0][0].todense()\n    [[1 1 1 1 1 1 1]]\n\nYou can also pass multiple records to `query` by simply increasing the\ndimension of the input to `query`. This will change the output data\nto have one extra dimension, representing the input query. (NOTE: When\nthen dimension is 1, a.k.a. a single sparse row, this extra dimension won't\nbe added.) Here's the output when `query` is passed a 2-dimensional input:\n\n    [\n      [((\u003c1x7 sparse matrix ...\u003e, 'label'), 1.0)],\n      [((\u003c1x7 sparse matrix ...\u003e, 'extra'), 0.8),\n       ((\u003c1x7 sparse matrix ...\u003e, 'another'), 0.3)]\n    ]\n\nHere, you can see an extra dimension, one for each query point passed\nto `query`. The data structure for each query point result is the same\nas the 1-Dimensional output.\n\n## Main Interface\n\nMost of the parameters are supplied at class init time:\n\n    LSH( hash_size,\n         input_dim,\n         num_of_hashtables=1,\n         storage_config=None,\n         matrices_filename=None,\n         overwrite=False)\n\nParameters:\n\n    hash_size:\n        The length of the resulting binary hash. This controls how many \"buckets\"\n        there will be for items to be sorted into.\n\n    input_dim:\n        The dimension of the input vector. This needs to be the same as the input\n        points.\n\n    num_hashtables = 1:\n        (optional) The number of hash tables used. More hashtables increases the\n        probability of hash-collisions and the more similar items are likely\n        to be found for a query item. Increase this to get better accuracy\n        at the expense of increased memory usage.\n\n    storage = None:\n        (optional) A dict representing the storage backend and configuration\n        options. The following storage backends are supported with the following\n        configurations:\n            In-Memory Python Dictionary:\n                {\"dict\": None} # Takes no options\n            Redis:\n                {\"redis\": {\"host\": \"127.0.0.1\", \"port\": 6379, \"db\": 0}\n            LevelDB:\n                {\"leveldb\":{\"db\": \"ldb\"}}\n                Where \"ldb\" specifies the directory to store the LevelDB database.\n                (In this case it will be `./ldb/`)\n            Berkeley DB:\n                {\"berkeleydb\":{\"filename\": \"./db\"}}\n                Where \"filename\" is the location of the database file.\n\n    matrices_filename = None:\n        (optional) Specify the path to the .npz file random matrices are stored\n        or to be stored if the file does not exist yet. If you change the input\n        dimensions or the number of hashtables, you'll need to set the following\n        option, overwrite, to True, or delete this file.\n\n    overwrite = False:\n        (optional) Whether to overwrite the matrices file if it already exists.\n\n### Index (Add points to hash table):\n\n- To index a data point of a given `LSH` instance:\n\n    lsh.index(input_point, extra_data=None)\n\nParameters:\n\n    input_point:\n        The input data point is an array or tuple of numbers of input_dim.\n\n    extra_data = None:\n        (optional) Extra data to be added along with the input_point.\n        This can be used to hold values like class labels, URIs, titles, etc.\n\nThis function returns nothing.\n\n### Query (Search for similar points)\n\nTo query a data point against a given `LSH` instance:\n\n    lsh.query(query_point, num_results=None, distance_func=\"euclidean\")\n\nParameters:\n\n    query_point:\n        The query data point is a sparse CSR matrix.\n\n    num_results = None:\n        (optional) Integer, specifies the max amount of results to be\n        returned. If not specified all candidates will be returned as a\n        list in ranked order.\n        NOTE: You do not save processing by limiting the results. Currently,\n        a similarity ranking and sort is done on all items in the hashtable\n        regardless if this parameter.\n\n    distance_func = \"euclidean\":\n        (optional) Distance function to use to rank the candidates. By default\n        euclidean distance function will be used.\n\nReturns a list of tuples, each of which has the original input point (which\nwill be a tuple of csr-matrix, extra_data or just the csr-matrix if no extra\ndata was supplied) and a similarity score.\n\n\n[sparselsh-pypi]: https://pypi.org/project/sparselsh/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrandonrobertz%2Fsparselsh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrandonrobertz%2Fsparselsh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrandonrobertz%2Fsparselsh/lists"}