{"id":17968337,"url":"https://github.com/guofei9987/pylshash","last_synced_at":"2025-03-25T09:30:45.262Z","repository":{"id":45400568,"uuid":"513469459","full_name":"guofei9987/pyLSHash","owner":"guofei9987","description":"Locality Sensitive Hashing, fuzzy-hash， min-hash, simhash, aHash, pHash, dHash。基于 Hash值的图片相似度、文本相似度","archived":false,"fork":false,"pushed_at":"2023-12-25T11:42:08.000Z","size":263,"stargazers_count":58,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-20T00:38:43.939Z","etag":null,"topics":["fuzzy-hash","imagehash","lsh","lsh-algorithm","lsh-implementation","min-hash"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guofei9987.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-07-13T10:01:32.000Z","updated_at":"2025-03-14T06:59:25.000Z","dependencies_parsed_at":"2023-11-23T06:38:14.913Z","dependency_job_id":"b41cabda-f6e1-43ca-b714-1b1d8f83983a","html_url":"https://github.com/guofei9987/pyLSHash","commit_stats":{"total_commits":78,"total_committers":4,"mean_commits":19.5,"dds":"0.33333333333333337","last_synced_commit":"bb12e3a44e568e280c9f2176a9963be82f56d4b0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guofei9987%2FpyLSHash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guofei9987%2FpyLSHash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guofei9987%2FpyLSHash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guofei9987%2FpyLSHash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guofei9987","download_url":"https://codeload.github.com/guofei9987/pyLSHash/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245435034,"owners_count":20614817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fuzzy-hash","imagehash","lsh","lsh-algorithm","lsh-implementation","min-hash"],"created_at":"2024-10-29T14:21:01.509Z","updated_at":"2025-03-25T09:30:45.242Z","avatar_url":"https://github.com/guofei9987.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [pyLSHash](https://github.com/guofei9987/pyLSHash)\n\n[![PyPI](https://img.shields.io/pypi/v/pyLSHash)](https://pypi.org/project/pyLSHash/)\n[![Python package](https://github.com/guofei9987/pyLSHash/actions/workflows/python-package.yml/badge.svg)](https://github.com/guofei9987/pyLSHash/actions/workflows/python-package.yml)\n[![codecov](https://codecov.io/gh/guofei9987/pyLSHash/branch/main/graph/badge.svg)](https://codecov.io/gh/guofei9987/pyLSHash)\n[![License](https://img.shields.io/pypi/l/pyLSHash.svg)](https://github.com/guofei9987/pyLSHash/blob/master/LICENSE)\n![Python](https://img.shields.io/badge/python-\u003e=3.5-green.svg)\n![Platform](https://img.shields.io/badge/platform-windows%20|%20linux%20|%20macos-green.svg)\n[![stars](https://img.shields.io/github/stars/guofei9987/pyLSHash?style=social)](https://github.com/guofei9987/pyLSHash/fork)\n\n\nA fast Python implementation of locality sensitive hashing.\n\n\n\n| Algorithm  | Function                                                                                                                                           | Application                         | Features                                                                              |\n|------------|----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------|\n| fuzzy-hash | Map text or string or file to 64-bits (or other) hash values. Similar contents hash similar hash values                                            | Fast compare similar contents       | Suitable for text/string/file                                                         |\n| min-hash   | Map sets to signature matrices and find similar sets by calculating Jaccard similarity                                                             | Similarity retrieval                | Suitable for text, network, audio, and other data                                     |\n| SimHash    | Convert high-dimensional data such as text and images into fixed-length vectors, and map similar vectors to the same bucket through hash functions | Text and image similarity retrieval | Suitable for high-dimensional data                                                    |\n| aHash      | Compress images to a fixed size and map similar images to the same bucket through hash functions                                                   | Similar image retrieval             | Has some robustness to scaling and slight deformations                                |\n| dHash      | Convert images to grayscale and calculate difference values, then map similar images to the same bucket through hash functions                     | Similar image retrieval             | Has some robustness to scaling and slight deformations                                |\n| pHash      | Convert images to DCT coefficients and map similar images to the same bucket through hash functions                                                | Similar image retrieval             | Has some robustness to scaling, brightness, translation, rotation, and noise addition |\n| LSH        | Map high-dimensional vectors to low-dimensional space and map similar vectors to the same bucket through hash functions                            | Fast search for approximate vectors | Suitable for large-scale high-dimensional data                                        |\n\n\n\n\n\n## Highlights\n\n- Fast hash calculation for large amount of high dimensional data through the use of `numpy` arrays.\n- Built-in support for persistency through Redis.\n- Multiple hash indexes support.\n- Built-in support for common distance/objective functions for ranking outputs.\n\n## Installation\n\n\n`pyLSHash` depends on the following libraries:\n\n- numpy\n- redis (if persistency through Redis is needed)\n\n\nTo install:\n\n\n```bash\n$ pip install pyLSHash\n```\n\n## Quickstart\n\n### fuzzy-hash\n\n```python\nsentence1 = '''\n近期，有一部热播硬核电视剧引发全网关注。与其他硬核电视剧不同的是，这部电视剧真的硬“核”，含“核”量高达100%。\n这就是《许你万家灯火》——首部全景反映我国核电工业发展历程的电视剧。\n《许你万家灯火》极具年代感，这是因为取景地之一是中国核动力的发源地——九〇九基地。\n中国第一代核潜艇研发实验基地\n也是中国核动力研究设计院的前身\n\n剧组在基地里面内置景，1:1复刻了主要场景十余个，\n包括第一座陆上模式堆主控室、核电大院、核电办公楼、核电家属楼、零号点、医院、图书馆、大礼堂等，\n高度还原了老一辈核工业人的研发和生活环境。\n\n而《许你万家灯火》的创作题材，便是中国完全自主知识产权的三代核电技术——“华龙一号”。\n从核潜艇研发起步的中国核工业\n如何实现拥有世界一流核电站的梦想？\n'''\n\nsentence2 = '''\n你好：\n近期，有一部热播硬核节目引发全网关注。与其他硬核节目不同的是，这部电视剧真的硬“核”，含“核”量高达100%。\n这就是《许你万家灯火》——首部全景反映我国核电工业发展历程的电视剧。\n《许你万家灯火》极具年代感，这是因为取景地之一是中国核动力的发源地——九〇九基地。\n中国第一代核潜艇研发实验基地\n也是中国核动力研究设计院的前身\n剧组在基地里面内置景，1:1复刻了主要场景十余个，\n包括第一座陆上模式堆主控室、核电大院、核电办公楼、核电家属楼、零号点、医院、图书馆、大礼堂等，\n高度还原了老一辈核工业人的研发和生活环境。\n而《许你万家灯火》的创作题材，便是中国完全自主知识产权的三代核电技术——“华龙一号”。\n从核潜艇研发起步的中国核工业\n如何实现拥有世界一流核电站的梦想？\n感谢！\n'''\n\nfrom pyLSHash import FuzzyHash\n\nfuzzy_hash = FuzzyHash()\n\nhash1 = fuzzy_hash.get_hash(sentence1.encode('utf-8'))\nhash2 = fuzzy_hash.get_hash(sentence2.encode('utf-8'))\n\nprint(hash1)\nprint(hash2)\n\ncorr = fuzzy_hash.compare(hash1, hash2)\nprint('corr = {}%'.format(corr))\n```\n\u003eb'24:NCRqxthHLDYTvxiiIhNM+Nkr6gy8C4xB6YR514cLCxd6tXKlru2uEj:tBHATdN+OuNOZrIxnAa'\nb'24:TsoR7RmxthHLDYTvxiiIhNM+Nkr6gy8o4xB6YR514cLCxd6tXilru2uEUv:fR7RmBHATdN+OulOZrIxdA7'\ncorr = 86%\n\nLook at [examples/example_fuzzy_hash.py](examples/example_fuzzy_hash.py)\n\n\n### SimHash\n\n```py\nfrom pyLSHash import SimHash, hamming\n\nsim_hash = SimHash()\n\nsh1 = sim_hash.get_hash(sentence1)\nsh2 = sim_hash.get_hash(sentence2)\n\ncorr = 1 - hamming(sh1, sh2) / sim_hash.len_hash\nprint(sh1)\nprint(sh2)\nprint('corr = {}'.format(corr))\n```\n\n\u003e957004571726091744  \n943493772323861728  \ncorr = 0.890625  \n\n\nLook at [examples/example_simhash.py](examples/example_simhash.py)\n\n### minHash\n\n\n```python\nfrom pyLSHash import min_hash\n\nk = 3  # minHash 值的维度\n\nx1 = [1, 1, 0, 0, 0, 1, 1, 1, 1, 0]\nx2 = [1, 0, 0, 0, 0, 1, 1, 1, 1, 0]\n\nn = len(x1)  # 向量的维度\nmin_hash_val1 = min_hash.get_min_hash(x1, n, k)\nmin_hash_val2 = min_hash.get_min_hash(x2, n, k)\nprint(min_hash_val1)\nprint(min_hash_val2)\n```\n\n\u003e[1, 0, 0]  \n[1, 0, 0]  \n\n\nLook at [examples/example_min_hash.py](examples/example_min_hash.py)\n\n\n### aHash/dHash/pHash\n\naHash\n```python\na_hash_img1 = img_hash.a_hash(PIL.Image.open(img1))\na_hash_img2 = img_hash.a_hash(PIL.Image.open(img2))\nhamming_distance = hamming(a_hash_img1, a_hash_img2)\n```\n\n\ndHash\n```python\nd_hash_img1 = img_hash.d_hash(PIL.Image.open(img1))\nd_hash_img2 = img_hash.d_hash(PIL.Image.open(img2))\nhamming_distance = hamming(d_hash_img1, d_hash_img2)\n```\n\n\npHash\n```python\np_hash_img1 = img_hash.p_hash(PIL.Image.open(img1))\np_hash_img2 = img_hash.p_hash(PIL.Image.open(img2))\nhamming_distance = hamming(p_hash_img1, p_hash_img2)\n```\n\noutputs:\n\u003e[aHash]: img1 = 0xffc3c3db819f0000, img2 = 0xffc3c3cb819f0000  \nhamming_distance = 1  \n[dHash]: img1 = 0x7ffae0c63d188743, img2 = 0x7ffae0c23d188743  \nhamming_distance = 1  \n[pHash]: img1 = 0xa8a0008200000000, img2 = 0xa8a0008200000000  \nhamming_distance = 0  \n\n\n\nLook at [examples/example_img_hash.py](examples/example_img_hash.py)\n\n\n\n## LSHash\nTo create 6-bit hashes for input data of 8 dimensions:\n\n\n```python\nfrom pyLSHash import LSHash\n\nlsh = LSHash(hash_size=6, input_dim=8)\nlsh.index([1, 2, 3, 4, 5, 6, 7, 8])\nlsh.index([2, 3, 4, 5, 6, 7, 8, 9])\n# attach extra_data\nlsh.index([2, 3, 4, 5, 6, 7, 8, 9], extra_data=\"some vector info\")\nlsh.index([10, 12, 99, 1, 5, 31, 2, 3])\n\nres = lsh.query([1, 2, 3, 4, 5, 6, 7, 7])\n```\n\n\u003e[((1, 2, 3, 4, 5, 6, 7, 8), 1.0),\n((2, 3, 4, 5, 6, 7, 8, 9), 11)]\n\n### User defined distance function\n\n```python\ndef l1norm_dist(x, y):\n    return sum(abs(x - y))\n\n\nres2 = lsh.query([1, 2, 3, 4, 5, 6, 7, 7], dist_func=l1norm_dist)\n\nprint(res2)\n```\n\n\n### Use Redis\n\n```python\nfrom pyLSHash import LSHash\n\nlsh = LSHash(hash_size=6, input_dim=8\n             , storage_instance=RedisStorage({'host': 'localhost', 'port': 6379, 'decode_responses': True}))\n\nlsh.index([1, 2, 3, 4, 5, 6, 7, 8])\nlsh.index([2, 3, 4, 5, 6, 7, 8, 9])\n# attach extra_data\nlsh.index([2, 3, 4, 5, 6, 7, 8, 9], extra_data=\"some vector info\")\nlsh.index([10, 12, 99, 1, 5, 31, 2, 3])\nlsh.index([10, 12, 99, 1, 5, 31, 2, 3])\n\nres = lsh.query([1, 2, 3, 4, 5, 6, 7, 7])\n```\n\n### Use other database as storage\n\n```python\nfrom pyLSHash import LSHash\nfrom pyLSHash.storage import StorageBase\nimport redis\nimport json\n\n\nclass MyStorage(StorageBase):\n    def __init__(self):\n        self.storage = redis.StrictRedis(host='localhost', port=6379, decode_responses=True)\n\n    def keys(self, pattern=\"*\"):\n        return self.storage.keys(pattern)\n\n    def set_val(self, key, val):\n        self.storage.set(key, val)\n\n    def get_val(self, key):\n        return self.storage.get(key)\n\n    def append_val(self, key, val):\n        self.storage.rpush(key, json.dumps(val))\n\n    def get_list(self, key):\n        res_list = [json.loads(val) for val in self.storage.lrange(key, 0, -1)]\n        return tuple((tuple(item[0]), item[1]) for item in res_list)\n\n    def clear(self):\n        for key in self.storage.keys():\n            self.storage.delete(key)\n\n\nlsh = LSHash(hash_size=6, input_dim=8\n             , storage_instance=MyStorage())\n\nlsh.index([1, 2, 3, 4, 5, 6, 7, 8])\nlsh.index([2, 3, 4, 5, 6, 7, 8, 9])\nlsh.index([2, 3, 4, 5, 6, 7, 8, 9], extra_data=\"some vector info\")\nlsh.index([10, 12, 99, 1, 5, 31, 2, 3])\nlsh.index([10, 12, 99, 1, 5, 31, 2, 3])\n\nres = lsh.query([1, 2, 3, 4, 5, 6, 7, 7])\n```\n\n\n### save\u0026load model\n\n```python\nlsh.save_uniform_planes(\"filename.pkl\")\nlsh.load_uniform_planes(\"filename.pkl\")\n```\n\nclear indexed data\n```python\nlsh.clear_storage()\n```\n\n## Other examples\n\n- Examples for min-hash ：[examples/example_min_hash.py](examples/example_min_hash.py)\n- Examples for SimHash ：[examples/example_simhash.py](examples/example_simhash.py), [examples/example_simhash2.py](examples/example_simhash2.py)\n- Examples for aHash ：[examples/example_img_hash.py](examples/example_img_hash.py)\n- Examples for dHash ： [examples/example_img_hash.py](examples/example_img_hash.py)\n- Examples for pHash ：[examples/example_img_hash.py](examples/example_img_hash.py)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguofei9987%2Fpylshash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguofei9987%2Fpylshash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguofei9987%2Fpylshash/lists"}