{"id":22960164,"url":"https://github.com/qyokizzzz/simhash","last_synced_at":"2025-04-02T02:43:08.745Z","repository":{"id":193478267,"uuid":"525860532","full_name":"Qyokizzzz/simhash","owner":"Qyokizzzz","description":"The extended version of simhash supports fingerprint extraction of documents and  images.","archived":false,"fork":false,"pushed_at":"2022-08-22T16:05:57.000Z","size":564,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-07T17:44:45.647Z","etag":null,"topics":["document-search","fingerprint","image-deduplication","image-search","simhash"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Qyokizzzz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-17T15:51:21.000Z","updated_at":"2024-09-02T02:55:18.000Z","dependencies_parsed_at":null,"dependency_job_id":"da3061a5-2cfb-4bac-a933-d7fa1de74b49","html_url":"https://github.com/Qyokizzzz/simhash","commit_stats":null,"previous_names":["qyokizzzz/simhash"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qyokizzzz%2Fsimhash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qyokizzzz%2Fsimhash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qyokizzzz%2Fsimhash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qyokizzzz%2Fsimhash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Qyokizzzz","download_url":"https://codeload.github.com/Qyokizzzz/simhash/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246746879,"owners_count":20827061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["document-search","fingerprint","image-deduplication","image-search","simhash"],"created_at":"2024-12-14T18:31:05.013Z","updated_at":"2025-04-02T02:43:08.724Z","avatar_url":"https://github.com/Qyokizzzz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simhash\n\nSimhash is an algorithm for extracting document fingerprints, which can evaluate the similarity of two documents by calculating the Hamming Distance between the simhash values of them.\n\nI extended simhash to add the function of processing images.\n\nBy comparing the similarity between images, you can implement the functions of image deduplication, image search and so on.\n\n## Purpose\n\nWhen we work with images, we tend to use deep learning methods, but algorithm engineers and computing resources are expensive.\n\nSo, if your business wants to add image search to the search engine of your application, simhash will be a good choice.\n\n## Installation\n\n## Introduction\n\nUsing Simhash to customized yourself application.\n\nUsing Scheduler to call some encapsulated functions like image-deduper, image-search, extract simhash for image and document.\n\nUsing CorpusReader to read documents and load trie of stopwords.\n\nUsing Dictionary to convert a document to word bag vector.\n\n## API\n\n### Simhash\n\nPure simhash, not coupled with any dependencies.\n\n__init__(hashify: Callable, n: int, hamming_dist_threshold: int):\n\n    hashify(word: Optional[Union[str, int]]) -\u003e str: A customizable callback function to specify the hash algorithm.\n    n: Specifies how many bits of simhash value to generate, it should be the same as the number of hash strings generated by hashify.\n    hamming_dist_threshold: Specify a similarity threshold.\n\ngenerate(characteristic_idx_list: List[int]) -\u003e str:\n\n    characteristic_idx_list: The index(image hist or document bag vector) that best reflects the characteristics of documents or images.\n    return: simhash value\n\ncalc_hamming_dist(simhash1: str, simhash2: str) -\u003e int:\n\n    Calculate Hamming distance between two simhash.\n\nsegment(simhash: str) -\u003e List[str]:\n\n    Divide the simhash value into n segments.\n    n = hamming_dist_threshold\n\n### ComHistsGetter\n\nCommon hist is required for handle image when CharacteristicIdxListExtractor is TF-IDF.\n\nIf you want to handle image, you need use it to extract a common image hist list for TF-IDF.\n\nComHistsGetter is not required. You can also get common hist from the database or other places.\n\n__init__(extractor: Callable):\n\n    extractor(img: Mat) -\u003e List[int]: A customizable callback function to determine how to extract image features.\n\nget_common_hist(dirname: str, p: float) -\u003e List[List[int]]:\n    \n    dirname: Common image path\n    p: Sampling rate\n    return: common image hist List\n\n### TFIDF\n\nIt is an algorithm for extracting topics from documents.\n\nHere, I use TFIDF as an algorithm to extract the features that best reflect the document or image.\n\nYou can also use other algorithms to implement CharacteristicIdxListExtractor interface to replace TFIDF.\n\n__init__(top_n: int = 15):\n\n    top_n: The number of topics\n\nget_characteristic_idx_list(doc_bow: List[int], common_doc_bows: List[List[int]]) -\u003e List[int]:\n\n    doc_bow: The document vector of the topic to be extracted may be a word bag or an image hist.\n    common_doc_bows: A list in the form of [doc_bow1, doc_bow2, ...] for storing general documents.\n\n### About Callbacks\n\nWhere needing to pass in hashify and extractor, you can pass in any signature-compatible callback function.\n\nhashify(word: Optional[Union[str, int]]) -\u003e str:\n\n    Required argument.\n    Built in hashify_by_murmurhash_64bits and hashify_by_murmurhash_128bits for use.\n    word: It can be the index of the word bag vector and image hist, or a word.\n\nextractor(img: Mat) -\u003e List[int]:\n\n    Required for handle image.\n    Built in hsv_extractor for get HSV hist from image.\n    img: Ignored.\n    \n\n### Scheduler\n\nIt is similar to a Director, encapsulating commonly used methods, and unlike Simhash, it is coupled to the system's built-in dependencies.\n\n__init__(\n    ce: CharacteristicIdxListExtractor,\n    hashify: Callable = hashify_by_murmurhash_128bits,\n    extractor: Callable = hsv_extractor,\n    n: int = 128,\n    hamming_dist_threshold: int = 4\n):\n\nhandle_img(img: Mat, common_hists: List[List[int]]) -\u003e str:\n\n    Generate simhash for image.\n    img: Ignored.\n    common_hists: Ignored.\n    return: Ignored.\n\nhandle_doc(doc: List[List[str]], common_docs: List[List[List[str]]]) -\u003e str:\n\n    Generate simhash for document.\n    doc: Ignored.\n    common_docs: Ignored.\n    return: Ignored.\n\ngenerate_for_img_list(\n    img_list: List[Mat],\n    common_hists: List[List[int]]\n) -\u003e List[str]:\n\n    Generate simhash for image List.\n    img_list: Ignored.\n    common_hists: Ignored.\n    return: Ignored.\n\nimg_deduper(\n    signatures: List[str],\n    simhash_list: List[str]\n) -\u003e List[Tuple[Tuple[str, str], Tuple[str, str], int]]:\n\n    Using for image deduplication.\n    signatures: Identifying each simhash, In the image-deduper example, I used the file name as the signature, but this is not necessary. You can also use the id in the database as the signature.\n    simhash_list: Ignored.\n    return: All duplicate images, and list item is ((signature1, simhash1), (signature2, simhash2), hamming_dist), you can decide for yourself how to deal with these duplicate images.\n\nsave_simhash(signature: str, simhash: str) -\u003e None:\n\n    Divide simhash into n(n = hamming_dist_threshold) segments.\n    And store each segment as key and (signature, simhash) as value in dict.\n    Like: Dict[segment, (signature, simhash)].\n    signature: Ignored.\n    simhash: Ignored.\n\nsave_simhash_list(signatures: List[str], simhash_list: List[str]) -\u003e None:\n    \n    Call save_simhash for list.\n    signatures: Ignored.\n    simhash_list: Ignored.\n\nimg_search(simhash: str) -\u003e List[Any]:\n\n    Using save_simhash_list store a image library, and using img_search to searching.\n    simhash: Ignored.\n    return: Candidate list, like [(signature1, simhash1), (signature2, simhash2), ...], and list item sorted by hamming dist.\n\n### Ellipsis\n\nFor more api details, please read the source code.\n\n## Examples\n\n[image search](examples/image-search.py)\n\n[image deduper](examples/image-deduper.py)\n\n[generate simhash for document](examples/gen_doc_simhash.py)\n\n## Principle\n\n### Workflow\n\nInputting document, and convert it to document bag vector. -\u003e doc_bow, like [5, 3, 10, 11]. Each number represents how many times the word appears in the document.\n\nExtracting some characteristic words(indexes) of the document(doc_bow). -\u003e characteristic_idx_list, like [idx1, idx2, ...].\n\nHashify each item of characteristic_idx_list. -\u003e characteristic_hash_list, like [hash1, hash2, ...].\n\nFor each hash(like 010101101010101010101010...) in characteristic_hash_list, for each bit in hash, if bit \u003e 0 return 1, else return -1. -\u003e tmp, like [[1, -1, -1, ...], [-1, 1, 1, ...], ...].\n\nGetting the sum(tmp) result. -\u003e weights, like [10, -11, 9, 8, -2, 0, ...].\n\nFor each weight, if weight \u003e 0 return 1 else return 0. -\u003e simhash, like '0101010101010101010100101...'.\n\n### Extend\n\nThe hist of a picture is similar to the meaning of the bag of words, so simhash can also be used as an extraction algorithm for image fingerprints.\n\n### Customization\n\nThe project uses a low-coupling architecture, so it is very flexible to use, and you can customize your own feature extractor to replace the default values.\n\nThere are few optimization items enabled in the project. You can use C\\C++ or Java to reproduce it, so that simhash can reach the industrial available running speed.\n\n![Class diagram](class-diagram.jpg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqyokizzzz%2Fsimhash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqyokizzzz%2Fsimhash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqyokizzzz%2Fsimhash/lists"}