{"id":33936680,"url":"https://github.com/serega/gaoya","last_synced_at":"2026-03-17T20:04:35.297Z","repository":{"id":40520997,"uuid":"334790307","full_name":"serega/gaoya","owner":"serega","description":"Locality Sensitive Hashing","archived":false,"fork":false,"pushed_at":"2023-07-12T02:45:17.000Z","size":242,"stargazers_count":77,"open_issues_count":8,"forks_count":9,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-14T02:31:01.867Z","etag":null,"topics":["locality-sensitive-hashing","lsh","minhash","search","simhash","similarity"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serega.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-01T00:43:04.000Z","updated_at":"2025-12-04T11:30:03.000Z","dependencies_parsed_at":"2022-08-09T22:40:25.713Z","dependency_job_id":null,"html_url":"https://github.com/serega/gaoya","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/serega/gaoya","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serega%2Fgaoya","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serega%2Fgaoya/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serega%2Fgaoya/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serega%2Fgaoya/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serega","download_url":"https://codeload.github.com/serega/gaoya/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serega%2Fgaoya/sbom","scorecard":{"id":812032,"data":{"date":"2025-08-11","repo":{"name":"github.com/serega/gaoya","commit":"b0d67acdf5da12f2c51b97043113609f87e92967"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/11 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/create-py-mac-release.yaml:1","Warn: no topLevel permission defined: .github/workflows/create-py-release-manylinux.yaml:1","Warn: no topLevel permission defined: .github/workflows/create-py-window-release.yaml:1","Warn: no topLevel permission defined: .github/workflows/rust.yml:1","Warn: no topLevel permission defined: .github/workflows/test-python.yaml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/create-py-mac-release.yaml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-mac-release.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/create-py-mac-release.yaml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-mac-release.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/create-py-mac-release.yaml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-mac-release.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/create-py-mac-release.yaml:35: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-mac-release.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/create-py-release-manylinux.yaml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-release-manylinux.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/create-py-release-manylinux.yaml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-release-manylinux.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/create-py-window-release.yaml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-window-release.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/create-py-window-release.yaml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-window-release.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/create-py-window-release.yaml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-window-release.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/create-py-window-release.yaml:33: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/create-py-window-release.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/rust.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/rust.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test-python.yaml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/test-python.yaml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/test-python.yaml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/test-python.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test-python.yaml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/serega/gaoya/test-python.yaml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/test-python.yaml:27","Warn: pipCommand not pinned by hash: .github/workflows/test-python.yaml:28","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   6 third-party GitHubAction dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 26 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-23T13:24:50.565Z","repository_id":40520997,"created_at":"2025-08-23T13:24:50.566Z","updated_at":"2025-08-23T13:24:50.566Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30630038,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-17T17:32:55.572Z","status":"ssl_error","status_checked_at":"2026-03-17T17:32:38.732Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["locality-sensitive-hashing","lsh","minhash","search","simhash","similarity"],"created_at":"2025-12-12T14:29:19.822Z","updated_at":"2026-03-17T20:04:35.292Z","avatar_url":"https://github.com/serega.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gaoya\n\n\n## About\nThis project implements Locality Sensitive Hashing algorithms and data structures for indexing and querying text documents.\nThe primary use cases for Gaoya are deduplication and clustering. \n\n## Main Features\n* 64,32,16,8 bit minhash\n* 64,128 bit simhash\n* Fast implementation in Rust\n* Multi-threaded thanks to [rayon](https://github.com/rayon-rs/rayon)\n* Python bindings\n\n\n\n### Python Example\n\n```python\n\u003e\u003e\u003e import gaoya\n\u003e\u003e\u003e index = gaoya.minhash.MinHashStringIndex(hash_size=32, \n                                             jaccard_threshold=0.5, \n                                             num_bands=42, \n                                             band_size=3,\n                                             num_hashes=42*3,\n                                             analyzer='word', \n                                             lowercase=True, \n                                             ngram_range=(1,1))\n\u003e\u003e\u003e corpus = [\n...     'This is the first document.',\n...     'This document is the second document.',\n...     'And this is the third document.',\n...     'Is this the first document?',\n...     'This not the first nor the second nor the third, but the fourth document'\n... ]\n\u003e\u003e\u003e \n\u003e\u003e\u003e for i, doc in enumerate(corpus): index.insert_document(i, doc)\n... \n\u003e\u003e\u003e index.query('This is the first document.')\n[0, 1, 2, 3]\n\u003e\u003e\u003e \n```\n\n### Installation\n```\n$ pip3 install gaoya\n```\n\n### Examples\n[Document Deduplication with Gaoya](https://github.com/serega/gaoya/blob/master/py-gaoya/examples/deduplication_scholarly_articles_gaoya.ipynb)\n\n\n### Rust Example\n\n```rust\nuse gaoya::minhash::{MinHashIndex, MinHasher32, MinHasher} ;\nuse gaoya::text::whitespace_split;\nuse fxhash::FxHashSet;\nlet corpus = [\n    \"This is the first document.\",\n    \"This document is the second document.\",\n    \"And this is the third document.\",\n    \"Is this the first document?\",\n    \"This not the first nor the second nor the third, but the fourth document\"];\nlet (num_bands, band_width) = (42, 3);\nlet minhasher = MinHasher32::new(num_bands * band_width);\nlet mut index = MinHashIndex::new(num_bands, band_width, 0.5);\nfor (i, doc) in corpus.iter().enumerate() {\n    index.insert(i, minhasher.create_signature(whitespace_split(\u0026doc.to_lowercase())));\n}\nfor (i, doc) in corpus.iter().enumerate() {\n    if i \u003c 4 {\n        let mut expected = FxHashSet::default();\n        expected.extend(vec![0, 1, 2, 3].into_iter());\n        let signature = minhasher.create_signature(whitespace_split(\u0026doc.to_lowercase()));\n        assert_eq!(index.query_owned(\u0026signature), expected);\n    } else {\n        let mut expected = FxHashSet::default();\n        expected.insert(4);\n        let signature = minhasher.create_signature(whitespace_split(\u0026doc.to_lowercase()));\n        assert_eq!(index.query_owned(\u0026signature), expected);\n    }\n}\n\n```\n\n\n## References\n[[1] Chapter 3, Mining of Massive Datasets](http://www.mmds.org)\n\n[[2] Similarity Estimation Techniques from Rounding Algorithms](https://www.cs.princeton.edu/courses/archive/spr04/cos598B/bib/CharikarEstim.pdf)\n\n[[3] Detecting Near-Duplicates for Web Crawling](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/33026.pdf)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserega%2Fgaoya","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserega%2Fgaoya","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserega%2Fgaoya/lists"}