{"id":20122881,"url":"https://github.com/freaky/gcstool","last_synced_at":"2025-10-29T15:07:51.737Z","repository":{"id":142942518,"uuid":"128490385","full_name":"Freaky/gcstool","owner":"Freaky","description":"A small tool for creating and searching Golomb Compressed Sets","archived":false,"fork":false,"pushed_at":"2022-07-21T09:05:13.000Z","size":109,"stargazers_count":13,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-06T16:49:00.645Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Freaky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-07T02:46:35.000Z","updated_at":"2021-02-15T05:35:40.000Z","dependencies_parsed_at":"2023-05-22T14:00:33.220Z","dependency_job_id":null,"html_url":"https://github.com/Freaky/gcstool","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Freaky/gcstool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freaky%2Fgcstool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freaky%2Fgcstool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freaky%2Fgcstool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freaky%2Fgcstool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Freaky","download_url":"https://codeload.github.com/Freaky/gcstool/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Freaky%2Fgcstool/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262017021,"owners_count":23245576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T19:41:37.872Z","updated_at":"2025-10-29T15:07:46.691Z","avatar_url":"https://github.com/Freaky.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"gcstool\n-------\n\nA small (prerelease) tool for creating and searching [Golomb Compressed Sets][1].\n\n\n### What?\n\nGolomb Compressed Sets are similar to [Bloom filters][2] - they're space-efficient\ndata structures that let you test whether a given element is a member of a set.\n\nLike Bloom filters, they have a controllable rate of false-positives - they may\nconsider an element a member of a set even if it's never been seen before - while\nhaving no false negatives.  If a GCS hasn't seen it, it's not on the list.\n\n\n### Why?\n\nLet's illustrate with a real-world problem: checking against leaked password lists.\n\nSay you have a copy of [pwned-passwords-2.0.txt][3], and you want to check\nyour users passwords against the list when they sign up.  Unfortunately it's\nnearly 30GB, and not in a format particularly suited for searching.\n\nYou cut it down quite a bit simply by converting the SHA1 hashes in it to\nbinary, and even truncating them in the process, but even being really aggressive\nand spending just 6 bytes per entry (for a false-positive rate of 1 in 500,000)\nonly gets you down to 3 GB.\n\nLet's see what this tool can do:\n\n    # Use precomputed hashes directly with -H hex\n    % gcstool -H hex create -p 500000 pwned-passwords-2.0.txt pwned-passwords-2.0-p500k.gcs\n    (5 minutes and 3.8 GiB of RAM later)\n    % du -h pwned-passwords-2.0-p500k.gcs\n    1.2G    pwned-passwords-2.0-p500k.gcs\n\nAn equivalent Bloom filter would consume [1.6 GiB][4].  Not too shabby.\n\nBut 1 in 500,000 passwords being randomly rejected for no good reason is a bit crap.\nYour users deserve better than that, surely.  You could double-check against the\n[pwned passwords API][5], but then we're leaking a little bit about our users\npasswords to a third party and that makes me itch.  What if we could just up the\nerror rate so it's negligible?\n\n    % gcstool -H hex create -p 50000000 pwned-passwords-2.0.txt pwned-passwords-2.0-p50m.gcs\n    (5 minutes and 3.8 GiB of RAM later)\n    % du -h pwned-passwords-2.0-p50m.gcs\n    1.6G    pwned-passwords-2.0-p50m.gcs\n\nContrast this with a [2.15 GiB Bloom filter][5].\n\nWe can now query a database for for known passwords:\n\n    % gcstool query pwned-passwords-2.0-p50m.gcs\n    Ready for queries on 501636842 items with a 1 in 50000000 false-positive rate.  ^D to exit.\n    \u003e password\n    Found in 0.3ms\n    \u003e not a leaked password\n    Not found in 1.7ms\n    \u003e I love dogs\n    Found in 0.7ms\n    \u003e I guess it works\n    Not found in 0.1ms\n\nYay.\n\nIntegrating it into your website is left as an exercise right now.  Eventually I'll\nfactor this out into a library and sort out a Rubygem.\n\n\n### How?\n\nGolomb Compressed Sets are surprisingly simple:\n\nChoose a false positive rate `p`.  For each of your `n` elements, hash to an integer\n`mod pn`.  Then sort the list of integers, and convert each entry to a diff from the\nprevious one, so you end up with a list of offsets.\n\nThis is the set - just a list of offsets.  You can prove to yourself that this works\nbecause you've distributed your n items in n*p buckets, so a bucket has just a\n1-in-p probability of being filled by accident by any given item.\n\nNow it's just a matter of storing each offset efficiently.  We use a subset variant of\n[Golomb coding][6] called Rice coding - divmod by p, store the quotient in unary\n(0 = 0, 1 = 1, 2 = 11, 3 = 111, etc), and store the remainder in log2(p) bits.\n\nThen we simply store an index indicating what and where every `n`th element is so we\ncan jump in the middle of the set to find a random item.  In this version of the GCS,\nwe simply store pairs of 64-bit integers indicating the number and the offset.\n\nMore complex index encoding is possible to make it smaller, but since a default index\nfor even 500 million items is just 16MB it barely seems worth the effort.\n\n\n[1]: http://giovanni.bajo.it/post/47119962313/golomb-coded-sets-smaller-than-bloom-filters\n[2]: https://en.wikipedia.org/wiki/Bloom_filter\n[3]: https://haveibeenpwned.com/Passwords\n[4]: https://hur.st/bloomfilter/?n=501652074\u0026p=500000\u0026m=\u0026k=\n[5]: https://hur.st/bloomfilter/?n=501652074\u0026p=50M\u0026m=\u0026k=\n[6]: https://en.wikipedia.org/wiki/Golomb_coding\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreaky%2Fgcstool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreaky%2Fgcstool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreaky%2Fgcstool/lists"}