{"id":13711921,"url":"https://github.com/kristoff-it/redis-cuckoofilter","last_synced_at":"2025-08-20T16:32:42.462Z","repository":{"id":44894549,"uuid":"98114033","full_name":"kristoff-it/redis-cuckoofilter","owner":"kristoff-it","description":"Hashing-function agnostic Cuckoo filters for Redis","archived":false,"fork":false,"pushed_at":"2020-03-05T21:10:34.000Z","size":6650,"stargazers_count":231,"open_issues_count":3,"forks_count":22,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-12-10T02:21:15.384Z","etag":null,"topics":["bloom-filter","c","cuckoo-filter","probabilistic-data-structures","redis","zig"],"latest_commit_sha":null,"homepage":"","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kristoff-it.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-23T17:28:09.000Z","updated_at":"2024-11-28T16:33:28.000Z","dependencies_parsed_at":"2022-08-17T21:41:05.142Z","dependency_job_id":null,"html_url":"https://github.com/kristoff-it/redis-cuckoofilter","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kristoff-it%2Fredis-cuckoofilter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kristoff-it%2Fredis-cuckoofilter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kristoff-it%2Fredis-cuckoofilter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kristoff-it%2Fredis-cuckoofilter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kristoff-it","download_url":"https://codeload.github.com/kristoff-it/redis-cuckoofilter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230438185,"owners_count":18225870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloom-filter","c","cuckoo-filter","probabilistic-data-structures","redis","zig"],"created_at":"2024-08-02T23:01:12.980Z","updated_at":"2024-12-19T13:08:21.855Z","avatar_url":"https://github.com/kristoff-it.png","language":"Zig","funding_links":[],"categories":["Database","Libraries","Data \u0026 Science"],"sub_categories":["Database"],"readme":"\n\u003ch1 align=\"center\"\u003eredis-cuckoofilter\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/kristoff-it/redis-cuckoofilter/releases/latest\"\u003e\u003cimg src=\"https://badgen.net/github/release/kristoff-it/redis-cuckoofilter\"/\u003e\u003c/a\u003e\n    \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://badgen.net/github/license/kristoff-it/zig-cuckoofilter\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/kristoff-it/zig-cuckoofilter\"\u003e\u003cimg src=\"https://badgen.net/badge/based%20on/zig-cuckoofilter\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://twitter.com/croloris\"\u003e\u003cimg src=\"https://badgen.net/badge/twitter/@croloris/1DA1F2?icon\u0026label\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\tHashing-function agnostic Cuckoo filters for Redis.\n\u003c/p\u003e\n\nWhat's a Cuckoo Filter?\n-----------------------\nCuckoo filters are a probabilistic data structure that allows you to test for \nmembership of an element in a set without having to hold the whole set in \nmemory.\n\nThis is done at the cost of having a probability of getting a false positive \nresponse, which, in other words, means that they can only answer \"Definitely no\" \nor \"Probably yes\". The false positive probability is roughly inversely related \nto how much memory you are willing to allocate to the filter.\n\nThe most iconic data structure used for this kind of task are Bloom filters \nbut Cuckoo filters boast both better practical performance and efficiency, and, \nmore importantly, the ability of **deleting elements from the filter**. \n\nBloom filters only support insertion of new items.\nSome extensions of Bloom filters have the ability of deleting items but they \nachieve so at the expense of precision or memory usage, resulting in a far worse \ntradeoff compared to what Cuckoo filters offer.\n\n\nWhat Makes This Redis Module Interesting\n----------------------------------------\nCuckoo filters offer a very interesting division of labour between server and \nclients.\n\nSince Cuckoo filters rely on a single hashing of the original item you want to \ninsert, it is possible to off-load that part of the computation to the client. \nIn practical terms it means that instead of sending the whole item to Redis, the\nclients send `hash` and `fingeprint` of the original item.\n\n### What are the advantages of doing so?\n\t\n- You need to push trough the cable a constant amount of data per item instead \n  of N bytes *(Redis is a remote service afterall, you're going through a UNIX \n  socket at the very least)*.\n- To perform well, Cuckoo filters rely on a good choice of fingerprint for each \n  item and it should not be left to the library.\n- **The hash function can be decided by you, meaning that this module is \n  hashing-function agnostic**.\n\nThe last point is the most important one. \nIt allows you to be more flexible in case you need to reason about item hashes \nacross different clients potentially written in different languages. \n\nAdditionally, different hashing function families specialize on different use \ncases that might interest you or not. For example some work best for small data \n(\u003c 7 bytes), some the opposite. Some focus more on performance at the expense of \nmore collisions, while some others behave better than the rest on peculiar \nplatforms.\n\n[This blogpost](http://aras-p.info/blog/2016/08/09/More-Hash-Function-Tests/) \nshows a few benchmarks of different hashing function families.\n\nConsidering all of that, the choice of hashing and fingerprinting functions has \nto be up to you.\n\n*For the internal partial hashing that has to happen when reallocating a \nfingerprint server-side, this implementation uses FNV1a which is robust and fast \nfor 1 byte inputs (the size of a fingerprint).*\n\n*Thanks to how Cuckoo filters work, that choice is completely transparent to the \nclients.*\n\nInstallation \n------------\n\n1. Download a precompiled binary from the \n   [Release section](https://github.com/kristoff-it/redis-cuckoofilter/releases/) \n   of this repo or compile it yourself (instructions at the end of this README).\n\n2. Put `libredis-cuckoofilter.so` module in a folder readable by your Redis \n   server.\n\n3. To try out the module you can send \n   `MODULE LOAD /path/to/libredis-cuckoofilter.so` using redis-cli or a client of \n   your choice.\n\n4. Once you save on disk a key containing a Cuckoo filter you will need to add \n   `loadmodule /path/to/libredis-cuckoofilter.so` to your `redis.conf`, otherwise \n   Redis will not load complaining that it doesn't know how to read some data \n   from the `.rdb` file.\n\n\nQuickstart\n----------\n\n```\nredis-cli\u003e MODULE LOAD /path/to/libredis-cuckoofilter.so\nOK\n\nredis-cli\u003e CF.INIT test 64K\nOK \n \nredis-cli\u003e CF.ADD test 5366164415461427448 97\nOK\n\nredis-cli\u003e CF.CHECK test 5366164415461427448 97\n(integer) 1\n\nredis-cli\u003e CF.REM test 5366164415461427448 97\nOK \n\nredis-cli\u003e CF.CHECK test 5366164415461427448 97\n(integer) 0\n```\n\nClient-side quickstart\n----------------------\n```python\nimport redis\n\nr = redis.Redis()\n\n# Load the module if you haven't done so already\nr.execute_command(\"module\", \"load\", \"/path/to/libredis-cuckoofilter.so\")\n\n# Create a filter\nr.execute_command(\"cf.init\", \"test\", \"64k\")\n\n# Define a fingerprinting function, for hashing we'll use python's builtin `hash()` \ndef fingerprint(x):\n  return ord(x[0]) # takes the first byte and returns its numerical value\n\nitem = \"banana\"\n\n# Add an item to the filter\nr.execute_command(\"cf.add\", \"test\", hash(item), fingerprint(item))\n\n# Check for its presence\nr.execute_command(\"cf.check\", \"test\", hash(item), finterprint(item)) # =\u003e true\n\n# Check for a non-existing item\nr.execute_command(\"cf.check\", \"test\", hash(\"apple\"), fingerprint(\"apple\")) # =\u003e false\n```\n\nFingerprint size and error rates\n--------------------------------\nIn Cuckoo filters the number of bytes that we decide to use as fingerprint\nwill directly impact the maximum false positive error rate of a given filter.\nThis implementation supports 1, 2 and 4-byte wide fingerprints.\n\n### 1 (3% error)\nError % -\u003e `3.125e-02 (~0.03, i.e. 3%)`\n\n### 2 (0.01% error)\nError % -\u003e `1.22070312e-04 (~0.0001, i.e. 0.01%))`\n\n### 4 (0.0000001% error)\nError % -\u003e `9.31322574e-10 (~0.000000001, i.e. 0.0000001%)`\n\n\nComplete command list\n---------------------\n\n### - `CF.SIZEFOR universe [fpsize] [EXACT]`\n#### Complexity: O(1)\n#### Example: `CF.SIZEFOR 1000 2 EXACT`\nReturns the correct size for a filter that must hold at most `universe` items.\nDefault `fpsize` is 1, specify a different value if you need an error rate lower\nthan 3%.\nCuckoo filters should never be filled over 80% of their maximum theoretical capacity\nboth for performance reasons and because a filter that approaces 100% fill rate will\nstart refusing inserts with a `ERR too full` error.\nThis command will automatically pad `universe` for you. Use `EXACT` if you don't want \nthat behavior.\n\n### - `CF.CAPACITY size [fpsize]`\n#### Complexity: O(1)\n#### Example: `CF.CAPACITY 4G 2`\nReturns the theoretical maximum number of items that can be added to a filter of given\n`size` and `fpsize`. Default `fpsize` is 1.\n\n\n### - `CF.INIT key size [fpsize]`\n#### Complexity: O(size)\n#### Example: `CF.INIT mykey 64K`\nInstantiates a new filter. Use `CF.SIZEFOR` to know the correct value for `size`.\nSupported sizes are a power of 2 in this range: `1K .. 8G`.\nDefault error rate is 3%, use `fpsize` to specify a different target error rate.\n\n### - `CF.ADD key hash fp`\n#### Complexity: O(1) \n#### Example `CF.ADD mykey 100 97`\nAdds a new item to the filter. Both `hash` and `fp` must be numbers.\nIn particular, `hash` has to be a 64bit representable number, while `fp`\nshould be a `fpsize` representable number. As an example, a filter with \n`fpsize` set to `1` will cause the maximum recommended value of `fp` to be `255`.\nThe `fp` argument is a `u32` so `(2^32)-1` is its maximum valid value, but when\n`fpsize` is lower than `4`, high bits will be truncated (e.g. `-1 == 255` when \n`fpsize == 1`).\n\nYou can use both signed and unsigned values as long as you are consistent\nin their use. Internally all values will be transalted to unsigned.\nIf a filter is undersized/overfilled or you are adding multiple copies of \nthe same item or, worse, you're not properely handling information entropy, \nthis command will return `ERR too full`.\nRead the extented example in \n  [kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter) \nto learn more about misusage scenarios.\n\n### - `CF.REM key hash fp`\n#### Complexity: O(1)\n#### Example `CF.REM mykey 100 97`\nDeletes an item. Accepts the same arguments as `CF.ADD`. \nWARNING: this command must be used to only delete items that were\npreviously inserted. Trying to delete non-existing items will corrupt the \nfilter and cause it to lockdown. When that happens all command will start\nreturning `ERR broken`, because at that point it will be impossible to \nknow what the correct state would be. Incurring in `ERR broken` is \na usage error and should never happen. Read the extented example in \n  [kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter) \nto learn more about misusage scenarios.\n\n### - `CF.CHECK key hash fp`\n#### Complexity: O(1)\n#### Example `CF.CHECK mykey 100 97`\nChecks if an item is present in the filter or not. Returns `1` for the \npositive case and `0` otherwise. Accepts the same arguments as `CF.ADD`.\n\n### - `CF.COUNT key`\n#### Complexity: O(1)\n#### Example: `CF.COUNT mykey`\nReturns the number of items present in the filter.\n\n### - `CF.ISBROKEN key`\n#### Complexity: O(1)\n#### Example: `CF.ISBROKEN mykey`\nReturns `1` if the filter was broken because of misusage of `CF.REM`,\nreturns `0` otherwise. A broken filter cannot be fixed and will start\nreturning `ERR broken` from most comamnds.\n\n\n### - `CF.ISTOOFULL key`\n#### Complexity: O(1)\n#### Example: `CF.ISTOOFULL mykey`\nReturns `1` if the filter is too full, returns `0` otherwise.\nThis command can return `1` even if you never received a \n`ERR too full` from a call to `CF.ADD`. \nRead the extented example in \n  [kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter) \nto learn more about misusage scenarios.\n\n### - `CF.FIXTOOFULL key`\n#### Complexity: O(1) big constant\n#### Example: `CF.FIXTOOFULL mykey`\nIf you are adding and also **deleting** items from the filter\nbut in a moment of *congestion* you ended up ovferfilling the filter,\nthis command can help re-distribute some items to fix the situation.\nIt's not a command you should ever rely on because it should never \nbe needed if you properly sized your filter using `CF.SIZEFOR`.\nRead the extented example in \n  [kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter) \nto learn more about misusage scenarios.\n\nAdvanced usage\n--------------\nCheckout \n  [kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter) \nfor more information about advanced usage of Cuckoo filters and \nhow to deal (and most importantly, prevent) failure scenarios.\n\nPlanned Features\n----------------\n\n- Advanced client-side syncrhonization\n    Given that now the logic is bundled in zig-cuckoofilter and that\n    it can now be used by any C ABI compatible target (checkout the \n    repo for examples in C, JS, Python and Go), combined with Streams\n    it would be possible to keep a client-side Cuckoo filter synced\n    with one in Redis, allowing clients to keep reads locally and \n    asyncrhonously sync with Redis to obtain new updates to the filter.\n\nCompiling \n---------\nDownload the latest Zig compiler version from http://ziglang.org.\n\n### To compile for your native platform\n```sh\n$ zig build-lib -dynamic -isystem src --release-fast src/redis-cuckoofilter.zig\n```\n\n### To cross-compile\n```sh\n$ zig build-lib -dynamic -isystem src --release-fast -target x86_64-linux --library c src/redis-cuckoofilter.zig\n```\nUse `zig targets` for the complete list of available targets.\n\nLicense\n-------\n\nMIT License\n\nCopyright (c) 2019 Loris Cro\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkristoff-it%2Fredis-cuckoofilter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkristoff-it%2Fredis-cuckoofilter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkristoff-it%2Fredis-cuckoofilter/lists"}