{"id":15470480,"url":"https://github.com/mpope9/efuse_filter","last_synced_at":"2025-04-22T12:09:41.003Z","repository":{"id":48076641,"uuid":"393515146","full_name":"mpope9/efuse_filter","owner":"mpope9","description":"Erlang NIF for Binary Fuse Filter. Fast and Smaller Than Xor Filters.","archived":false,"fork":false,"pushed_at":"2021-08-17T19:35:31.000Z","size":998,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T12:08:58.891Z","etag":null,"topics":["bloom-filter","elixir","erlang","fuse-filter","nif","probabilistic-data-structures","xor-filter"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpope9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"patreon":"mpope"}},"created_at":"2021-08-06T22:09:49.000Z","updated_at":"2024-11-14T07:31:44.000Z","dependencies_parsed_at":"2022-08-12T18:01:23.066Z","dependency_job_id":null,"html_url":"https://github.com/mpope9/efuse_filter","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpope9%2Fefuse_filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpope9%2Fefuse_filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpope9%2Fefuse_filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpope9%2Fefuse_filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpope9","download_url":"https://codeload.github.com/mpope9/efuse_filter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250237832,"owners_count":21397401,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloom-filter","elixir","erlang","fuse-filter","nif","probabilistic-data-structures","xor-filter"],"created_at":"2024-10-02T02:04:57.427Z","updated_at":"2025-04-22T12:09:40.986Z","avatar_url":"https://github.com/mpope9.png","language":"C","funding_links":["https://patreon.com/mpope"],"categories":[],"sub_categories":[],"readme":"efuse_filter\n====\n\nA dependency free [Binary Fuse Filter](https://github.com/FastFilter/xor_singleheader) Erlang NIF.\n\nThey're faster and smaller than Bloom, Cuckoo, and Xor filters.\n\nThis library is almost API compatible with the [exor_filter](https://github.com/mpope9/exor_filter) library, and can be used as a replacement without too much hassle.\n\n## Table Of Contents\n* [Benchmarks](#benchmarks)\n* [Installation](#installation)\n* [Example Usage](#example-usage)\n   * [Basic Usage](#basic-usage)\n   * [Elixir Example](#elixir-example)\n   * [Incremental Initialization](#incremental-initialization)\n* [Custom Return Values](#custom-return-values)\n* [Custom Hashing](#custom-hashing)\n* [Serialization](#serialization)\n\n## Benchmarks\n![Benchmark Graph](/images/results.png)\n\nThis was benchmarked with the [exor_benchmark](https://github.com/mpope9/exor_bechmark) suite that compares several Erlang and Elixir bloom / xor / fuse filter implementations. The full benchmark and results can be found there.\n\n## Installation\n\nThis library requires Erlang version 24+.\n\n[This library is available on hex.pm](https://hex.pm/packages/efuse_filter).\n\n### rebar3\n```erlang\n%% rebar.config\n\n{deps, [\n  %% hex.pm\n  {efuse_filter, \"0.1.0\"},\n\n  %% git\n  {efuse_filter, {git, \"git://github.com/mpope9/efuse_filter.git\", {tag, \"0.1.0\"}}}\n]}.\n```\n\n### mix\n```elixir\n## mix.exs\n\ndefp deps do\n  [\n    {:efuse_filter, \"~\u003e 0.1.0\"}\n  ]\nend\n```\n\n## Example Usage\n\n### Basic Usage\n```erlang\nFilter = fuse8:new([\"cat\", \"dog\", \"mouse\"]),\ntrue = fuse8:contain(Filter, \"cat\"),\nfalse = fuse8:contain(Filter, \"goose\").\n```\n\n### Elixir Example\n```Elixir\nalias :fuse8 as: Fuse8\n\n# ...\n\ntrue =\n    [1, 2, 3, 4]\n    |\u003e Fuse8.new()\n    |\u003e Fuse8.contain(1)\n```\n\n### Incremental Initialization\n\nThis builds a filter over multiple calls. Filters are immutable. Once finalized, more elements cannot be added.\n\nThe `exor_filter` stores elements using a compressed bitmap. This library uses the optimized Erlang `sets` version 2 module, and keeps them in Erlang memory. This library will have a higher memory footprint when initializing a filter incrementally compared to the `exor_filter`.\n\n```erlang\nFilter0 = fuse8:new_empty(),\n\n%% Add multiple elements.\nFilter1 = fuse8:add(Filter0, [1, 2]),\nFilter2 = fuse8:add(Filter1, [3, 4]),\n\n%% Add single element.\nFilter3 = fuse8:add(Filter2, 5),\n\nFilter4 = fuse8:finalize(Filter3),\n\ntrue = fuse8:contain(Filter4, 1),\nfalse = fuse8:contain(Filter4, 6).\n```\n\n## Custom Return Values\n`fuse8:contain/3` can return a custom value instead of `false` if the required item isn't present in the filter:\n\n```erlang\nFilter = fuse8:new([\"Ricky Bobby\", \"Cal Naughton Jr.\"]),\ntrue = fuse8:contain(Filter, \"Ricky Bobby\", {error, not_found}),\n{error, not_found} = fuse8:contain(Filter, \"Reese Bobby\", {error, not_found}).\n```\n\n## Custom Hashing\nBy default this library uses the [`erlang:phash2/1`](https://erlang.org/doc/man/erlang.html#phash2-1) function. If you want to use your own custom hashing, pass `none` to the `fuse8:new/2` function. Values passed to `fuse8:/new` and `fuse8:contain` need to be pre-hashed, and those functions will return a `{error, pre_hashed_values_should_be_ints}` error of non-integer values are passed.\n\nExample usage:\n```erlang\nPreHashedList = [...],\nFilter = fuse8:new(PreHashedList, none),\ntrue = fuse8:contain(Filter, hd(PreHashedList)).\n```\n\nUsing your own hashing method has benefits. If the filter is sent between services other platforms might not have `erlang:phash2/1` available. Also for larger filters, a hashing function with a larger key space might be desired. Keys in the C code are mapped to a `uint64_t`. Keep that in mind when choosing the hashing function.\n\nThe `exor_filter` supports passing a function to do the hashing. This library does not.\n\n## Serialization\nFilters can be serialized using `fuse8:to_bin/1` and `fuse8:from_bin/1` and `fuse8:from_bin/2`.\n\nFunctions are provided to the filter in binary form, instead of a nif reference. This can be useful to interop with other platforms / systems. The bin returned can be used with `fuse8:contain` for ease of use. Example usage:\n\nExample usage:\n```erlang\nFilter     = fuse8:new([\"test1\", \"test2\", \"test3\"]),\nBinFilter  = fuse8:to_bin(Filter),\ntrue       = fuse8:contain(BinFilter, \"test1\"),\nFilter2    = fuse8:from_bin(BinFilter),\ntrue       = fuse8:contain(Filter2, \"test1\").\n```\n\nTo use custom hashing with a serialized filter, the following `fuse8:contain/4` method is provided:\n```erlang\nFilter     = fuse8:new([\"test1\", \"test2\", \"test3\"]),\nBinFilter  = fuse8:to_bin(Filter),\ntrue       = fuse8:contain(BinFilter, \"test1\", false, none),\nFilter2    = fuse8:from_bin(BinFilter, none),\ntrue       = fuse8:contain(Filter2, \"test1\").\n```\n\nBuild\n====\n\n```bash\n$ rebar3 compile\n```\n\nTest\n====\n\n```bash\n$ rebar3 eunit\n$ rebar3 cover\n```\n\nDocs\n====\n\n```\n$ rebar3 edoc\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpope9%2Fefuse_filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpope9%2Fefuse_filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpope9%2Fefuse_filter/lists"}