{"id":15068173,"url":"https://github.com/peterrk/pagebloomfilter","last_synced_at":"2025-04-10T04:36:05.613Z","repository":{"id":154011455,"uuid":"625248338","full_name":"PeterRK/PageBloomFilter","owner":"PeterRK","description":"May be fastest bloom filter in C++/Go/Java/C#/Python/Rust","archived":false,"fork":false,"pushed_at":"2024-05-02T10:09:28.000Z","size":415,"stargazers_count":135,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-05-02T22:54:04.626Z","etag":null,"topics":["bloom-filter","cpp","csharp","golang","hash","high-performance","java","python","rosetta-code","rust"],"latest_commit_sha":null,"homepage":"","language":"Assembly","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PeterRK.png","metadata":{"files":{"readme":"README-CN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-08T14:22:57.000Z","updated_at":"2024-06-19T10:02:45.656Z","dependencies_parsed_at":"2024-06-19T10:02:35.343Z","dependency_job_id":"d7b48cfd-547a-46eb-a2e5-fcb2bdd1605e","html_url":"https://github.com/PeterRK/PageBloomFilter","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterRK%2FPageBloomFilter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterRK%2FPageBloomFilter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterRK%2FPageBloomFilter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PeterRK%2FPageBloomFilter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PeterRK","download_url":"https://codeload.github.com/PeterRK/PageBloomFilter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248158724,"owners_count":21057189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloom-filter","cpp","csharp","golang","hash","high-performance","java","python","rosetta-code","rust"],"created_at":"2024-09-25T01:31:58.779Z","updated_at":"2025-04-10T04:36:05.580Z","avatar_url":"https://github.com/PeterRK.png","language":"Assembly","readme":"# PageBloomFilter\n\n采用分页设计的布隆过滤器，兼顾存储密度与访问性能。\n\n## C++\n```cpp\nauto bf = NEW_BLOOM_FILTER(500, 0.01);\nif (bf.set(\"Hello\")) {\n    std::cout \u003c\u003c \"set new Hello\" \u003c\u003c std::endl;\n}\nif (bf.test(\"Hello\")) {\n    std::cout \u003c\u003c \"find Hello\" \u003c\u003c std::endl;\n}\n```\n在AESNI指令加持下，性能一骑绝尘，标准版也足够惊艳。\n```\n// U7-155H \u0026 Clang-18\npbf-set:        10.9247 ns/op\npbf-test:        6.0765 ns/op\npbf-aesni-set:   8.3275 ns/op\npbf-aesni-test:  4.3405 ns/op\nlibbf-set:      36.6518 ns/op\nlibbf-test:     31.7608 ns/op\nlibbloom-set:   33.0665 ns/op\nlibbloom-test:  13.5359 ns/op\n```\n\n## Go\n\n```go\n// import \"github.com/PeterRK/PageBloomFilter/go\"\n// 有效容量500，假阳率0.01\nbf := pbf.NewBloomFilter(500, 0.01)\nif bf.Set(\"Hello\") {\n    fmt.Println(\"set new Hello\")\n}\nif bf.Test(\"Hello\") {\n    fmt.Println(\"find Hello\")\n}\n```\n\n除了原生实现，在AMD64环境中还提供基于**函数注入技术**的实现，具体而言就是将C函数编译后注入到Go程序中以免除CGO的调用开销。在Xeon-8374C上测试50万元素，Go注入版较原生版有一倍左右的性能提升，仅比C++版略慢。\n\n```\nname   old time/op  new time/op  delta\nSet4   53.6ns ± 6%  26.5ns ± 6%  -50.52%  (p=0.000 n=20+20)\nTest4  40.5ns ± 5%  21.2ns ± 5%  -47.63%  (p=0.000 n=20+18)\nSet5   56.4ns ± 5%  28.0ns ± 5%  -50.34%  (p=0.000 n=20+19)\nTest5  41.5ns ± 3%  18.8ns ± 7%  -54.72%  (p=0.000 n=20+19)\nSet6   57.6ns ± 5%  29.1ns ± 5%  -49.44%  (p=0.000 n=20+20)\nTest6  42.2ns ± 4%  18.5ns ± 7%  -56.22%  (p=0.000 n=20+18)\nSet7   58.8ns ± 4%  30.8ns ± 9%  -47.68%  (p=0.000 n=20+20)\nTest7  43.9ns ± 6%  18.9ns ± 8%  -56.98%  (p=0.000 n=20+19)\nSet8   58.4ns ± 9%  32.4ns ± 5%  -44.53%  (p=0.000 n=20+19)\nTest8  44.8ns ± 2%  18.4ns ± 7%  -58.86%  (p=0.000 n=19+20)\n```\n\nAMD64环境中注入版默认开启，编译前最好先执行[go-inject.sh](go/go-inject.sh)生成新的注入函数。注入函数生成脚本依赖clang和binutils，以及python，建议在Linux环境执行。\n\n[测评](https://gist.github.com/PeterRK/b0df9e80caaaee1e9349e295cb435a67) 表明本实现比知名的 [bits-and-blooms](https://github.com/bits-and-blooms/bloom)和[Tyler Treat版](https://github.com/tylertreat/BoomFilters)要快2倍：\n```\n// i7-10710U \u0026 Go-1.20\nBenchmarkPageBloomFilterSet-6        1000000            32.70 ns/op\nBenchmarkPageBloomFilterTest-6       1000000            20.23 ns/op\nBenchmarkBitsAndBloomSet-6           1000000           120.5  ns/op\nBenchmarkBitsAndBloomTest-6          1000000            81.46 ns/op\nBenchmarkTylerTreatSet-6             1000000            98.30 ns/op\nBenchmarkTylerTreatTest-6            1000000            60.69 ns/op\n\n// U7-155H \u0026 Go-1.20\nBenchmarkPageBloomFilterSet-16       1000000            13.95 ns/op\nBenchmarkPageBloomFilterTest-16      1000000             8.40 ns/op\nBenchmarkBitsAndBloomSet-16          1000000            44.57 ns/op\nBenchmarkBitsAndBloomTest-16         1000000            37.94 ns/op\nBenchmarkTylerTreatSet-16            1000000            43.94 ns/op\nBenchmarkTylerTreatTest-16           1000000            20.80 ns/op\n\n// U7-155H \u0026 Go-1.24 (to fix: injection is broken since Go-1.21)\nBenchmarkPageBloomFilterSet-16       1000000            26.35 ns/op\nBenchmarkPageBloomFilterTest-16      1000000            23.97 ns/op\n```\n\n## Java\n```java\nPageBloomFilter bf = PageBloomFilter.New(500, 0.01);\nbyte[] hello = \"Hello\".getBytes(\"UTF-8\");\nif (bf.set(hello)) {\n    System.out.println(\"set new Hello\");\n}\nif (bf.test(hello)) {\n    System.out.println(\"find Hello\");\n}\n```\n[测评](java/src/test/java/com/github/peterrk/pbf/BloomFilterBenchmark.java) 表明本实现比Google的[Guava](https://github.com/google/guava)要快很多，而有时稍逊于Alexandr Nikitin的[bloom-filter-scala](https://github.com/alexandrnikitin/bloom-filter-scala)。由于缺少针对性优化，Java版没有Go版快。\n```\n// i7-10710U \u0026 OpenJDK-17\npbfSet       50.962 ns/op\npbfTest      40.465 ns/op\nguavaSet    133.514 ns/op\nguavaTest   112.318 ns/op\nnikitinSet   86.931 ns/op\nnikitinTest  62.133 ns/op\n\n// U7-155H \u0026 OpenJDK-21\npbfSet       24.562 ns/op\npbfTest      20.511 ns/op\nguavaSet     44.889 ns/op\nguavaTest    45.652 ns/op\nnikitinSet   22.474 ns/op\nnikitinTest  18.489 ns/op\n```\n\n## C#\n```csharp\nvar bf = PageBloomFilter.New(500, 0.01);\nvar hello = Encoding.ASCII.GetBytes(\"Hello\")\nif (bf.Set(hello)) {\n    Console.WriteLine(\"set new Hello\");\n}\nif (bf.Test(hello)) {\n    Console.WriteLine(\"find Hello\");\n}\n```\nC#版代码和Java版高度一致，不过跑出来要慢不少。性能胜过[BloomFilter.NetCore](https://github.com/vla/BloomFilter.NetCore)。\n```\n// i7-10710U \u0026 .NET-7\npbf-set:  83.461274 ns/op\npbf-test: 74.953785 ns/op\n\n// U7-155H \u0026 .NET-9\npbf-set:     28.63103 ns/op\npbf-test:    22.88545 ns/op\nbf.net-set:  41.66280 ns/op\nbf.net-test: 40.12608 ns/op\n```\n\n## Python\n```python\nbf = pbf.create(500, 0.01)\nif bf.set(\"Hello\"):\n    print(\"set new Hello\")\nif bf.test(\"Hello\"):\n    print(\"find Hello\")\n```\nPython版基于C扩展实现，虽然还是慢，不过足以吊打[pybloom](https://github.com/jaybaird/python-bloomfilter)。\n```\n// i7-10710U \u0026 Python-3.11\npbf-set:       307.835638 ns/op\npbf-test:      289.679349 ns/op\npybloom-set:  2770.372372 ns/op\npybloom-test: 2417.377588 ns/op\n\n// U7-155H \u0026 Python-3.12\npbf-set:  127.227066 ns/op\npbf-test: 101.274176 ns/op\n```\n\n## Rust\n```rust\nlet mut bf = pbf::new_bloom_filter(500, 0.01);\nlet hello = \"Hello\".as_bytes();\nif (bf.set(hello)) {\n    println!(\"set new Hello\");\n}\nif (bf.test(hello)) {\n    println!(\"find Hello\");\n}\n```\nRust版也缺少针对性优化，照样快过Java。看上去比[fastbloom](https://github.com/tomtomwombat/fastbloom)和[rust-bloom-filter](https://github.com/jedisct1/rust-bloom-filter)强。\n```\n// i7-10710U \u0026 Rust-1.65\npbf-set:  45.99ns/op\npbf-test: 27.81ns/op\n\n// U7-155H \u0026 Rust-1.80\npbf-set:        19.85ns/op\npbf-test:       12.50ns/op\nfastbloom-set:  19.97ns/op\nfastbloom-test: 14.93ns/op\nrbf-set:        36.51ns/op\nrbf-test:       24.93ns/op\n```\n\n## 横向比较\n![](images/U7-155H.png)\n将在U7-155H上的测试数据放到一起看，可以得到性能排位：C++，Go，Rust，Java，C#，Python。\n\n## 序列化与反序列化\n不同语言实现的数据结构是一致（除了C++的aesni加强版），可以跨语言使用。虽然这里不提供专门的序列化和反序列化API，但也很容易实现：保存和加载`way`、`page_level`、`unique_cnt`三个参数，以及`data`的位图即可。其中`way`和`page_level`是个很小的整数， 可以分别用4bit表示。\n```cpp\n// C++\nauto bf = pbf::New(500, 0.01);\nauto bf2 = pbf::New(bf-\u003eway(), bf-\u003epage_level(), bf-\u003edata(), bf-\u003edata_size(), bf-\u003eunique_cnt());\n\n// 示例格式（并非标准）\nstruct Pack {\n    uint32_t way : 4;\n    uint32_t page_level : 4;\n    uint32_t unique_cnt : 24;\n    uint32_t data_size;\n    uint8_t data[0];\n};\n```\n```go\n// GO\nbf := pbf.NewBloomFilter(500, 0.01)\nbf2 := pbf.CreatePageBloomFilter(bf.Way(), bf.PageLevel(), bf.Data(), bf.Unique())\n```\n```java\n// Java\nPageBloomFilter bf = PageBloomFilter.New(500, 0.01);\nPageBloomFilter bf2 = PageBloomFilter.New(bf.getWay(), bf.getPageLevel(), bf.getData(), bf.getUniqueCnt());\n```\n```csharp\n// C#\nvar bf = PageBloomFilter.New(500, 0.01);\nvar bf2 = PageBloomFilter.New(bf.Way, bf.PageLevel, bf.Data, bf.UniqueCnt);\n```\n```python\n# Python\nbf = pbf.create(500, 0.01)\nbf2 = pbf.restore(bf.way, bf.page_level, bf.data, bf.unique_cnt)\n```\n```rust\n// Rust\nlet mut bf = pbf::new_bloom_filter(500, 0.01);\nlet mut bf2 = pbf::restore_pbf(bf.get_way(), bf.get_page_level(), bf.get_data(), bf.get_unique_cnt());\n```\n\n## 详细测试\n![](images/Xeon-8374C.png)\n在Xeon-8374C上测试50万元素，平均每次操作小于25ns，SIMD能有效加速查询操作。\n\n![](images/EPYC-7K83.png)\n在EPYC-7K83上测试表现略逊，SIMD加速效果不明显。\n\n![](images/Xeon-8475B.png)\n在Xeon-8475B上测试SIMD模式，使用aesni-hash可获得显著加速（**小于7ns的test操作**）。\n\n![](images/EPYC-9T24.png)\n在EPYC-9T24上测试SIMD模式，使用aesni-hash也可获得显著加速，但没有Intel平台上显著。\n\n## 理论分析\n\n### 每元素字节数与假阳率的关系\n![](images/byte.png)\n\n### 容积率与假阳率的关系\n![](images/ratio.png)\n\n---\n[【中文】](README-CN.md) [【英文】](README.md)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterrk%2Fpagebloomfilter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeterrk%2Fpagebloomfilter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterrk%2Fpagebloomfilter/lists"}