{"id":47717232,"url":"https://github.com/lemire/constmap","last_synced_at":"2026-04-04T21:01:06.243Z","repository":{"id":347842555,"uuid":"1195465008","full_name":"lemire/constmap","owner":"lemire","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-30T14:46:27.000Z","size":20,"stargazers_count":132,"open_issues_count":1,"forks_count":6,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-03T20:59:24.224Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lemire.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-29T17:47:53.000Z","updated_at":"2026-04-03T20:07:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lemire/constmap","commit_stats":null,"previous_names":["lemire/constmap"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/lemire/constmap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fconstmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fconstmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fconstmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fconstmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lemire","download_url":"https://codeload.github.com/lemire/constmap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fconstmap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31413946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T20:09:54.854Z","status":"ssl_error","status_checked_at":"2026-04-04T20:09:44.350Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-02T19:03:07.233Z","updated_at":"2026-04-04T21:01:06.137Z","avatar_url":"https://github.com/lemire.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# constmap\n\nA fast, compact, immutable map from strings to `uint64` values in Go. It uses the binary fuse filter construction to store key-value pairs in a compact array where lookup requires only one hash computation, three array accesses, and two XOR operations.\n\nThe data structure is ideal when you have a known set of string keys at construction time and need fast, memory-efficient lookups afterward.\n\n## Reference\n\nThis implementation is based on the binary fuse filter algorithm described in:\n\n\u003e Thomas Mueller Graf and Daniel Lemire, [Binary Fuse Filters: Fast and Smaller Than Xor Filters](https://arxiv.org/abs/2201.01174), *ACM Journal of Experimental Algorithmics*, Volume 27, 2022. DOI: [10.1145/3510449](https://doi.org/10.1145/3510449)\n\nSee also the earlier xor filter paper:\n\n\u003e Thomas Mueller Graf and Daniel Lemire, [Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters](https://arxiv.org/abs/1912.08258), *ACM Journal of Experimental Algorithmics*, Volume 25, 2020. DOI: [10.1145/3376122](https://doi.org/10.1145/3376122)\n\n## Installation\n\n```\ngo get github.com/lemire/constmap\n```\n\n\n## Usage\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/lemire/constmap\"\n)\n\nfunc main() {\n\tkeys := []string{\"apple\", \"banana\", \"cherry\"}\n\tvalues := []uint64{100, 200, 300}\n\n\tcm, err := constmap.New(keys, values)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tfmt.Println(cm.Map(\"banana\")) // 200\n}\n```\n\nThe `keys` and `values` slices must have equal length, and keys must be unique. After construction, the `ConstMap` is immutable. Looking up a key that was not in the original set returns an undefined value.\n\n### Verified Lookups\n\nIf you need to detect missing keys, use `VerifiedConstMap`. It stores an additional fingerprint per key and returns the sentinel `NotFound` for keys not in the original set:\n\n```go\nvm, err := constmap.NewVerified(keys, values)\nif err != nil {\n\tlog.Fatal(err)\n}\n\nfmt.Println(vm.Map(\"banana\")) // 200\nfmt.Println(vm.Map(\"grape\"))  // constmap.NotFound (0xFFFFFFFFFFFFFFFF)\n```\n\nThis doubles memory usage (~18 bytes/key instead of ~9) but lookup remains fast.\n\n## Serialization\n\nA `ConstMap` can be serialized to disk and loaded back later, avoiding the cost of reconstruction. The binary format includes a FNV-1a checksum to detect corruption.\n\n```go\n// Save to file.\nerr := cm.SaveToFile(\"mymap.cmap\")\n\n// Load from file.\ncm, err := constmap.LoadFromFile(\"mymap.cmap\")\n```\n\nFor streaming use, `WriteTo` and `ReadFrom` work with any `io.Writer` / `io.Reader`:\n\n```go\n// Write to any io.Writer.\nn, err := cm.WriteTo(w)\n\n// Read from any io.Reader.\nvar cm constmap.ConstMap\nn, err := cm.ReadFrom(r)\n```\n\n## Running Tests\n\n```\ngo test -v\n```\n\n## Performance gains\n\nThe construction time is higher (as expected for any compact data structure), but lookups are optimized for speed. I ran benchmarks on my Apple M4 Max processor to compare constmap lookups against Go's built-in `map[string]uint64`. The test uses 1 million keys.\n\n| Data Structure    | Lookup Time | Memory Usage |\n|-------------------|-------------|--------------|\n| ConstMap          | 7.6 ns/op   | 9 bytes/key  |\n| VerifiedConstMap  | 13 ns/op    | 18 bytes/key |\n| Go Map            | 23 ns/op    | 56 bytes/key |\n\n## Benchmarks\n\nThe benchmark suite compares `ConstMap` against Go's built-in `map[string]uint64` using 1,000,000 keys. To run:\n\n```\ngo test -bench=. -benchmem\n```\n\nThere are three benchmarks:\n\n- **BenchmarkConstMap** -- lookup throughput for `ConstMap.Map()`\n- **BenchmarkVerifiedConstMap** -- lookup throughput for `VerifiedConstMap.Map()`\n- **BenchmarkGoMap** -- lookup throughput for Go's built-in map\n\nFor stable, reproducible results:\n\n```\ngo test -bench=. -benchmem -count=5 -benchtime=3s\n```\n\nThe `-count=5` flag runs each benchmark five times so you can assess variance. The `-benchtime=3s` flag gives each iteration more time to stabilize.\n\n## Memory Usage\n\nRun the memory comparison test to see the retained memory of each data structure with 1,000,000 keys:\n\n```\ngo test -run TestMemoryUsage -v\n```\n\nThe `ConstMap` stores approximately 1.125 x *n* x 8 bytes (roughly 9 bytes per key), the `VerifiedConstMap` uses twice that (~18 bytes/key), while Go's `map[string]uint64` typically uses around 50-60 bytes per key for keys of this size.\n\n## How It Works\n\nGiven *n* key-value pairs, the algorithm:\n\n1. Hashes each key (using [xxhash](https://github.com/cespare/xxhash)) and maps it to three positions in an array of size ~1.125*n* using overlapping segments.\n2. Uses a peeling process to find an ordering where each key can be assigned to one of its three positions uniquely.\n3. Walks the ordering in reverse, setting each array cell so that `array[h0] XOR array[h1] XOR array[h2] == value` for every key.\n\nLookup computes the same three positions and XORs the three array cells to recover the value. This gives O(1) lookup with minimal memory overhead.\n\nCompared to xor filters which divide the array into three equal blocks (~1.23*n* overhead), binary fuse filters use overlapping segments for better locality and a lower space overhead (~1.125*n*), and they construct faster.\n\n## License\n\nApache License 2.0. See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Fconstmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemire%2Fconstmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Fconstmap/lists"}