{"id":31359368,"url":"https://github.com/perbu/cdb","last_synced_at":"2025-09-26T23:25:08.342Z","repository":{"id":313350437,"uuid":"1050970111","full_name":"perbu/cdb","owner":"perbu","description":"CDB library for Go with memory map and 64b support.","archived":false,"fork":false,"pushed_at":"2025-09-05T12:37:45.000Z","size":59,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-05T14:30:32.900Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perbu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-05T08:29:32.000Z","updated_at":"2025-09-05T12:37:48.000Z","dependencies_parsed_at":"2025-09-05T14:31:53.844Z","dependency_job_id":"9e59d16a-ab44-4963-9d1b-1f1a5e985358","html_url":"https://github.com/perbu/cdb","commit_stats":null,"previous_names":["perbu/cdb"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/perbu/cdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fcdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fcdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fcdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fcdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perbu","download_url":"https://codeload.github.com/perbu/cdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fcdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277156894,"owners_count":25770721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-26T02:00:09.010Z","response_time":78,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-26T23:25:07.465Z","updated_at":"2025-09-26T23:25:08.337Z","avatar_url":"https://github.com/perbu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CDB - 64-bit Constant Database\n\nA native Go implementation of CDB,Constant Database, with 64-bit support and memory-mapped reading. Originates\nfrom [github.com/colinmarc/cdb](https://github.com/colinmarc/cdb). Although there is little left of the original code,\nthe algorithm is the same.\n\nOriginally, CDB was described as:\n\u003e CDB is a fast, reliable, simple package for creating and reading constant databases. Its database structure provides\n\u003e several features:\n\u003e - **Fast lookups**: A successful lookup in a large database normally takes just two disk accesses. An unsuccessful\n    lookup takes only one.\n\u003e - **Low overhead**: A database uses 4096 bytes for the index, plus 16 bytes per hash table entry, plus the space for\n    keys and data.\n\u003e - **Large file support**: This 64-bit implementation can handle databases up to 8 exabytes (2^63 bytes). There are no\n    other restrictions; records don't even have to fit into memory.\n\u003e - **Machine-independent format**: Databases are stored in a consistent binary format across platforms.\n\nWith mmap reads, a further improvement is gained. Care should be taken when using this on many large databases, as the\nmemory pressure will be different from what you might be used to.\n\n## Features\n\n- **64-bit only**: Simplified implementation supporting only 64-bit databases. These are only marginally larger than the\n  32-bit equivalent and have no size restrictions.\n- **Memory-mapped reads**: Zero-copy access using mmap for optimal read performance. Reduces allocations by 90%.\n- **Native Go iterators**: Support for Go 1.23+ `range` syntax over keys, values, and key-value pairs\n- **Buffered writes**: 64KB write buffer for efficient database creation\n\n## Quick Start\n\n```go\npackage main\n\nimport (\n\t\"log\"\n\n\t\"github.com/perbu/cdb\"\n)\n\nfunc main() {\n\t// Create a new database\n\twriter, err := cdb.Create(\"/tmp/example.cdb\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\t// Write some key/value pairs\n\twriter.Put([]byte(\"Alice\"), []byte(\"Practice\"))\n\twriter.Put([]byte(\"Bob\"), []byte(\"Hope\"))\n\twriter.Put([]byte(\"Charlie\"), []byte(\"Horse\"))\n\n\t// Freeze the database and open for reads\n\tdb, err := writer.Freeze()\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer db.Close()\n\n\t// Read a value\n\tvalue, err := db.Get([]byte(\"Alice\"))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tlog.Println(string(value)) // Output: Practice\n\n\t// Iterate over all key-value pairs (Go 1.23+)\n\tfor key, value := range db.All() {\n\t\tlog.Printf(\"%s: %s\", key, value)\n\t}\n\n\t// Iterate over just keys\n\tfor key := range db.Keys() {\n\t\tlog.Printf(\"Key: %s\", key)\n\t}\n\n\t// Iterate over just values  \n\tfor value := range db.Values() {\n\t\tlog.Printf(\"Value: %s\", value)\n\t}\n}\n```\n\n## File Format\n\nThis implementation uses a 64-bit CDB format:\n\n- **Index**: 4096 bytes at file start (256 tables × 16 bytes each)\n- **Data section**: Key-value pairs with 64-bit length prefixes (16 bytes per record header)\n- **Hash tables**: Linear probing collision resolution with 64-bit offsets\n\n## Performance\n\nThe performance goal was to get rid of the context switching and allocations that came with the original\nCDB implementation. A read through a memory map will have no slowdown if the content is in the page cache already,\navoiding both the seek and read syscalls.\n\nThe most important metric for me is the time to iterate over the database. At below 2 ns, it is hard to see how it can\nbe faster. The benchmarks show a clear lead to Apple Silicon, likely because of lower memory latency.\n\n```\ngoos: darwin\ngoarch: arm64\npkg: github.com/perbu/cdb\ncpu: Apple M4\nBenchmarkGet-10                         54848398                22.04 ns/op            0 B/op          0 allocs/op\nBenchmarkMmapIteratorAll-10             628005786                1.896 ns/op           0 B/op          0 allocs/op\nBenchmarkMmapIteratorKeys-10            708844383                1.678 ns/op           0 B/op          0 allocs/op\nBenchmarkMmapIteratorValues-10          708912603                1.677 ns/op           0 B/op          0 allocs/op\nBenchmarkPut-10                          1731052               715.6 ns/op           730 B/op          8 allocs/op\n```\n\nPerformance on a 64-bit Linux machine is similar:\n\n```\ngoos: linux\ngoarch: amd64\npkg: github.com/perbu/cdb\ncpu: AMD Ryzen 7 9800X3D 8-Core Processor\nBenchmarkGet-16                         45158083                26.57 ns/op            0 B/op          0 allocs/op\nBenchmarkMmapIteratorAll-16             445780636                2.675 ns/op           0 B/op          0 allocs/op\nBenchmarkMmapIteratorKeys-16            452996073                2.677 ns/op           0 B/op          0 allocs/op\nBenchmarkMmapIteratorValues-16          451605426                2.650 ns/op           0 B/op          0 allocs/op\nBenchmarkPut-16                          1572086               745.9 ns/op           734 B/op          8 allocs/op```\n```\n## API Reference\n\n### Writing\n\n```go\npackage main\n\nimport (\n\t\"log\"\n\t\"os\"\n\n\t\"github.com/perbu/cdb\"\n)\n\nfunc main() {\n\tpath := \"/tmp/example.cdb\"\n\t\n\t// Create new database file\n\twriter, err := cdb.Create(path)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\t\n\t// Add key-value pair\n\tkey := []byte(\"example\")\n\tvalue := []byte(\"data\")\n\terr = writer.Put(key, value)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\t\n\t// Finalize and return reader\n\tdb, err := writer.Freeze()\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer db.Close()\n\t\n\t// Alternative: use custom WriteSeeker\n\tfile, err := os.Create(\"/tmp/custom.cdb\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer file.Close()\n\t\n\twriter2, err := cdb.NewWriter(file)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\t\n\terr = writer2.Put([]byte(\"test\"), []byte(\"value\"))\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\t\n\terr = writer2.Close()\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n}\n```\n\n### Reading\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\n\t\"github.com/perbu/cdb\"\n)\n\nfunc main() {\n\tpath := \"/tmp/example.cdb\"\n\t\n\t// Open with memory mapping\n\tdb, err := cdb.Open(path)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer db.Close()\n\t\n\t// Lookup value\n\tkey := []byte(\"example\")\n\tvalue, err := db.Get(key)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Printf(\"Value: %s\\n\", value)\n\t\n\t// Get file size\n\tsize := db.Size()\n\tfmt.Printf(\"Database size: %d bytes\\n\", size)\n\t\n\t// Alternative: Create from open file\n\tfile, err := os.Open(path)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer file.Close()\n\t\n\tdb2, err := cdb.Mmap(file)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer db2.Close()\n}\n```\n\n### Iteration (Go 1.23+)\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/perbu/cdb\"\n)\n\nfunc main() {\n\t// Create a sample database first\n\twriter, err := cdb.Create(\"/tmp/iter_example.cdb\")\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\t\n\twriter.Put([]byte(\"key1\"), []byte(\"value1\"))\n\twriter.Put([]byte(\"key2\"), []byte(\"value2\"))\n\twriter.Put([]byte(\"key3\"), []byte(\"value3\"))\n\t\n\tdb, err := writer.Freeze()\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tdefer db.Close()\n\t\n\t// Iterate key-value pairs\n\tfmt.Println(\"Key-Value pairs:\")\n\tfor key, value := range db.All() {\n\t\tfmt.Printf(\"  %s: %s\\n\", key, value)\n\t}\n\t\n\t// Iterate keys only\n\tfmt.Println(\"Keys:\")\n\tfor key := range db.Keys() {\n\t\tfmt.Printf(\"  %s\\n\", key)\n\t}\n\t\n\t// Iterate values only\n\tfmt.Println(\"Values:\")\n\tfor value := range db.Values() {\n\t\tfmt.Printf(\"  %s\\n\", value)\n\t}\n}\n```\n\n---\n\nBased on the original [CDB specification](http://cr.yp.to/cdb.html) by D. J. Bernstein.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperbu%2Fcdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperbu%2Fcdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperbu%2Fcdb/lists"}