{"id":17281369,"url":"https://github.com/mr-karan/barreldb","last_synced_at":"2026-03-01T20:34:18.534Z","repository":{"id":64940543,"uuid":"570417683","full_name":"mr-karan/barreldb","owner":"mr-karan","description":"A disk based KV store (based on Bitcask implementation)","archived":false,"fork":false,"pushed_at":"2023-12-04T11:10:40.000Z","size":121,"stargazers_count":200,"open_issues_count":1,"forks_count":10,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-02-17T22:47:49.232Z","etag":null,"topics":["bitcask","key-value","kv-store"],"latest_commit_sha":null,"homepage":"https://mrkaran.dev/posts/barreldb/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mr-karan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-25T06:18:21.000Z","updated_at":"2026-02-07T22:09:12.000Z","dependencies_parsed_at":"2024-06-20T08:23:08.844Z","dependency_job_id":"3c13dfc2-beda-496d-9e07-3efa8e2b20aa","html_url":"https://github.com/mr-karan/barreldb","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mr-karan/barreldb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fbarreldb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fbarreldb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fbarreldb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fbarreldb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mr-karan","download_url":"https://codeload.github.com/mr-karan/barreldb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mr-karan%2Fbarreldb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29983211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-01T16:35:47.903Z","status":"ssl_error","status_checked_at":"2026-03-01T16:35:44.899Z","response_time":124,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcask","key-value","kv-store"],"created_at":"2024-10-15T09:45:41.359Z","updated_at":"2026-03-01T20:34:18.503Z","avatar_url":"https://github.com/mr-karan.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"./_docs/logo.png\" alt=\"logo\" width=\"15%\" /\u003e\n\u003c/p\u003e\n\n# barreldb\n\n_A disk based key-value store based on [Bitcask](https://en.wikipedia.org/wiki/Bitcask)_.\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/mr-karan/barreldb.svg)](https://pkg.go.dev/github.com/mr-karan/barreldb)\n[![Go Report Card](https://goreportcard.com/badge/github.com/mr-karan/barreldb)](https://goreportcard.com/report/github.com/mr-karan/barreldb)\n[![GitHub Actions](https://github.com/mr-karan/barreldb/actions/workflows/release.yml/badge.svg)](https://github.com/mr-karan/barreldb/actions/workflows/release.yml)\n\n---\n\nBarrelDB is a Golang implementation of [Bitcask by Riak](https://riak.com/assets/bitcask-intro.pdf) paper and aims to closely follow the spec.\n\nBitcask is based on a log-structured hash table to store key-value data on disk. It opens a \"datafile\" (term used for a Bitcask DB file) in an _append-only_ mode and all the writes are sequentially written to this file. Additionally, it also updates an in-memory hash table which maps the key with the offset of the record in the file. This clever yet simple design decision makes it possible to retrieve records from the disk using a _single_ disk seek.\n\n### Benefits of this approach\n\n- Low Latency: Write queries are handled with a single O(1) disk seek. Keys lookup happen in memory using a hash table lookup. This makes it possible to achieve low latency even with a lot of keys/values in the database. Bitcask also relies on the filesystem read-ahead cache for a faster reads.\n- High Throughput: Since the file is opened in \"append only\" mode, it can handle large volumes of write operations with ease. \n- Predictable performance: The DB has a consistent performance even with growing number of records. This can be seen in benchmarks as well.\n- Crash friendly: Bitcask commits each record to the disk and also generates a \"hints\" file which makes it easy to recover in case of a crash.\n- Elegant design: Bitcask achieves a lot just by keeping the architecture simple and relying on filesystem primitives for handling complex scenarios (for eg: backup/recovery, cache etc).\n- Ability to handle datasets larger than RAM.\n\n### Limitations\n\n- The main limitation is that all the keys must fit in RAM since they're held inside as an in-memory hash table. A potential workaround for this could be to shard the keys in multiple buckets. Incoming records can be hashed into different buckets based on the key. A shard based approach allows each bucket to have limited RAM usage.\n\n## Internals\n\nYou can refer to [Writing a disk-based key-value store in Golang](https://mrkaran.dev/posts/barreldb) blog post to read about the internals of Bitcask which also explains how BarrelDB works.\n\n## Usage\n\n### Library\n\n\n```go\nimport (\n\t\"github.com/mr-karan/barreldb\"\n)\n\nbarrel, _ := barrel.Init(barrel.WithDir(\"data/\"))\n\n// Set a key.\nbarrel.Put(\"hello\", []byte(\"world\"))\n\n// Fetch the key.\nv, _ := barrel.Get(\"hello\")\n\n// Delete a key.\nbarrel.Delete(\"hello\")\n\n// Set with expiry.\nbarrel.PutEx(\"hello\", []byte(\"world\"), time.Second * 3)\n```\n\nFor a complete example, visit [examples](./examples/main.go).\n\n### Redis Client\n\n`barreldb` implements the API over a simple Redis-compatible server (`barreldb`):\n\n```\n127.0.0.1:6379\u003e set hello world\nOK\n127.0.0.1:6379\u003e get hello\n\"world\"\n127.0.0.1:6379\u003e set goodbye world 10s\nOK\n127.0.0.1:6379\u003e get goodbye\n\"world\"\n127.0.0.1:6379\u003e get goodbye\nERR: invalid key: key is already expired\n```\n\n## API\n\n| Method                                       | Description                                                                                              |\n| -------------------------------------------- | -------------------------------------------------------------------------------------------------------- |\n| `Init(...cfg) *Barrel`                       | Returns a new instance of `Barrel`. Additional options can be passed like `WithDir`, `WithReadOnly` etc. |\n| `Put(string, []byte) error`                  | Store a key and value in the datastore.                                                                  |\n| `PutEx(string, []byte, time.Duration) error` | Store a key and value with expiry in the datastore.                                                      |\n| `Get(string) []byte,error`                   | Retrieve a value by key from the datastore.                                                              |\n| `Delete(string) error`                       | Delete a key from the datastore.                                                                         |\n| `Keys() []string`                            | List all keys in the datastore.                                                                          |\n| `Len() int`                                  | Return the total count of keys in the datastore.                                                         |\n| `Fold(func(string) error) error`             | Fold over all K/V pairs in a Bitcask datastore. Calls a function on each key inside the datastore.       |\n| `Sync() error`                               | Force any writes to sync to disk.                                                                        |\n| `Shutdown() error`                           | Close a data store and flush all pending writes. Removes any lock on the data directory as well.         |\n\n## Benchmarks\n\nUsing `make bench`:\n\n```\ngo test -bench=. -benchmem ./...\nHELLO\ngoos: linux\ngoarch: amd64\npkg: github.com/mr-karan/barreldb\ncpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz\nBenchmarkPut/DisableSync-8                385432              3712 ns/op        1103.48 MB/s          88 B/op          4 allocs/op\nBenchmarkPut/AlwaysSync-8                    222           5510563 ns/op           0.74 MB/s         115 B/op          4 allocs/op\nBenchmarkGet-8                            840627              1304 ns/op        3142.20 MB/s        4976 B/op          5 allocs/op\nPASS\nok      github.com/mr-karan/barreldb 10.751s\n```\n\nUsing `redis-benchmark`:\n\n```\n$ redis-benchmark -p 6379 -t set -n 10000 -r 100000000\nSummary:\n  throughput summary: 140845.06 requests per second\n  latency summary (msec):\n          avg       min       p50       p95       p99       max\n        0.196     0.016     0.175     0.255     1.031     2.455\n\n$ redis-benchmark -p 6379 -t set -n 200000 -r 100000000\nSummary:\n  throughput summary: 143678.17 requests per second\n  latency summary (msec):\n          avg       min       p50       p95       p99       max\n        0.184     0.016     0.183     0.223     0.455     2.183\n\n$ redis-benchmark -p 6379 -t get -n 100000 -r 100000000\nSummary:\n  throughput summary: 170068.03 requests per second\n  latency summary (msec):\n          avg       min       p50       p95       p99       max\n        0.153     0.040     0.143     0.199     0.367     1.447\n```\n\n## References\n\n- [Bitcask paper](https://riak.com/assets/bitcask-intro.pdf)\n- [Highscalability article on Bitcask](http://highscalability.com/blog/2011/1/10/riaks-bitcask-a-log-structured-hash-table-for-fast-keyvalue.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmr-karan%2Fbarreldb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmr-karan%2Fbarreldb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmr-karan%2Fbarreldb/lists"}