{"id":22152295,"url":"https://github.com/auula/prologic-bitcask","last_synced_at":"2025-03-24T13:26:02.176Z","repository":{"id":103285032,"uuid":"461458264","full_name":"auula/prologic-bitcask","owner":"auula","description":null,"archived":false,"fork":false,"pushed_at":"2022-02-20T10:53:42.000Z","size":2000,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T18:32:49.528Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/auula.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-20T10:53:14.000Z","updated_at":"2022-02-20T10:53:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"874dd667-5576-43f2-b606-bf7704dee24b","html_url":"https://github.com/auula/prologic-bitcask","commit_stats":null,"previous_names":[],"tags_count":56,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/auula%2Fprologic-bitcask","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/auula%2Fprologic-bitcask/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/auula%2Fprologic-bitcask/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/auula%2Fprologic-bitcask/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/auula","download_url":"https://codeload.github.com/auula/prologic-bitcask/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245277751,"owners_count":20589195,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T00:49:42.092Z","updated_at":"2025-03-24T13:26:02.170Z","avatar_url":"https://github.com/auula.png","language":"Go","funding_links":["https://www.patreon.com/prologic"],"categories":[],"sub_categories":[],"readme":"# bitcask\n\n[![Build Status](https://ci.mills.io/api/badges/prologic/bitcask/status.svg)](https://ci.mills.io/prologic/bitcask)\n[![Go Report Card](https://goreportcard.com/badge/git.mills.io/prologic/bitcask)](https://goreportcard.com/report/git.mills.io/prologic/bitcask)\n[![Go Reference](https://pkg.go.dev/badge/git.mills.io/prologic/bitcask.svg)](https://pkg.go.dev/git.mills.io/prologic/bitcask)\n\nA high performance Key/Value store written in [Go](https://golang.org) with a predictable read/write performance and high throughput. Uses a [Bitcask](https://en.wikipedia.org/wiki/Bitcask) on-disk layout (LSM+WAL) similar to [Riak](https://riak.com/)\n\nFor a more feature-complete Redis-compatible server, distributed key/value store have a look at [Bitraft](https://git.mills.io/prologic/bitraft) which uses this library as its backend. Use [Bitcask](https://git.mills.io/prologic/bitcask) as a starting point or if you want to embed in your application, use [Bitraft](https://git.mills.io/prologic/bitraft) if you need a complete server/client solution with high availability with a Redis-compatible API.\n\n## Features\n\n* Embedded (`import \"git.mills.io/prologic/bitcask\"`)\n* Builtin CLI (`bitcask`)\n* Builtin Redis-compatible server (`bitcaskd`)\n* Predictable read/write performance\n* Low latency\n* High throughput (See: [Performance](README.md#Performance) )\n\n## Is Bitcask right for my project?\n\n__NOTE__: Please read this carefully to identify whether using Bitcask is\n          suitable for your needs.\n\n`bitcask` is a **great fit** for:\n\n- Storing hundreds of thousands to millions of key/value pairs based on\n  default configuration. With the default configuration (_configurable_)\n  of 64 bytes per key and 64kB values, 1M keys would consume roughly ~600-700MB\n  of memory ~65-70GB of disk storage. These are all configurable when you\n  create a new database with `bitcask.Open(...)` with functional-style options\n  you can pass with `WithXXX()`.\n\n- As the backing store to a distributed key/value store. See for example the\n  [bitraft](https://git.mills.io/prologic/bitraft) as an example of this.\n\n- For high performance, low latency read/write workloads where you cannot fit\n  a typical hash-map into memory, but require the highest level of performance\n  and predicate read latency. Bitcask ensures only 1 read/write IOPS are ever\n  required for reading and writing key/value pairs.\n\n- As a general purpose embedded key/value store where you would have used\n  [BoltDB](https://github.com/boltdb/bolt),\n  [LevelDB](https://github.com/syndtr/goleveldb),\n  [BuntDB](https://github.com/tidwall/buntdb)\n  or similar...\n\n`bitcask` is not suited for:\n\n- Storing billions of records\n  The reason for this is the key-space is held in memory using a highly\n  performant and memory optimized adaptive radix tree thanks to\n  [go-adaptive-radix-tree](github.com/plar/go-adaptive-radix-tree) _however_\n  this means the more keys you have in your key space, the more memory is\n  consumed. Consider using a disk-backed B-Tree like [BoltDB](https://github.com/boltdb/bolt)\n  or [LevelDB](https://github.com/syndtr/goleveldb) if you intend to store a\n  large quantity of key/value pairs.\n\n\u003e Note however that storing large amounts of data in terms of value(s) is\n\u003e totally fine. In other wise thousands to millions of keys with large values\n\u003e will work just fine.\n\n- Write intensive workloads. Due to the [Bitcask design](https://riak.com/assets/bitcask-intro.pdf?source=post_page---------------------------)\n  heavy write workloads that lots of key/value pairs will over time cause\n  problems like \"Too many open files\" (#193) errors to occur. This can be mitigated by\n  periodically compacting the data files by issuing a `.Merge()` operation however\n  if key/value pairs do not change or are never deleted, as-in only new key/value\n  pairs are ever written this will have no effect. Eventually you will run out\n  of file descriptors!\n\n\u003e You should consider your read/write workloads carefully and ensure you set\n\u003e appropriate file descriptor limits with `ulimit -n` that suit your needs.\n\n## Development\n\n```sh\n$ git clone https://git.mills.io/prologic/bitcask.git\n$ make\n```\n\n## Install\n\n```sh\n$ go get git.mills.io/prologic/bitcask\n```\n\n## Usage (library)\n\nInstall the package into your project:\n\n```sh\n$ go get git.mills.io/prologic/bitcask\n```\n\n```go\npackage main\n\nimport (\n\t\"log\"\n\t\"git.mills.io/prologic/bitcask\"\n)\n\nfunc main() {\n    db, _ := bitcask.Open(\"/tmp/db\")\n    defer db.Close()\n    db.Put([]byte(\"Hello\"), []byte(\"World\"))\n    val, _ := db.Get([]byte(\"Hello\"))\n    log.Printf(string(val))\n}\n```\n\nSee the [GoDoc](https://godoc.org/git.mills.io/prologic/bitcask) for further\ndocumentation and other examples.\n\n## Usage (tool)\n\n```sh\n$ bitcask -p /tmp/db set Hello World\n$ bitcask -p /tmp/db get Hello\nWorld\n```\n\n## Usage (server)\n\nThere is also a builtin very  simple Redis-compatible server called `bitcaskd`:\n\n```sh\n$ ./bitcaskd ./tmp\nINFO[0000] starting bitcaskd v0.0.7@146f777              bind=\":6379\" path=./tmp\n```\n\nExample session:\n\n```sh\n$ telnet localhost 6379\nTrying ::1...\nConnected to localhost.\nEscape character is '^]'.\nSET foo bar\n+OK\nGET foo\n$3\nbar\nDEL foo\n:1\nGET foo\n$-1\nPING\n+PONG\nQUIT\n+OK\nConnection closed by foreign host.\n```\n\n## Docker\n\nYou can also use the [Bitcask Docker Image](https://cloud.docker.com/u/prologic/repository/docker/prologic/bitcask):\n\n```sh\n$ docker pull prologic/bitcask\n$ docker run -d -p 6379:6379 prologic/bitcask\n```\n\n## Performance\n\nBenchmarks run on a 11\" MacBook with a 1.4Ghz Intel Core i7:\n\n```sh\n$ make bench\n...\ngoos: darwin\ngoarch: amd64\npkg: git.mills.io/prologic/bitcask\n\nBenchmarkGet/128B-4         \t  316515\t      3263 ns/op\t  39.22 MB/s\t     160 B/op\t       1 allocs/op\nBenchmarkGet/256B-4         \t  382551\t      3204 ns/op\t  79.90 MB/s\t     288 B/op\t       1 allocs/op\nBenchmarkGet/512B-4         \t  357216\t      3835 ns/op\t 133.51 MB/s\t     576 B/op\t       1 allocs/op\nBenchmarkGet/1K-4           \t  274958\t      4429 ns/op\t 231.20 MB/s\t    1152 B/op\t       1 allocs/op\nBenchmarkGet/2K-4           \t  227764\t      5013 ns/op\t 408.55 MB/s\t    2304 B/op\t       1 allocs/op\nBenchmarkGet/4K-4           \t  187557\t      5534 ns/op\t 740.15 MB/s\t    4864 B/op\t       1 allocs/op\nBenchmarkGet/8K-4           \t  153546\t      7652 ns/op\t1070.56 MB/s\t    9472 B/op\t       1 allocs/op\nBenchmarkGet/16K-4          \t  115549\t     10272 ns/op\t1594.95 MB/s\t   18432 B/op\t       1 allocs/op\nBenchmarkGet/32K-4          \t   69592\t     16405 ns/op\t1997.39 MB/s\t   40960 B/op\t       1 allocs/op\n\nBenchmarkPut/128BNoSync-4   \t  123519\t     11094 ns/op\t  11.54 MB/s\t      49 B/op\t       2 allocs/op\nBenchmarkPut/256BNoSync-4   \t   84662\t     13398 ns/op\t  19.11 MB/s\t      50 B/op\t       2 allocs/op\nBenchmarkPut/1KNoSync-4     \t   46345\t     24855 ns/op\t  41.20 MB/s\t      58 B/op\t       2 allocs/op\nBenchmarkPut/2KNoSync-4     \t   28820\t     43817 ns/op\t  46.74 MB/s\t      68 B/op\t       2 allocs/op\nBenchmarkPut/4KNoSync-4     \t   13976\t     90059 ns/op\t  45.48 MB/s\t      89 B/op\t       2 allocs/op\nBenchmarkPut/8KNoSync-4     \t    7852\t    155101 ns/op\t  52.82 MB/s\t     130 B/op\t       2 allocs/op\nBenchmarkPut/16KNoSync-4    \t    4848\t    238113 ns/op\t  68.81 MB/s\t     226 B/op\t       2 allocs/op\nBenchmarkPut/32KNoSync-4    \t    2564\t    391483 ns/op\t  83.70 MB/s\t     377 B/op\t       3 allocs/op\n\nBenchmarkPut/128BSync-4     \t     260\t   4611273 ns/op\t   0.03 MB/s\t      48 B/op\t       2 allocs/op\nBenchmarkPut/256BSync-4     \t     265\t   4665506 ns/op\t   0.05 MB/s\t      48 B/op\t       2 allocs/op\nBenchmarkPut/1KSync-4       \t     256\t   4757334 ns/op\t   0.22 MB/s\t      48 B/op\t       2 allocs/op\nBenchmarkPut/2KSync-4       \t     255\t   4996788 ns/op\t   0.41 MB/s\t      92 B/op\t       2 allocs/op\nBenchmarkPut/4KSync-4       \t     222\t   5136481 ns/op\t   0.80 MB/s\t      98 B/op\t       2 allocs/op\nBenchmarkPut/8KSync-4       \t     223\t   5530824 ns/op\t   1.48 MB/s\t      99 B/op\t       2 allocs/op\nBenchmarkPut/16KSync-4      \t     213\t   5717880 ns/op\t   2.87 MB/s\t     202 B/op\t       2 allocs/op\nBenchmarkPut/32KSync-4      \t     211\t   5835948 ns/op\t   5.61 MB/s\t     355 B/op\t       3 allocs/op\n\nBenchmarkScan-4             \t  568696\t      2036 ns/op\t     392 B/op\t      33 allocs/op\nPASS\n```\n\nFor 128B values:\n\n* ~300,000 reads/sec\n* ~90,000 writes/sec\n* ~490,000 scans/sec\n\nThe full benchmark above shows linear performance as you increase key/value sizes.\n\n## Support\n\nSupport the ongoing development of Bitcask!\n\n**Sponsor**\n\n- Become a [Sponsor](https://www.patreon.com/prologic)\n\n## Contributors\n\nThank you to all those that have contributed to this project, battle-tested it,\nused it in their own projects or products, fixed bugs, improved performance\nand even fix tiny typos in documentation! Thank you and keep contributing!\n\nYou can find an [AUTHORS](/AUTHORS) file where we keep a list of contributors\nto the project. If you contribute a PR please consider adding your name there.\n\n## Related Projects\n\n- [bitraft](https://git.mills.io/prologic/bitraft) -- A Distributed Key/Value store (_using Raft_) with a Redis compatible protocol.\n- [bitcaskfs](https://git.mills.io/prologic/bitcaskfs) -- A FUSE file system for mounting a Bitcask database.\n- [bitcask-bench](https://git.mills.io/prologic/bitcask-bench) -- A benchmarking tool comparing Bitcask and several other Go key/value libraries.\n\n## License\n\nbitcask is licensed under the term of the [MIT License](https://git.mills.io/prologic/bitcask/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fauula%2Fprologic-bitcask","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fauula%2Fprologic-bitcask","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fauula%2Fprologic-bitcask/lists"}