https://github.com/ldmtam/go-bitcask
A Log-Structured Hash Table for Fast Key/Value Data implemented in Golang
https://github.com/ldmtam/go-bitcask
bitcask go high-throughput low-latency storage-engine
Last synced: about 1 year ago
JSON representation
A Log-Structured Hash Table for Fast Key/Value Data implemented in Golang
- Host: GitHub
- URL: https://github.com/ldmtam/go-bitcask
- Owner: ldmtam
- Created: 2023-12-31T06:10:37.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-21T04:30:31.000Z (over 2 years ago)
- Last Synced: 2025-03-30T01:51:43.028Z (about 1 year ago)
- Topics: bitcask, go, high-throughput, low-latency, storage-engine
- Language: Go
- Homepage:
- Size: 56.6 KB
- Stars: 14
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# GO-BITCASK
### What is Go-Bitcask
A Log-Structured Hash Table for Fast Key/Value Data implemented in Golang
Go-Bitcask storage engine features:
1. Low latency per item read or written
- Write latency: ~80µs
- Read latency: ~17µs
2. High throughput, especially when writing an incoming stream of random items
- Write throughput (per second): ~330_000
- Read throughput (per second): ~1_000_000
3. Crash friendliness, both in terms of fast recovery and not losing data
4. Ease of backup and restore
5. Relatively simple, understandable code structure and data format with high unit-test coverage (84%)
For more details, please reference to [Bitcask whitepaper](https://riak.com/assets/bitcask-intro.pdf)
### Usage
Create new instance of **go-bitcask**
```
db, err := gobitcask.New(
WithDirName(dirName),
WithSegmentSize(1024 * 1024 * 1024), // 1 GB per segment file
WithMergeOpt(&MergeOption{
Interval: 6 * time.Hour, // run compaction every 6 hours
MinFiles: 5, // at least 5 data files before merging
})
)
if err != nil {
log.Fatalf("create gobitcask instance failed: %v", err)
}
defer db.Close()
```
Store key/value pair to storage
```
err := db.Put([]byte("key1"), []byte("val1"))
if err != nil {
log.Fatalf("store data to gobitcask failed: %v", err)
}
```
Get value from storage by particular key
```
val, err := db.Get([]byte("key1"))
if err != nil && err != gobitcask.ErrKeyNotFound {
log.Fatalf("get data from gobitcask failed: %v", err)
}
fmt.Println(string(val))
```
Delete a key/value pair by key
```
err := db.Delete([]byte("key1"))
if err != nil {
log.Fatalf("delete data from gobitcask failed: %v", err)
}
```
List all keys
```
keys, err := db.ListKeys()
if err != nil {
log.Fatalf("list keys from gobitcask failed: %v", err)
}
```
Fold over all key/value pairs
```
err := db.Fold(func(key, val []byte) error {
fmt.Printf("key: %v, val: %v\n", string(key), string(val))
return nil
})
if err != nil {
log.Fatalf("fold over all key/value pairs failed: %v", err)
}
```
### Benchmark
Machine information: Macbook Pro 2021 (16 inch), M1 Pro, 16 GB RAM, 512 GB SSD
| Action | Num of keys | Key size (byte) | Value size (byte) | Duration (second) | RAM usage (MB)
| ------------- | ----------- |---------------- |------------------ |-------------------|----------------
| Put | 1 000 000 | 8 | 128 | 2.809 | 636
| Put | 1 000 000 | 16 | 512 | 3.331 | 1514
| Put | 10 000 000 | 8 | 128 | 30.047 | 5697
| Random Get | 1 000 000 | 8 | 128 | 0.769 | N/A
| Random Get | 1 000 000 | 16 | 512 | 0.867 | N/A
| Random Get | 10 000 000 | 8 | 128 | 8.81 | N/A