Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zuoyebang/bitalosdb
Bitalosdb is a high-performance KV storage engine.
https://github.com/zuoyebang/bitalosdb
database distributed-storage high-performance io-architecture kvstore nosql storage-engine
Last synced: 4 days ago
JSON representation
Bitalosdb is a high-performance KV storage engine.
- Host: GitHub
- URL: https://github.com/zuoyebang/bitalosdb
- Owner: zuoyebang
- License: apache-2.0
- Created: 2024-02-26T09:34:34.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-07-24T08:27:33.000Z (6 months ago)
- Last Synced: 2025-01-12T00:07:06.153Z (11 days ago)
- Topics: database, distributed-storage, high-performance, io-architecture, kvstore, nosql, storage-engine
- Language: Go
- Homepage: https://bitalos.zuoyebang.com
- Size: 2.39 MB
- Stars: 162
- Watchers: 16
- Forks: 17
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### Bitalosdb is a high-performance KV storage engine. [中文版](./README_CN.md)
## Introduction
- Bitalosdb is a high-performance KV storage engine (self-developed), which is based on the creative IO architecture and storage technology and focuses on solving the problem of LSM-Tree read and write amplification. As a substitute for Rocksdb, Bitalosdb can improve read and write performance greatly.
## Team
- Produced: Zuoyebang Company - Platform technical team
- Author: Xu Ruibo([email protected])
- Contributors: Xing Fu([email protected]), Li Jingchen([email protected]), Lu Wenwei([email protected]), Liu Fang([email protected])
## Key Technology
- Bithash (KV separation technology), significantly reduce write amplification. For bithash, time complexity of retrieval is O(1). GC can be completed independently, and value and index are decoupled.
- Bitalostree (high-performance compression index technology), basically eliminate read amplification. If B+ Tree has a number of huge Pages, write amplification is a severe problem. With a creative index compression technology, Bitalostree eliminates B+ Tree write amplification, and improves the read performance.
- Bitalostable (cold and hot data separation technology), stores cold data, which is calculated according to the data scale and access frequency. Storage engine writes cold data to Bitalostable when QPS becomes low. Improve data compression, reduce index memory consumption, and achieve more rational resource utilization. (open source stable edition has basic features, enterprise edition supports more comprehensive hot and cold separation).
## Performance
- This benchmark is based on Bitloasdb version (v5.0) and rocksdb stable version (v7.6.0).
### Hardware
```
CPU: Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
Memory: 384GB
Disk: 2*3.5TB SSD
```
- Disk(File) IOPS(BW)
|BlockSize| Write | RandWrite | Read | RandRead
|---------|----------|----------|------|-----
| 4KB | 294K(1150MiB/s) | 232K(905MiB/s) | 446K(1742MiB/s) | 446K(1743MiB/s)
| 8KB | 266K(2080MiB/s) | 244K(1902MiB/s) | 309K(2414MiB/s) | 404K(3159MiB/s)
### Program
- Benchmark thread number: 8
- Program cpu cgroup: 8 core
- Comparison standard: QPS on single-core (multi-core QPS / core number), single-core performance reflects cost advantage better.
### Data
- Key-value spec: key-size=32B、value-size=1KB
- Comparison dimensions: Total data size(25/50/100GB) * IO ratio(100% random write, 100% random read, 50% random write + 50% random read, 30% random write + 70% random read)
### Config
- rocksdb
```
Memtable：256MB
WAL：enable
Cache：8GB
TargetFileSize：256M
L0CompactTrigger：8
L0StopWritesTrigger：24
```
- bitalosdb
```
Memtable：256MB
WAL：enable
Cache：disable
```
### Result
- QPS ([Horizontal](./docs/benchmark-qps.png))
## Document
Technical architecture and documentation, refer to the official website: [bitalos.zuoyebang.com](https://bitalos.zuoyebang.com)