https://github.com/databendlabs/openkv

LSM based key-value store in rust, design for cloud
https://github.com/databendlabs/openkv

Last synced: 4 days ago
JSON representation

LSM based key-value store in rust, design for cloud

• Host: GitHub
• URL: https://github.com/databendlabs/openkv
• Owner: datafuselabs
• License: apache-2.0
• Created: 2022-02-08T12:38:53.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2022-02-27T02:54:58.000Z (over 2 years ago)
• Last Synced: 2024-06-12T08:56:43.201Z (5 months ago)
• Size: 1.15 MB
• Stars: 88
• Watchers: 15
• Forks: 9
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# openkv

LSM based key-value store in rust, design for cloud.

This project is in **Alpha** phase.

API and data layout will change rapidly.

# Goal

- [ ] Transactional key-value store:

    Provide transactional write, and snapshot read.

- [ ] Pluggable WAL: using local fs, raft log, or Kafka as a WAL provider.

- [ ] Minimize ser/de cost: 

    Access to serialized data in SSTable without deserializing.

- [ ] Design for large datasets:

    Reduce unnecessary IO with an efficient data index.

    Reduce in-memory data size by combining sparse-index and bloom filter.

- [ ] Flexible compaction policy:

    Reduces unnecessary compaction by comparing SSTable cardinality signature.

    Reduces IO consumption with partially SSTable merge down.

    Reclaims space quickly with cross-level SSTable merge.

- [ ] Design for the cloud:

    Stores less accessed SSTable on S3.

- [ ] Internal sharding

    A flattened structure reduces write/read amplification.

    A span with a heavy load will be pushed down more frequently than other spans.