Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/al8n/skl
A lock-free thread-safe arena based Skiplist impelementation for building memtable.
https://github.com/al8n/skl
bitcask bitcask-storage-engine embedded inmemory inmemory-db key-value key-value-database key-value-store lock-free memtable mvcc rust skiplist sstable thread-safe write-ahead-log write-ahead-logging
Last synced: 5 days ago
JSON representation
A lock-free thread-safe arena based Skiplist impelementation for building memtable.
- Host: GitHub
- URL: https://github.com/al8n/skl
- Owner: al8n
- License: apache-2.0
- Created: 2022-02-13T06:28:10.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-10-24T15:15:36.000Z (27 days ago)
- Last Synced: 2024-10-25T07:46:00.227Z (26 days ago)
- Topics: bitcask, bitcask-storage-engine, embedded, inmemory, inmemory-db, key-value, key-value-database, key-value-store, lock-free, memtable, mvcc, rust, skiplist, sstable, thread-safe, write-ahead-log, write-ahead-logging
- Language: Rust
- Homepage:
- Size: 627 KB
- Stars: 49
- Watchers: 2
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
- License: LICENSE-APACHE
Awesome Lists containing this project
README
SKL
[
][Github-url]
[
][CI-url]
[
][codecov-url]
[
][doc-url]
[
][crates-url]
[
][crates-url]
1. A lock-free thread-safe concurrent `SkipMap` implementation based on ARENA skiplist which helps develop MVCC memtable for LSM-Tree.
2. A lock-free thread-safe concurrent memory map based on-disk `SkipMap`.
## Installation
- Only use heap backend (suppport `no_std`)
```toml
[dependencies]
skl = "0.20"
```
- Enable memory map backend
```toml
[dependencies]
skl = { version = "0.20", features = ["memmap"] }
```
## Features
- **MVCC and 3D access**: Builtin MVCC (multiple versioning concurrency control) and key-value-version access support.
- **Lock-free and Concurrent-Safe:** `SkipMap` provide lock-free operations, ensuring efficient concurrent access without the need for explicit locking mechanisms.
- **Extensible for Key-Value Database Developers:** Designed as a low-level crate, `SkipMap` offer a flexible foundation for key-value database developers. You can easily build your own memtable or durable storage using these structures.
- **Memory Efficiency:** These data structures are optimized for minimal memory overhead. They operate around references, avoiding unnecessary allocations and deep copies, which can be crucial for efficient memory usage.
- **Segment tracker:** Builtin segment recollection support, a lock-free freelist helps reuse free segments.
- **Prefix compression:**
- Same key will only be stored once.
- Keys with common prefix will be stored once (longest one must be inserted first).
- (experimental) Keys are sub-slice of negeibours will be stored once (require `CompressionPolicy::High`).
- **Discard tracker:** Builtin discard tracker, which will track discarded bytes to help end-users decide when to compact or rewrite.
- **Efficient Iteration:** Enjoy fast forward and backward iteration through the elements in your `SkipMap`. Additionally, bounded iterators are supported, allowing you to traverse only a specified range of elements efficiently.
- **Snapshot Support:** Create snapshots of your `SkipMap`, offering a read-only view of the contents at a specific moment in time. Snapshots provide a consistent view of the data, enabling implementations of transactional semantics and other use cases where data consistency is crucial.
- **Memory Management Options:**
- **Heap Allocation:** Memory allocation is handled by Rust's allocator, ensuring all data resides in RAM.
- **Mmap:** Data can be mapped to a disk file by the operating system, making it suitable for durable storage.
- **Mmap Anonymous:** Mapped to anonymous memory (virtual memory) by the OS, ideal for large in-memory memtables, optimizing memory utilization.
## Examples
Please see [examples](https://github.com/al8n/skl/tree/main/examples) folder for more details.
## Q & A
- Does the on-disk version `SkipMap` ensure crash safety or power failure resilience?
No, If you really need a crash safe, power failure resilience, concurrent-safe and durable ordered write-ahead log implementation,
see [`orderwal`](https://github.com/al8n/orderwal) project.
On-disk version `SkipMap` does not ensure crash safety or power failure resilience. Hence, it is not recommended to directly
use the `SkipMap` as a durable database. It is recommended to use the on-disk version `SkipMap` as a final frozen file for quick lookup.
## Related projects
- [`aol`](https://github.com/al8n/aol): Yet another generic purpose, append-only write-ahead log implementation.
- [`orderwal`](https://github.com/al8n/orderwal): A generic-purpose, atomic, ordered, zero-copy, concurrent-safe, pre-allocate style (memory map) write-ahead-log for developing databases.
## Tests
- `test`:
```sh
cargo test --all-features
```
- `miri` (Stack Borrows)
```sh
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-disable-isolation -Zmiri-symbolic-alignment-check" \
RUSTFLAGS = "--cfg all_skl_tests" \
cargo miri test --all-features
```
- `miri` (Tree Borrows)
```sh
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-disable-isolation -Zmiri-symbolic-alignment-check -Zmiri-tree-borrows" \
RUSTFLAGS = "--cfg all_skl_tests" \
cargo miri test --all-features
```
## Support Platforms
See `cross` section in [GitHub CI file](https://github.com/al8n/skl/blob/main/.github/workflows/ci.yml#L69).
## Pedigree
This code is inspired and modified based on Cockroachdb's pebble arenaskl and Dgraph's badger skl code:
https://github.com/cockroachdb/pebble/tree/master/internal/arenaskl
https://github.com/dgraph-io/badger/tree/master/skl
The pebble's arenaskl code is based on Andy Kimball's arenaskl code:
https://github.com/andy-kimball/arenaskl
The arenaskl code is based on the skiplist found in Badger, a Go-based
KV store:
https://github.com/dgraph-io/badger/tree/master/skl
The skiplist in Badger is itself based on a C++ skiplist built for
Facebook's RocksDB:
https://github.com/facebook/rocksdb/tree/master/memtable
#### License
`skl` is under the terms of both the MIT license and the
Apache License (Version 2.0).
See [LICENSE-APACHE](LICENSE-APACHE), [LICENSE-MIT](LICENSE-MIT) for details.
Copyright (c) 2022 Al Liu.
[Github-url]: https://github.com/al8n/skl/
[CI-url]: https://github.com/al8n/skl/actions/workflows/ci.yml
[doc-url]: https://docs.rs/skl
[crates-url]: https://crates.io/crates/skl
[codecov-url]: https://app.codecov.io/gh/al8n/skl/
[rustc-url]: https://github.com/rust-lang/rust/blob/master/RELEASES.md
[license-apache-url]: https://opensource.org/licenses/Apache-2.0
[license-mit-url]: https://opensource.org/licenses/MIT