Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/al8n/skl

A lock-free thread-safe arena based Skiplist impelementation for building memtable.
https://github.com/al8n/skl

bitcask bitcask-storage-engine embedded inmemory inmemory-db key-value key-value-database key-value-store lock-free memtable mvcc rust skiplist sstable thread-safe write-ahead-log write-ahead-logging

Last synced: 2 days ago
JSON representation

A lock-free thread-safe arena based Skiplist impelementation for building memtable.

Host: GitHub
URL: https://github.com/al8n/skl
Owner: al8n
License: apache-2.0
Created: 2022-02-13T06:28:10.000Z (almost 3 years ago)
Default Branch: main
Last Pushed: 2024-12-25T16:58:57.000Z (6 days ago)
Last Synced: 2024-12-25T19:53:40.314Z (6 days ago)
Topics: bitcask, bitcask-storage-engine, embedded, inmemory, inmemory-db, key-value, key-value-database, key-value-store, lock-free, memtable, mvcc, rust, skiplist, sstable, thread-safe, write-ahead-log, write-ahead-logging
Language: Rust
Homepage:
Size: 765 KB
Stars: 54
Watchers: 2
Forks: 6
Open Issues: 4
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Funding: .github/FUNDING.yml
- License: LICENSE-APACHE

Awesome Lists containing this project

README

SKL

[][Github-url]
LoC
[][CI-url]
[][codecov-url]

[][doc-url]
[][crates-url]
[][crates-url]

1. A lock-free thread-safe concurrent `SkipMap` implementation based on ARENA skiplist which helps develop MVCC memtable for LSM-Tree.
2. A lock-free thread-safe concurrent memory map based on-disk `SkipMap`.

## Installation

- Only use heap backend (suppport `no_std`)

```toml
[dependencies]
skl = "0.22"
```

- Enable memory map backend

```toml
[dependencies]
skl = { version = "0.22", features = ["memmap"] }
```

## Features

- **MVCC and 3D access**: Builtin MVCC (multiple versioning concurrency control) and key-value-version access support.
- **Lock-free and Concurrent-Safe:** `SkipMap` provide lock-free operations, ensuring efficient concurrent access without the need for explicit locking mechanisms.
- **Extensible for Key-Value Database Developers:** Designed as a low-level crate, `SkipMap` offer a flexible foundation for key-value database developers. You can easily build your own memtable or durable storage using these structures.
- **Memory Efficiency:** These data structures are optimized for minimal memory overhead. They operate around references, avoiding unnecessary allocations and deep copies, which can be crucial for efficient memory usage.
- **Segment tracker:** Builtin segment recollection support, a lock-free freelist helps reuse free segments.
- **Prefix compression:**
- Same key will only be stored once.
- Keys with common prefix will be stored once (longest one must be inserted first).
- (experimental) Keys are sub-slice of negeibours will be stored once (require `CompressionPolicy::High`).
- **Discard tracker:** Builtin discard tracker, which will track discarded bytes to help end-users decide when to compact or rewrite.
- **Efficient Iteration:** Enjoy fast forward and backward iteration through the elements in your `SkipMap`. Additionally, bounded iterators are supported, allowing you to traverse only a specified range of elements efficiently.
- **Snapshot Support:** Create snapshots of your `SkipMap`, offering a read-only view of the contents at a specific moment in time. Snapshots provide a consistent view of the data, enabling implementations of transactional semantics and other use cases where data consistency is crucial.
- **Memory Management Options:**
- **Heap Allocation:** Memory allocation is handled by Rust's allocator, ensuring all data resides in RAM.
- **Mmap:** Data can be mapped to a disk file by the operating system, making it suitable for durable storage.
- **Mmap Anonymous:** Mapped to anonymous memory (virtual memory) by the OS, ideal for large in-memory memtables, optimizing memory utilization.

## Examples

Please see [examples](https://github.com/al8n/skl/tree/main/examples) folder for more details.

## Q & A

- Does the on-disk version `SkipMap` ensure crash safety or power failure resilience?

No, If you really need a crash safe, power failure resilience, concurrent-safe and durable ordered write-ahead log implementation,
see [`orderwal`](https://github.com/al8n/orderwal) project.

On-disk version `SkipMap` does not ensure crash safety or power failure resilience. Hence, it is not recommended to directly
use the `SkipMap` as a durable database. It is recommended to use the on-disk version `SkipMap` as a final frozen file for quick lookup.

## Related projects

- [`aol`](https://github.com/al8n/aol): Yet another generic purpose, append-only write-ahead log implementation.
- [`orderwal`](https://github.com/al8n/orderwal): A generic-purpose, atomic, ordered, zero-copy, concurrent-safe, pre-allocate style (memory map) write-ahead-log for developing databases.

## Tests

- `test`:

```sh
cargo test --all-features
```

- `miri` (Stack Borrows)

```sh
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-disable-isolation -Zmiri-symbolic-alignment-check" \
RUSTFLAGS = "--cfg all_skl_tests" \
cargo miri test --all-features
```

- `miri` (Tree Borrows)

```sh
MIRIFLAGS="-Zmiri-strict-provenance -Zmiri-disable-isolation -Zmiri-symbolic-alignment-check -Zmiri-tree-borrows" \
RUSTFLAGS = "--cfg all_skl_tests" \
cargo miri test --all-features
```

## Support Platforms

See `cross` section in [GitHub CI file](https://github.com/al8n/skl/blob/main/.github/workflows/ci.yml#L69).

## Pedigree

This code is inspired and modified based on Cockroachdb's pebble arenaskl and Dgraph's badger skl code:

https://github.com/cockroachdb/pebble/tree/master/internal/arenaskl

https://github.com/dgraph-io/badger/tree/master/skl

The pebble's arenaskl code is based on Andy Kimball's arenaskl code:

https://github.com/andy-kimball/arenaskl

The arenaskl code is based on the skiplist found in Badger, a Go-based
KV store:

https://github.com/dgraph-io/badger/tree/master/skl

The skiplist in Badger is itself based on a C++ skiplist built for
Facebook's RocksDB:

https://github.com/facebook/rocksdb/tree/master/memtable

#### License

`skl` is under the terms of both the MIT license and the
Apache License (Version 2.0).

See [LICENSE-APACHE](LICENSE-APACHE), [LICENSE-MIT](LICENSE-MIT) for details.

[Github-url]: https://github.com/al8n/skl/
[CI-url]: https://github.com/al8n/skl/actions/workflows/ci.yml
[doc-url]: https://docs.rs/skl
[crates-url]: https://crates.io/crates/skl
[codecov-url]: https://app.codecov.io/gh/al8n/skl/
[rustc-url]: https://github.com/rust-lang/rust/blob/master/RELEASES.md
[license-apache-url]: https://opensource.org/licenses/Apache-2.0
[license-mit-url]: https://opensource.org/licenses/MIT