https://github.com/jamesgober/lsm-db

LSM-tree storage engine library for Rust - memtable, WAL, SSTables, compaction, bloom filters.
https://github.com/jamesgober/lsm-db

compaction database lsm-tree reps rust storage-engine

Last synced: 3 days ago
JSON representation

LSM-tree storage engine library for Rust - memtable, WAL, SSTables, compaction, bloom filters.

• Host: GitHub
• URL: https://github.com/jamesgober/lsm-db
• Owner: jamesgober
• License: apache-2.0
• Created: 2026-05-29T23:47:54.000Z (about 1 month ago)
• Default Branch: main
• Last Pushed: 2026-06-06T19:01:32.000Z (about 1 month ago)
• Last Synced: 2026-06-06T21:04:27.707Z (about 1 month ago)
• Topics: compaction, database, lsm-tree, reps, rust, storage-engine
• Language: Rust
• Size: 63.5 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE-APACHE

Awesome Lists containing this project

README

          


    

    


    lsm-db

    


    LSM-TREE STORAGE ENGINE





    

    

    

    

    








    


        lsm-db is a log-structured merge-tree storage engine: the write path that powers RocksDB, LevelDB, Cassandra, and ScyllaDB, packaged as a clean Rust library. Writes go to an in-memory memtable backed by a durable log; when the memtable fills it is flushed to an immutable sorted run on disk; background compaction merges those runs to keep reads fast and space bounded.

    

    


        It is built from the portfolio's own primitives rather than re-deriving them: durability comes from wal-db and point-read filtering from bloom-lib. That keeps the engine small and lets each primitive be audited and benchmarked once.

    

    


        The common case is open / put / get / scan. Compaction strategy, level sizing, and write-buffer tuning live behind a builder.

    

    


    


    


        MSRV is 1.85+ (Rust 2024 edition). Durable writes via wal-db. Background compaction. Bloom-filtered reads.

    

    


        Status: 1.0 — stable. The public API is frozen until 2.0, and the on-disk run format is frozen for the 1.x series (since 0.3.0; see docs/SSTABLE_FORMAT.md). Crash-safe writes (durability) and bloom-filtered reads (bloom) ship as opt-in features. See CHANGELOG.md for detail.

    








What it does


- **Memtable** — in-memory sorted write buffer; flushes to an immutable sorted run when full

- **Multiple sorted runs** — each flush appends a run; reads merge across all of them, newest first

- **Background compaction** — a dedicated thread merges runs to bound read amplification, concurrent with reads and writes

- **Frozen on-disk format** — block-structured runs with per-block CRC32C integrity; specified in [`docs/SSTABLE_FORMAT.md`](./docs/SSTABLE_FORMAT.md)

- **Crash recovery** — a manifest records the live runs; a crash mid-flush or mid-compaction recovers to a consistent state

- **Tombstone deletes** — deletes mask older values and resolve away during compaction

- **Range scans** — merge the buffer and every run into one sorted stream

- **Grouped writes** — apply a batch atomically with respect to concurrent readers

- **Crash-safe writes** — under the `durability` feature, every write hits a `wal-db` log before acknowledgment and is replayed on open (no acknowledged write lost across a crash)

- **Bloom-filtered reads** — under the `bloom` feature, a per-run filter lets a point read skip any run that can't contain the key (negative lookups read no data blocks)

- **Block cache** — a shared cache of decoded run blocks; a repeat point read over a hot working set does no I/O, checksum, or parse

- **Shared, thread-safe handle** — one engine, many threads, behind an `Arc`




## Installation

```toml

[dependencies]

lsm-db = "1.0"

# Crash-safe writes (write-ahead log) and/or bloom-filtered point reads:

lsm-db = { version = "1.0", features = ["durability", "bloom"] }

```




## Quick Start

```rust

use lsm_db::Lsm;

fn main() -> Result<(), Box> {

    // Open (or create) a database backed by a directory.

    let db = Lsm::open("my-db")?;

    // Keys and values are arbitrary bytes.

    db.put(b"user:1", b"alice")?;

    db.put(b"user:2", b"bob")?;

    // Point reads return owned values.

    assert_eq!(db.get(b"user:1")?, Some(b"alice".to_vec()));

    // Deletes mask the key.

    db.delete(b"user:1")?;

    assert_eq!(db.get(b"user:1")?, None);

    // Range scans walk keys in sorted order.

    db.put(b"user:1", b"alice")?;

    for (key, value) in db.scan(b"user:".to_vec()..b"user;".to_vec())? {

        println!("{} = {}", String::from_utf8_lossy(&key), String::from_utf8_lossy(&value));

    }

    // Force the buffer to disk; it will be there on the next open.

    db.flush()?;

    Ok(())

}

```

Tuning lives behind [`LsmConfig`](./docs/API.md#lsmconfig); grouped writes behind [`Batch`](./docs/API.md#batch). See [`docs/API.md`](./docs/API.md) for the full reference and the [`examples/`](./examples) directory for runnable programs.




## Status

This is v1.0.0 — the first **stable** release. The **public API is frozen until 2.0** and the on-disk format is **frozen for the 1.x series**. The engine is feature-complete, hardened against hostile input, and soak-tested single- and multi-threaded: multiple on-disk runs, background compaction, crash recovery, crash-safe writes (durability), bloom-filtered point reads (bloom), and a block cache, behind the Tier-1 API (open/put/get/delete/scan). See docs/API.md and docs/PERFORMANCE.md.






## Where It Fits

`lsm-db` is a storage engine. It builds on:

- [`wal-db`](https://github.com/jamesgober/wal-db) — memtable durability and crash recovery

- [`bloom-lib`](https://github.com/jamesgober/bloom-lib) — SSTable point-read filtering

- Hive DB — a candidate storage engine behind the `StorageEngine` trait

It stays foreign-compatible: usable standalone as an embedded key-value store.




## Cross-Platform Support

**Tier 1 Support:**

- Linux (x86_64, aarch64)

- macOS (x86_64, Apple Silicon)

- Windows (x86_64)

Behavior is verified on each target by the CI matrix.




## Contributing

Before opening a PR, `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `cargo test --all-features` must be clean. Hot-path changes require a `criterion` benchmark; correctness-critical paths require property and/or `loom` tests.






    
License

    
Licensed under either of

    


        


Apache License, Version 2.0 — see LICENSE-APACHE



        


MIT License — see LICENSE-MIT



    

    
at your option.





  


  COPYRIGHT © 2026 JAMES GOBER.