Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/jamesgober/iqdb-persist

Disk persistence, WAL, and crash recovery for iQDB indexes - part of the iQDB family.
https://github.com/jamesgober/iqdb-persist

iqdb persistence reps rust vector-database wal

Last synced: 13 days ago
JSON representation

Disk persistence, WAL, and crash recovery for iQDB indexes - part of the iQDB family.

Awesome Lists containing this project

README 

          


    Rust logo

    


    iqdb-persist

    


    iQDB DISK PERSISTENCE






    Crates.io

    Downloads

    docs.rs

    CI

    MSRV










    


        iqdb-persist is what moves iQDB from demo-only to actually usable: it adds durable snapshot save and load to any index, behind a versioned file header, a CRC32 integrity check, and an atomic write that never corrupts an existing good file.

    


    


        It is the embedded persistence layer, generic over any type that implements iqdb_index::Index, and designed to sit on the storage-io substrate (the renamed fsys-rs) rather than touching files directly.

    


    


    


    


        MSRV is 1.87+ (Rust 2024 edition). Atomic saves. Versioned, portable on-disk format. CRC32 integrity.

    


    


        Status: stable (v1.0.0). The public API and on-disk format are frozen under the SemVer 1.x guarantee — no breaking changes before 2.0. See CHANGELOG.md and the ROADMAP.

    











What it does



- **Atomic snapshot save/load** — write-to-temp + `fsync` + atomic rename + directory `fsync`; an interrupted write never corrupts an existing good file.

- **Versioned header** — magic bytes, format version, index-type tag, dim, metric, vector count. All sizes are fixed-width little-endian `u64`, so a file is portable across 32- and 64-bit hosts.

- **CRC32 integrity** — computed over the payload; a single-bit flip surfaces as `ChecksumMismatch` on load, never a panic or a silently-wrong result.

- **Write-ahead log & crash recovery** — with `wal_enabled`, every `insert` / `delete` is logged and `fsync`ed *before* it touches memory, then replayed onto the snapshot on `load`. A crash mid-append leaves a torn tail that recovery detects and discards.

- **Optional compression** — Zstd or LZ4 on the snapshot payload, behind the `zstd` / `lz4` cargo features. The CRC32 covers the compressed bytes, so corruption is caught before decompression.

- **Generic over the index** — `PersistedIndex` wraps any concrete index; the framing lives here, the payload bytes live in the index's `Persistable` impl.







## Installation



```toml

[dependencies]

iqdb-persist = "1.0"

iqdb-index   = "1.0"

iqdb-types   = "1.0"

```



Snapshot compression is opt-in via cargo features (off by default):



```toml

iqdb-persist = { version = "1.0", features = ["zstd", "lz4"] }

```







## Quick start



```rust

use iqdb_persist::{PersistConfig, PersistedIndex};



// `MyIndex: iqdb_index::Index + iqdb_persist::Persistable`

let cfg = PersistConfig::new("/var/lib/app/index.iqdb");



// Wrap an in-memory index and save it atomically.

let wrapped = PersistedIndex::open_with(my_index, cfg.clone())?;

wrapped.save()?;



// Later — reconstruct it from disk (verifies magic, version, type, CRC32).

let restored: PersistedIndex = PersistedIndex::load(cfg)?;

let index = restored.index();

```



For durable, crash-recoverable mutation, set `cfg.wal_enabled = true` and

mutate through the wrapper — each op is logged before it is applied, and

`checkpoint()` folds the log back into a fresh snapshot:



```rust

let mut db = PersistedIndex::open_with(my_index, cfg.clone())?; // writes base snapshot + opens WAL

db.insert(id, vector, metadata)?;   // logged + fsynced, then applied

db.delete(&other_id)?;

db.checkpoint()?;                    // snapshot the state, truncate the WAL

// after a crash: PersistedIndex::load(cfg) replays the WAL onto the snapshot

```



An index opts in by implementing the two-method `Persistable` trait. A

complete, runnable version lives in

[`examples/save_and_load.rs`](./examples/save_and_load.rs) — run it

with `cargo run --example save_and_load`. Full reference:

[`docs/API.md`](./docs/API.md).







## Status



This is v1.0.0 — stable. Atomic snapshots + CRC32 (v0.2), the write-ahead log with replay and crash recovery (v0.3), and optional Zstd/LZ4 snapshot compression (v0.4) are complete; the API and on-disk format were frozen at v0.5 and the parse/recovery paths adversarially hardened; v0.6 added property tests for the core invariants. The public API and on-disk format are now committed under the SemVer 1.x guarantee — no breaking changes before 2.0. The external `storage-io` substrate is out of scope for 1.0, deferred behind the internal storage seam (an internal swap when it lands, not an API change). See the ROADMAP.









## Where It Fits



`iqdb-persist` is the embedded persistence crate of the iQDB family. It builds on:



- `iqdb-types` — core vocabulary (`DistanceMetric`, `IqdbError`)

- `iqdb-index` — the `Index` / `IndexCore` traits it wraps as persistable



Snapshot file I/O goes through a tiny internal `Storage` seam so the future `storage-io` substrate (the `fsys-rs` rename) can drop in unchanged; today it ships one impl over `std::fs`, and the WAL appends through its own `std::fs` handle. Adopting `storage-io` is deferred until that rename lands — an internal swap behind the seam, not an API change.







## Contributing



See dev/DIRECTIVES.md for engineering standards and the definition of done. Before a PR: `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `cargo test --all-features` must be clean.









    
License


    
Licensed under either of


    

    
at your option.







  


  COPYRIGHT © 2026 JAMES GOBER.