{"id":51514697,"url":"https://github.com/jamesgober/iqdb-persist","last_synced_at":"2026-07-08T10:01:43.843Z","repository":{"id":362801374,"uuid":"1254209784","full_name":"jamesgober/iqdb-persist","owner":"jamesgober","description":"Disk persistence, WAL, and crash recovery for iQDB indexes - part of the iQDB family.","archived":false,"fork":false,"pushed_at":"2026-06-06T00:08:21.000Z","size":32,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-06T02:12:40.443Z","etag":null,"topics":["iqdb","persistence","reps","rust","vector-database","wal"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jamesgober.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-30T09:23:32.000Z","updated_at":"2026-06-06T00:08:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jamesgober/iqdb-persist","commit_stats":null,"previous_names":["jamesgober/iqdb-persist"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/jamesgober/iqdb-persist","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fiqdb-persist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fiqdb-persist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fiqdb-persist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fiqdb-persist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jamesgober","download_url":"https://codeload.github.com/jamesgober/iqdb-persist/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesgober%2Fiqdb-persist/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35260671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iqdb","persistence","reps","rust","vector-database","wal"],"created_at":"2026-07-08T10:01:40.139Z","updated_at":"2026-07-08T10:01:43.827Z","avatar_url":"https://github.com/jamesgober.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n    \u003cimg width=\"99\" alt=\"Rust logo\" src=\"https://raw.githubusercontent.com/jamesgober/rust-collection/72baabd71f00e14aa9184efcb16fa3deddda3a0a/assets/rust-logo.svg\"\u003e\n    \u003cbr\u003e\n    \u003cb\u003eiqdb-persist\u003c/b\u003e\n    \u003cbr\u003e\n    \u003csub\u003e\u003csup\u003eiQDB DISK PERSISTENCE\u003c/sup\u003e\u003c/sub\u003e\n\u003c/h1\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://crates.io/crates/iqdb-persist\"\u003e\u003cimg alt=\"Crates.io\" src=\"https://img.shields.io/crates/v/iqdb-persist\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://crates.io/crates/iqdb-persist\"\u003e\u003cimg alt=\"Downloads\" src=\"https://img.shields.io/crates/d/iqdb-persist?color=%230099ff\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://docs.rs/iqdb-persist\"\u003e\u003cimg alt=\"docs.rs\" src=\"https://img.shields.io/docsrs/iqdb-persist\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/jamesgober/iqdb-persist/actions\"\u003e\u003cimg alt=\"CI\" src=\"https://github.com/jamesgober/iqdb-persist/actions/workflows/ci.yml/badge.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/rust-lang/rfcs/blob/master/text/2495-min-rust-version.md\"\u003e\u003cimg alt=\"MSRV\" src=\"https://img.shields.io/badge/MSRV-1.87%2B-blue\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\u003cdiv align=\"left\"\u003e\n    \u003cp\u003e\n        \u003cstrong\u003eiqdb-persist\u003c/strong\u003e is what moves iQDB from demo-only to actually usable: it adds durable snapshot \u003cstrong\u003esave\u003c/strong\u003e and \u003cstrong\u003eload\u003c/strong\u003e to any index, behind a versioned file header, a CRC32 integrity check, and an atomic write that never corrupts an existing good file.\n    \u003c/p\u003e\n    \u003cp\u003e\n        It is the embedded persistence layer, generic over any type that implements \u003ccode\u003eiqdb_index::Index\u003c/code\u003e, and designed to sit on the \u003ccode\u003estorage-io\u003c/code\u003e substrate (the renamed \u003ccode\u003efsys-rs\u003c/code\u003e) rather than touching files directly.\n    \u003c/p\u003e\n    \u003cbr\u003e\n    \u003chr\u003e\n    \u003cp\u003e\n        \u003cstrong\u003eMSRV is 1.87+\u003c/strong\u003e (Rust 2024 edition). Atomic saves. Versioned, portable on-disk format. CRC32 integrity.\n    \u003c/p\u003e\n    \u003cblockquote\u003e\n        \u003cstrong\u003eStatus: stable (v1.0.0).\u003c/strong\u003e The public API and on-disk format are frozen under the SemVer 1.x guarantee — no breaking changes before 2.0. See \u003ca href=\"./CHANGELOG.md\"\u003e\u003ccode\u003eCHANGELOG.md\u003c/code\u003e\u003c/a\u003e and the \u003ca href=\"./dev/ROADMAP.md\"\u003e\u003ccode\u003eROADMAP\u003c/code\u003e\u003c/a\u003e.\n    \u003c/blockquote\u003e\n\u003c/div\u003e\n\n\u003chr\u003e\n\u003cbr\u003e\n\n\u003ch2\u003eWhat it does\u003c/h2\u003e\n\n- **Atomic snapshot save/load** \u0026mdash; write-to-temp + `fsync` + atomic rename + directory `fsync`; an interrupted write never corrupts an existing good file.\n- **Versioned header** \u0026mdash; magic bytes, format version, index-type tag, dim, metric, vector count. All sizes are fixed-width little-endian `u64`, so a file is portable across 32- and 64-bit hosts.\n- **CRC32 integrity** \u0026mdash; computed over the payload; a single-bit flip surfaces as `ChecksumMismatch` on load, never a panic or a silently-wrong result.\n- **Write-ahead log \u0026 crash recovery** \u0026mdash; with `wal_enabled`, every `insert` / `delete` is logged and `fsync`ed *before* it touches memory, then replayed onto the snapshot on `load`. A crash mid-append leaves a torn tail that recovery detects and discards.\n- **Optional compression** \u0026mdash; Zstd or LZ4 on the snapshot payload, behind the `zstd` / `lz4` cargo features. The CRC32 covers the compressed bytes, so corruption is caught before decompression.\n- **Generic over the index** \u0026mdash; `PersistedIndex\u003cI: Index + Persistable\u003e` wraps any concrete index; the framing lives here, the payload bytes live in the index's `Persistable` impl.\n\n\u003cbr\u003e\n\n## Installation\n\n```toml\n[dependencies]\niqdb-persist = \"1.0\"\niqdb-index   = \"1.0\"\niqdb-types   = \"1.0\"\n```\n\nSnapshot compression is opt-in via cargo features (off by default):\n\n```toml\niqdb-persist = { version = \"1.0\", features = [\"zstd\", \"lz4\"] }\n```\n\n\u003cbr\u003e\n\n## Quick start\n\n```rust\nuse iqdb_persist::{PersistConfig, PersistedIndex};\n\n// `MyIndex: iqdb_index::Index + iqdb_persist::Persistable`\nlet cfg = PersistConfig::new(\"/var/lib/app/index.iqdb\");\n\n// Wrap an in-memory index and save it atomically.\nlet wrapped = PersistedIndex::open_with(my_index, cfg.clone())?;\nwrapped.save()?;\n\n// Later — reconstruct it from disk (verifies magic, version, type, CRC32).\nlet restored: PersistedIndex\u003cMyIndex\u003e = PersistedIndex::load(cfg)?;\nlet index = restored.index();\n```\n\nFor durable, crash-recoverable mutation, set `cfg.wal_enabled = true` and\nmutate through the wrapper — each op is logged before it is applied, and\n`checkpoint()` folds the log back into a fresh snapshot:\n\n```rust\nlet mut db = PersistedIndex::open_with(my_index, cfg.clone())?; // writes base snapshot + opens WAL\ndb.insert(id, vector, metadata)?;   // logged + fsynced, then applied\ndb.delete(\u0026other_id)?;\ndb.checkpoint()?;                    // snapshot the state, truncate the WAL\n// after a crash: PersistedIndex::load(cfg) replays the WAL onto the snapshot\n```\n\nAn index opts in by implementing the two-method `Persistable` trait. A\ncomplete, runnable version lives in\n[`examples/save_and_load.rs`](./examples/save_and_load.rs) \u0026mdash; run it\nwith `cargo run --example save_and_load`. Full reference:\n[`docs/API.md`](./docs/API.md).\n\n\u003cbr\u003e\n\n## Status\n\nThis is \u003ccode\u003ev1.0.0\u003c/code\u003e — \u003cstrong\u003estable\u003c/strong\u003e. Atomic snapshots + CRC32 (v0.2), the write-ahead log with replay and crash recovery (v0.3), and optional Zstd/LZ4 snapshot compression (v0.4) are complete; the API and on-disk format were frozen at v0.5 and the parse/recovery paths adversarially hardened; v0.6 added property tests for the core invariants. The public API and on-disk format are now committed under the SemVer 1.x guarantee — no breaking changes before 2.0. The external `storage-io` substrate is out of scope for 1.0, deferred behind the internal storage seam (an internal swap when it lands, not an API change). See the \u003ca href=\"./dev/ROADMAP.md\"\u003e\u003ccode\u003eROADMAP\u003c/code\u003e\u003c/a\u003e.\n\n\u003chr\u003e\n\u003cbr\u003e\n\n## Where It Fits\n\n`iqdb-persist` is the embedded persistence crate of the iQDB family. It builds on:\n\n- `iqdb-types` \u0026mdash; core vocabulary (`DistanceMetric`, `IqdbError`)\n- `iqdb-index` \u0026mdash; the `Index` / `IndexCore` traits it wraps as persistable\n\nSnapshot file I/O goes through a tiny internal `Storage` seam so the future `storage-io` substrate (the `fsys-rs` rename) can drop in unchanged; today it ships one impl over `std::fs`, and the WAL appends through its own `std::fs` handle. Adopting `storage-io` is deferred until that rename lands — an internal swap behind the seam, not an API change.\n\n\u003cbr\u003e\n\n## Contributing\n\nSee \u003ca href=\"./dev/DIRECTIVES.md\"\u003e\u003ccode\u003edev/DIRECTIVES.md\u003c/code\u003e\u003c/a\u003e for engineering standards and the definition of done. Before a PR: `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `cargo test --all-features` must be clean.\n\n\u003cbr\u003e\n\n\u003cdiv id=\"license\"\u003e\n    \u003ch2\u003eLicense\u003c/h2\u003e\n    \u003cp\u003eLicensed under either of\u003c/p\u003e\n    \u003cul\u003e\n        \u003cli\u003e\u003cb\u003eApache License, Version 2.0\u003c/b\u003e \u0026mdash; \u003ca href=\"./LICENSE-APACHE\"\u003eLICENSE-APACHE\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003cb\u003eMIT License\u003c/b\u003e \u0026mdash; \u003ca href=\"./LICENSE-MIT\"\u003eLICENSE-MIT\u003c/a\u003e\u003c/li\u003e\n    \u003c/ul\u003e\n    \u003cp\u003eat your option.\u003c/p\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ch2\u003e\u003c/h2\u003e\n  \u003csup\u003eCOPYRIGHT \u003csmall\u003e\u0026copy;\u003c/small\u003e 2026 \u003cstrong\u003eJAMES GOBER.\u003c/strong\u003e\u003c/sup\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesgober%2Fiqdb-persist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesgober%2Fiqdb-persist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesgober%2Fiqdb-persist/lists"}