{"id":13632250,"url":"https://github.com/surrealdb/surrealkv","last_synced_at":"2026-02-14T11:26:33.320Z","repository":{"id":188177367,"uuid":"658762448","full_name":"surrealdb/surrealkv","owner":"surrealdb","description":"A low-level, versioned, embedded, ACID-compliant, key-value database for Rust","archived":false,"fork":false,"pushed_at":"2026-02-10T14:23:46.000Z","size":2303,"stargazers_count":481,"open_issues_count":12,"forks_count":32,"subscribers_count":18,"default_branch":"main","last_synced_at":"2026-02-10T16:25:59.237Z","etag":null,"topics":["embedded-database","key-value-database","key-value-store","kv-store","surreal","surrealdb","surrealkv"],"latest_commit_sha":null,"homepage":"https://surrealdb.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/surrealdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-06-26T12:47:29.000Z","updated_at":"2026-02-10T11:18:22.000Z","dependencies_parsed_at":"2026-01-20T05:13:13.280Z","dependency_job_id":null,"html_url":"https://github.com/surrealdb/surrealkv","commit_stats":null,"previous_names":["surrealdb/surrealkv"],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/surrealdb/surrealkv","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealkv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealkv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealkv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealkv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/surrealdb","download_url":"https://codeload.github.com/surrealdb/surrealkv/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealkv/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29443447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T10:51:12.367Z","status":"ssl_error","status_checked_at":"2026-02-14T10:50:52.088Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embedded-database","key-value-database","key-value-store","kv-store","surreal","surrealdb","surrealkv"],"created_at":"2024-08-01T22:02:57.771Z","updated_at":"2026-02-14T11:26:33.314Z","avatar_url":"https://github.com/surrealdb.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# SurrealKV\n\n[![License](https://img.shields.io/badge/license-Apache_License_2.0-00bfff.svg?style=flat-square)](https://github.com/surrealdb/surrealkv)\n\nSurrealKV is a versioned, embedded key-value store built on an LSM (Log-Structured Merge) tree architecture, with support for time-travel queries.\n\nIt is designed specifically for use within SurrealDB, with the goal of reducing dependency on external storage engine (RocksDB). This approach allows the storage layer to evolve in alignment with SurrealDB’s requirements and access patterns.\n\n## Features\n\n- **ACID Compliance**: Full support for Atomicity, Consistency, Isolation, and Durability\n- **Snapshot Isolation**: MVCC support with non-blocking concurrent reads and writes\n- **Durability Levels**: Immediate and Eventual durability modes\n- **Time-Travel Queries**: Built-in versioning with point-in-time reads and historical queries\n- **Checkpoint and Restore**: Create consistent snapshots for backup and recovery\n- **Value Log (Wisckey)**: Ability to store large values separately, with garbage collection\n\n## Quick Start\n\n```rust\nuse surrealkv::{Tree, TreeBuilder};\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    // Create a new LSM tree using TreeBuilder\n    let tree = TreeBuilder::new()\n        .with_path(\"path/to/db\".into())\n        .build()?;\n\n    // Start a read-write transaction\n    let mut txn = tree.begin()?;\n\n    // Set some key-value pairs\n    txn.set(b\"hello\", b\"world\")?;\n\n    // Commit the transaction (async)\n    txn.commit().await?;\n\n    Ok(())\n}\n```\n\n## Configuration\n\nSurrealKV can be configured through various options when creating a new LSM tree:\n\n### Basic Configuration\n\n```rust\nuse surrealkv::TreeBuilder;\n\nlet tree = TreeBuilder::new()\n    .with_path(\"path/to/db\".into())           // Database directory path\n    .with_max_memtable_size(100 * 1024 * 1024) // 100MB memtable size\n    .with_block_size(4096)                    // 4KB block size\n    .with_level_count(7)                      // Number of levels in LSM tree\n    .build()?;\n```\n\n**Options:**\n- `with_path()` - Database directory where SSTables and WAL files are stored\n- `with_max_memtable_size()` - Size threshold for memtable before flushing to SSTable\n- `with_block_size()` - Size of data blocks in SSTables (affects read performance)\n- `with_level_count()` - Number of levels in the LSM tree structure\n\n### Compression Configuration\n\nSurrealKV supports per-level compression for SSTable data blocks, allowing different compression algorithms for different LSM levels. By default, no compression is used.\n\n```rust\nuse surrealkv::{CompressionType, Options, TreeBuilder};\n\n// Default: No compression (for maximum write performance)\nlet tree = TreeBuilder::new()\n    .with_path(\"path/to/db\".into())\n    .build()?;\n\n// Explicitly disable compression (same as default)\nlet opts = Options::new()\n    .with_path(\"path/to/db\".into())\n    .without_compression();\n\nlet tree = TreeBuilder::with_options(opts).build()?;\n\n// Per-level compression configuration\nlet opts = Options::new()\n    .with_path(\"path/to/db\".into())\n    .with_compression_per_level(vec![\n        CompressionType::None,        // L0: No compression for speed\n        CompressionType::SnappyCompression, // L1+: Snappy compression\n    ]);\n\nlet tree = TreeBuilder::with_options(opts).build()?;\n\n// Convenience: No compression on L0, Snappy on other levels\nlet opts = Options::new()\n    .with_path(\"path/to/db\".into())\n    .with_l0_no_compression();\n\nlet tree = TreeBuilder::with_options(opts).build()?;\n```\n\n**Options:**\n- `without_compression()` - Disable compression for all levels (default behavior)\n- `with_compression_per_level()` - Set compression type per level (vector index = level number)\n- `with_l0_no_compression()` - Convenience method for no compression on L0, Snappy compression on other levels\n\n**Compression Types:**\n- `CompressionType::None` - No compression (fastest writes, largest files)\n- `CompressionType::SnappyCompression` - Snappy compression (good balance of speed and compression ratio)\n\n### Value Log Configuration\n\nThe Value Log (VLog) separates large values from the LSM tree for more efficient storage and compaction.\n\n```rust\nuse surrealkv::{TreeBuilder, VLogChecksumLevel};\n\nlet tree = TreeBuilder::new()\n    .with_path(\"path/to/db\".into())\n    .with_enable_vlog(true)                     // Enable VLog\n    .with_vlog_value_threshold(1024)            // Values \u003e 1KB go to VLog\n    .with_vlog_max_file_size(256 * 1024 * 1024) // 256MB VLog file size\n    .with_vlog_checksum_verification(VLogChecksumLevel::Full)\n    .build()?;\n```\n\n**Options:**\n- `with_enable_vlog()` - Enable/disable Value Log for large value storage\n- `with_vlog_value_threshold()` - Size threshold in bytes; values larger than this are stored in VLog (default: 1KB)\n- `with_vlog_max_file_size()` - Maximum size of VLog files before rotation (default: 256MB)\n- `with_vlog_checksum_verification()` - Checksum verification level (`Disabled` or `Full`)\n\n\n### Versioning Configuration\n\nEnable time-travel queries to read historical versions of your data:\n\n```rust\nuse surrealkv::{Options, TreeBuilder};\n\nlet opts = Options::new()\n    .with_path(\"path/to/db\".into())\n    .with_versioning(true, 0);  // Enable versioning, retention_ns = 0 means no limit\n\nlet tree = TreeBuilder::with_options(opts).build()?;\n```\n\n**Note:** Versioning requires VLog to be enabled. When you call `with_versioning(true, retention_ns)`, VLog is automatically enabled and configured appropriately.\n\n## Transaction Operations\n\n### Basic Operations\n\n```rust\nuse surrealkv::TreeBuilder;\n\n#[tokio::main]\nasync fn main() -\u003e Result\u003c(), Box\u003cdyn std::error::Error\u003e\u003e {\n    let tree = TreeBuilder::new()\n        .with_path(\"path/to/db\".into())\n        .build()?;\n\n    // Write Transaction\n    {\n        let mut txn = tree.begin()?;\n        \n        // Set multiple key-value pairs\n        txn.set(b\"foo1\", b\"bar1\")?;\n        txn.set(b\"foo2\", b\"bar2\")?;\n        \n        // Commit changes (async)\n        txn.commit().await?;\n    }\n\n    // Read Transaction\n    {\n        let txn = tree.begin()?;\n        \n        if let Some(value) = txn.get(b\"foo1\")? {\n            println!(\"Value: {:?}\", value);\n        }\n    }\n\n    Ok(())\n}\n```\n\n**Note:** The transaction API accepts flexible key and value types through the `IntoBytes` trait. You can use `\u0026[u8]`, `\u0026str`, `String`, `Vec\u003cu8\u003e`, or `Bytes` for both keys and values.\n\n### Transaction Modes\n\nSurrealKV supports three transaction modes for different use cases:\n\n```rust\nuse surrealkv::Mode;\n\n// Read-write transaction (default)\nlet mut txn = tree.begin()?;\n\n// Read-only transaction - prevents any writes\nlet txn = tree.begin_with_mode(Mode::ReadOnly)?;\n\n// Write-only transaction - optimized for writes, no reads allowed\nlet mut txn = tree.begin_with_mode(Mode::WriteOnly)?;\n```\n\n### Range Operations\n\nRange operations use a cursor-based iterator API for efficient iteration over key ranges:\n\n```rust\nlet txn = tree.begin()?;\n\n// Range scan between keys (inclusive start, exclusive end)\nlet mut iter = txn.range(b\"key1\", b\"key5\")?;\niter.seek_first()?;\nwhile iter.valid() {\n    let key = iter.key();\n    let value = iter.value()?;\n    println!(\"{:?} = {:?}\", key, value);\n    iter.next()?;\n}\n\n// Backward iteration\nlet mut iter = txn.range(b\"key1\", b\"key5\")?;\niter.seek_last()?;\nwhile iter.valid() {\n    let key = iter.key();\n    let value = iter.value()?;\n    println!(\"{:?} = {:?}\", key, value);\n    iter.prev()?;\n}\n\n// Delete a key\nlet mut txn = tree.begin()?;\ntxn.delete(b\"key1\")?;\ntxn.commit().await?;\n```\n\n**Note:** Range iterators support both forward (`next()`) and backward (`prev()`) iteration using the cursor-based API.\n\n### Durability Levels\n\nControl the durability guarantees for your transactions:\n\n```rust\nuse surrealkv::Durability;\n\nlet mut txn = tree.begin()?;\n\n// Eventual durability (default) - faster, data written to OS buffer\ntxn.set_durability(Durability::Eventual);\n\n// Immediate durability - slower, fsync before commit returns\ntxn.set_durability(Durability::Immediate);\n\ntxn.set(b\"key\", b\"value\")?;\ntxn.commit().await?;\n```\n\n**Durability Levels:**\n- `Eventual`: Commits are guaranteed to be persistent eventually. Data is written to the kernel buffer but not fsynced before returning from `commit()`. This is the default and provides the best performance.\n- `Immediate`: Commits are guaranteed to be persistent as soon as `commit()` returns. Data is fsynced to disk before returning. This is slower but provides the strongest durability guarantees.\n\n\n## Time-Travel Queries\n\nTime-travel queries allow you to read historical versions of your data at specific points in time.\n\n### Enabling Versioning\n\n```rust\nuse surrealkv::{Options, TreeBuilder};\n\nlet opts = Options::new()\n    .with_path(\"path/to/db\".into())\n    .with_versioning(true, 0);  // retention_ns = 0 means no retention limit\n\nlet tree = TreeBuilder::with_options(opts).build()?;\n```\n\n### Writing Versioned Data\n\n```rust\n// Write data with explicit timestamps\nlet mut tx = tree.begin()?;\ntx.set_at(b\"key1\", b\"value_v1\", 100)?;\ntx.commit().await?;\n\n// Update with a new version at a later timestamp\nlet mut tx = tree.begin()?;\ntx.set_at(b\"key1\", b\"value_v2\", 200)?;\ntx.commit().await?;\n```\n\n### Point-in-Time Reads\n\nQuery data as it existed at a specific timestamp:\n\n```rust\nlet tx = tree.begin()?;\n\n// Get value at specific timestamp\nlet value = tx.get_at(b\"key1\", 100)?;\nassert_eq!(value.unwrap().as_ref(), b\"value_v1\");\n\n// Get value at later timestamp\nlet value = tx.get_at(b\"key1\", 200)?;\nassert_eq!(value.unwrap().as_ref(), b\"value_v2\");\n```\n\n### Retrieving All Versions\n\nUse the unified `history()` API to iterate over all historical versions of keys in a range.\nThis API uses streaming iteration (no memory collection) and works with both LSM and B+tree backends:\n\n```rust\nlet tx = tree.begin()?;\nlet mut iter = tx.history(b\"key1\", b\"key2\")?;\n\niter.seek_first()?;\nwhile iter.valid() {\n    let key = iter.key();\n    let timestamp = iter.timestamp();\n\n    if iter.is_tombstone() {\n        println!(\"Key {:?} deleted at timestamp {}\", key, timestamp);\n    } else {\n        let value = iter.value()?;\n        println!(\"Key {:?} = {:?} at timestamp {}\", key, value, timestamp);\n    }\n    iter.next()?;\n}\n```\n\nFor more control, use `history_with_options()`:\n\n```rust\nuse surrealkv::HistoryOptions;\n\nlet tx = tree.begin()?;\nlet opts = HistoryOptions::new()\n    .with_tombstones(true)  // Include deleted entries\n    .with_limit(100);       // Limit to 100 unique keys\n\nlet mut iter = tx.history_with_options(b\"key1\", b\"key2\", \u0026opts)?;\niter.seek_first()?;\nwhile iter.valid() {\n    let key = iter.key();\n    let timestamp = iter.timestamp();\n    if iter.is_tombstone() {\n        println!(\"Key {:?} deleted at timestamp {}\", key, timestamp);\n    } else {\n        let value = iter.value()?;\n        println!(\"Key {:?} = {:?} at timestamp {}\", key, value, timestamp);\n    }\n    iter.next()?;\n}\n```\n\n## Advanced Read Options\n\nUse `ReadOptions` for fine-grained control over read operations:\n\n```rust\nuse surrealkv::ReadOptions;\n\nlet tx = tree.begin()?;\n\n// Range query with bounds using setter methods\nlet mut options = ReadOptions::new();\noptions.set_iterate_lower_bound(Some(b\"a\".to_vec()));\noptions.set_iterate_upper_bound(Some(b\"z\".to_vec()));\n\n// Use cursor-based iteration\nlet mut iter = tx.range_with_options(\u0026options)?;\niter.seek_first()?;\nwhile iter.valid() {\n    let key = iter.key();\n    let value = iter.value()?;\n    println!(\"{:?} = {:?}\", key, value);\n    iter.next()?;\n}\n\n// Point-in-time read (requires versioning enabled)\nlet value = tx.get_at(b\"key1\", 12345)?;\n```\n\n## Checkpoint and Restore\n\nCreate consistent point-in-time snapshots of your database for backup and recovery.\n\n### Creating Checkpoints\n\n```rust\nlet tree = TreeBuilder::new()\n    .with_path(\"path/to/db\".into())\n    .build()?;\n\n// Insert some data\nlet mut txn = tree.begin()?;\ntxn.set(b\"key1\", b\"value1\")?;\ntxn.set(b\"key2\", b\"value2\")?;\ntxn.commit().await?;\n\n// Create checkpoint\nlet checkpoint_dir = \"path/to/checkpoint\";\nlet metadata = tree.create_checkpoint(\u0026checkpoint_dir)?;\n\nprintln!(\"Checkpoint created at timestamp: {}\", metadata.timestamp);\nprintln!(\"Sequence number: {}\", metadata.sequence_number);\nprintln!(\"SSTable count: {}\", metadata.sstable_count);\nprintln!(\"Total size: {} bytes\", metadata.total_size);\n```\n\n### Restoring from Checkpoint\n\n```rust\n// Restore database to checkpoint state\ntree.restore_from_checkpoint(\u0026checkpoint_dir)?;\n\n// Data is now restored to the checkpoint state\n// Any data written after checkpoint creation is discarded\n```\n\n**What's included in a checkpoint:**\n- All SSTables from all levels\n- Current WAL segments\n- Level manifest\n- VLog directories (if VLog is enabled)\n- Checkpoint metadata\n\n**Note:** Restoring from a checkpoint discards any pending writes in the active memtable and returns the database to the exact state when the checkpoint was created.\n\n## Platform Compatibility\n\n### ✅ Supported Platforms\n- **Linux** (x86_64, aarch64): Full support including all features and tests\n- **macOS** (x86_64, aarch64): Full support including all features and tests\n\n### ❌ Not Supported\n- **WebAssembly (WASM)**: Not supported due to fundamental incompatibilities:\n  - Requires file system access not available in WASM environments\n  - Write-Ahead Log (WAL) and Value Log (VLog) operations are not compatible\n  - System-level I/O operations are not available\n\n- **Windows** (x86_64): Basic functionality supported, but some features are limited:\n  - File operations are not thread safe (TODO)\n  - Some advanced file system operations may have reduced functionality\n  - Performance may be lower compared to Unix-like systems\n\n## History\n\nSurrealKV has undergone a significant architectural evolution to address scalability challenges:\n\n### Previous Design (VART-based)\nThe original implementation used a **versioned adaptive radix trie (VART)** architecture with the following components:\n\n- **In-Memory Index**: Versioned adaptive radix trie using [vart](https://github.com/surrealdb/vart) for key-to-offset mappings\n- **Sequential Log Storage**: Append-only storage divided into segments with binary record format\n- **Memory Limitations**: The entire index had to reside in memory, limiting scalability for large datasets\n\n**Why the Change?**\nThe VART-based design had fundamental scalability limitations:\n- **Memory Constraint**: The entire index must fit in memory, making it unsuitable for datasets larger than available RAM\n- **Recovery Overhead**: Startup required scanning all log segments to rebuild the in-memory index\n- **Write Amplification**: Each update created new versions, leading to memory pressure\n\n### Current Design (LSM Tree)\nThe new LSM (Log-Structured Merge) tree architecture provides:\n\n- **Better Scalability**: Supports datasets much larger than available memory\n- **Leveled Compaction**: Score-based compaction strategy for efficient space utilization\n\nThis architectural change enables SurrealKV to handle larger than memory datasets.\n\nFor detailed architecture documentation, see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).\n\n## References\nSurrealKV draws inspiration from design ideas used in [RocksDB](https://github.com/facebook/rocksdb) and [Pebble](https://github.com/cockroachdb/pebble)\n\n\n## License\n\nLicensed under the Apache License, Version 2.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsurrealdb%2Fsurrealkv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsurrealdb%2Fsurrealkv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsurrealdb%2Fsurrealkv/lists"}