{"id":31703605,"url":"https://github.com/surrealdb/surrealmx","last_synced_at":"2025-10-08T22:06:17.152Z","repository":{"id":231294891,"uuid":"780799742","full_name":"surrealdb/surrealmx","owner":"surrealdb","description":"An embedded, in-memory, lock-free, transaction-based, key-value database engine","archived":false,"fork":false,"pushed_at":"2025-10-06T21:32:05.000Z","size":246,"stargazers_count":22,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-10-06T22:42:39.328Z","etag":null,"topics":["database","key-value","key-value-store","rust","surreal","surrealdb","surrealmx"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/surrealmx","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/surrealdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-02T07:22:03.000Z","updated_at":"2025-10-06T22:20:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"3d24557f-805b-408f-8b53-7763a57816aa","html_url":"https://github.com/surrealdb/surrealmx","commit_stats":null,"previous_names":["surrealdb/memodb","surrealdb/surrealmx"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/surrealdb/surrealmx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealmx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealmx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealmx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealmx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/surrealdb","download_url":"https://codeload.github.com/surrealdb/surrealmx/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/surrealdb%2Fsurrealmx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000733,"owners_count":26082851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","key-value","key-value-store","rust","surreal","surrealdb","surrealmx"],"created_at":"2025-10-08T22:06:15.967Z","updated_at":"2025-10-08T22:06:17.137Z","avatar_url":"https://github.com/surrealdb.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cbr\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://surrealdb.com#gh-dark-mode-only\" target=\"_blank\"\u003e\n        \u003cimg width=\"200\" src=\"/img/white/logo.svg\" alt=\"SurrealMX Logo\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://surrealdb.com#gh-light-mode-only\" target=\"_blank\"\u003e\n        \u003cimg width=\"200\" src=\"/img/black/logo.svg\" alt=\"SurrealMX Logo\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003eAn embedded, in-memory, lock-free, transaction-based, key-value database engine.\u003c/p\u003e\n\n\u003cbr\u003e\n\n\u003cp align=\"center\"\u003e\n\t\u003ca href=\"https://github.com/surrealdb/surrealmx\"\u003e\u003cimg src=\"https://img.shields.io/badge/status-stable-ff00bb.svg?style=flat-square\"\u003e\u003c/a\u003e\n\t\u0026nbsp;\n\t\u003ca href=\"https://docs.rs/surrealmx/\"\u003e\u003cimg src=\"https://img.shields.io/docsrs/surrealmx?style=flat-square\"\u003e\u003c/a\u003e\n\t\u0026nbsp;\n\t\u003ca href=\"https://crates.io/crates/surrealmx\"\u003e\u003cimg src=\"https://img.shields.io/crates/v/surrealmx?style=flat-square\"\u003e\u003c/a\u003e\n\t\u0026nbsp;\n\t\u003ca href=\"https://github.com/surrealdb/surrealmx\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache_License_2.0-00bfff.svg?style=flat-square\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n#### Features\n\n- In-memory database\n- Multi-version concurrency control\n- Rich transaction support with rollbacks\n- Multiple concurrent readers without locking\n- Multiple concurrent writers without locking\n- Support for serializable, snapshot isolated transactions\n- Atomicity, Consistency and Isolation from ACID\n- Optional persistence with configurable modes:\n  - Support for synchronous and asynchronous append-only logging\n  - Support for periodic full-datastore snapshots\n  - Support for fsync on every commit, or periodically in the background\n  - Support for LZ4 snapshot file compression\n\n#### Quick start\n\n```rust\nuse surrealmx::{Database, DatabaseOptions};\n\nfn main() {\n    // Create a database with custom settings\n    let opts = DatabaseOptions { pool_size: 128, ..Default::default() };\n    let db: Database\u003c\u0026str, \u0026str\u003e = Database::new_with_options(opts);\n\n    // Start a write transaction\n    let mut tx = db.transaction(true);\n    tx.put(\"key\", \"value\").unwrap();\n    tx.commit().unwrap();\n\n    // Read the value back\n    let mut tx = db.transaction(false);\n    assert_eq!(tx.get(\"key\").unwrap(), Some(\"value\"));\n    tx.cancel().unwrap();\n}\n```\n\n#### Manual cleanup and garbage collection\n\nBackground worker threads perform cleanup and garbage collection at regular\nintervals. These workers can be disabled through `DatabaseOptions` by setting\n`enable_cleanup` or `enable_gc` to `false`. When disabled, the tasks can be\ntriggered manually using the `run_cleanup` and `run_gc` methods.\n\n```rust\nuse surrealmx::{Database, DatabaseOptions};\n\nfn main() {\n    // Create a database with custom settings\n    let opts = DatabaseOptions { enable_gc: false, enable_cleanup: false, ..Default::default() };\n    let db: Database\u003c\u0026str, \u0026str\u003e = Database::new_with_options(opts);\n\n    // Start a write transaction\n    let mut tx = db.transaction(true);\n    tx.put(\"key\", \"value1\").unwrap();\n    tx.commit().unwrap();\n\n\t// Start a write transaction\n    let mut tx = db.transaction(true);\n    tx.put(\"key\", \"value2\").unwrap();\n    tx.commit().unwrap();\n\n\t// Manually remove unused transaction stale versions\n    db.run_cleanup();\n\t\n\t// Manually remove old queue entries\n    db.run_gc();\n}\n```\n\n#### Persistence modes\n\nMemoDB supports optional persistence with two modes:\n\n##### Full persistence (AOL + Snapshots) - Default\n\nProvides maximum durability by logging every change to an append-only log and taking periodic snapshots.\n\n```rust\nuse surrealmx::{Database, DatabaseOptions, PersistenceOptions, AolMode, SnapshotMode};\nuse std::time::Duration;\n\nfn main() -\u003e std::io::Result\u003c()\u003e {\n    let db_opts = DatabaseOptions::default();\n    let persistence_opts = PersistenceOptions::new(\"./data\")\n        .with_aol_mode(AolMode::SynchronousOnCommit)\n        .with_snapshot_mode(SnapshotMode::Interval(Duration::from_secs(60)));\n    \n    let db: Database\u003cString, String\u003e = Database::new_with_persistence(db_opts, persistence_opts)?;\n    \n    let mut tx = db.transaction(true);\n    tx.put(\"key\".to_string(), \"value\".to_string())?;\n    tx.commit()?; // Changes immediately written to AOL\n    \n    Ok(())\n}\n```\n\n##### Snapshot-only persistence\n\nProvides good performance with periodic durability by taking snapshots without logging individual changes.\n\n```rust\nuse surrealmx::{Database, DatabaseOptions, PersistenceOptions, AolMode, SnapshotMode};\nuse std::time::Duration;\n\nfn main() -\u003e std::io::Result\u003c()\u003e {\n    let db_opts = DatabaseOptions::default();\n    let persistence_opts = PersistenceOptions::new(\"./snapshot_data\")\n        .with_aol_mode(AolMode::Never) // Disable AOL, use only snapshots\n        .with_snapshot_mode(SnapshotMode::Interval(Duration::from_secs(30)));\n    \n    let db: Database\u003cString, String\u003e = Database::new_with_persistence(db_opts, persistence_opts)?;\n    \n    let mut tx = db.transaction(true);\n    tx.put(\"key\".to_string(), \"value\".to_string())?;\n    tx.commit()?; // Changes only persisted during snapshots\n    \n    Ok(())\n}\n```\n\n##### Configuration Options\n\n###### AOL Modes\n- **`AolMode::Never`**: Disables append-only logging entirely (default)\n- **`AolMode::SynchronousOnCommit`**: Writes changes to AOL immediately on every commit (maximum durability)\n- **`AolMode::AsynchronousAfterCommit`**: Writes changes to AOL asynchronously after every commit (better performance)\n\n###### Snapshot Modes\n- **`SnapshotMode::Never`**: Disables snapshots entirely (default)\n- **`SnapshotMode::Interval(Duration)`**: Takes snapshots at the specified interval\n\n###### Fsync Modes\n- **`FsyncMode::Never`**: Never calls fsync - fastest but least durable (default)\n- **`FsyncMode::EveryAppend`**: Calls fsync after every AOL append - slowest but most durable\n- **`FsyncMode::Interval(Duration)`**: Calls fsync at most once per interval - balanced approach\n\n###### Compression Support\n- **`CompressionMode::None`**: No compression applied to snapshots (default)\n- **`CompressionMode::Lz4`**: Fast LZ4 compression for snapshots (reduces storage size)\n\n##### Advanced Configuration Example\n\n```rust\nuse surrealmx::{Database, DatabaseOptions, PersistenceOptions, AolMode, SnapshotMode, FsyncMode, CompressionMode};\nuse std::time::Duration;\n\nfn main() -\u003e std::io::Result\u003c()\u003e {\n    let db_opts = DatabaseOptions::default();\n    let persistence_opts = PersistenceOptions::new(\"./advanced_data\")\n        .with_aol_mode(AolMode::AsynchronousAfterCommit) // Async AOL writes\n        .with_snapshot_mode(SnapshotMode::Interval(Duration::from_secs(300))) // Snapshot every 5 minutes\n        .with_fsync_mode(FsyncMode::Interval(Duration::from_secs(1))) // Fsync every second\n        .with_compression(CompressionMode::Lz4); // Enable LZ4 compression\n    \n    let db: Database\u003cString, String\u003e = Database::new_with_persistence(db_opts, persistence_opts)?;\n    \n    let mut tx = db.transaction(true);\n    tx.put(\"key\".to_string(), \"value\".to_string())?;\n    tx.commit()?; // Changes written asynchronously to AOL, fsync'd every second\n    \n    Ok(())\n}\n```\n\n**Trade-offs:**\n- **AOL + Snapshots**: Maximum durability, slower writes, larger storage\n- **Snapshot-only**: Better performance, risk of data loss between snapshots, smaller storage\n- **Synchronous AOL**: Immediate durability, slower commit times\n- **Asynchronous AOL**: Better performance, small risk of data loss on system crash\n- **Frequent fsync**: Higher durability, reduced performance\n- **LZ4 Compression**: Smaller storage footprint, slight CPU overhead\n\n#### Historical reads\n\nMemoDB's MVCC (Multi-Version Concurrency Control) design allows you to read data as it existed at any point in time. This enables powerful use cases like:\n\n- **Audit trails**: See what data looked like at specific timestamps\n- **Time-travel debugging**: Examine application state at the time of an issue\n- **Consistent reporting**: Generate reports based on a snapshot of data from a specific point in time\n- **Conflict resolution**: Compare different versions of data to understand changes\n\n```rust\nuse surrealmx::Database;\n\nfn main() {\n    let db: Database\u003c\u0026str, \u0026str\u003e = Database::new();\n    \n    // Insert some initial data\n    let mut tx = db.transaction(true);\n    tx.put(\"user:1\", \"Alice\").unwrap();\n    tx.commit().unwrap();\n    \n    // Capture timestamp after first commit\n    let version_1 = db.oracle.current_timestamp();\n    \n    // Wait a moment to ensure different timestamps\n    std::thread::sleep(std::time::Duration::from_millis(1));\n    \n    // Make some changes\n    let mut tx = db.transaction(true);\n    tx.set(\"user:1\", \"Alice Smith\").unwrap(); // Update name\n    tx.put(\"user:2\", \"Bob\").unwrap();         // Add new user\n    tx.commit().unwrap();\n    \n    // Read historical data\n    let mut tx = db.transaction(false);\n    \n    // Read current state\n    assert_eq!(tx.get(\"user:1\").unwrap(), Some(\"Alice Smith\"));\n    assert_eq!(tx.get(\"user:2\").unwrap(), Some(\"Bob\"));\n    \n    // Read state as it was at version_1 (before changes)\n    assert_eq!(tx.get_at_version(\"user:1\", version_1).unwrap(), Some(\"Alice\"));\n    assert_eq!(tx.get_at_version(\"user:2\", version_1).unwrap(), None);\n    \n    // Range operations also support historical reads\n    let historical_keys = tx.keys_at_version(\"user:0\"..\"user:9\", None, None, version_1).unwrap();\n    assert_eq!(historical_keys, vec![\"user:1\"]);\n    \n    tx.cancel().unwrap();\n}\n```\n\n**Available historical read methods:**\n- `get_at_version(key, version)`: Read a single key's value at a specific version\n- `keys_at_version(range, skip, limit, version)`: Get keys in range at a specific version\n- `scan_at_version(range, skip, limit, version)`: Get key-value pairs at a specific version\n- `total_at_version(range, skip, limit, version)`: Count keys at a specific version\n\n#### Isolation levels\n\nMemoDB supports two isolation levels to balance between performance and consistency guarantees:\n\n##### Snapshot Isolation (Default)\n\nProvides excellent performance with strong consistency guarantees. Transactions see a consistent snapshot of the database as it existed when the transaction began.\n\n- **Read consistency**: All reads within a transaction see the same consistent view\n- **Write isolation**: Changes from other transactions are not visible until they commit\n- **No dirty reads**: Never see uncommitted changes from other transactions\n- **No non-repeatable reads**: Reading the same key multiple times returns the same value\n\n```rust\nuse surrealmx::Database;\n\nfn main() {\n    let db: Database\u003c\u0026str, i32\u003e = Database::new();\n    \n    // Snapshot isolation (default behavior)\n    let mut tx1 = db.transaction(true);\n    let mut tx2 = db.transaction(false); // Start tx2 before tx1 commits\n    \n    tx1.put(\"counter\", 1).unwrap();\n    tx1.commit().unwrap();\n    \n    // tx2 started before tx1 committed, so it doesn't see the change\n    assert_eq!(tx2.get(\"counter\").unwrap(), None);\n    tx2.cancel().unwrap();\n}\n```\n\n##### Serializable Snapshot Isolation\n\nProvides the strongest consistency guarantee by detecting read-write conflicts and aborting transactions that would violate serializability.\n\n- **All Snapshot Isolation guarantees**: Plus additional conflict detection\n- **Read-write conflict detection**: Prevents phantom reads and write skew\n- **Serializable execution**: Equivalent to running transactions one at a time\n- **Higher abort rate**: More transactions may need to retry due to conflicts\n\n```rust\nuse surrealmx::{Database, Error};\n\nfn main() {\n    let db: Database\u003c\u0026str, i32\u003e = Database::new();\n    \n    // Initialize data\n    let mut tx = db.transaction(true);\n    tx.put(\"x\", 0).unwrap();\n    tx.put(\"y\", 0).unwrap();\n    tx.commit().unwrap();\n    \n    // Two concurrent transactions that would cause write skew\n    let mut tx1 = db.transaction(true); // Uses SerializableSnapshotIsolation internally\n    let mut tx2 = db.transaction(true);\n    \n    // tx1 reads x and writes to y\n    let x_val = tx1.get(\"x\").unwrap().unwrap();\n    tx1.set(\"y\", x_val + 1).unwrap();\n    \n    // tx2 reads y and writes to x  \n    let y_val = tx2.get(\"y\").unwrap().unwrap();\n    tx2.set(\"x\", y_val + 1).unwrap();\n    \n    // First transaction commits successfully\n    tx1.commit().unwrap();\n    \n    // Second transaction detects conflict and aborts\n    match tx2.commit() {\n        Err(Error::KeyReadConflict) =\u003e {\n            // Transaction must be retried\n            println!(\"Transaction aborted due to read conflict, retrying...\");\n        }\n        _ =\u003e panic!(\"Expected read conflict\"),\n    }\n}\n```\n\n**When to use each isolation level:**\n- **Snapshot Isolation**: Most applications, high-performance scenarios, read-heavy workloads\n- **Serializable Snapshot Isolation**: Financial applications, inventory management, any scenario requiring strict serializability\n\n#### Range operations\n\nMemoDB provides powerful range-based operations for scanning, counting, and iterating over keys. All range operations support:\n\n- **Forward and reverse iteration**\n- **Skip and limit parameters** for pagination  \n- **Historical versions** for time-travel queries\n- **Efficient range scans** using the underlying B+ tree structure\n\n##### Basic range scanning\n\n```rust\nuse surrealmx::Database;\n\nfn main() {\n    let db: Database\u003c\u0026str, \u0026str\u003e = Database::new();\n    \n    // Insert test data\n    let mut tx = db.transaction(true);\n    for i in 1..=10 {\n        tx.put(\u0026format!(\"key:{:02}\", i), \u0026format!(\"value:{}\", i)).unwrap();\n    }\n    tx.commit().unwrap();\n    \n    let mut tx = db.transaction(false);\n    \n    // Get all keys in range\n    let keys = tx.keys(\"key:03\"..\"key:08\", None, None).unwrap();\n    assert_eq!(keys, vec![\"key:03\", \"key:04\", \"key:05\", \"key:06\", \"key:07\"]);\n    \n    // Get key-value pairs in range\n    let pairs = tx.scan(\"key:03\"..\"key:06\", None, None).unwrap();\n    assert_eq!(pairs, vec![\n        (\"key:03\", \"value:3\"),\n        (\"key:04\", \"value:4\"), \n        (\"key:05\", \"value:5\")\n    ]);\n    \n    // Count keys in range\n    let count = tx.total(\"key:00\"..\"key:99\", None, None).unwrap();\n    assert_eq!(count, 10);\n    \n    tx.cancel().unwrap();\n}\n```\n\n##### Pagination and reverse iteration\n\n```rust\nuse surrealmx::Database;\n\nfn main() {\n    let db: Database\u003c\u0026str, i32\u003e = Database::new();\n    \n    // Insert test data\n    let mut tx = db.transaction(true);\n    for i in 1..=100 {\n        tx.put(\u0026format!(\"item:{:03}\", i), i).unwrap();\n    }\n    tx.commit().unwrap();\n    \n    let mut tx = db.transaction(false);\n    \n    // Paginated forward scan: skip 10, take 5\n    let page1 = tx.scan(\"item:000\"..\"item:999\", Some(10), Some(5)).unwrap();\n    assert_eq!(page1.len(), 5);\n    assert_eq!(page1[0].0, \"item:011\");\n    assert_eq!(page1[4].0, \"item:015\");\n    \n    // Reverse iteration: get last 3 items\n    let last_items = tx.scan_reverse(\"item:000\"..\"item:999\", None, Some(3)).unwrap();\n    assert_eq!(last_items.len(), 3);\n    assert_eq!(last_items[0].0, \"item:100\"); // First item is the highest key\n    assert_eq!(last_items[2].0, \"item:098\"); // Last item is lower\n    \n    tx.cancel().unwrap();\n}\n```\n\n##### Historical range operations\n\n```rust\nuse surrealmx::Database;\n\nfn main() {\n    let db: Database\u003c\u0026str, \u0026str\u003e = Database::new();\n    \n    // Insert initial data\n    let mut tx = db.transaction(true);\n    tx.put(\"a\", \"1\").unwrap();\n    tx.put(\"b\", \"2\").unwrap();\n    tx.commit().unwrap();\n    let version_1 = db.oracle.current_timestamp();\n    \n    // Wait a moment to ensure different timestamps\n    std::thread::sleep(std::time::Duration::from_millis(1));\n    \n    // Add more data\n    let mut tx = db.transaction(true);\n    tx.put(\"c\", \"3\").unwrap();\n    tx.put(\"d\", \"4\").unwrap();\n    tx.commit().unwrap();\n    \n    let mut tx = db.transaction(false);\n    \n    // Current state: all 4 keys\n    let current_keys = tx.keys(\"a\"..\"z\", None, None).unwrap();\n    assert_eq!(current_keys, vec![\"a\", \"b\", \"c\", \"d\"]);\n    \n    // Historical state: only first 2 keys\n    let historical_keys = tx.keys_at_version(\"a\"..\"z\", None, None, version_1).unwrap();\n    assert_eq!(historical_keys, vec![\"a\", \"b\"]);\n    \n    // Count at different versions\n    let current_count = tx.total(\"a\"..\"z\", None, None).unwrap();\n    let historical_count = tx.total_at_version(\"a\"..\"z\", None, None, version_1).unwrap();\n    assert_eq!(current_count, 4);\n    assert_eq!(historical_count, 2);\n    \n    tx.cancel().unwrap();\n}\n```\n\n**Available range operation methods:**\n\n**Current version:**\n- `keys(range, skip, limit)` / `keys_reverse(...)`: Get keys in range\n- `scan(range, skip, limit)` / `scan_reverse(...)`: Get key-value pairs in range\n- `total(range, skip, limit)`: Count keys in range\n\n**Historical versions:**\n- `keys_at_version(range, skip, limit, version)` / `keys_at_version_reverse(...)`\n- `scan_at_version(range, skip, limit, version)` / `scan_at_version_reverse(...)`\n- `total_at_version(range, skip, limit, version)`\n\n**Range parameters:**\n- `range`: Rust range syntax (`\"start\"..\"end\"`) - start inclusive, end exclusive\n- `skip`: Optional number of items to skip (for pagination)\n- `limit`: Optional maximum number of items to return\n- `version`: Specific version timestamp for historical operations\n\n#### Project History\n\n**Note:** This project was originally developed under the name `memodb`. It has been renamed to `surrealmx` to better reflect its evolution and alignment with the SurrealDB ecosystem.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsurrealdb%2Fsurrealmx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsurrealdb%2Fsurrealmx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsurrealdb%2Fsurrealmx/lists"}