{"id":21452945,"url":"https://github.com/arindas/riakv","last_synced_at":"2025-07-14T23:31:04.672Z","repository":{"id":47612901,"uuid":"398649703","full_name":"arindas/riakv","owner":"arindas","description":"Log structured append-only key-value store from Rust In Action with some enhancements.","archived":false,"fork":false,"pushed_at":"2021-09-15T08:04:08.000Z","size":1916,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-03-07T11:32:10.487Z","etag":null,"topics":["append-only","key-value-store","log-structured"],"latest_commit_sha":null,"homepage":"https://arindas.github.io/riakv/libriakv/index.html","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arindas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-21T20:22:58.000Z","updated_at":"2022-10-29T10:55:44.000Z","dependencies_parsed_at":"2022-07-21T16:34:39.429Z","dependency_job_id":null,"html_url":"https://github.com/arindas/riakv","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arindas%2Friakv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arindas%2Friakv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arindas%2Friakv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arindas%2Friakv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arindas","download_url":"https://codeload.github.com/arindas/riakv/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226002954,"owners_count":17558152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["append-only","key-value-store","log-structured"],"created_at":"2024-11-23T04:33:17.426Z","updated_at":"2024-11-23T04:33:17.912Z","avatar_url":"https://github.com/arindas.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `riakv`\n[![ci-tests](https://github.com/arindas/riakv/actions/workflows/ci-tests.yml/badge.svg)](https://github.com/arindas/riakv/actions/workflows/ci-tests.yml)\n[![rustdoc](https://github.com/arindas/riakv/actions/workflows/rustdoc.yml/badge.svg)](https://github.com/arindas/riakv/actions/workflows/rustdoc.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n \nLog structured, append only, key value store implementation from [Rust In Action](https://www.manning.com/books/rust-in-action) with some enhancements.\n\n## Features\n\n- [x] Persistent key value store with a hash table index\n- [x] `crc32` checksum validation for every key value pair stored.\n- [x] Optionally, persistent index for fast loading\n- [x] Exhaustive, comprehensive tests\n\n## Design enhancements\n\n### Generic storage\nThe underlying storage used by the key value store is completely generic. It is subject to `Read + Write + Seek` trait bounds.\n```rust\n#[derive(Debug)]\npub struct RiaKV\u003cF\u003e\nwhere\n    F: Read + Write + Seek,\n{\n    f: F,\n    pub index: HashMap\u003cByteString, u64\u003e,\n}\n```\n\nThis allows for creation of key value store instances from an in-memory buffer:\n```rust\nimpl RiaKV\u003cio::Cursor\u003cVec\u003cu8\u003e\u003e\u003e {\n    pub fn open_from_in_memory_buffer(capacity: usize) -\u003e Self {\n        RiaKV {\n            f: io::Cursor::new(vec![0; capacity]),\n            index: HashMap::new(),\n        }\n    }\n}\n```\n\n### Refactors in iteration over key value pairs stored in file\nInstead of duplicating iteration code in `RiaKV::find` and `RiaKV::load`, we refactor the loop\ninto `RiaKV::for_each`. This method accepts a callback to operate on the key value pair\nreceived in every iteration. Finally, the callback returns an enum which specifies how to\nupdate the store hashtable index using the key value pair.\n\nThis is implemented as follows. First, we have an index operation type:\n```rust\npub enum IndexOp {\n    Insert(KeyValuePair, u64),\n    Delete(KeyValuePair, u64),\n    End,\n    Nop,\n}\n```\n\nNext we define the for each function:\n```rust\npub fn for_each_kv_entry_in_storage\u003cFunc\u003e(\u0026mut self, mut callback: Func) -\u003e io::Result\u003c()\u003e\n    where\n        Func: FnMut(KeyValuePair, u64) -\u003e IndexOp,\n    { ...\n```\n\nAs we can see the callback take the key value pair, and its position in the file, and\nreturns an index operation. Now in the iteration step, we match on the value returned\nby the callback and perform the required index operation.\n\n```rust\n    loop { \n\n        ...\n        \n        match callback(kv, position) {\n                IndexOp::Insert(kv, position) =\u003e {\n                    self.index.insert(kv.key, position);\n                }\n                IndexOp::Delete(kv, _) =\u003e {\n                    self.index.remove(\u0026kv.key);\n                }\n                IndexOp::Nop =\u003e {}\n                IndexOp::End =\u003e {\n                    break;\n                }\n            }\n        }\n        \n        ...\n```\n\nThis function is used as follows:\n```rust\npub fn load(\u0026mut self) -\u003e io::Result\u003c()\u003e {\n    self.for_each_kv_entry_in_storage(|kv, position| {\n        if kv.value.len() \u003e 0 {\n            IndexOp::Insert(kv, position)\n        } else {\n            IndexOp::Delete(kv, position)\n        }\n    })\n}\n\n// ...\n\npub fn find(\u0026mut self, target: \u0026ByteStr) -\u003e io::Result\u003cOption\u003c(u64, ByteString)\u003e\u003e {\n    let mut found: Option\u003c(u64, ByteString)\u003e = None;\n\n    self.for_each_kv_entry_in_storage(|kv, position| {\n        if kv.key == target.to_vec() {\n            found = Some((position, kv.value));\n            IndexOp::End\n        } else {\n            IndexOp::Nop\n        }\n    })?;\n\n    Ok(found)\n}\n```\n\n## Building\n\nAfter cloning the repository, simply run `cargo build --release` from the project root. This project\nprovides two binaries `riakv_mem` and `riakv_disk`. The `_*` suffix specifies whether the index is\npersisted in the disk or not.\n\n# License\n\n`riakv` is licensed under the MIT License. See [LICENSE](./LICENSE) for the full license text.\n\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Farindas%2Friakv.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Farindas%2Friakv?ref=badge_large)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farindas%2Friakv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farindas%2Friakv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farindas%2Friakv/lists"}