{"id":20710328,"url":"https://github.com/kavirajk/kvs","last_synced_at":"2025-06-11T04:11:45.840Z","repository":{"id":71631309,"uuid":"258560714","full_name":"kavirajk/kvs","owner":"kavirajk","description":"A simple Key Value store","archived":false,"fork":false,"pushed_at":"2020-05-20T21:25:58.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-17T21:07:57.883Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kavirajk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-24T16:08:18.000Z","updated_at":"2024-11-06T13:54:23.000Z","dependencies_parsed_at":"2023-05-13T01:45:09.952Z","dependency_job_id":null,"html_url":"https://github.com/kavirajk/kvs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kavirajk%2Fkvs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kavirajk%2Fkvs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kavirajk%2Fkvs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kavirajk%2Fkvs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kavirajk","download_url":"https://codeload.github.com/kavirajk/kvs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242981017,"owners_count":20216337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T02:11:17.707Z","updated_at":"2025-03-11T06:18:54.953Z","avatar_url":"https://github.com/kavirajk.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PNA Rust Project 2: Log-structured file I/O\n\n**Task**: Create a _persistent_ key/value store that _can be accessed from the\ncommand line_.\n\n**Goals**:\n\n- Handle and report errors robustly\n- Use serde for serialization\n- Write data to disk as a log using standard file APIs\n- Read the state of the key/value store from disk\n- Map in-memory key-indexes to on-disk values\n- Periodically compact the log to remove stale data\n\n**Topics**: log-structured file I/O, bitcask, the `failure` crate, `Read` /\n`Write` traits, the `serde` crate.\n\n- [Introduction](#user-content-introduction)\n- [Terminology](#user-content-terminology)\n- [Project spec](#user-content-project-spec)\n- [Project setup](#user-content-project-setup)\n- [Part 1: Error handling](#user-content-part-1-error-handling)\n- [Part 2: How the log behaves](#user-content-part-2-how-the-log-behaves)\n- [Part 3: Writing to the log](#user-content-part-3-writing-to-the-log)\n- [Part 4: Reading from the log](#user-content-part-4-reading-from-the-log)\n- [Part 5: Storing log pointers in the index](#user-content-part-5-storing-log-pointers-in-the-index)\n- [Part 6: Stateless vs. stateful `KvStore`](#user-content-part-6-stateless-vs-stateful-kvstore)\n- [Part 7: Compacting the log](#user-content-part-7-compacting-the-log)\n\n\n## Introduction\n\nIn this project you will create a simple on-disk key/value store that can be\nmodified and queried from the command line. It will use a simplification of the\nstorage algorithm used by [bitcask], chosen for its combination of simplicity\nand effectiveness. You will start by maintaining a _log_ (sometimes called a\n[\"write-ahead log\"][wal] or \"WAL\") on disk of previous write commands that is\nevaluated on startup to re-create the state of the database in memory. Then you\nwill extend that by storing only the keys in memory, along with offsets into the\non-disk log. Finally, you will introduce log compaction so that it does not grow\nindefinitely. At the end of this project you will have built a simple, but\nwell-architected database using Rust file APIs.\n\n[wal]: https://en.wikipedia.org/wiki/Write-ahead_logging\n[bitcask]: https://github.com/basho/bitcask\n\n\n\u003c!--\n## Basic database architecture\n\nTODO\n\n- Basic description and terminology of log, memtable, blocks, etc\n- good opportunity for a diagram\n- find a good background reading\n- using the os page cache for caching\n--\u003e\n\n## Terminology\n\nSome terminology we will use in this course. It is the same as or inspired by\n[bitcask]. Different databases will have slightly different terminology.\n\n- _command_ - A request or the representation of a request made to the database.\n  These are issued on the command line or over the network. They have an\n  in-memory representation, a textual representation, and a machine-readable\n  serialized representation.\n- _log_ - An on-disk sequence of commands, in the order originally received and\n  executed. Our database's on-disk format is almost entirely made up of logs. It\n  will be simple, but also surprisingly efficient.\n- _log pointer_ - A file offset into the log. Sometimes we'll just call this a\n  \"file offset\".\n- _log compaction_ - As writes are issued to the database they sometimes\n  invalidate old log entries. For example, writing key/value `a = 0` then\n  writing `a = 1`, makes the first log entry for \"a\" useless. Compaction \u0026mdash;\n  in our database at least \u0026mdash; is the process of reducing the size of the\n  database by remove stale commands from the log.\n- _in-memory index_ (or _index_) - A map of keys to log pointers. When a read\n  request is issued, the in-memory index is searched for the appropriate log\n  pointer, and when it is found the value is retrieved from the on-disk log. In\n  our key/value store, like in bitcask, the index for the _entire database_ is\n  stored in memory.\n- _index file_ - The on-disk representation of the in-memory index. Without this\n  the log would need to be completely replayed to restore the state of the\n  in-memory index each time the database is started.\n\n\n## Project spec\n\nThe cargo project, `kvs`, builds a command-line key-value store client called\n`kvs`, which in turn calls into a library called `kvs`.\n\nThe `kvs` executable supports the following command line arguments:\n\n- `kvs set \u003cKEY\u003e \u003cVALUE\u003e`\n\n  Set the value of a string key to a string.\n  Print an error and return a non-zero exit code on failure.\n\n- `kvs get \u003cKEY\u003e`\n\n  Get the string value of a given string key.\n  Print an error and return a non-zero exit code on failure.\n\n- `kvs rm \u003cKEY\u003e`\n\n  Remove a given key.\n  Print an error and return a non-zero exit code on failure.\n\n- `kvs -V`\n\n  Print the version\n\nThe `kvs` library contains a type, `KvStore`, that supports the following\nmethods:\n\n- `KvStore::set(\u0026mut self, key: String, value: String) -\u003e Result\u003c()\u003e`\n\n  Set the value of a string key to a string.\n  Return an error if the value is not written successfully.\n\n- `KvStore::get(\u0026mut self, key: String) -\u003e Result\u003cOption\u003cString\u003e\u003e`\n\n  Get the string value of a string key.\n  If the key does not exist, return `None`.\n  Return an error if the value is not read successfully.\n\n- `KvStore::remove(\u0026mut self, key: String) -\u003e Result\u003c()\u003e`\n\n  Remove a given key.\n  Return an error if the key does not exist or is not removed successfully.\n\n- `KvStore::open(path: impl Into\u003cPathBuf\u003e) -\u003e Result\u003cKvStore\u003e`\n\n  Open the KvStore at a given path.\n  Return the KvStore.\n\nWhen setting a key to a value, `kvs` writes the `set` command to disk in a\nsequential log, then stores the log pointer (file offset) of that command in the\nin-memory index from key to pointer. When removing a key, similarly, `kvs`\nwrites the `rm` command in the log, then removes the key from the in-memory\nindex.  When retrieving a value for a key with the `get` command, it searches\nthe index, and if found then loads from the log the command at the corresponding\nlog pointer, evaluates the command and returns the result.\n\nOn startup, the commands in the log are traversed from oldest to newest, and the\nin-memory index rebuilt.\n\nWhen the size of the uncompacted log entries reach a given threshold, `kvs`\ncompacts it into a new log, removing redundent entries to reclaim disk space.\n\nNote that our `kvs` project is both a stateless command-line program, and a\nlibrary containing a stateful `KvStore` type: for CLI use the `KvStore` type\nwill load the index, execute the command, then exit; for library use it will\nload the index, then execute multiple commands, maintaining the index state,\nuntil it is dropped.\n\n\n## Project setup\n\nContinuing from your previous project, delete your previous `tests` directory and\ncopy this project's `tests` directory into its place. Like the previous project,\nthis project should contain a library and an executable, both named `kvs`.\n\nYou need the following dev-dependencies in your `Cargo.toml`:\n\n```toml\n[dev-dependencies]\nassert_cmd = \"0.11.0\"\npredicates = \"1.0.0\"\ntempfile = \"3.0.7\"\nwalkdir = \"2.2.7\"\n```\n\nAs with the previous project, go ahead and write enough empty or panicking\ndefinitions to make the test cases build.\n\n_Do that now._\n\n\n## Part 1: Error handling\n\nIn this project it will be possible for the code to fail due to I/O errors. So\nbefore we get started implementing a database we need to do one more thing that\nis crucial to Rust projects: decide on an error handling strategy.\n\n\u003c!-- TODO outline strategies? --\u003e\n\nRust's error handling is powerful, but involves a lot of boilerplate to use\ncorrectly. For this project the [`failure`] crate will provide the tools to\neasily handle errors of all kinds.\n\n[`failure`]: https://docs.rs/failure/0.1.5/failure/\n\nThe [failure guide][fg] describes [several] error handling patterns.\n\n[fg]: https://boats.gitlab.io/failure/\n[several]: https://boats.gitlab.io/failure/guidance.html\n\nPick one of those strategies and, in your library, either define your own error\ntype or import `failure`s `Error`. This is the error type you will use in all of\nyour `Result`s, converting error types from other crates to your own with the\n`?` operator.\n\nAfter that, define a type alias for `Result` that includes your concrete error\ntype, so that you don't need to type `Result\u003cT, YourErrorType\u003e` everywhere, but\ncan simply type `Result\u003cT\u003e`. This is a common Rust pattern.\n\nFinally, import those types into your executable with `use` statements, and\nchange `main`s function signature to return `Result\u003c()\u003e`. All functions in your\nlibrary that may fail will pass these `Results` back down the stack all the way\nto `main`, and then to the Rust runtime, which will print an error.\n\nRun `cargo check` to look for compiler errors, then fix them. For now it's\nok to end `main` with `panic!()` to make the project build.\n\n_Set up your error handling strategy before continuing._\n\nAs with the previous project, you'll want to create placeholder data structures\nand methods so that the tests compile. Now that you have defined an error type\nthis should be straightforward. Add panics anywhere necessary to get the test\nsuite to compile (`cargo test --no-run`).\n\n\n\u003c!--\n## Aside: The history of Rust error handling\n--\u003e\n\n_Note: Error-handling practices in Rust are still evolving. This course\ncurrently uses the [`failure`] crate to make defining error types easier. While\n`failure` has a good design, its use is [arguably not a best practice][nbp]. It\nmay not continue to be viewed favorably by Rust experts. Future iterations\nof the course will likely not use `failure`. In the meantime, it is fine, and\npresents an opportunity to learn more of the history and nuance of Rust error\nhandling._\n\n[nbp]: https://github.com/rust-lang-nursery/rust-cookbook/issues/502#issue-387418261\n\n\u003c!--\nRust error handling has a long and winding history. Expert Rust programmers will\nbe aware of it, as that history informs and explains modern Rust error handling.\n\nTODO\n--\u003e\n\n\n## Part 2: How the log behaves\n\nNow we are finally going to begin implementing the beginnings of a real database\nby reading and writing from disk. You will use [`serde`] to serialize the \"set\"\nand \"rm\" commands to a string, and the standard file I/O APIs to write it to\ndisk.\n\n[`serde`]: https://serde.rs/\n\nThis is the basic behavior of `kvs` with a log:\n\n- \"set\"\n  - The user invokes `kvs set mykey myvalue`\n   - `kvs` creates a value representing the \"set\" command, containing its key and\n    value\n  - It then serializes that command to a `String`\n  - It then appends the serialized command to a file containing the log\n  - If that succeeds, it exits silently with error code 0\n  - If it fails, it exits by printing the error and returning a non-zero error code\n- \"get\"\n  - The user invokes `kvs get mykey`\n  - `kvs` reads the entire log, one command at a time, recording the \n   affected key and file offset of the command to an in-memory _key -\u003e log\n    pointer_ map\n  - It then checks the map for the log pointer\n  - If it fails, it prints \"Key not found\", and exits with exit code 0\n  - If it succeeds\n    - It deserializes the command to get the last recorded value of the key\n    - It prints the value to stdout and exits with exit code 0\n- \"rm\"\n  - The user invokes `kvs rm mykey`\n  - Same as the \"get\" command, `kvs` reads the entire log to build the in-memory\n    index\n  - It then checks the map if the given key exists\n  - If the key does not exist, it prints \"Key not found\", and exits with a\n    non-zero error code\n  - If it succeeds\n    - It creates a value representing the \"rm\" command, containing its key\n    - It then appends the serialized command to the log\n    - If that succeeds, it exits silently with error code 0\n\nThe log is a record of the transactions committed to the database. By\n\"replaying\" the records in the log on startup we reconstruct the previous state\nof the database.\n\nIn this iteration you may store the value of the keys directly in memory (and\nthus never read from the log after initial startup and log replay). In a future\niteration you will store only \"log pointers\" (file offsets) into the log.\n\n\n## Part 3: Writing to the log\n\nYou will start by implementing the \"set\" flow. There are a number of steps here.\nMost of them are straightforward to implement and you can verify you've done so\nby running the appropriate `cli_*` test cases.\n\n`serde` is a large library, with many options, and supporting many serialization\nformats. Basic serialization and deserialization only requires annotating\nyour data structure correctly, and calling a function to write it\neither to a `String` or a stream implementing `Write`.\n\nYou need to pick a serialization format. Think about the properties you want in\nyour serialization format \u0026mdash; do you want to prioritize performance? Do you\nwant to be able to read the content of the log in plain text? It's your choice,\nbut maybe you should include a comment in the code explaining it.\n\nOther things to consider include: where is the system performing buffering and\nwhere do you need buffering? What is the impact of buffering on subsequent\nreads? When should you open and close file handles? For each command? For the\nlifetime of the `KvStore`?\n\nSome of the APIs you will call may fail, and return a `Result` of some error type.\nMake sure that your calling functions return a `Result` of _your own_ error type,\nand that you convert between the two with `?`.\n\nIt is similar to implementing the \"rm\" command, but you should additionally\ncheck if the key exists before writing the command to the log. As we have two\ndifferent commands that must be distinguished, you may use variants of a single\nenum type to represent each command. `serde` just works perfectly with enums.\n\nYou may implement the \"set\" and \"rm\" commands now, focusing on the `set` / `rm`\ntest cases, or you can proceed to the next section to read about the \"get\"\ncommand. It may help to keep both in mind, or to implement them both\nsimultaneously. It is your choice.\n\n\n## Part 4: Reading from the log\n\nNow it's time to implement \"get\". In this part, you don't need to store\nlog pointers in the index, we will leave the work to the next part. Instead,\njust read each command in the log on startup, executing them to save every key\nand value in the memory. Then read from the memory.\n\nShould you read all records in the log into memory at once and then replay\nthem into your map type; or should you read them one at a time while\nreplaying them into your map? Should you read into a buffer before deserializing\nor deserialize from a file stream? Think about the memory usage of your approach.\nThink about the way reading from I/O streams interacts with the kernel.\n\nRemember that \"get\" may not find a value and that case has to be handled\nspecially. Here, our API returns `None` and our command line client prints\na particular message and exits with a zero exit code.\n\nThere's one complication to reading the log, and you may have already considered\nit while writing the \"set\" code: how do you distinguish between each record in\nthe log? That is, how do you know when to stop reading one record, and start\nreading the next? Do you even need to? Maybe serde will deserialize a record\ndirectly from an I/O stream and stop reading when it's done, leaving the\nfile cursor in the correct place to read subsequent records. Maybe serde will\nreport an error when it sees two records back-to-back. Maybe you need to insert\nadditional information to distinguish the length of each record. Maybe not.\n\n[`serde`]: https://serde.rs/\n\n_Implement \"get\" now_.\n\n\n## Part 5: Storing log pointers in the index\n\nAt this point most, if not all (besides the compaction test), other test suite should all pass. The changes\nintroduced in the next steps are simple optimizations, necessary for fast\nperformance and reduced storage. As you implement them, pay attention to what\nexactly they are optimizing for.\n\nAs we've described, the database you are building maintains an in-memory index\nof all keys in the database. That index maps from string keys to log pointers,\nnot the values themselves.\n\nThis change introduces the need to perform reads from the log\nat arbitrary offsets. Consider how that might impact the way\nyou manage file handles.\n\n_If, in the previous steps, you elected to store the string values directly in\nmemory, now is the time to update your code to store log pointers instead,\nloading from disk on demand._\n\n\n## Part 6: Stateless vs. stateful `KvStore`\n\nRemember that our project is both a library and a command-line program.\nThey have sligtly different requirements: the `kvs` CLI commits a single change\nto disk, then exits (it is stateless); the `KvStore` type commits\nchanges to disk, then stays resident in memory to service future\nqueries (it is stateful).\n\nIs your `KvStore` stateful or stateless?\n\n_Make your `KvStore` retain the index in memory so it doesn't need to\nre-evaluate it for every call to `get`._\n\n\n## Part 7: Compacting the log\n\nAt this point the database works just fine, but the log grows indefinitely. That\nis appropriate for some databases, but not the one we're building \u0026mdash; we\nwant to minimize disk usage as much as we can.\n\nSo the final step in creating your database is to compact the log. Consider\nthat as the log grows that multiple entries may set the value of a given key.\nConsider also that only the most recent command that modified a given key has\nany effect on the current value of that key:\n\n| idx | command |\n|:---:|:--------|\n| 0 | ~Command::Set(\"key-1\", \"value-1a\")~ |\n| 20 | Command::Set(\"key-2\", \"value-2\") |\n| | ... |\n| 100 | Command::Set(\"key-1\", \"value-1b\") |\n\nIn this example obviously the command at index 0 is redundant, so it doesn't\nneed to be stored. Log compaction then is about rebuilding the log to remove\nredundancy:\n\n| idx | command |\n|:---:|:--------|\n| 0 | Command::Set(\"key-2\", \"value-2\") |\n| | ... |\n| 99 | Command::Set(\"key-1\", \"value-1b\") |\n\nHere's the basic algorithm you will use:\n\n\u003c!-- TODO: Think about this. should the algorithm be specified? what _is_ a\ngood heuristic to rebuild the log? always rebuild the entire log? --\u003e\n\n_How_ you re-build the log is up to you. Consider questions like: what is the\nnaive solution? How much memory do you need? What is the minimum amount of\ncopying necessary to compact the log? Can the compaction be done in-place? How\ndo you maintain data-integrity if compaction fails?\n\nSo far we've been refering to \"the log\", but in actuallity it is common for a\ndatabase to store many logs, in different files. You may find it easier to\ncompact the log if you split your log across files.\n\n_Implement log compaction for your database._\n\nCongratulations! You have written a fully-functional database.\n\nIf you are curious, now is a good time to start comparing the performance of\nyour key/value store to others, like [sled], [bitcask], [badger], or [RocksDB].\nYou might enjoy investigating their architectures, thinking about how theirs\ncompare to yours, and how architecture affects performance. The next few\nprojects will give you opportunities to optimize.\n\n[sled]: https://github.com/spacejam/sled\n[badger]: https://github.com/dgraph-io/badger\n[RocksDB]: https://rocksdb.org/\n\nNice coding, friend. Enjoy a nice break.\n\n\n\u003c!--\n\n- Potential readings\n  - error handling https://github.com/rust-lang-nursery/rust-cookbook/issues/502#issue-387418261\n\n## TODOs\n\n- should flushing the log be part of the main project or an extension?\n- check terminology\n  - what's the correct term for the in-memory representation of the executed log?\n- is there a term for converting a log to it's permanent format?\n- custom main error handling\n- limits on k/v size?\n- maintaining data integrity on failure\n  - _must_ call flush at least\n- todo: `Result\u003cOption\u003cString\u003e\u003e` vs `Result\u003cString\u003e`\n  - is \"not found\" exit code 0 or !0?\n- error context\n- serialize directly to file stream\n- need readings for WAL\n- add code samples and digarams to illustrate the text and\n  be less monotonous\n- maintain the index file!\n- specify where data should be stored\n- caching the index\n\n--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkavirajk%2Fkvs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkavirajk%2Fkvs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkavirajk%2Fkvs/lists"}