{"id":16797065,"url":"https://github.com/jdockerty/squirrel","last_synced_at":"2026-03-06T11:32:33.353Z","repository":{"id":233975871,"uuid":"734335796","full_name":"jdockerty/squirrel","owner":"jdockerty","description":"A bitcask inspired, replicated key-value store.","archived":false,"fork":false,"pushed_at":"2024-07-13T14:33:36.000Z","size":74,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-30T21:38:44.525Z","etag":null,"topics":["bitcask-like","key-value","replicated-cache","rust","rust-learning"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jdockerty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-21T12:33:24.000Z","updated_at":"2024-07-13T14:33:39.000Z","dependencies_parsed_at":"2024-07-13T15:47:56.727Z","dependency_job_id":"6bf88615-86e9-4ff4-a909-f3f237687d64","html_url":"https://github.com/jdockerty/squirrel","commit_stats":null,"previous_names":["jdockerty/squirrel"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jdockerty/squirrel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdockerty%2Fsquirrel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdockerty%2Fsquirrel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdockerty%2Fsquirrel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdockerty%2Fsquirrel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jdockerty","download_url":"https://codeload.github.com/jdockerty/squirrel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdockerty%2Fsquirrel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30173694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T07:56:45.623Z","status":"ssl_error","status_checked_at":"2026-03-06T07:55:55.621Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcask-like","key-value","replicated-cache","rust","rust-learning"],"created_at":"2024-10-13T09:21:02.450Z","updated_at":"2026-03-06T11:32:33.303Z","avatar_url":"https://github.com/jdockerty.png","language":"Rust","readme":"# squirrel\n\n_As in \"[to squirrel away][squirrel-away]\"._\n\nA (replicated) persistent key-value store which uses a simple implementation of [bitcask](https://github.com/basho/bitcask/blob/develop/doc/bitcask-intro.pdf) as the underlying storage mechanism.\n\n## How it works\n\nThis follows a simple structure and exposes a very common API surface: `set(k,v)`, `get(k)`, and `remove(k)`.\n\n### Get\n\nA `get(k)` operation will read from the running store via its `keydir`. The value\n**must** exist here in order for it to be returned[^1], as the `keydir` contains a\nmapping of all known values and their respective offsets in the active or compacted\nlog files.\n\n[^1]: When a call to `open` is made, the directory is scanned for any log files,\nwhich means that in the event of a crash or restart the `keydir` is always rebuilt\nto its prior state.\n\nIf a key exists, its containing log file is opened, the offset is seeked to, and the\nentry deserialised for return.\n\nIn the event of a `None` value, this signifies either a tombstone value (from prior removal)\nor a key which has never existed. In either case, the value does not exist in the\n`keydir` so no value is returned.\n\n### Set\n\n```mermaid\nsequenceDiagram\n    participant client\n    participant squirrel\n    participant WAL as active log file\n    participant keydir\n\n    client-\u003e\u003esquirrel: set(k, v)\n    squirrel-\u003e\u003eWAL: Append 'LogEntry'\n    note right of WAL: LogEntry consists of\u003cbr/\u003ekey (k), value (v)\u003cbr/\u003e and metadata\n    WAL--\u003e\u003esquirrel: Acknowledged\n    squirrel-\u003e\u003ekeydir: Upsert KeydirEntry\n    note right of keydir: Maps key to \u003cbr/\u003elog file and offset\n    keydir--\u003e\u003esquirrel: Acknowledged\n```\n\n\u003e [!NOTE]\n\u003e For now, both keys and values are restricted to `String` types.\n\n### Remove\n\nA call to `remove(k)` is similar to `get(k)`, except a tombstone value, represented\nby `None`, is appended to the active log file and updated in the `keydir`.\n\nThe tombstone value signifies that the entry should be dropped on the next compaction\ncycle[^2]. This means that the value will no longer be present afterwards.\n\n[^2]: Compaction is not a background job, it is a simple check over `MAX_LOG_FILE_SIZE`\nafter either a `set` or `remove` operation, as these cause an append to the active log file.\n\nAttempting to remove a key which does not exist will result in an error.\n\n### Client/Server\n\nTL;DR it uses gRPC with `tonic`.\n\nSee [`client.rs`](./src/client.rs)/[`server.rs`](./src/server.rs)/[`proto`](./proto/) definitions.\n\n### Replication\n\nA simplistic strategy for replication is achieved via the [`replication`](./src/replication/mod.rs) crate.\n\n## Notes\n\nThis was initially built through my implementation of the PingCAP talent plan course for building a key-value store in Rust:\n\n- [Course](https://github.com/pingcap/talent-plan/tree/master/courses/rust#the-goal-of-this-course)\n- [Lesson plan](https://github.com/pingcap/talent-plan/blob/master/courses/rust/docs/lesson-plan.md#pna-rust-lesson-plan)\n\nAnd has since grown into my own toy project.\n\n[squirrel-away]: https://dictionary.cambridge.org/dictionary/english/squirrel-away\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdockerty%2Fsquirrel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjdockerty%2Fsquirrel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdockerty%2Fsquirrel/lists"}