{"id":50493560,"url":"https://github.com/henryqingmo/kubekv","last_synced_at":"2026-06-02T05:01:17.573Z","repository":{"id":349460590,"uuid":"1200876236","full_name":"henryqingmo/KubeKV","owner":"henryqingmo","description":"Go implementation of the Raft consensus algorithm with simulated RPC, leader election, and fault-tolerant log replication.","archived":false,"fork":false,"pushed_at":"2026-05-07T06:21:23.000Z","size":8971,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-07T08:24:25.211Z","etag":null,"topics":["concensus","etcd","fault-tolerance","go","k8s","kubernetes","raft","raft-consensus-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/henryqingmo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-03T23:46:06.000Z","updated_at":"2026-05-07T06:26:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/henryqingmo/KubeKV","commit_stats":null,"previous_names":["henryqingmo/fault-tolerant-kv-raft","henryqingmo/raft","henryqingmo/kubekv"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/henryqingmo/KubeKV","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/henryqingmo%2FKubeKV","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/henryqingmo%2FKubeKV/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/henryqingmo%2FKubeKV/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/henryqingmo%2FKubeKV/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/henryqingmo","download_url":"https://codeload.github.com/henryqingmo/KubeKV/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/henryqingmo%2FKubeKV/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33806987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-02T02:00:07.132Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concensus","etcd","fault-tolerance","go","k8s","kubernetes","raft","raft-consensus-algorithm"],"created_at":"2026-06-02T05:01:15.107Z","updated_at":"2026-06-02T05:01:17.553Z","avatar_url":"https://github.com/henryqingmo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KubeKV\n\nA fault-tolerant, linearizable key-value store built on a from-scratch Raft consensus implementation in Go. Long-term goal: a minimal etcd — the kind of thing Kubernetes uses for cluster state.\n\n## What it does\n\n- **Raft consensus** — leader election, log replication, and term-based safety\n- **Log compaction** — snapshotting with `InstallSnapshot` so logs don't grow unbounded\n- **Linearizable KV** — versioned `Put`/`Get` operations; stale reads are rejected\n- **Partition tolerance** — correctly handles split-brain, heals on reconnect, checks term + leadership before committing\n\n## Architecture\n\n```mermaid\nflowchart TD\n    subgraph client[\"Client Layer\"]\n        C[\"Clerk\\nclient.go\"]\n    end\n\n    subgraph kv[\"KV Layer  — kvraft1/\"]\n        KVS[\"KVServer\\nserver.go\"]\n        RSM[\"RSM  reader loop\\nrsm/rsm.go\"]\n        Store[\"In-memory KV store\\nversioned map[key]→{val, version}\"]\n    end\n\n    subgraph raft[\"Raft Layer  — raft1/\"]\n        RL[\"Raft leader\\nraft.go\"]\n        RF1[\"Raft follower\"]\n        RF2[\"Raft follower\"]\n        P[\"Persister\\nraft state + snapshot\"]\n    end\n\n    C --\u003e|\"Get / Put RPC\"| KVS\n    KVS --\u003e|\"Submit(op)\\nblocks until committed\"| RSM\n    RSM --\u003e|\"Start(cmd)\\nstartCh → immediate AppendEntries\"| RL\n    RL --\u003e|\"AppendEntries RPC\"| RF1\n    RL --\u003e|\"AppendEntries RPC\"| RF2\n    RL -.-\u003e|\"InstallSnapshot RPC\\nlagging follower\"| RF1\n    RL --\u003e|\"applyCh ApplyMsg\\ncommitted index + command\"| RSM\n    RSM --\u003e|\"DoOp()\"| Store\n    RSM -.-\u003e|\"Snapshot()\\nwhen log ≥ 90% maxraftstate\"| RL\n    RL --\u003e|\"persist()\"| P\n```\n\nEach write goes through the Raft log. Reads block until the server confirms it's still the leader (no stale reads). The RSM layer owns the `applyCh` drain loop — it executes committed ops, wakes blocked `Submit()` callers via a per-index channel, and triggers snapshotting when the log approaches `maxraftstate`. On log growth, the KV layer serializes its state and Raft discards the log prefix, shipping the snapshot to lagging followers via `InstallSnapshot`.\n\n## Key design decisions\n\n**Versioned puts over CAS** — every key carries a monotonic version. `Put(key, val, version)` is rejected if the version doesn't match, giving clients optimistic concurrency without distributed locks. This maps cleanly to how etcd's watch + revision model works.\n\n**Snapshot-driven log truncation** — when the KV layer signals a snapshot, Raft replaces the log prefix with `(lastIncludedIndex, lastIncludedTerm)` and ships the snapshot to lagging followers via `InstallSnapshot`. This is the same mechanism etcd uses to onboard new members without replaying the full history.\n\n**Fast path on `Start()`** — `Start()` signals a dedicated channel that immediately triggers `AppendEntries` to all peers, rather than waiting for the next heartbeat tick. Cuts commit latency under load.\n\n## Running\n\n```bash\ncd src/raft1\ngo test -run TestBasicAgree\ngo test -run TestSnapshotInstall\ngo test -run TestConcurrentClients\n```\n\n## Where this is headed\n\nThe immediate target is a standalone gRPC server that speaks a subset of the etcd v3 API — enough to swap in as a Kubernetes datastore for a small cluster. That requires:\n\n- [ ] gRPC transport replacing the in-process RPC shim\n- [ ] Persistent WAL on disk (currently in-memory via `Persister`)\n- [ ] Watch streams (key-range notifications on commit)\n- [ ] Membership changes (Raft joint consensus, §6 of the paper)\n- [ ] Read index / lease-based reads for lower-latency queries\n\n## References\n\n- [In Search of an Understandable Consensus Algorithm](https://raft.github.io/raft.pdf) — Ongaro \u0026 Ousterhout\n- [etcd internals](https://etcd.io/docs/v3.5/learning/design-learner/) — membership and storage design\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhenryqingmo%2Fkubekv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhenryqingmo%2Fkubekv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhenryqingmo%2Fkubekv/lists"}