{"id":39642281,"url":"https://github.com/feichai0017/NoKV","last_synced_at":"2026-01-19T23:01:37.698Z","repository":{"id":252733915,"uuid":"841059560","full_name":"feichai0017/NoKV","owner":"feichai0017","description":"High-performance distributed KV storage based on LSM Tree","archived":false,"fork":false,"pushed_at":"2026-01-18T07:43:50.000Z","size":363503,"stargazers_count":153,"open_issues_count":5,"forks_count":13,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-01-18T15:50:09.419Z","etag":null,"topics":["database","distributed-systems","golang","kv-store","lsm-tree","mvcc","raft","redis"],"latest_commit_sha":null,"homepage":"https://feichai0017.github.io/NoKV/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feichai0017.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-08-11T14:19:45.000Z","updated_at":"2026-01-18T13:03:01.000Z","dependencies_parsed_at":"2024-09-14T20:48:17.436Z","dependency_job_id":"4d709aca-7fec-4402-a309-73b5dd11fbd3","html_url":"https://github.com/feichai0017/NoKV","commit_stats":null,"previous_names":["feichai0017/cloud_distributed_file-system","feichai0017/norag","feichai0017/nokv"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/feichai0017/NoKV","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feichai0017%2FNoKV","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feichai0017%2FNoKV/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feichai0017%2FNoKV/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feichai0017%2FNoKV/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feichai0017","download_url":"https://codeload.github.com/feichai0017/NoKV/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feichai0017%2FNoKV/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28588968,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T20:45:59.482Z","status":"ssl_error","status_checked_at":"2026-01-19T20:45:41.500Z","response_time":67,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","distributed-systems","golang","kv-store","lsm-tree","mvcc","raft","redis"],"created_at":"2026-01-18T09:00:59.976Z","updated_at":"2026-01-19T23:01:37.692Z","avatar_url":"https://github.com/feichai0017.png","language":"Go","readme":"# 🚀 NoKV – High-Performance Distributed KV Engine\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./img/logo.svg\" width=\"220\" alt=\"NoKV Logo\" /\u003e\n\n  \u003cp\u003e\n    \u003c!-- Build / Quality --\u003e\n    \u003ca href=\"https://github.com/feichai0017/NoKV/actions\"\u003e\n      \u003cimg alt=\"CI\" src=\"https://img.shields.io/github/actions/workflow/status/feichai0017/NoKV/go.yml?branch=main\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://codecov.io/gh/feichai0017/NoKV\"\u003e\n      \u003cimg alt=\"Coverage\" src=\"https://img.shields.io/codecov/c/gh/feichai0017/NoKV\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://goreportcard.com/report/github.com/feichai0017/NoKV\"\u003e\n      \u003cimg alt=\"Go Report Card\" src=\"https://img.shields.io/badge/go%20report-A+-brightgreen\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://pkg.go.dev/github.com/feichai0017/NoKV\"\u003e\n      \u003cimg alt=\"Go Reference\" src=\"https://img.shields.io/badge/go.dev-reference-007d9c?logo=go\u0026logoColor=white\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/avelino/awesome-go#databases-implemented-in-go\"\u003e\n      \u003cimg alt=\"Mentioned in Awesome\" src=\"https://awesome.re/mentioned-badge.svg\" /\u003e\n    \u003c/a\u003e\n  \u003c/p\u003e\n\n  \u003cp\u003e\n    \u003c!-- Meta --\u003e\n    \u003cimg alt=\"Go Version\" src=\"https://img.shields.io/badge/go-1.24%2B-00ADD8?logo=go\u0026logoColor=white\" /\u003e\n    \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/license-Apache--2.0-yellow\" /\u003e\n    \u003ca href=\"https://deepwiki.com/feichai0017/NoKV\"\u003e\n      \u003cimg alt=\"DeepWiki\" src=\"https://img.shields.io/badge/DeepWiki-Ask-6f42c1\" /\u003e\n    \u003c/a\u003e\n  \u003c/p\u003e\n\n  \u003cp\u003e\u003cstrong\u003eLSM Tree • ValueLog • MVCC • Multi-Raft Regions • Redis-Compatible\u003c/strong\u003e\u003c/p\u003e\n\u003c/div\u003e\n\n\nNoKV is a Go-native storage engine that mixes RocksDB-style manifest discipline with Badger-inspired value separation. You can embed it locally, drive it via multi-Raft regions, or front it with a Redis protocol gateway—all from a single topology file.\n\n---\n\n## ✨ Feature Highlights\n\n- 🚀 **Dual runtime modes** – call `NoKV.Open` inside your process or launch `nokv serve` for a distributed deployment, no code changes required.\n- 🔁 **Hybrid LSM + ValueLog** – WAL → MemTable → SST pipeline for latency, with a ValueLog to keep large payloads off the hot path.\n- ⚡ **MVCC-native transactions** – snapshot isolation, conflict detection, TTL, and iterators built into the core (no external locks).\n- 🧠 **Multi-Raft regions** – `raftstore` manages per-region raft groups, WAL/manifest pointers, and tick-driven leader elections.\n- 🛰️ **Redis gateway** – `cmd/nokv-redis` exposes RESP commands (SET/GET/MGET/NX/XX/TTL/INCR...) on top of raft-backed storage.\n- 🔍 **Observability first** – `nokv stats`, expvar endpoints, hot key tracking, RECOVERY/TRANSPORT metrics, and ready-to-use recovery scripts.\n- 🧰 **Single-source config** – `raft_config.json` feeds local scripts, Docker Compose, Redis gateway, and CI so there’s zero drift.\n\n---\n\n## 🚦 Quick Start\n\nStart an end-to-end playground with either the local script or Docker Compose. Both spin up a three-node Raft cluster (plus the optional TSO) and expose the Redis-compatible gateway.\n\n```bash\n# Option A: local processes\n./scripts/run_local_cluster.sh --config ./raft_config.example.json\n# In another shell: launch the Redis gateway on top of the running cluster\ngo run ./cmd/nokv-redis --addr 127.0.0.1:6380 --raft-config raft_config.example.json\n\n# Option B: Docker Compose (cluster + gateway + TSO)\ndocker compose up --build\n# Tear down\ndocker compose down -v\n```\n\nOnce the cluster is running you can point any Redis client at `127.0.0.1:6380` (or the address exposed by Compose).\n\nFor quick CLI checks:\n\n```bash\n# Inspect stats from an existing workdir\ngo run ./cmd/nokv stats --workdir ./artifacts/cluster/store-1\n```\n\nMinimal embedded snippet:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\tNoKV \"github.com/feichai0017/NoKV\"\n)\n\nfunc main() {\n\topt := NoKV.NewDefaultOptions()\n\topt.WorkDir = \"./workdir-demo\"\n\n\tdb := NoKV.Open(opt)\n\tdefer db.Close()\n\n\tkey := []byte(\"hello\")\n\tif err := db.Set(key, []byte(\"world\")); err != nil {\n\t\tlog.Fatalf(\"set failed: %v\", err)\n\t}\n\n\tentry, err := db.Get(key)\n\tif err != nil {\n\t\tlog.Fatalf(\"get failed: %v\", err)\n\t}\n\tfmt.Printf(\"value=%s\\n\", entry.Value)\n\tentry.DecrRef()\n}\n```\n\n\u003e ℹ️ `run_local_cluster.sh` rebuilds `nokv`, `nokv-config`, `nokv-tso`, seeds manifests via `nokv-config manifest`, and parks logs under `artifacts/cluster/store-\u003cid\u003e/server.log`. Use `Ctrl+C` to exit cleanly; if the process crashes, wipe the workdir (`rm -rf ./artifacts/cluster`) before restarting to avoid WAL replay errors.\n\n---\n\n## 🧭 Topology \u0026 Configuration\n\nEverything hangs off a single file: [`raft_config.example.json`](./raft_config.example.json).\n\n```jsonc\n\"stores\": [\n  { \"store_id\": 1, \"listen_addr\": \"127.0.0.1:20170\", ... },\n  { \"store_id\": 2, \"listen_addr\": \"127.0.0.1:20171\", ... },\n  { \"store_id\": 3, \"listen_addr\": \"127.0.0.1:20172\", ... }\n],\n\"regions\": [\n  { \"id\": 1, \"range\": [-inf,\"m\"), peers: 101/201/301, leader: store 1 },\n  { \"id\": 2, \"range\": [\"m\",+inf), peers: 102/202/302, leader: store 2 }\n]\n```\n\n- **Local scripts** (`run_local_cluster.sh`, `serve_from_config.sh`, `bootstrap_from_config.sh`) ingest the same JSON, so local runs match production layouts.\n- **Docker Compose** mounts the file into each container; manifests, transports, and Redis gateway all stay in sync.\n- Need more stores or regions? Update the JSON and re-run the script/Compose—no code changes required.\n- Programmatic access: import `github.com/feichai0017/NoKV/config` and call `config.LoadFile` / `Validate` for a single source of truth across tools.\n\n### 🧬 Tech Stack Snapshot\n\n| Layer | Tech/Package | Why it matters |\n| --- | --- | --- |\n| Storage Core | `lsm/`, `wal/`, `vlog/` | Hybrid log-structured design with manifest-backed durability and value separation. |\n| Concurrency | `mvcc/`, `txn.go`, `oracle` | Timestamp oracle + lock manager for MVCC transactions and TTL-aware reads. |\n| Replication | `raftstore/*` | Multi-Raft orchestration (regions, peers, router, schedulers, gRPC transport). |\n| Tooling | `cmd/nokv`, `cmd/nokv-config`, `cmd/nokv-redis` | CLI, config helper, Redis-compatible gateway share the same topology file. |\n| Observability | `stats`, `hotring`, expvar | Built-in metrics, hot-key analytics, and crash recovery traces. |\n\n---\n\n## 🧱 Architecture Overview\n\n```mermaid\ngraph TD\n    Client[Client API / Txn] --\u003e|Set/Get| DBCore\n    DBCore --\u003e|Append| WAL\n    DBCore --\u003e|Insert| MemTable\n    DBCore --\u003e|ValuePtr| ValueLog\n    MemTable --\u003e|Flush Task| FlushMgr\n    FlushMgr --\u003e|Build SST| SSTBuilder\n    SSTBuilder --\u003e|LogEdit| Manifest\n    Manifest --\u003e|Version| LSMLevels\n    LSMLevels --\u003e|Compaction| Compactor\n    FlushMgr --\u003e|Discard Stats| ValueLog\n    ValueLog --\u003e|GC updates| Manifest\n    DBCore --\u003e|Stats/HotKeys| Observability\n```\n\nKey ideas:\n- **Durability path** – WAL first, memtable second. ValueLog writes occur before WAL append so crash replay can fully rebuild state.\n- **Metadata** – manifest stores SST topology, WAL checkpoints, and vlog head/deletion metadata.\n- **Background workers** – flush manager handles `Prepare → Build → Install → Release`, compaction reduces level overlap, and value log GC rewrites segments based on discard stats.\n- **Transactions** – MVCC timestamps ensure consistent reads; commit reuses the same write pipeline as standalone writes.\n\nDive deeper in [docs/architecture.md](docs/architecture.md).\n\n---\n\n## 🧩 Module Breakdown\n\n| Module | Responsibilities | Source | Docs |\n| --- | --- | --- | --- |\n| WAL | Append-only segments with CRC, rotation, replay (`wal.Manager`). | [`wal/`](./wal) | [WAL internals](docs/wal.md) |\n| LSM | MemTable, flush pipeline, leveled compactions, iterator merging. | [`lsm/`](./lsm) | [Memtable](docs/memtable.md)\u003cbr\u003e[Flush pipeline](docs/flush.md)\u003cbr\u003e[Cache](docs/cache.md) |\n| Manifest | VersionEdit log + CURRENT handling, WAL/vlog checkpoints, Region metadata. | [`manifest/`](./manifest) | [Manifest semantics](docs/manifest.md) |\n| ValueLog | Large value storage, GC, discard stats integration. | [`vlog.go`](./vlog.go), [`vlog/`](./vlog) | [Value log design](docs/vlog.md) |\n| Transactions | MVCC `oracle`, managed/unmanaged transactions, iterator snapshots. | [`txn.go`](./txn.go) | [Transactions \u0026 MVCC](docs/txn.md) |\n| RaftStore | Multi-Raft Region management, hooks, metrics, transport. | [`raftstore/`](./raftstore) | [RaftStore overview](docs/raftstore.md) |\n| HotRing | Hot key tracking, throttling helpers. | [`hotring/`](./hotring) | [HotRing overview](docs/hotring.md) |\n| Observability | Periodic stats, hot key tracking, CLI integration. | [`stats.go`](./stats.go), [`cmd/nokv`](./cmd/nokv) | [Stats \u0026 observability](docs/stats.md)\u003cbr\u003e[CLI reference](docs/cli.md) |\n| Filesystem | mmap-backed file helpers shared by SST/vlog. | [`file/`](./file) | [File abstractions](docs/file.md) |\n\nEach module has a dedicated document under `docs/` describing APIs, diagrams, and recovery notes.\n\n---\n\n\n## 📡 Observability \u0026 CLI\n\n- `Stats.StartStats` publishes metrics via `expvar` (flush backlog, WAL segments, value log GC stats, txn counters).\n- `cmd/nokv` gives you:\n  - `nokv stats --workdir \u003cdir\u003e [--json] [--no-region-metrics]`\n  - `nokv manifest --workdir \u003cdir\u003e`\n  - `nokv regions --workdir \u003cdir\u003e [--json]`\n  - `nokv vlog --workdir \u003cdir\u003e`\n- `hotring` continuously surfaces hot keys in stats + CLI so you can pre-warm caches or debug skewed workloads.\n\nMore in [docs/cli.md](docs/cli.md) and [docs/testing.md](docs/testing.md#4-observability-in-tests).\n\n---\n\n## 🔌 Redis Gateway\n\n- `cmd/nokv-redis` exposes a RESP-compatible endpoint. In embedded mode (`--workdir`) every command runs inside local MVCC transactions; in distributed mode (`--raft-config`) calls are routed through `raftstore/client` and committed with TwoPhaseCommit so NX/XX, TTL, arithmetic and multi-key writes match the single-node semantics.\n- TTL metadata is stored under `!redis:ttl!\u003ckey\u003e` and is automatically cleaned up when reads detect expiration.\n- `--metrics-addr` publishes `NoKV.Redis` statistics via expvar and `--tso-url` can point to an external TSO service (otherwise a local oracle is used).\n- A ready-to-use cluster configuration is available at `raft_config.example.json`, matching both `scripts/run_local_cluster.sh` and the Docker Compose setup.\n\n\u003e For the complete command matrix, configuration and deployment guides, see [docs/nokv-redis.md](docs/nokv-redis.md).\n\n---\n\n## 📄 License\n\nApache-2.0. See [LICENSE](LICENSE).\n","funding_links":[],"categories":["数据库","Database","Data Integration Frameworks"],"sub_categories":["Go中实现的数据库","Databases Implemented in Go"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeichai0017%2FNoKV","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeichai0017%2FNoKV","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeichai0017%2FNoKV/lists"}