{"id":19907273,"url":"https://github.com/captainirs/sharded-kvs","last_synced_at":"2026-04-15T13:31:37.690Z","repository":{"id":250672325,"uuid":"778946888","full_name":"CaptainIRS/sharded-kvs","owner":"CaptainIRS","description":"A Model Sharded Key-Value Store Implementation using Go and Kubernetes","archived":false,"fork":false,"pushed_at":"2025-08-04T10:03:54.000Z","size":365,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-27T13:24:16.534Z","etag":null,"topics":["go","grpc","helm","kubernetes","sharding"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CaptainIRS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-28T18:13:12.000Z","updated_at":"2025-08-04T10:03:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"71bf56a8-a9b3-4be0-8fa9-211faa1b73b5","html_url":"https://github.com/CaptainIRS/sharded-kvs","commit_stats":null,"previous_names":["captainirs/sharded-kvs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CaptainIRS/sharded-kvs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainIRS%2Fsharded-kvs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainIRS%2Fsharded-kvs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainIRS%2Fsharded-kvs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainIRS%2Fsharded-kvs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CaptainIRS","download_url":"https://codeload.github.com/CaptainIRS/sharded-kvs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CaptainIRS%2Fsharded-kvs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31842849,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T13:28:40.153Z","status":"ssl_error","status_checked_at":"2026-04-15T13:28:29.396Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","grpc","helm","kubernetes","sharding"],"created_at":"2024-11-12T20:39:53.216Z","updated_at":"2026-04-15T13:31:37.672Z","avatar_url":"https://github.com/CaptainIRS.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eSharded Key-Value Store\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n    A Model Sharded Key-Value Store Implementation using Go and Kubernetes\u003cbr\u003e\u003cbr\u003e\n    \u003cimg width=\"708\" alt=\"Screenshot 2024-07-29 at 12 07 53 PM\" src=\"https://github.com/user-attachments/assets/5c290093-17d0-4560-97f8-03e629d65f5a\"\u003e\n\u003c/p\u003e\n\n## Features\n* **Sharding**: The key-value store is sharded across multiple nodes using [Consistent Hashing with Bounded Loads](https://research.google/blog/consistent-hashing-with-bounded-loads/).\n* **Replication**: The key-value store supports replication across multiple replicas for each shard. The number of replicas can be configured (default: 3). It uses the [Raft Consensus Algorithm](https://raft.github.io/).\n* **Load Balancing**: The key-value store uses a round-robin load balancer to distribute the requests across the replicas using the [NGINX Ingress Controller](https://kubernetes.github.io/ingress-nginx/). Leader forwarding is used to forward the requests to the leader replica.\n* **Scalability**: The key-value store can be deployed on Kubernetes with a configurable number of shards and replicas and can be changed dynamically using the `KVStore` custom resource managed by the `kvctl` controller.\n* **Upgradability**: The key-value store can be upgraded without any downtime using the [RollingUpdate](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/) strategy in Kubernetes. (Run `make sync` after updating the source code).\n* **Fault Simulation**: The system can simulate network partitions and shard failures using [Chaos Mesh](https://chaos-mesh.org/). Visit http://chaos-dashboard.svc.localho.st/chaos-mesh/ to access the Chaos Mesh dashboard and experiment with different fault scenarios.\n* **Tracing**: The system provides logs of traffic among all the shards logged to the standard output by capturing packets using eBPF. The logs can be viewed using the `stern` tool. (Run `stern -l group=shard-0 -n=kvs -t=short` to view the logs of the shard-0 replica group).\n* **Portability**: The system can be run in any platform (Windows, macOS, Linux) with no changes to the codebase since it is built using Go and Kubernetes.\n\n\n## Prerequisites (development)\n1. Install [Go](https://go.dev/doc/install). Make sure to set the PATH environment variable correctly.\n2. Install the [Protobuf Compiler](https://grpc.io/docs/protoc-installation).\n3. Install the Go Protobuf Compiler plugins:\n    ```bash\n    go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28\n    go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2\n    ```\n\n## Prerequisites (deployment)\n1. Install [Docker](https://docs.docker.com/get-docker/).\n2. Install [Docker Buildx](https://github.com/docker/buildx?tab=readme-ov-file#installing).\n3. Install [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation).\n4. Install [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl).\n5. Install `stern`:\n    ```bash\n    go install github.com/stern/stern@latest\n    ```\n\n## Repository Structure\n\n- `cmd/`: Contains the code for CLI binaries that users can use to interact with the key-value store.\n- `internal/`: Contains the core implementation of the key-value store.\n  - `internal/protos/`: Contains the generated Go code for the Protobuf messages and services.\n    - `fsm.proto`: Contains the proto definition for the Raft state machine.\n    - `kv.proto`: Contains the proto definition for the key-value store service through which clients can interact with the key-value store.\n    - `shard.proto`: Contains the proto definition for the shard service through which replica groups can interact with each other (for forwarding request to the shard containing the required shard).\n    - `replica.proto`: Contains the proto definition for the replica service through which replicas can interact with each other (for leader forwarding).\n  - `internals/raft`: Contains the implementation of the Raft state machine.\n  - `internals/capture`: Contains the code for tracing network requests by capturing with eBPF.\n- `protos/`: Contains the Protobuf definitions for the messages and services used in the key-value store.\n\n## Usage\n\n### Makefile Targets\n\nRun `make \"target\"` where `\"target\"` is one of the following:\n- `deploy`: Deploy the system (Key-Value Store Server) in Kubernetes.\n- `client`: Run the client.\n- `clean`: Remove the system from Kubernetes.\n- `sync`: Sync any changes in the system to Kubernetes.\n- `dashboard`: Open the Kubernetes dashboard.\n- `proto`: Generate the Go code from the Protobuf definitions.\n- `fmt`: Format the Go code and helm templates before committing.\n\nInside the `kvctl` folder, run the following to setup the controller.\n- `make generate manifests`\n- `make docker-build IMG=controller:latest`\n- `kind load docker-image controller:latest -n sharded-kvs-cluster`\n- `make deploy IMG=controller:latest`\n- Edit `kvctl/config/samples/kvctl_v1_kvstore.yaml` for the `KVStore` config\n- `kubectl apply -f kvctl/config/samples/kvctl_v1_kvstore.yaml`\n\n### Viewing Logs/Traces\n\n* To view all logs of the cluster:\n    ```console\n    $ stern -n kvs -t=short\n    ```\n* To view logs of a specific replica group (e.g., shard-0):\n    ```console\n    $ stern -l group=shard-0 -n kvs -t=short\n    ```\n* To view logs of a specific replica (e.g., shard-0-replica-0):\n    ```console\n    $ stern -l shard-0-replica-0 -n kvs -t=short\n    ```\n* To remove logs of heartbeat messages add `-e=\"Sending heartbeat\"` to the command:\n    ```console\n    $ stern -n kvs -t=short -e=\"Sending heartbeat\"\n    ```\n\n## System Architecture\n\n![sharded-kvs drawio](https://github.com/user-attachments/assets/f60bd480-27a9-426f-9de5-a199a26da5ed)\n\n## Kubernetes Architecture\n\n```mermaid\ngraph LR;\n client([client]).-\u003eingress[Ingress];\n ingress--\u003e|routing rule|service0[Service\u003cbr\u003eLB];\n ingress--\u003e|routing rule|service1[Service\u003cbr\u003eLB];\n ingress--\u003e|routing rule|service2[Service\u003cbr\u003eLB];\n subgraph shard0[shard-0 StatefulSet]\n service0--\u003epod00[Pod\u003cbr\u003ereplica-0];\n service0--\u003epod01[Pod\u003cbr\u003ereplica-1];\n service0--\u003epod02[Pod\u003cbr\u003ereplica-2];\n end\n subgraph shard1[shard-1 StatefulSet]\n service1--\u003epod10[Pod\u003cbr\u003ereplica-0];\n service1--\u003epod11[Pod\u003cbr\u003ereplica-1];\n service1--\u003epod12[Pod\u003cbr\u003ereplica-2];\n end\n subgraph shard2[shard-2 StatefulSet]\n service2--\u003epod20[Pod\u003cbr\u003ereplica-0];\n service2--\u003epod21[Pod\u003cbr\u003ereplica-1];\n service2--\u003epod22[Pod\u003cbr\u003ereplica-2];\n end\n classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;\n classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;\n classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;\n class ingress,service0,ingress1,service1,ingress2,service2,pod01,pod02,pod00,pod11,pod12,pod10,pod21,pod22,pod20 k8s;\n class client plain;\n class shard0,shard1,shard2 cluster;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptainirs%2Fsharded-kvs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaptainirs%2Fsharded-kvs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaptainirs%2Fsharded-kvs/lists"}