{"id":19341474,"url":"https://github.com/lizhaoliu/konsen","last_synced_at":"2025-06-28T13:32:28.490Z","repository":{"id":89222453,"uuid":"257473228","full_name":"lizhaoliu/konsen","owner":"lizhaoliu","description":"Strongly consistent distributed key-value store based on Raft algorithm","archived":false,"fork":false,"pushed_at":"2022-09-16T07:13:06.000Z","size":273,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-23T03:38:53.139Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lizhaoliu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-21T03:48:47.000Z","updated_at":"2023-05-25T20:35:31.000Z","dependencies_parsed_at":"2023-06-14T12:00:35.786Z","dependency_job_id":null,"html_url":"https://github.com/lizhaoliu/konsen","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lizhaoliu/konsen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lizhaoliu%2Fkonsen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lizhaoliu%2Fkonsen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lizhaoliu%2Fkonsen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lizhaoliu%2Fkonsen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lizhaoliu","download_url":"https://codeload.github.com/lizhaoliu/konsen/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lizhaoliu%2Fkonsen/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262438052,"owners_count":23311076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T03:30:58.730Z","updated_at":"2025-06-28T13:32:28.474Z","avatar_url":"https://github.com/lizhaoliu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# konsen\n### Overview\nKonsen is an implementation of [Raft](https://raft.github.io/raft.pdf) consensus protocol based upon replicated state\nmachine. Konson offers strong consistency and partition tolerance. For a cluster of N nodes, it can tolerate up to N/2\nfailures.\n### Features\n- [x] Consensus with replicated state machine.\n- [x] Leader election.\n- [x] Distributed key-value store.\n- [ ] Distributed transactions.\n### Build Local Cluster\n#### Cluster configuration\nEdit the cluster config in `conf/cluster.yml`, for example: \n```yaml\nservers:\n  node1: 192.168.86.25:10001\n  node2: 192.168.86.25:10002\n  node3: 192.168.86.25:10003\n  node4: 192.168.86.25:10004\n  node5: 192.168.86.25:10005\nhttpServers:\n  node1: 192.168.86.25:20001\n  node2: 192.168.86.25:20002\n  node3: 192.168.86.25:20003\n  node4: 192.168.86.25:20004\n  node5: 192.168.86.25:20005\n```\nThis creates a cluster of 5 nodes, where \"httpServers\" define the HTTP servers that accept client request such as writes\nand reads.\n#### (Optional) Regenerate protobuf code\n```shell script\nprotoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative proto/*.proto\n```\n#### Build cluster\n```shell script\npython build_cluster.py\n```\nThis will build a release for each node in the cluster config. \n```\noutput/\n  node1/\n    bootstrap.sh\n    cluster.yml\n    konsen\n  node2/\n    bootstrap.sh\n    cluster.yml\n    konsen\n  node3/\n    bootstrap.sh\n    cluster.yml\n    konsen\n  ...\n```\n### Benchmark\n#### Setup\n* go version go1.14.2 linux/amd64.\n* 5 nodes on local machine, AMD 3900x 12 cores + 32GB Ram + 256GB SSD.\n* Ubuntu 19.10.\n* Benchmark tool: [Vegeta](https://github.com/tsenart/vegeta).\n#### Write \nAll write requests will be redirected to the leader node: leader writes to local log and then replicates it to all (at\nleast majority of) nodes such that it applies to local state machine, and responds to the request.\n```shell script\nRequests      [total, rate, throughput]  2500, 500.21, 495.11\nDuration      [total, attack, wait]      5.049347643s, 4.997898254s, 51.449389ms\nLatencies     [mean, 50, 95, 99, max]    82.190365ms, 79.205946ms, 113.411489ms, 126.342645ms, 154.046802ms\nBytes In      [total, mean]              0, 0.00\nBytes Out     [total, mean]              37500, 15.00\nSuccess       [ratio]                    100.00%\nStatus Codes  [code:count]               200:2500\n```\n#### Read\nAll read requests will be redirected to the leader node.\n```shell script\nRequests      [total, rate, throughput]  50000, 9994.47, 9990.42\nDuration      [total, attack, wait]      5.004796803s, 5.002768278s, 2.028525ms\nLatencies     [mean, 50, 95, 99, max]    484.303µs, 218.459µs, 662.541µs, 9.176152ms, 42.321458ms\nBytes In      [total, mean]              0, 0.00\nBytes Out     [total, mean]              0, 0.00\nSuccess       [ratio]                    100.00%\nStatus Codes  [code:count]               200:50000  \n```\n### TODO\n- [ ] Unit tests.\n- [ ] Supervisors.\n- [ ] Log compaction.\n- [ ] Cluster resize.\n- [ ] HTTP server.\n- [ ] CI/CD pipeline.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flizhaoliu%2Fkonsen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flizhaoliu%2Fkonsen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flizhaoliu%2Fkonsen/lists"}