{"id":21749296,"url":"https://github.com/iifawzi/tf-raft","last_synced_at":"2025-07-06T14:40:26.888Z","repository":{"id":206080403,"uuid":"714417434","full_name":"iifawzi/tf-raft","owner":"iifawzi","description":"A non-production-ready reliable, and fault-tolerant distributed key-value store based on the RAFT Consensus Protocol.","archived":false,"fork":false,"pushed_at":"2025-02-01T21:01:47.000Z","size":291,"stargazers_count":25,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T07:16:15.074Z","etag":null,"topics":["consensus","distributed","educational","raft","replicated-state-machine","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iifawzi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-04T20:52:21.000Z","updated_at":"2025-04-02T22:29:37.000Z","dependencies_parsed_at":"2024-11-24T17:24:22.114Z","dependency_job_id":"1f92ee42-a4a2-42bd-b092-8387de1e8514","html_url":"https://github.com/iifawzi/tf-raft","commit_stats":null,"previous_names":["iifawzi/tf-raft"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/iifawzi/tf-raft","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iifawzi%2Ftf-raft","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iifawzi%2Ftf-raft/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iifawzi%2Ftf-raft/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iifawzi%2Ftf-raft/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iifawzi","download_url":"https://codeload.github.com/iifawzi/tf-raft/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iifawzi%2Ftf-raft/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263920984,"owners_count":23530299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consensus","distributed","educational","raft","replicated-state-machine","typescript"],"created_at":"2024-11-26T08:17:30.041Z","updated_at":"2025-07-06T14:40:26.851Z","avatar_url":"https://github.com/iifawzi.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tf-raft: Distributed key-value store for educational fun!\n\n![Untitled-2023-12-02-0248](https://github.com/iifawzi/tf-raft/assets/46695441/118eb8d8-7796-4b02-8f4c-0fb82bca1af3)\n\n## About the Project\n\ntf-raft is a \u003cb\u003enon-production-ready\u003c/b\u003e reliable, and fault-tolerant distributed key-value store based on the RAFT Consensus Protocol. It is designed for educational purposes, providing a hands-on experience with distributed systems.\n\nThe system supports three types of data stores: `HASH`, `SET`, and `STRING`.\n\n#### tl;dr: why not production ready?\n\ntf-raft is not production-ready due to some unimplemented features, such as leadership transfer, only-once semantics, and log compaction. However, log replication, leader elections, and membership changes are fully implemented, functional, and thoroughly tested. The project is shared openly to facilitate exploration and understanding of the RAFT Protocol within the TypeScript/JavaScript community, with potential future work to complete the missing features.\n\n## Raft Consensus Protocol\n\nThe implementation of tf-raft is based on the [CONSENSUS: Bridging theory and practice](https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pdf) dissertation. Raft is a consensus protocol designed to be easy to understand. In essence, Raft ensures that a distributed system reaches a consensus on a single value even if some of the nodes in the system fail or behave maliciously.\n\nIn short, Raft achieves consensus through a leader-follower model, where one node serves as the leader and others as followers. The leader is responsible for coordinating the consensus process, and all updates go through the leader to ensure consistency.\n\ntf-raft implements the three core components of the RAFT Consensus Protocol:\n\n1. **Leader Election:** The process by which a leader is chosen among the nodes.\n2. **Log Replication:** Ensuring that the logs across nodes are consistent through replication.\n3. **Cluster Membership Changes (one member at a time):** Handling dynamic changes in the cluster, such as adding or removing nodes \n\nThe core of tf-raft is fully isolated and independent from the infrastructure, relying on ports and adapters for high flexibility.\ntf-raft currently supports `gRPC` and `In-Memory` adapters for the network layer \u0026 `In-Memory` and `JSON-Based` adapters for volatile and non-volatile states, respectively.\n\n## Store Commands\n\n![Dec-02-2023 03-19-42](https://github.com/iifawzi/tf-raft/assets/46695441/f929ca76-17b4-4186-8785-bd9536eb1cca)\n\nBelow are the commands supported by tf-raft, along with their descriptions and example usage:\n\n| Command                                        | Description                                     | Example Usage                                         |\n| ---------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------- |\n| `SET KEY VALUE`                                | Set the value of a key                          | `SET my_key 42`                                       |\n| `GET KEY`                                      | Retrieve the value of a key                     | `GET my_key`                                          |\n| `DEL KEY`                                      | Delete a key and its associated value           | `DEL my_key`                                          |\n| `HDEL hash_name key1 [key2 key3 ...]`          | Delete one or more fields from a hash           | `HDEL my_hash field1 field2`                          |\n| `HSET hash_name key1:value1 [key2:value2 ...]` | Set multiple field values in a hash             | `HSET user_info username:john email:john@example.com` |\n| `HGET hash_name key`                           | Get the value associated with a field in a hash | `HGET user_info username`                             |\n| `SDEL set_name value1 [value2 value ...]`      | Remove one or more members from a set           | `SDEL my_set member1 member2`                         |\n| `SHAS set_name value`                          | Check if a value is a member of a set           | `SHAS my_set member`                                  |\n| `SSET set_name value1 [value2 ...]`            | Add one or more members to a set                | `SSET my_set value1 value2`                           |\n\nPlease note that tf-raft is not intended for production use and serves solely as an educational tool.\n\n## Installation \u0026 Usage\n\n### Install Dependencies\n\n```bash\nnpm install\n```\n\n### Start the tf-raft Cluster\n\n```bash\nnpm run start [memory, RPC] [number_of_nodes]\n```\n\n\u003cul\u003e\n  \u003cli\u003e\u003cstrong\u003eOptional Parameters:\u003c/strong\u003e\u003c/li\u003e\n  \u003cul\u003e\n    \u003cli\u003e\u003ccode\u003ememory\u003c/code\u003e or \u003ccode\u003eRPC\u003c/code\u003e: Specify the protocol for network communication (default is \u003ccode\u003ememory\u003c/code\u003e).\u003c/li\u003e\n    \u003cli\u003e\u003ccode\u003enumber_of_nodes\u003c/code\u003e: Specify the number of nodes in the cluster (default is 3). The number must be 3 or larger.\u003c/li\u003e\n  \u003c/ul\u003e\n\u003c/ul\u003e\n\nAfter running the start command, a command-line prompt will be available to issue the commands mentioned above.\n\n## Development areas\n\n### How it Works?\n\nThe tf-raft implementation is organized into distinct folders to enhance clarity and maintainability. The core implementation resides in the Core folder, featuring the essential components of the RAFT consensus protocol. Additionally, network and persistence adapters are located in the adapters folder, while the logic governing the key-value store is encapsulated within the store folder.\n\ntf-raft supports currently comes with two clusters, namely `MemoryCluster` and `gRPCCluster`, The `MemoryCluster` has a virtual network to facilitate communication among peers and nodes.\n\nWithin the context of tf-raft, the term \"peers\" refers to the clients of a node. Communication with any node is achieved through its associated peer. The concept of peers aligns with RAFT terminology, where, for a given node, all other nodes in the system are considered peers.\n\n### How to create a node / Peer?\n\nLet's say we want to create a node using the `Memory` Protocol, this can be achieved by firstly creating the adapters, `MemoryServer` and `LocalStateManager` then inject it into the RaftNode create method.\n\n\n```ts\nexport class MemoryCluster implements RaftCluster {\n  constructor(private nodesNumber: number) {\n    this.nodesNumber = nodesNumber;\n  }\n\n  public connections: PeerConnection[] = [];\n  public async start() {\n    const network = MemoryNetwork.getNetwork();\n\n    // LEADER NODE\n    const nodeIdentifier1 = \"NODE\";\n    const server1 = new MemoryServer();\n    network.addServer(nodeIdentifier1, server1);\n    const state1 = new LocalStateManager(nodeIdentifier1);\n    await RaftNode.create(\n      nodeIdentifier1,\n      server1,\n      state1,\n      \"MEMORY\",\n      true\n    );\n\n    const node1Connection = PeerFactory(\"MEMORY\", nodeIdentifier1);\n    this.connections.push(node1Connection);\n    setTimeout(async () =\u003e {\n      for ( let i = 0; i \u003c this.nodesNumber - 1; i++) {\n        const nodeIdentifier = \"NODE\" + i;\n        const server = new MemoryServer();\n        network.addServer(nodeIdentifier, server);\n        const state = new LocalStateManager(nodeIdentifier);\n        await RaftNode.create(\n          nodeIdentifier,\n          server,\n          state,\n          \"MEMORY\"\n        );\n        const nodeConnection = PeerFactory(\"MEMORY\", nodeIdentifier);\n        this.connections.push(nodeConnection);\n        server1.AddServer({ newServer: nodeIdentifier });\n      }\n    }, 310);\n  }\n}\n\n```\n\n\u003col\u003e\n  \u003cli\u003e\u003cstrong\u003eNode Identification:\u003c/strong\u003e\n    \u003cul\u003e\n      \u003cli\u003econst identifier = \"NODE\": Assigns a unique identifier to the node, crucial for distinguishing it within the cluster.\u003c/li\u003e\n    \u003c/ul\u003e\n  \u003c/li\u003e\n  \u003cli\u003e\u003cstrong\u003eAdapters Creation:\u003c/strong\u003e\n    \u003cul\u003e\n      \u003cli\u003eCreate the necessary adapters, \u003ccode\u003eMemoryServer\u003c/code\u003e for network communication and \u003ccode\u003eLocalStateManager\u003c/code\u003e for local state management.\u003c/li\u003e\n    \u003c/ul\u003e\n  \u003c/li\u003e\n  \u003cli\u003e\u003cstrong\u003eRegister Server in Network: (Only used for memory protocol) \u003c/strong\u003e\n    \u003cul\u003e\n      \u003cli\u003e\u003ccode\u003enetwork.addServer(identifier, server1);\u003c/code\u003e: Registers the newly created server in the network with the assigned identifier.\u003c/li\u003e\n    \u003c/ul\u003e\n  \u003c/li\u003e\n  \u003cli\u003e\u003cstrong\u003eLocal State Manager Setup:\u003c/strong\u003e\n    \u003cul\u003e\n      \u003cli\u003e\u003ccode\u003econst state1 = new LocalStateManager(identifier)\u003c/code\u003e: Instantiates a \u003ccode\u003eLocalStateManager\u003c/code\u003e to manage the local state of the node, associated with the given identifier.\u003c/li\u003e\n    \u003c/ul\u003e\n  \u003c/li\u003e\n  \u003cli\u003e\u003cstrong\u003eNode Creation:\u003c/strong\u003e\n    \u003cul\u003e\n      \u003cli\u003e\u003ccode\u003eawait RaftNode.create(identifier, server1, state1, \"MEMORY\", true)\u003c/code\u003e: Invokes the \u003ccode\u003ecreate\u003c/code\u003e method of \u003ccode\u003eRaftNode\u003c/code\u003e, initializing a new node with the specified identifier, server, state manager, protocol (\"MEMORY\" in this case), and an optional parameter indicating whether the node should start as a leader (\u003ccode\u003etrue\u003c/code\u003e).\u003c/li\u003e\n    \u003c/ul\u003e\n  \u003c/li\u003e\n  \u003cli\u003e\u003cstrong\u003eLeader Startup Considerations:\u003c/strong\u003e\n    \u003cul\u003e\n      \u003cli\u003eThe last parameter (\u003ccode\u003etrue\u003c/code\u003e) is crucial when a node is intended to be a leader. It helps distinguish nodes that initiate the random election timeout and transition to the \u003ccode\u003ecandidate\u003c/code\u003e state from those waiting for a leader to communicate with them (steady state). This prevents unnecessary conversions of newly added followers to candidates while the leader is syncing with them.\u003c/li\u003e\n    \u003c/ul\u003e\n  \u003c/li\u003e\n\u003c/ol\u003e\n\nfor the Peers creation, you can simply depend on the factory:\n\nhttps://github.com/iifawzi/tf-raft/blob/96d8d738b5db0b22771fda7cd909c09735eb60c6/clusters/memory.cluster.ts#L30\n\nhttps://github.com/iifawzi/tf-raft/blob/96d8d738b5db0b22771fda7cd909c09735eb60c6/factories/peer.factory.ts#L5-L18\n\n### Testing\n\nThe foundational logic, coupled with the memory adapters, has undergone comprehensive testing using JEST, achieving near 99% test coverage.\n\nTo run the tests, execute the following command:\n```bash\nnpm run test\n```\n\nPlease note that due to the usage of json-db for persistent storage, occasional unexpected behavior may occur, leading to the temporary deletion and restoration of the entire stored data for seconds. This unpredictability might result in test failures.\n\nFor thorough testing, it is better to run the tests multiple times. If encountering errors such as \"can't read term of undefined,\" it indicates a momentary disappearance of persisted data. Running the tests again should mitigate this issue.\n\n### Useful References for implementation\n\nbesides the dissertation, it was super useful going through the discussions in the raft-dev group, many of the questions that mind come to your mind while implementing this, has been already discussed in the group. [Raft development Group](https://groups.google.com/g/raft-de)\n\nbonus if you're arabic speaker: the distributed systems' list by Ahmed Fraghal [Distributed Systems in arabic](https://www.youtube.com/watch?v=s_p3I5CMGJw)\nand actually it was the first time I hear about raft, in this series.\n\n## License\n\n[MIT](LICENSE)\n\n## Copyright\n\nCopyright (c) 2023 Fawzi Abdulfattah\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiifawzi%2Ftf-raft","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiifawzi%2Ftf-raft","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiifawzi%2Ftf-raft/lists"}