{"id":20018543,"url":"https://github.com/rupeshtr78/raftleader","last_synced_at":"2025-10-07T09:59:46.276Z","repository":{"id":45458336,"uuid":"207624696","full_name":"rupeshtr78/raftleader","owner":"rupeshtr78","description":"Raft leader election process in Hyperledger Fabric.","archived":false,"fork":false,"pushed_at":"2019-09-10T21:11:43.000Z","size":228,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-08T14:07:35.572Z","etag":null,"topics":["blockchain-technology","consensus-algorithm","etcd","etcd-raft","fabric","hyperledger","hyperledger-fabric","leader-election-algorithm","orderer","raft","raft-consensus-algorithm"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rupeshtr78.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-10T17:42:48.000Z","updated_at":"2024-09-09T01:00:54.000Z","dependencies_parsed_at":"2022-07-14T13:20:52.713Z","dependency_job_id":null,"html_url":"https://github.com/rupeshtr78/raftleader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Fraftleader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Fraftleader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Fraftleader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rupeshtr78%2Fraftleader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rupeshtr78","download_url":"https://codeload.github.com/rupeshtr78/raftleader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252414208,"owners_count":21744064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain-technology","consensus-algorithm","etcd","etcd-raft","fabric","hyperledger","hyperledger-fabric","leader-election-algorithm","orderer","raft","raft-consensus-algorithm"],"created_at":"2024-11-13T08:23:07.340Z","updated_at":"2025-10-07T09:59:41.257Z","avatar_url":"https://github.com/rupeshtr78.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Raft leader election process in Hyperledger Fabric.\n\nIn this article we will explore how the leader election process in raft works by deploying Fabric samples first network using with raft ordering service. \n\n```bash\ncd first-network\n# Raft\n./byfn.sh generate -o etcdraft\n./byfn.sh up -o etcdraft\n```\n\nMake sure the the genesis block , channel config is created with orderer type = etcdraft.\n\n![](images/genesisblock.png)\n\nFor this exercise we will create a simple grafana dashboard to see the cluster size and number of active nodes and the current raft leader by channel.\n\n![](images/grafana01.png)\n\n```\n Elected leader 3 at term 2 channel=mychannel\n```\n\n\n\n- At any given time raft  nodes  is in one of these states:\n\n  - - **Pre-Candidate:** The pre-candidate first asks other nodes whether pre-candidate can get enough votes to win an election. If not, it would remain as follower.\n\n    - **Candidate** : candidate state, is used during the election process to elect a new leader. \n\n    - **Leader** : handles all client requests (if a client contacts a follower, the follower redirects it to the leader).\n\n    - **Follower** : Passive: they issue no requests on their own but simply respond to requests from leaders and candidates. If a follower receives no communication from leader ,it becomes a pre-candidate and initiates an election.\n\n      Lets see if we can find out the the process of leader election from the fabric orderer logs.\n\n## Raft Leader Election\n\nIn a raft cluster election at a  high level these are the sequence of steps.\n\n**Election**\n\n1. To begin an election, a follower transitions to **pre**-**candidate** state. \n\n2. After receiving majority pre-vote it becomes **candidate** and starts the election.\n\n3. It then votes for **itself** and issues RequestVote RPCs in parallel to each to the other servers in the cluster. \n\n4. A **candidate** continues in this state until one of three things happens: (a) it wins the election, (b) another server establishes itself as leader,or (c) a period of time goes by with no winner.\n\n5. A candidate node wins an election and becomes **leader** if it receives votes from a **majority** of the servers in the full cluster for the same term. \n\n   \n\nBased on above steps lets see how the orderer3 became the leader for mychannel from the its logs.\n\nBelow is the extract of *docker logs orderer3.example.com*\n\n```bash\n#Step0 Node3 starts election becomes Pre-Candidate\nStep -\u003e INFO 048 3 is starting a new election at term 1 channel=mychannel node=3\nbecomePreCandidate -\u003e INFO 049 3 became pre-candidate at term 1 channel=mychannel node=3\npoll -\u003e INFO 04a 3 received MsgPreVoteResp from 3 at term 1 channel=mychannel node=3\n\n#Step2 Node3 Sends MsgPreVote to all nodes in cluster.\ncampaign -\u003e INFO 04b 3 [logterm: 1, index: 5] sent MsgPreVote request to 1 at term 1 channel=mychannel node=3\ncampaign -\u003e INFO 04c 3 [logterm: 1, index: 5] sent MsgPreVote request to 2 at term 1 channel=mychannel node=3\ncampaign -\u003e INFO 04d 3 [logterm: 1, index: 5] sent MsgPreVote request to 4 at term 1 channel=mychannel node=3\ncampaign -\u003e INFO 04e 3 [logterm: 1, index: 5] sent MsgPreVote request to 5 at term 1 channel=mychannel node=3\n\n#Node 3 Starts Receving Response from other Nodes\npoll -\u003e INFO 04f 3 received MsgPreVoteResp from 5 at term 1 channel=mychannel node=3\nstepCandidate -\u003e INFO 050 3 [quorum:3] has received 2 MsgPreVoteResp votes and 0 vote rejections channel=mychannel node=3\npoll -\u003e INFO 051 3 received MsgPreVoteResp from 2 at term 1 channel=mychannel node=3\n\n#Step3 Node3 receives majority 3 / 5 MsgPreVoteResp from all nodes\nstepCandidate -\u003e INFO 052 3 [quorum:3] has received 3 MsgPreVoteResp votes and 0 vote rejections channel=mychannel node=3\n\n# Step4 Node3 incrementes term 1 to term 2 becomes candidate.\nbecomeCandidate -\u003e INFO 053 3 became candidate at term 2 channel=mychannel node=3\n# Step5 Node3 votes for itself\npoll -\u003e INFO 054 3 received MsgVoteResp from 3 at term 2 channel=mychannel node=3\n\n# Step6 Node3 sends MsgVote request to to other nodes.\ncampaign -\u003e INFO 055 3 [logterm: 1, index: 5] sent MsgVote request to 1 at term 2 channel=mychannel node=3\ncampaign -\u003e INFO 056 3 [logterm: 1, index: 5] sent MsgVote request to 2 at term 2 channel=mychannel node=3\ncampaign -\u003e INFO 057 3 [logterm: 1, index: 5] sent MsgVote request to 4 at term 2 channel=mychannel node=3\ncampaign -\u003e INFO 058 3 [logterm: 1, index: 5] sent MsgVote request to 5 at term 2 channel=mychannel node=3\n\n# Step7  Node 3 starts receving MsgVoteResp from other nodes\npoll -\u003e INFO 059 3 received MsgVoteResp from 4 at term 2 channel=mychannel node=3\nstepCandidate -\u003e INFO 05a 3 [quorum:3] has received 2 MsgVoteResp votes and 0 vote rejections channel=mychannel node=3\npoll -\u003e INFO 05b 3 received MsgVoteResp from 5 at term 2 channel=mychannel node=3\n\n# Step8 Node3 receives majority votes 3 / 5 \nstepCandidate -\u003e INFO 05c 3 [quorum:3] has received 3 MsgVoteResp votes and 0 vote rejections channel=mychannel node=3\n\n# Step9 Node3 becomes elected as leader\nbecomeLeader -\u003e INFO 05d 3 became leader at term 2 channel=mychannel node=3\n\nrun -\u003e INFO 05e raft.node: 3 elected leader 3 at term 2 channel=mychannel node=3\nrun -\u003e INFO 05f Leader 3 is present, quit campaign channel=mychannel node=3\nserveRequest -\u003e INFO 060 Raft leader changed: 0 -\u003e 3 channel=mychannel node=3\n\n# Step10 Node3 starts receiving requests as leader\nserveRequest -\u003e INFO 061 Start accepting requests as Raft leader at block [0] channel=mychannel node=3\npropose -\u003e INFO 062 Created block [1], there are 0 blocks in flight channel=mychannel node=3\n\n```\n\n\n\n#### Leader election after node failure\n\nLets simulate a node failure by stopping the Node3 which is the leader for mychannel.\n\n- docker stop orderer3.example.com.\n- Grafana dashboard we can see the number of nodes active is now 4.\n- And the **new leader** has been elected  for mychannel **Node1** which is orderer.example.com.\n\n![](images/newleader.png)\n\nLets inspect the docker logs orderer.example.com in detail.\n\n```bash\nrun -\u003e INFO 095 raft.node: 1 lost leader 3 at term 4 channel=mychannel node=1\n\n#Step1 Node1 Starts election Process\nStep -\u003e INFO 08e 1 is starting a new election at term 4 channel=mychannel node=1\nbecomePreCandidate -\u003e INFO 08f 1 became pre-candidate at term 4 channel=mychannel node=1\npoll -\u003e INFO 090 1 received MsgPreVoteResp from 1 at term 4 channel=mychannel node=1\ncampaign -\u003e INFO 091 1 [logterm: 4, index: 12] sent MsgPreVote request to 5 at term 4 channel=mychannel node=1\ncampaign -\u003e INFO 092 1 [logterm: 4, index: 12] sent MsgPreVote request to 2 at term 4 channel=mychannel node=1\ncampaign -\u003e INFO 093 1 [logterm: 4, index: 12] sent MsgPreVote request to 3 at term 4 channel=mychannel node=1\ncampaign -\u003e INFO 094 1 [logterm: 4, index: 12] sent MsgPreVote request to 4 at term 4 channel=mychannel node=1\nrun -\u003e INFO 095 raft.node: 1 lost leader 4 at term 4 channel=mychannel node=1\n\n\n```\n\nWe can see that the process repeats with Node1 becoming precandidate then candidate and then gets elected.\n\n- Now let us rejoin the node we stopped by starting the docker container.\n- After rejoining the node tries start and election by becoming pre-candidate  \n- But fails to get pre-votes majority and becomes a follower and elects the existing leader.\n\n```bash\nStep -\u003e INFO 0b5 3 is starting a new election at term 5 channel=mychannel node=3\nbecomePreCandidate -\u003e INFO 0b6 4 became pre-candidate at term 5 channel=mychannel node=3\n\ncampaign -\u003e INFO 0bf 3 [logterm: 4, index: 12] sent MsgPreVote request to 1 at term 5 channel=mychannel node=3\ncampaign -\u003e INFO 0c0 3 [logterm: 4, index: 12] sent MsgPreVote request to 2 at term 5 channel=mychannel node=3\ncampaign -\u003e INFO 0c1 3 [logterm: 4, index: 12] sent MsgPreVote request to 3 at term 5 channel=mychannel node=3\ncampaign -\u003e INFO 0c2 3 [logterm: 4, index: 12] sent MsgPreVote request to 5 at term 5 channel=mychannel node=3\npoll -\u003e INFO 0be 3 received MsgPreVoteResp from 3 at term 5 channel=mychannel node=3\nbecomeFollower -\u003e INFO 0c3 3 became follower at term 5 channel=mychannel node=3\nrun -\u003e INFO 0c4 raft.node: 3 elected leader 1 at term 5 channel=mychannel node=3\n```\n\nLets look at the logs of one of the other nodes orderer2.example.com during the same time period.\n\n```\nStep -\u003e INFO 05c 2 [logterm: 2, index: 7, vote: 0] ignored MsgPreVote from 3 [logterm: 2, index: 7] at term 2: lease is not expired (remaining ticks: 10) channel=byfn-sys-channel node=2\nStep -\u003e INFO 05d 2 [logterm: 5, index: 13, vote: 0] rejected MsgPreVote from 3 [logterm: 4, index: 12] at term 5 channel=mychannel node=2\n\n```\n\nWe can see that the pre-vote request from Node3 was rejected.\n\n\n\n#### Metrics\n\nFor this exercise we configured Prometheus and Grafana to explore cluster and consensus  related metrics. Refer this [link](https://hyperledger-fabric.readthedocs.io/en/latest/metrics_reference.html) for all available metrics.\n\nBefore starting network add the below lines under orderer-base environment variables.\n\n```bash\n      - ORDERER_OPERATIONS_LISTENADDRESS=0.0.0.0:8443\n      - ORDERER_METRICS_PROVIDER=prometheus \n```\n\nConfigure the [prometheus.yml](prometheus/prometheus.yml) as per your model.\n\n```bash\ndocker-compose -f docker-compose-prom.yaml up -d\n```\n\nMake sure the Prometheus is running after the deployment. http://serverIP:9090/graph\n\n![](images/prometheus.png)\n\nNow lets configure grafana\n\nAdd Prometheus as data source to grafana . http://serverIP:3000/datasources\n\n![](images/datasource.png)\n\nWe will create a simple grafana dashboard for this exercise.\n\n```bash\nconsensus_etcdraft_is_leader\u003e0\nconsensus_etcdraft_cluster_size{channel=\"byfn-sys-channel\"}\ncount(consensus_etcdraft_is_leader{channel=\"byfn-sys-channel\"})\n```\n\n![](images/grafana01.png)\n\n\n\n##### Reference\n\nhttps://kubernetes.io/blog/2019/08/30/announcing-etcd-3-4/\n\nhttps://raft.github.io/raft.pdf\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frupeshtr78%2Fraftleader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frupeshtr78%2Fraftleader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frupeshtr78%2Fraftleader/lists"}