https://github.com/adithya-s-k/pyraft
raft algorithm implemented using python and FASTAPI
https://github.com/adithya-s-k/pyraft
fastapi
Last synced: 6 months ago
JSON representation
raft algorithm implemented using python and FASTAPI
- Host: GitHub
- URL: https://github.com/adithya-s-k/pyraft
- Owner: adithya-s-k
- License: mit
- Created: 2024-03-19T06:38:16.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-05-19T08:16:32.000Z (over 1 year ago)
- Last Synced: 2025-02-14T13:48:20.514Z (8 months ago)
- Topics: fastapi
- Language: Python
- Homepage:
- Size: 18.6 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## PyRaft
## Overview
raft algorithm implemented using python and FASTAPI aiming to create a detailed imitation of the mechanisms underlying Zookeeper and Kraft. This project is designed for those intrigued by the inner workings of distributed systems and consensus algorithms. It's an attempt to demystify and replicate the functionalities of these systems, not as a direct replacement, but as a learning tool and a tribute to their complex architectures.
## Objectives
The primary goals of PyRaft include:
- Gaining a deep understanding of the Raft Consensus algorithm, focusing on Leader Election, Log Replication, and Fault Tolerance.
- Exploring the ZAB (Zookeeper Atomic Broadcast) Consensus algorithm.
- Delving into the architectural nuances of Kraft and Zookeeper.
- Implementing an Event-Driven Architecture.
- Studying and replicating key aspects of Kafka's architecture.
```mermaid
graph TD;
subgraph Leader
A(Leader Election) --> B(Heartbeats)
end
subgraph Candidate
D(Start Election) --> E(Receive Votes)
end
subgraph Follower
G(Receive Heartbeats) --> H(Respond to Leader)
end
B --> C(Commit Log)
E --> C
H --> C
C -->|Apply| I(Change State)
I -->|Broadcast| A
```
This project implements a distributed node management system using Python and FastAPI. It focuses on maintaining a network of nodes, where each node can either be a leader or a follower. The system handles tasks like heartbeat management, metadata handling, and dynamic leader election.
## Features
- **Node Initialization**: Dynamically initialize nodes as leaders or followers based on configuration.
- **Heartbeat Mechanism**: Leaders regularly send heartbeats to followers to maintain the network's health.
- **Event Logging**: Each node maintains an event log, tracking significant actions and heartbeats.
- **Metadata Management**: Nodes store and update metadata information, crucial for network operation.
- **Dynamic Leader Election**: In case of leader failure, followers can elect a new leader.
- **Broker and Topic Management**: API endpoints to manage brokers, topics, and partitions within the network.
## Installation
1. Ensure Python 3.9 or higher is installed.
2. Install required packages:
```
pip install fastapi requests uvicorn
```
## Usage
To start a node:
```
python raft/node.py [port_number]
```
Replace `[port_number]` with the desired port number for the node.
API Documentation
# CRD API
## Create
#### Endpoint: `/register_broker/`
- **Method**: POST
- **Description**: Creates a new record under RegisterBrokerRecords.
- **Request Body**:
```
{
"brokerId": 5,
"brokerHost": "127.0.0.1",
"brokerPort": 8005,
"securityProtocol": "",
"rackId": "",
}
```
- **Response**: Returns `internal_uuid` of the created record.
#### Endpoint: `/register_topic/`
- **Method**: POST
- **Description**: Creates a new record under TopicRecord.
- **Request Body**:
```
{
"name": "Topic Name -1"
}
```
- **Response**: Returns `topicuuid` of the created record.
#### Endpoint: `/register_partition/`
- **Method**: POST
- **Description**: Creates a new record under PartitionRecord.
- **Request Body**:
```
{
"partitionId": 0,
"topicUUID": "",
"replicas": [],
"ISR": [],
"removingReplicas": [],
"addingReplicas": [],
"leader": "",
"partitionEpoch": 0
}
```
- **Response**: Returns `topicuuid` of the created record.
## Read
#### **Endpoint**: `/get_broker/`
- **Method**: GET
- **Description**: Retrieves single or multiple records of brokers.
- **Query Parameters**: `broker_id` (\*Optional).
- **Response**: JSON array of records / Single requested record.
#### **Endpoint**: `/get_topic/`
- **Method**: GET
- **Description**: Retrieves single or multiple records of topics.
- **Query Parameters**: `topic_name` (\*Optional).
- **Response**: JSON array of records / Single requested record.
#### **Endpoint**: `/get_partition/`
- **Method**: GET
- **Description**: Retrieves single or multiple records of partitions.
- **Query Parameters**: `partition_id` (\*Optional).
- **Response**: JSON array of records / Single requested record.
#### **Endpoint**: `/get_producer/`
- **Method**: GET (to retrieve all records) / POST (to retrieve selected record)
- **Description**: Retrieves single or multiple records of producers.
- **Request Body**: `{partitionId: 0, brokerId: "uuid"}` (\*Optional).
- **Response**: JSON array of records / Single requested record.
## Delete
#### **Endpoint**: `/delete_broker/{broker_id}`
- **Method**: DELETE
- **Description**: Deletes a specific record by its unique ID.
- **Path Parameter**: `id` - Unique ID of the entity to delete.
- **Response**: Deleted Record.
#### **Endpoint**: `/delete_topic/{topicName}`
- **Method**: DELETE
- **Description**: Deletes a specific record by its unique ID.
- **Path Parameter**: `id` - Unique ID of the entity to delete.
- **Response**: Deleted Record.
#### **Endpoint**: `/delete_partition/{partition_id}`
- **Method**: DELETE
- **Description**: Deletes a specific record by its unique ID.
- **Path Parameter**: `id` - Unique ID of the entity to delete.
- **Response**: Deleted Record.
## Broker Management API
### RegisterBroker
#### Endpoint: `/register_broker/`
- **Method**: POST
- **Description**: Creates a new record under RegisterBrokerRecords.
- **Request Body**:
```
{
"brokerId": 5,
"brokerHost": "127.0.0.1",
"brokerPort": 8005,
"securityProtocol": "",
"rackId": "",
}
```
- **Response**: Returns `internal_uuid` of the created record.
### RegisterBrokerChange
#### Endpoint: `/register_broker_change/`
- **Method**: POST
- **Description**: Changes specified record in `RegisterBrokerRecord`.
- **Request Body**:
```
{
"brokerId": 5,
"brokerHost": "127.0.0.1",
"brokerPort": "8005",
"securityProtocol": "",
"brokerStatus":"INIT"
}
```
- **Response**: Acknowledgement of change.
### MetadataFetch
#### To be implemented
## Client Management API
### RegisterProducer
- **Endpoint**: `/register_producer/`
- **Method**: POST
- **Description**: Registers a new producer in the system.
- **Request Body**:
```
{
"brokerId":"5",
"producerId":0,
"brokerEpoch":0
}
```
- **Response**: Acknowledgment of registration.
### MetadataFetch
- **Endpoint**: `/metadata_fetch_client/`
- **Method**: GET
- **Description**: Fetches topics, partition, and broker information records for clients.
- **Response**: Requested metadata information.
# Team Members
- Adithya S Kolavi
- Achala Nayak
- Srinidhi Somayaji P
- Monisha S