https://github.com/mahmoudasem337/distributed-system
Distributed Microservice system built on Spring Boot, demonstrating high performance using gRPC and reliable data handling via Kafka, Features deep observability via Prometheus and Grafana.
https://github.com/mahmoudasem337/distributed-system
backend docker docker-compose grafana-dashboard grpc java kafka load-testing nginx observability prometheus spring-boot
Last synced: 5 months ago
JSON representation
Distributed Microservice system built on Spring Boot, demonstrating high performance using gRPC and reliable data handling via Kafka, Features deep observability via Prometheus and Grafana.
- Host: GitHub
- URL: https://github.com/mahmoudasem337/distributed-system
- Owner: mahmoudasem337
- Created: 2025-09-03T17:16:04.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-11-20T12:39:03.000Z (7 months ago)
- Last Synced: 2025-11-20T14:29:33.625Z (7 months ago)
- Topics: backend, docker, docker-compose, grafana-dashboard, grpc, java, kafka, load-testing, nginx, observability, prometheus, spring-boot
- Language: Java
- Homepage:
- Size: 54.7 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Distributed System
This project is a robust, high-performance Microservices Distributed System using a hybrid **gRPC** and **REST** microservices, achieving asynchronous processing with **Kafka** message queuing, inspired by [Eng: Ahmed El Taweel](https://youtu.be/Ur6b1NWGbYE?si=F-2yqL7QPDIwyR6Z)
---
## Architecture
---
The application consists of two main services (Spring Boot):
***gRPC Service***: A gRPC service that takes two numbers and immediately sums them then forwards results to Kafka.
***REST Service***: A consumer REST service that reads messages from Kafka, adds the new value to an existing total, saves the updated value to an external file.
Communication Flow :
1. Client makes gRPC request to the First Service with two numbers
2. gRPC Service adds numbers and sends the result to Kafka
3. REST Service consumes messages and add the current summation to the latest in the file
---
# Features
## 1. gRPC Service
The system provides a **gRPC endpoint** that performs simple computation, summing two numbers. Beyond basic computation, the service is designed with strong **reliability and idempotency guarantees** to ensure safe operation in distributed environments.
### Key Features
- **Business-Level Idempotency**:
Each request includes a unique **UUID `requestId`**, which guarantees that duplicate client calls—whether due to retries, network issues, or user error—do not result in duplicate computation or duplicated Kafka messages.
- **Kafka Producer Idempotency**:
The Kafka producer is configured with **`enable.idempotence=true`**, ensuring that even if retries occur, the broker will prevent duplicate message delivery. This provides exactly-once delivery semantics for asynchronous messaging.
- **Retry Mechanism**:
Safe automatic retries are enabled to handle transient failures, such as temporary network glitches or Kafka broker unavailability, ensuring messages are reliably delivered without loss.
- **In-Memory Cache**:
A temporary in-memory cache stores recent processed requests, allowing the service to quickly detect duplicates and maintain consistency, even under high load or repeated client calls.
This combination of **idempotency, retries, and caching** ensures that the gRPC service remains **resilient, consistent, and reliable**, even in highly distributed and high-traffic environments.
## 2. REST Service
- Provides REST endpoints for saving summation results to external file for auditing or offline processing.
## 3. Load Testing with k6
The system includes robust load testing using **k6**, a modern load-testing tool designed to simulate realistic traffic patterns and measure system performance. This feature ensures the application can handle high loads and maintain reliability under stress.
### Key Features
- **Ramping Virtual Users**: Gradually increases the number of simulated users to observe system behavior under rising load, helping identify potential bottlenecks.
- **Randomized Endpoint Testing**: Simulates requests to different API endpoints randomly, ensuring all services are tested under realistic conditions.
- **Performance Thresholds**: Defines acceptable limits for error rates and response times, so the system is automatically flagged if performance degrades.
- **Realistic User Behavior**: Introduces random delays between requests to mimic real-world user interactions, rather than sending constant traffic.
- **Peak Load Simulation**: Can scale to thousands of concurrent users to test the system’s ability to handle extreme traffic spikes.
- **Comprehensive Metrics**: Measures throughput, latency, error rates, and other performance indicators, providing detailed insights for optimization.
Overall, k6 load testing helps ensure that services remain performant, resilient, and reliable before deployment to production.
## 4. Observability
The system uses a full observability stack to monitor performance, errors, and traces:
- **Grafana**: Dashboard for visualizing metrics, logs, and traces.
- **Prometheus**: Collects system and application metrics such as CPU usage, JVM heap, queue latency, error rate (5xx), and RPS.
- **Loki**: Centralized logging for debugging and tracking application events.
- **Tempo**: Distributed tracing for analyzing request flows and identifying bottlenecks.
**Metrics Captured:**
| Metric | Description |
|-------------------------|--------------------------------------------|
| CPU Usage | Tracks system CPU utilization |
| JVM Heap | Monitors Java memory consumption |
| Error Rate (5xx) | Tracks server-side errors |
| Requests per Second | Measures throughput |
| Queue Latency | Measures message queue delays |
---
## Screenshots
### Loki Logs Viewer
### Tempo Distributed Traces
### Grafana Dashboard
### Kafka Messege
### Docker Container
---
## Prerequisites
Make sure you have installed:
- Java 24
- Maven
- Docker & Docker Compose
- Kafka & ZooKeeper
- Node.js (for k6, optional if using CLI)
---
## Getting Started
### 1. Clone the Repository
```bash
git clone https://github.com/your-username/your-repo.git
cd your-repo
```
### 2. Start Spring Boot App
```bash
./mvnw clean install
./mvnw spring-boot:run
```
### 3. Start Zookeaper, Kafka
```bash
# Start ZooKeeper
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
# Start Kafka broker
.\bin\windows\kafka-server-start.bat .\config\server.properties
```
### 4. Start Observability Stack
```bash
docker-compose up -d
```
### 5. Run Load Testing with k6
```bash
k6 run load_test.js
```