{"id":41850757,"url":"https://github.com/mahmoudasem337/distributed-system","last_synced_at":"2026-01-25T10:21:16.791Z","repository":{"id":321281281,"uuid":"1049912988","full_name":"mahmoudasem337/distributed-system","owner":"mahmoudasem337","description":"Distributed Microservice system built on Spring Boot, demonstrating high performance using gRPC and reliable data handling via Kafka, Features deep observability via Prometheus and Grafana.","archived":false,"fork":false,"pushed_at":"2025-11-20T12:39:03.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-20T14:29:33.625Z","etag":null,"topics":["backend","docker","docker-compose","grafana-dashboard","grpc","java","kafka","load-testing","nginx","observability","prometheus","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mahmoudasem337.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-03T17:16:04.000Z","updated_at":"2025-11-20T12:39:06.000Z","dependencies_parsed_at":"2025-10-28T21:26:45.630Z","dependency_job_id":null,"html_url":"https://github.com/mahmoudasem337/distributed-system","commit_stats":null,"previous_names":["mahmoudasem337/distributed-system"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mahmoudasem337/distributed-system","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudasem337%2Fdistributed-system","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudasem337%2Fdistributed-system/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudasem337%2Fdistributed-system/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudasem337%2Fdistributed-system/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mahmoudasem337","download_url":"https://codeload.github.com/mahmoudasem337/distributed-system/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudasem337%2Fdistributed-system/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28751472,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T09:58:17.166Z","status":"ssl_error","status_checked_at":"2026-01-25T09:55:56.104Z","response_time":113,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","docker","docker-compose","grafana-dashboard","grpc","java","kafka","load-testing","nginx","observability","prometheus","spring-boot"],"created_at":"2026-01-25T10:21:16.160Z","updated_at":"2026-01-25T10:21:16.781Z","avatar_url":"https://github.com/mahmoudasem337.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed System\n\nThis project is a robust, high-performance Microservices Distributed System using a hybrid **gRPC** and **REST** microservices, achieving asynchronous processing with **Kafka** message queuing, inspired by [Eng: Ahmed El Taweel](https://youtu.be/Ur6b1NWGbYE?si=F-2yqL7QPDIwyR6Z)\n\n---\n\n## Architecture\n\u003cimg width=\"1266\" height=\"651\" alt=\"Architecture\" src=\"https://github.com/user-attachments/assets/5cd7f4b2-52e2-4154-a752-ee75488e609d\" /\u003e\n\n---\n\nThe application consists of two main services (Spring Boot):\n\n***gRPC Service***: A gRPC service that takes two numbers and immediately sums them then forwards results to Kafka.\n\n***REST Service***: A consumer REST service that reads messages from Kafka, adds the new value to an existing total, saves the updated value to an external file.\n\nCommunication Flow : \n\n1. Client makes gRPC request to the First Service with two numbers\n2. gRPC Service adds numbers and sends the result to Kafka\n3. REST Service consumes messages and add the current summation to the latest in the file\n\n---\n\n# Features\n## 1. gRPC Service\n\nThe system provides a **gRPC endpoint** that performs simple computation, summing two numbers. Beyond basic computation, the service is designed with strong **reliability and idempotency guarantees** to ensure safe operation in distributed environments.\n\n### Key Features\n\n- **Business-Level Idempotency**:  \n  Each request includes a unique **UUID `requestId`**, which guarantees that duplicate client calls—whether due to retries, network issues, or user error—do not result in duplicate computation or duplicated Kafka messages.\n- **Kafka Producer Idempotency**:  \n  The Kafka producer is configured with **`enable.idempotence=true`**, ensuring that even if retries occur, the broker will prevent duplicate message delivery. This provides exactly-once delivery semantics for asynchronous messaging.\n- **Retry Mechanism**:  \n  Safe automatic retries are enabled to handle transient failures, such as temporary network glitches or Kafka broker unavailability, ensuring messages are reliably delivered without loss.\n- **In-Memory Cache**:  \n  A temporary in-memory cache stores recent processed requests, allowing the service to quickly detect duplicates and maintain consistency, even under high load or repeated client calls.\n\nThis combination of **idempotency, retries, and caching** ensures that the gRPC service remains **resilient, consistent, and reliable**, even in highly distributed and high-traffic environments.\n\n## 2. REST Service\n\n- Provides REST endpoints for saving summation results to external file for auditing or offline processing.\n\n## 3. Load Testing with k6\n\nThe system includes robust load testing using **k6**, a modern load-testing tool designed to simulate realistic traffic patterns and measure system performance. This feature ensures the application can handle high loads and maintain reliability under stress.\n\n### Key Features\n\n- **Ramping Virtual Users**: Gradually increases the number of simulated users to observe system behavior under rising load, helping identify potential bottlenecks.  \n- **Randomized Endpoint Testing**: Simulates requests to different API endpoints randomly, ensuring all services are tested under realistic conditions.  \n- **Performance Thresholds**: Defines acceptable limits for error rates and response times, so the system is automatically flagged if performance degrades.  \n- **Realistic User Behavior**: Introduces random delays between requests to mimic real-world user interactions, rather than sending constant traffic.  \n- **Peak Load Simulation**: Can scale to thousands of concurrent users to test the system’s ability to handle extreme traffic spikes.  \n- **Comprehensive Metrics**: Measures throughput, latency, error rates, and other performance indicators, providing detailed insights for optimization.  \n\nOverall, k6 load testing helps ensure that services remain performant, resilient, and reliable before deployment to production.\n\n## 4. Observability\n\nThe system uses a full observability stack to monitor performance, errors, and traces:\n\n- **Grafana**: Dashboard for visualizing metrics, logs, and traces.\n- **Prometheus**: Collects system and application metrics such as CPU usage, JVM heap, queue latency, error rate (5xx), and RPS.\n- **Loki**: Centralized logging for debugging and tracking application events.\n- **Tempo**: Distributed tracing for analyzing request flows and identifying bottlenecks.\n\n**Metrics Captured:**\n\n| Metric                  | Description                                |\n|-------------------------|--------------------------------------------|\n| CPU Usage               | Tracks system CPU utilization              |\n| JVM Heap                | Monitors Java memory consumption           |\n| Error Rate (5xx)        | Tracks server-side errors                  |\n| Requests per Second     | Measures throughput                        |\n| Queue Latency           | Measures message queue delays              |\n\n---\n## Screenshots\n### Loki Logs Viewer\n\u003cimg width=\"1893\" height=\"920\" alt=\"Screenshot 2025-11-23 160843\" src=\"https://github.com/user-attachments/assets/dd40a7f2-abeb-4735-a1e9-7654c6a9ba6c\" /\u003e\n\n### Tempo Distributed Traces\n\u003cimg width=\"1891\" height=\"930\" alt=\"file_2025-11-22_19 54 40\" src=\"https://github.com/user-attachments/assets/894617f9-182b-47a1-98da-5685cdf6a127\" /\u003e\n\n### Grafana Dashboard\n\u003cimg width=\"1896\" height=\"918\" alt=\"file_2025-11-22_18 54 25\" src=\"https://github.com/user-attachments/assets/e0db15f1-9d2e-4b92-b268-edcfffee818d\" /\u003e\n\n### Kafka Messege\n\u003cimg width=\"1877\" height=\"917\" alt=\"file_2025-11-23_13 43 39\" src=\"https://github.com/user-attachments/assets/8a73f996-3e95-4ffc-bd20-efe2fd4b1b8f\" /\u003e\n\n### Docker Container\n\u003cimg width=\"1914\" height=\"1035\" alt=\"file_2025-11-23_13 42 11\" src=\"https://github.com/user-attachments/assets/fe0d902d-5949-4c38-bb05-ecc11289a510\" /\u003e\n\n---\n\n## Prerequisites\n\nMake sure you have installed:\n\n- Java 24\n- Maven\n- Docker \u0026 Docker Compose\n- Kafka \u0026 ZooKeeper\n- Node.js (for k6, optional if using CLI)\n  \n---\n\n## Getting Started\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/your-username/your-repo.git\ncd your-repo\n```\n\n### 2. Start Spring Boot App\n\n```bash\n./mvnw clean install\n./mvnw spring-boot:run\n```\n\n### 3. Start Zookeaper, Kafka\n```bash\n# Start ZooKeeper\n.\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties\n# Start Kafka broker\n.\\bin\\windows\\kafka-server-start.bat .\\config\\server.properties\n```\n\n### 4. Start Observability Stack\n```bash\ndocker-compose up -d\n```\n\n### 5. Run Load Testing with k6\n```bash\nk6 run load_test.js\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoudasem337%2Fdistributed-system","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahmoudasem337%2Fdistributed-system","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoudasem337%2Fdistributed-system/lists"}