{"id":23128078,"url":"https://github.com/rehan-adi/scalable-chat-app","last_synced_at":"2026-04-11T17:06:40.698Z","repository":{"id":254160006,"uuid":"845502341","full_name":"rehan-adi/Scalable-Chat-App","owner":"rehan-adi","description":"A scalable architecture for a real-time chat application, utilizing WebSocket for instant communication, Redis for message queuing, and Kafka for durable, scalable message storage and processing.","archived":false,"fork":false,"pushed_at":"2024-12-23T11:25:59.000Z","size":1159,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-09T17:16:00.438Z","etag":null,"topics":["express","kafka","postgresql","prisma","pubsub","redis","socket-io","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rehan-adi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-21T11:26:46.000Z","updated_at":"2024-12-23T11:26:02.000Z","dependencies_parsed_at":"2024-08-21T19:59:34.628Z","dependency_job_id":null,"html_url":"https://github.com/rehan-adi/Scalable-Chat-App","commit_stats":null,"previous_names":["rehan-adi/scalable-chat-app"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehan-adi%2FScalable-Chat-App","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehan-adi%2FScalable-Chat-App/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehan-adi%2FScalable-Chat-App/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehan-adi%2FScalable-Chat-App/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rehan-adi","download_url":"https://codeload.github.com/rehan-adi/Scalable-Chat-App/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128738,"owners_count":20888234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["express","kafka","postgresql","prisma","pubsub","redis","socket-io","typescript"],"created_at":"2024-12-17T09:17:32.124Z","updated_at":"2026-04-11T17:06:40.652Z","avatar_url":"https://github.com/rehan-adi.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scaling Node.js Socket.io Server  \n\nThis project demonstrates a scalable Socket.io server capable of handling a high volume of user connections and real-time communication. The system solves the challenges of horizontal scaling by introducing a message broker to synchronize messages across multiple server instances.  \n\n---\n\n## Problem  \n\nAs user traffic increases, a single server instance can no longer handle all connections. Horizontal scaling (adding more server instances) creates a new issue:  \n- Users connected to different server instances cannot exchange messages directly, as the servers are isolated.  \n\nHere’s an illustration of the problem:  \n![System Problem](./apps/web/public/images/problem.png)  \n\n---\n\n## Solution  \n\nA message broker is integrated into the architecture to synchronize messages between server instances. This setup ensures that users connected to different servers can communicate seamlessly, as if they were on the same instance.\n\n![System Architecture](./apps/web/public/images/system.png)  \n\n### Key Features of the System:  \n- **Real-time synchronization** across server instances.  \n- **Scalable architecture** for handling increased user traffic.  \n- **Reliable message delivery** with no loss, even during server instance failures.  \n\n### Technologies Used:  \n- **Node.js** for the server.  \n- **Socket.io** for real-time communication.  \n- **Redis** as the message broker.  \n\n---\n\n## Additional Problem: Database Write Load  \n\nWhen a large number of users send messages, the database write operations can become a bottleneck, significantly impacting performance. A direct approach to updating the database for each message is inefficient and can lead to high latency or crashes under heavy load.  \n\n---\n\n## Enhanced Solution: Kafka for Database Updates  \n\nTo handle the heavy database write load, Kafka, a high-throughput distributed messaging system, was integrated into the architecture.\n\n### How It Works:  \n1. **Message Producer**:  \n   - When a user sends a message, it is added to a Kafka topic named `MESSAGES`.  \n\n2. **Message Consumer**:  \n   - A Kafka consumer listens to the `MESSAGES` topic.  \n   - The consumer processes each message and writes it to the database in an optimized manner.\n\n3. **Database Write Optimization**:  \n   - Kafka’s high throughput ensures that messages are efficiently queued and processed without overloading the database.  \n   - This architecture enhances both application performance and database reliability.  \n\n---\n\n### Benefits of Kafka Integration:  \n- **High Throughput**: Kafka can handle a large volume of messages per second.  \n- **Reduced Latency**: By batching database writes, latency is minimized.  \n- **Scalability**: Kafka’s distributed nature allows the system to scale easily.  \n- **Fault Tolerance**: Kafka ensures message delivery even in the case of server failures.  \n\n### Updated Architecture:  \n\nClient A \u003c--\u003e Server 1\n|\nClient B \u003c--\u003e Server 2\n|\nRedis (Message Broker)\n|\nKafka (Message Queue)\n|\nDatabase (Persistent Storage)\n\n\n---\n\n## Conclusion  \n\nBy integrating Redis for real-time message synchronization and Kafka for database updates, this system achieves a scalable, efficient, and reliable architecture. It ensures seamless user communication and optimized database operations, even under heavy loads.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frehan-adi%2Fscalable-chat-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frehan-adi%2Fscalable-chat-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frehan-adi%2Fscalable-chat-app/lists"}