https://github.com/einzsweiz/realtimenotifications

Event-driven microservice for real-time notifications using WebSockets, Kafka, Celery, and Redis.
https://github.com/einzsweiz/realtimenotifications

celery event-driven-architecture fastapi kafka microservices mongodb rabbitmq real-time redis-cluster redis-pubsub websocket

Last synced: about 2 months ago
JSON representation

Event-driven microservice for real-time notifications using WebSockets, Kafka, Celery, and Redis.

• Host: GitHub
• URL: https://github.com/einzsweiz/realtimenotifications
• Owner: EinzSweiz
• Created: 2025-02-28T13:35:12.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-02-28T16:37:38.000Z (over 1 year ago)
• Last Synced: 2025-02-28T21:17:42.210Z (over 1 year ago)
• Topics: celery, event-driven-architecture, fastapi, kafka, microservices, mongodb, rabbitmq, real-time, redis-cluster, redis-pubsub, websocket
• Language: Python
• Homepage:
• Size: 22.5 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
📡 Event-Driven Notification Microservice

🚀 Overview

This microservice provides real-time notifications using WebSockets, Kafka, Redis, Celery, and RabbitMQ.

It allows users to receive instant updates via WebSockets while ensuring event-driven processing for creating, fetching, and reading notifications.

🌟 Key Features

WebSocket-based real-time notifications 📡

Event-driven architecture using Kafka 🎯

Asynchronous task execution with Celery & RabbitMQ ⚡

Redis Pub/Sub for WebSocket updates 🚀

MongoDB for notification storage 📂

Email notifications using Celery 📧

Fully Dockerized with docker-compose 🐳

🎯 System Architecture

sequenceDiagram

    participant Frontend as WebSocket Client

    participant FastAPI as WebSocket Server

    participant KafkaProducer as Kafka Producer

    participant Kafka as Kafka Broker

    participant KafkaConsumer as Kafka Consumer

    participant Redis as Redis Pub/Sub

    participant MongoDB as MongoDB

    participant Celery as Celery Worker

    participant Email as Email Service

    participant Prometheus as Metrics Collector

    participant Grafana as Monitoring Dashboard

    Frontend ->> FastAPI: Send Notification Event

    FastAPI ->> KafkaProducer: Forward Event to Kafka

    KafkaProducer ->> Kafka: Publish Event

    Kafka ->> KafkaConsumer: Consumer Reads Event

    KafkaConsumer ->> MongoDB: Store Notification

    KafkaConsumer ->> Redis: Publish WebSocket Update

    KafkaConsumer ->> Celery: Trigger Email Notification

    Celery ->> Email: Send Email

    Redis ->> FastAPI: Push WebSocket Update

    FastAPI ->> Frontend: Send Notification to Client

    KafkaConsumer ->> Prometheus: Expose Kafka Metrics

    Redis ->> Prometheus: Expose Redis Metrics

    Prometheus ->> Grafana: Provide Monitoring Data

🚀 How to Run the Project Locally

1️⃣ Clone the repository

git clone https://github.com/EinzSweiz/RealTimeNotifications.git

cd event-driven-notifications

docker-compose up --build  # or use Makefile automation

📡 WebSocket API Usage

🔌 Connecting to WebSocket

ws://localhost:8002/ws/{user_id}

📩 Sending Events

1️⃣ Create Notification

{

  "type": "notification",

  "event": "create",

  "message": "Your application was rejected",

  "notification_type": "proposal_rejected",

  "user_email": "user@example.com"

}

2️⃣ Fetch Notifications

{

  "type": "notification",

  "event": "fetch"

}

3️⃣ Mark Notification as Read

{

  "type": "notification",

  "event": "read",

  "notification_id": "some-uuid"

}

📊 Monitoring with Prometheus & Grafana

🔹 How We Collect Metrics

We implemented custom exporters for:

    Kafka Exporter → Collects Kafka consumer lag, partition offsets, and message rates.

    Redis Exporter → Tracks Pub/Sub events, cache hits/misses, and WebSocket updates.

    WebSocket Metrics → Monitors active connections and message flow.

    Celery Exporter → Measures task execution time and queue performance.

2️⃣ Access Prometheus Metrics

    Prometheus UI → http://localhost:9090

    

    To query specific metrics, use:

    up

    kafka_messages_total

    redis_pubsub_messages

    celery_task_duration_seconds

    websocket_active_connections

3️⃣ Access Grafana Dashboard

    Grafana UI → http://localhost:3000

    Default login:

        Username: admin

        Password: admin

    Add Prometheus as a data source in Grafana.

    

🛠 Running Tests

pytest tests/ or make test(Makefile automatization)

This will run unit tests for WebSocket, Kafka consumers, and DDD entities.

📊 Future Improvements

    Prometheus + Grafana for monitoring 📊 (DONE ✅)

    Add authentication & user management 🔒

    Deploy to AWS using EC2 ☁️

    

4️⃣ Available Metrics

    Kafka Events Processed: kafka_messages_total

    Redis WebSocket Updates: redis_pubsub_messages

    Celery Task Execution Time: celery_task_duration_seconds

    WebSocket Active Connections: websocket_active_connections

💡 Contributing

If you’d like to improve this project, feel free to fork & submit a PR!

🏆 Author

👨‍💻 [Riad Sultanov] – Email: riad.sultanov.1999@gmail.com

🚀 Feel free to star ⭐ this repository if you find it useful!