https://github.com/settorka/real-time-data-pipeline

Sink-source implementation for stream processing of JSON records
https://github.com/settorka/real-time-data-pipeline

docker golang kafka mongodb rest-api

Last synced: 9 days ago
JSON representation

Sink-source implementation for stream processing of JSON records

• Host: GitHub
• URL: https://github.com/settorka/real-time-data-pipeline
• Owner: settorka
• Created: 2024-05-31T04:06:18.000Z (6 months ago)
• Default Branch: main
• Last Pushed: 2024-09-14T22:30:39.000Z (2 months ago)
• Last Synced: 2024-09-15T06:22:38.002Z (2 months ago)
• Topics: docker, golang, kafka, mongodb, rest-api
• Language: Elixir
• Homepage:
• Size: 24.2 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Real-time data pipeline

Cloud-Native application for sink-source stream processing of JSON records

- Data Generator using Golang to concurrently generate JSON record. Can specify records/sec.

  - Rate can updated via HTTP POST request,

  - generator can be stopped/started via HTTP POST request.

- Kafka Pipeline to consume records upstream and sink them downstream to API consumers via message topics.

  

- REST APIs as Kafka Consumers which receive Kafka topics and post them to MongoDB

  - Python Flask

  - Ruby on Rails

  - Java Springboot

  - Elixir Phoenix

  - Express + Node.js

    

- MongoDB replicas to handle influx of post requests to DB,

  - deployed locally for test

  -   Was hosted on Azure Kubernetes Engine using

      - Terraform

      - Helm (bitnami for mongodb) + Kubernetes

      - https://learn.microsoft.com/en-us/azure/aks/mongodb-overview