https://github.com/sergio11/iot_event_streaming_architecture

🌐📈 Unlock IoT potential with this Event Streaming Architecture: Apache Kafka and MQTT power real-time data handling. Visualize, analyze, and scale effortlessly. 🚀💻📊
https://github.com/sergio11/iot_event_streaming_architecture

cadvisor-exporter docker docker-compose grafana kafka kafka-connect kafka-streams micrometer mongodb node-exporter prometheus spring-boot spring-boot-actuator zookeeper

Last synced: 4 months ago
JSON representation

🌐📈 Unlock IoT potential with this Event Streaming Architecture: Apache Kafka and MQTT power real-time data handling. Visualize, analyze, and scale effortlessly. 🚀💻📊

Host: GitHub
URL: https://github.com/sergio11/iot_event_streaming_architecture
Owner: sergio11
License: gpl-3.0
Created: 2019-06-07T20:48:49.000Z (almost 6 years ago)
Default Branch: master
Last Pushed: 2023-11-10T20:44:37.000Z (over 1 year ago)
Last Synced: 2023-11-10T21:32:42.255Z (over 1 year ago)
Topics: cadvisor-exporter, docker, docker-compose, grafana, kafka, kafka-connect, kafka-streams, micrometer, mongodb, node-exporter, prometheus, spring-boot, spring-boot-actuator, zookeeper
Language: Java
Homepage: https://medium.com/@sanchezsanchezsergio418/iot-event-streaming-architecture-fb790c634c2f
Size: 27.3 MB
Stars: 14
Watchers: 3
Forks: 2
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# QuantumFlow IoT: Smart IoT Event Streaming. Scaling with Apache Kafka and MQTT 🌐📊

Unlock the potential of the Internet of Things (IoT) with this cutting-edge Event Streaming Architecture. Leveraging the power of Apache Kafka and MQTT, this project is designed to handle IoT data at scale. 🚀

Ingest real-time data from IoT sensors across diverse locations, analyze metrics like temperature, humidity, pressure, and luminosity, and store them efficiently in a high-performance MongoDB database. 🌡️💧📈

The technology stack includes Spring Boot, Kafka Streams, Micrometer, and Grafana for real-time monitoring, making this architecture a powerhouse for IoT solutions. 💻📈🔍

With Docker containerization, deployment is a breeze. Explore this project, visualize IoT data, and gain insights into your sensor networks. 🐳🚀📊

Check out the detailed [Medium article]([link_to_medium_article](https://medium.com/@sanchezsanchezsergio418/iot-event-streaming-architecture-fb790c634c2f)) for a comprehensive overview. 📖

Thank you for visiting the Smart IoT Event Streaming GitHub repository! Empower your IoT endeavors and scale with confidence. 🌐📈💡

Project developed to practice what I have learned in the Udemy course [Apache Kafka Series - Kafka Connect Hands-on Learning](https://www.udemy.com/course/kafka-connect/) and
[Apache Kafka Series - Kafka Streams for Data Processing](https://www.udemy.com/course/kafka-streams/).

## Architecture Overview

The main goals of this architecture are the following:

* Ingest and store real-time data from IoT sensors located in various locations.
* Analyze and make aggregations through rotating time windows to know average temperature by sensor or place, in addition to humidity, pressure and luminosity.
* Store data after processing for subsequent monitoring in a NOSQL database with good performance for frequent writes for relatively homogeneous document sizes such as MongoDB.
* Visualization of the metrics of the sensors in real time and of the performance of the components of the architecture through Grafana and the consolidated documents in MongoDB through Mongo-Express.

## Used technology

* Spring Boot 2.3.3 / Apache Maven 3.6.3.
* Spring Boot Starter Actuator.
* Kafka Streams.
* Spring Kafka.
* Micrometer Registry Prometheus.
* Eclipse Paho MQTT Client.
* Kafka Connect.
* Kafka Rest Proxy
* lombok.
* Jackson.
* NodeExporter (Exporter for machine metrics).
* Prometheus.
* Grafana.
* Eclipse Mosquitto.
* MongoDB.
* Mongo DB Express (Web-based MongoDB admin interface, written with Node.js and express).
* Cadvisor (Analyzes resource usage and performance characteristics of running containers).
* kafka-exporter (Kafka exporter for Prometheus).

## Running Applications as Docker containers.

### Rake Tasks

The available tasks are detailed below (rake --task)

| Task | Description |
| ------ | ------ |
| check_deployment_file_task | Check Deployment File |
| check_docker_task | Check Docker and Docker Compose Task |
| cleaning_environment_task | Cleaning Evironment Task |
| deploy | Deploys the IoT Event Streaming Architecture and laun... |
| login | Authenticating with existing credentials |
| start | Start Containers |
| status | Status Containers |
| stop | Stop Containers |
| undeploy | UnDeploy IoT Event Streaming Architecture |

To start the platform make sure you have Ruby installed, go to the root directory of the project and run the `rake deploy` task, this task will carry out a series of preliminary checks, discard images and volumes that are no longer necessary and also proceed to download all the images and the initialization of the containers.

### Containers Ports

| Container | Port |
| ------ | ------ |
| kafka-topics-ui | localhost:8081 |
| kafka-connect-ui | localhost:8082 |
| zoonavigator-web | localhost:8083 |
| mongo-express | localhost:8084 |
| grafana | localhost:8085 |
| prometheus | localhost:8086 |
| kafka-rest-proxy | localhost:9999 |

## Some screenshots

### Deploy with Docker Compose.