{"id":22958005,"url":"https://github.com/sergio11/iot_event_streaming_architecture","last_synced_at":"2025-08-13T04:33:11.457Z","repository":{"id":45523975,"uuid":"190807153","full_name":"sergio11/iot_event_streaming_architecture","owner":"sergio11","description":"🌐📈 Unlock IoT potential with this Event Streaming Architecture: Apache Kafka and MQTT power real-time data handling. Visualize, analyze, and scale effortlessly. 🚀💻📊","archived":false,"fork":false,"pushed_at":"2025-03-13T17:52:05.000Z","size":28867,"stargazers_count":20,"open_issues_count":1,"forks_count":19,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-01T16:21:24.823Z","etag":null,"topics":["cadvisor-exporter","docker","docker-compose","grafana","kafka","kafka-connect","kafka-streams","micrometer","mongodb","node-exporter","prometheus","spring-boot","spring-boot-actuator","zookeeper"],"latest_commit_sha":null,"homepage":"https://medium.com/@sanchezsanchezsergio418/iot-event-streaming-architecture-fb790c634c2f","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sergio11.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-07T20:48:49.000Z","updated_at":"2025-03-15T17:50:16.000Z","dependencies_parsed_at":"2023-11-10T21:41:47.393Z","dependency_job_id":null,"html_url":"https://github.com/sergio11/iot_event_streaming_architecture","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sergio11/iot_event_streaming_architecture","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergio11%2Fiot_event_streaming_architecture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergio11%2Fiot_event_streaming_architecture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergio11%2Fiot_event_streaming_architecture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergio11%2Fiot_event_streaming_architecture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sergio11","download_url":"https://codeload.github.com/sergio11/iot_event_streaming_architecture/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sergio11%2Fiot_event_streaming_architecture/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270182806,"owners_count":24541311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cadvisor-exporter","docker","docker-compose","grafana","kafka","kafka-connect","kafka-streams","micrometer","mongodb","node-exporter","prometheus","spring-boot","spring-boot-actuator","zookeeper"],"created_at":"2024-12-14T17:34:45.656Z","updated_at":"2025-08-13T04:33:11.420Z","avatar_url":"https://github.com/sergio11.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IoT Event Streaming Architecture: Smart IoT Event Streaming. Scaling with Apache Kafka and MQTT 🌐📊\n\n\u003cimg width=\"300px\" align=\"left\" src=\"./iot_event_streaming_architecture_logo.webp\" /\u003e\n\nUnlock the potential of the Internet of Things (IoT) with this cutting-edge Event Streaming Architecture. Leveraging the power of Apache Kafka and MQTT, this project is designed to handle IoT data at scale. 🚀\n\nIngest real-time data from IoT sensors across diverse locations, analyze metrics like temperature, humidity, pressure, and luminosity, and store them efficiently in a high-performance MongoDB database. 🌡️💧📈\n\nThe technology stack includes Spring Boot, Kafka Streams, Micrometer, and Grafana for real-time monitoring, making this architecture a powerhouse for IoT solutions. 💻📈🔍\n\nWith Docker containerization, deployment is a breeze. Explore this project, visualize IoT data, and gain insights into your sensor networks. 🐳🚀📊\n\nCheck out the detailed [Medium article]([link_to_medium_article](https://medium.com/@sanchezsanchezsergio418/iot-event-streaming-architecture-fb790c634c2f)) for a comprehensive overview. 📖\n\nThank you for visiting the Smart IoT Event Streaming GitHub repository! Empower your IoT endeavors and scale with confidence. 🌐📈💡\n\nProject developed to practice what I have learned in the Udemy course [Apache Kafka Series - Kafka Connect Hands-on Learning](https://www.udemy.com/course/kafka-connect/) and \n[Apache Kafka Series - Kafka Streams for Data Processing](https://www.udemy.com/course/kafka-streams/).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Apache_Kafka-231F20?style=for-the-badge\u0026logo=apache-kafka\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Docker-2CA5E0?style=for-the-badge\u0026logo=docker\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/MongoDB-%234ea94b.svg?style=for-the-badge\u0026logo=mongodb\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Prometheus-000000?style=for-the-badge\u0026logo=prometheus\u0026labelColor=000000\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Grafana-F2F4F9?style=for-the-badge\u0026logo=grafana\u0026logoColor=orange\u0026labelColor=F2F4F9\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Spring-6DB33F?style=for-the-badge\u0026logo=spring\u0026logoColor=white\" /\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Spring_Boot-F2F4F9?style=for-the-badge\u0026logo=spring-boot\" /\u003e\n\u003c/p\u003e\n\n## ⚠️ Disclaimer:\n\nThis project, **IoT Event Streaming Architecture**, has been developed **for educational purposes only**. Its primary objective is to provide a hands-on learning experience for building scalable event streaming architectures and processing real-time IoT data using technologies like **Apache Kafka**, **MQTT**, **MongoDB**, **Grafana**, and **Spring Boot**.\n\nThis repository demonstrates how to ingest, process, and store data from IoT sensors, and it does not provide access to real-world IoT data or production-grade sensor networks. The project uses **simulated data** for demonstration purposes, allowing users to explore the design and architecture of IoT solutions without the need for live sensor deployments.\n\nThe name **IoT Event Streaming Architecture** was generated during a **brainstorming session** with the help of **AI**, and any resemblance to existing brands or services is purely **coincidental**.\n\nThe **logo of IoT Event Streaming Architecture** was **generated by ChatGPT** and is **free of copyright restrictions**. It is provided exclusively for **demonstration and educational purposes** and does not represent an actual brand or commercial entity.  \n\n## Architecture Overview\n\nThe main goals of this architecture are the following:\n\n* Ingest and store real-time data from IoT sensors located in various locations.\n* Analyze and make aggregations through rotating time windows to know average temperature by sensor or place, in addition to humidity, pressure and luminosity.\n* Store data after processing for subsequent monitoring in a NOSQL database with good performance for frequent writes for relatively homogeneous document sizes such as MongoDB.\n* Visualization of the metrics of the sensors in real time and of the performance of the components of the architecture through Grafana and the consolidated documents in MongoDB through Mongo-Express.\n\n\u003cimg width=\"auto\" src=\"./iot_streaming_architecture.png\" /\u003e\n\n\n## Used technology\n\n* Spring Boot 2.3.3 / Apache Maven 3.6.3.\n* Spring Boot Starter Actuator.\n* Kafka Streams.\n* Spring Kafka.\n* Micrometer Registry Prometheus.\n* Eclipse Paho MQTT Client.\n* Kafka Connect.\n* Kafka Rest Proxy\n* lombok.\n* Jackson.\n* NodeExporter (Exporter for machine metrics).\n* Prometheus.\n* Grafana.\n* Eclipse Mosquitto.\n* MongoDB.\n* Mongo DB Express (Web-based MongoDB admin interface, written with Node.js and express).\n* Cadvisor (Analyzes resource usage and performance characteristics of running containers).\n* kafka-exporter (Kafka exporter for Prometheus).\n\n## Running Applications as Docker containers.\n\n### Rake Tasks\n\nThe available tasks are detailed below (rake --task)\n\n| Task | Description |\n| ------ | ------ |\n| check_deployment_file_task | Check Deployment File |\n| check_docker_task | Check Docker and Docker Compose Task |\n| cleaning_environment_task | Cleaning Evironment Task |\n| deploy | Deploys the IoT Event Streaming Architecture and laun... |\n| login | Authenticating with existing credentials |\n| start | Start Containers |\n| status | Status Containers |\n| stop | Stop Containers |\n| undeploy | UnDeploy IoT Event Streaming Architecture |\n\n\nTo start the platform make sure you have Ruby installed, go to the root directory of the project and run the `rake deploy` task, this task will carry out a series of preliminary checks, discard images and volumes that are no longer necessary and also proceed to download all the images and the initialization of the containers.\n\n  ### Containers Ports\n\n| Container | Port |\n| ------ | ------ |\n| kafka-topics-ui | localhost:8081 |\n| kafka-connect-ui | localhost:8082 |\n| zoonavigator-web | localhost:8083 |\n| mongo-express | localhost:8084 |\n| grafana | localhost:8085 |\n| prometheus | localhost:8086 |\n| kafka-rest-proxy | localhost:9999 |\n\n## Some screenshots\n\n### Deploy with Docker Compose.\n\n\u003cimg width=\"auto\" src=\"./screenshots/platform_containers.PNG\" /\u003e\n\n### Viewing topics through Landoop Kafka Topics UI\n\n\u003cimg width=\"auto\" src=\"./screenshots/kafka_topics_1.PNG\" /\u003e\n\u003cimg width=\"auto\" src=\"./screenshots/kafka_topics_2.PNG\" /\u003e\n\n### Viewing Connect Topology through Landoop Kafka Connect UI\n\n\u003cimg width=\"auto\" src=\"./screenshots/kafka_connect_1.PNG\" /\u003e\n\u003cimg width=\"auto\" src=\"./screenshots/kafka_connect_2.PNG\" /\u003e\n\n### Viewing Zookeeper Nodes through ZooNavigator\n\n\u003cimg width=\"auto\" src=\"./screenshots/zoonavigator_1.PNG\" /\u003e\n\n### Viewing information consolidated and processed in MongoDB through Mongo Express.\n\n\u003cimg width=\"auto\" src=\"./screenshots/mongo_express_1.PNG\" /\u003e\n\u003cimg width=\"auto\" src=\"./screenshots/mongo_express_2.PNG\" /\u003e\n\u003cimg width=\"auto\" src=\"./screenshots/mongo_express_4.PNG\" /\u003e\n\u003cimg width=\"auto\" src=\"./screenshots/mongo_express_5.PNG\" /\u003e\n\n### Viewing the metrics of the IoT sensors simulated on the platform.\n\n\u003cimg width=\"auto\" src=\"./screenshots/grafana_1.PNG\" /\u003e\n\u003cimg width=\"auto\" src=\"./screenshots/grafana_2.PNG\" /\u003e\n\n### Viewing metrics about Kafka's performance.\n\u003cimg width=\"auto\" src=\"./screenshots/grafana_3.PNG\" /\u003e\n\n### Viewing platform container metrics.\n\u003cimg width=\"auto\" src=\"./screenshots/grafana_4.PNG\" /\u003e\n\n## ⚠️ Disclaimer:\n\nThis project, **IoT Event Streaming Architecture**, has been developed **for educational purposes only**. Its primary objective is to provide a hands-on learning experience for building scalable event streaming architectures and processing real-time IoT data using technologies like **Apache Kafka**, **MQTT**, **MongoDB**, **Grafana**, and **Spring Boot**.\n\nThis repository demonstrates how to ingest, process, and store data from IoT sensors, and it does not provide access to real-world IoT data or production-grade sensor networks. The project uses **simulated data** for demonstration purposes, allowing users to explore the design and architecture of IoT solutions without the need for live sensor deployments.\n\nThe name **IoT Event Streaming Architecture** was generated during a **brainstorming session** with the help of **AI**, and any resemblance to existing brands or services is purely **coincidental**.\n\nThe **logo of IoT Event Streaming Architecture** was **generated by ChatGPT** and is **free of copyright restrictions**. It is provided exclusively for **demonstration and educational purposes** and does not represent an actual brand or commercial entity.  \n\n## Visitors Count\n\n\u003cimg width=\"auto\" src=\"https://profile-counter.glitch.me/iot_event_streaming_architecture/count.svg\" /\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsergio11%2Fiot_event_streaming_architecture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsergio11%2Fiot_event_streaming_architecture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsergio11%2Fiot_event_streaming_architecture/lists"}