{"id":23673587,"url":"https://github.com/j-kulp/iot_pipeline","last_synced_at":"2026-05-12T07:38:57.132Z","repository":{"id":269589659,"uuid":"907861634","full_name":"j-kulp/IoT_pipeline","owner":"j-kulp","description":"A project to display real-time IoT sensor data, scalable and secure","archived":false,"fork":false,"pushed_at":"2024-12-24T18:54:56.000Z","size":20,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-19T17:21:19.838Z","etag":null,"topics":["bme280","dht11","esp32","fastapi","grafana-dashboard","grafana-influxdb","influxdb","mqtt-protocol"],"latest_commit_sha":null,"homepage":"https://j-kulp.github.io/portfolio/ESP32%20Pipeline.html","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/j-kulp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-24T15:20:07.000Z","updated_at":"2025-02-08T22:40:54.000Z","dependencies_parsed_at":"2024-12-24T17:32:47.055Z","dependency_job_id":null,"html_url":"https://github.com/j-kulp/IoT_pipeline","commit_stats":null,"previous_names":["j-kulp/iot_pipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-kulp%2FIoT_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-kulp%2FIoT_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-kulp%2FIoT_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/j-kulp%2FIoT_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/j-kulp","download_url":"https://codeload.github.com/j-kulp/IoT_pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239699593,"owners_count":19682577,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bme280","dht11","esp32","fastapi","grafana-dashboard","grafana-influxdb","influxdb","mqtt-protocol"],"created_at":"2024-12-29T12:30:00.366Z","updated_at":"2025-12-21T09:30:15.836Z","avatar_url":"https://github.com/j-kulp.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ESP32 MQTT, FastAPI, InfluxDB and Grafana Data Pipeline\n\nThis project involves using an **ESP32** to collect sensor data (e.g., from DHT11 or BME280 sensors), send it to an MQTT broker, and insert the data into an **InfluxDB** database. \nThe data is visualized in **Grafana** and can be tested via a **FastAPI** endpoint.\n\nThe project divides into two sections: the **ESP32 MQTT publisher** (```ESP32_MQTT_BM.ino```), and the **MQTT subscriber that pushes to InfluxDB** (```app.py```)\n\n![images/pipeline-diagram.png](https://github.com/j-kulp/portfolio/blob/e5d8aaf0b053426ca691c5e1b8806b83e4c09623/images/flow.png)\n\nI suggest following the order of installation here, starting with the ESP32 before finally moving on to the MQTT subscriber section.\n\n## Requirements\n\n### Hardware\n- **ESP32 microcontroller**\n  - **Sensors** like DHT11 or BME280 for environmental data\n- **MQTT Broker** (can be a cloud or local broker, e.g., Mosquitto)\n\n### Software \u0026 Libraries\n\n**ESP32**:\nArduino IDE or PlatformIO\nLibraries:\n  - **WiFi** (comes with the ESP32 Arduino core)\n  - **PubSubClient** (for MQTT communication)\n  - **ArduinoJson** (for JSON handling)\n  - **Adafruit BME280** (for sensor data collection)\n\n**Backend**:\n**Python 3.7+**\n- **FastAPI** (for serving the API)\n- **InfluxDB** (for storing sensor data)\n- **paho-mqtt** (for connecting to the MQTT broker)\n- **InfluxDB Client** (for writing data to InfluxDB)\n\n**Frontend**:\n- **Grafana** (for visualization)\n\n  \n=====================================================================\n\n=====================================================================\n\n## Installation and Setup - ESP32 with C++\n\n### Hardware Requirements\n- ESP32 development board (e.g., ESP32 DevKit v1)\n- BME280 sensor (can be connected via SPI)\n- Jumper wires for connections\n- Power supply for the ESP32\n\n\n#### Wiring Diagram\n\n|ESP32 Pin  |\tBME280 Pin | Description |\n|-----------|------------|---------|\n|GPIO 5\t| CS (Chip Select) |\tSPI Chip Select |\n|GPIO 18 | SCK (Clock) |\tSPI Clock |\n|GPIO 23 |\tMOSI\t| SPI MOSI (Master Out Slave In) |\n|GPIO 19 | MISO |\tSPI MISO (Master In Slave Out) |\n\nNote: The ESP32 uses SPI to communicate with the BME280 sensor in this project. Ensure the wiring matches the above table.\n\n### Software Requirements\nArduino IDE (or PlatformIO)\nLibraries:\n  - **WiFi** (comes with the ESP32 Arduino core)\n  - **PubSubClient** (for MQTT communication)\n  - **ArduinoJson** (for JSON handling)\n  - **Adafruit BME280** (for sensor data collection)\n\nYou can install the necessary libraries through the Arduino Library Manager or by using the Library Manager in the Arduino IDE.\n\n### Installation and Setup\n\n### 1: Install Arduino IDE and Configure ESP32\n- Install the **Arduino IDE**.\n- Install the ESP32 board support in the Arduino IDE:\n  - Open Arduino IDE\n  - Go to File \u003e Preferences and add the following URL to the \"Additional Boards Manager URLs\" field:\n    ``` https://dl.espressif.com/dl/package_esp32_index.json ```\n  - Go to Tools \u003e Board \u003e Boards Manager, search for ESP32, and install it.\n\n### 2: Install Required Libraries\n  - Open Arduino IDE \u003e Sketch \u003e Include Library \u003e Manage Libraries.\n  - Install the following libraries:\n    - **WiFi** (for connecting the ESP32 to a Wi-Fi network)\n    - **PubSubClient** (for MQTT communication)\n    - **ArduinoJson** (for JSON data handling)\n    - **Adafruit BME280** (for reading data from the BME280 sensor)\n\n### 3: Upload the Code to the ESP32\n- Connect the ESP32 board to your computer via USB.\n- Open the provided code in the Arduino IDE.\n- Configure the Wi-Fi and MQTT settings:\n  - Set your Wi-Fi SSID and password in the code.\n  - Set your MQTT broker's IP_ADDRESS, username, and password.\n- Select the appropriate Port and ESP32 board from Tools \u003e Board.\n- Upload the code to the ESP32.\n\n### 4: Monitor the Serial Output\n- Open the Serial Monitor in Arduino IDE to view connection status and data being published to the MQTT broker.\n- Set the baud rate to 115200 for correct serial communication.\n\n### 5. MQTT Setup\nYou can use any MQTT broker, such as Mosquitto, HiveMQ, or a cloud-based service like AWS IoT or ThingSpeak. I used Mosquitto.\nThe default topic where the sensor data is published is ```sensor/data```, but you can change this in the code.\n\nExample MQTT Message:\n\n``` json\n{\n  \"deviceID\": \"DEVICE_NAME\",\n  \"sensorName\": \"BME280\",\n  \"temperature\": 25.3,\n  \"humidity\": 60.5,\n  \"pressure\": 1012.3\n}\n```\n\n### Customizing the Code\n- **Device Name**: Update the device variable to your preferred device name.\n- **Sensor Type**: The current code is configured for a BME280 sensor, but it can easily be adapted to work with other sensors by modifying the sensorName and sensor-specific code.\n- **Publish Interval**: Adjust the publishInterval to control how often data is sent to the MQTT broker.\n\n### Troubleshooting\n\n- **ESP32 Not Connecting to Wi-Fi**: \n  - Double-check the SSID and password in the code.\n  - Ensure your ESP32 is within range of the Wi-Fi network and has adequate power.\n  \n- **MQTT Connection Issues**:\n  - Ensure the MQTT broker is running and accessible.\n  - Verify the MQTT username and password are correct.\n  - Check the MQTT topic name to ensure the ESP32 is publishing to the correct one.\n\n- **Sensor Reading Failures**:\n  - Ensure the BME280 sensor is correctly wired.\n  - Verify the sensor is powered properly (3.3V and GND).\n  - Check for any loose connections or wiring issues.\n\n\n=====================================================================\n\n=====================================================================\n\n## Installation and Setup - Backend/Laptop in Python\n\n### 1. Install Mosquitto MQTT Broker\n\nMosquitto is an open-source MQTT broker that you can run locally or on a server to handle messages between the ESP32 and the FastAPI server. Here's how to install and start the service:\n\n- **Download and install Mosquitto**:\n  - You can download the latest version of Mosquitto from the official site: [Mosquitto Download](https://mosquitto.org/download/).\n  - Follow the installation instructions for your operating system.\n\n- **Start the Mosquitto MQTT Broker**:\n  Once Mosquitto is installed, open a terminal/command prompt and run the following command to start the broker:\n\n  ```bash\n  net start mosquitto\n  ```\n  This command starts the Mosquitto MQTT broker on your local machine.\n\n### 2. Install InfluxDB\nInfluxDB is a time-series database that will store the sensor data from the ESP32. To set up InfluxDB, follow these steps:\n\n- **Download and install InfluxDB**:\n  - Download the latest version of InfluxDB from the official site: [InfluxDB Download](https://www.influxdata.com/influxdata-downloads/)\n  - Follow the installation instructions for your operating system.\n\n- Start the InfluxDB service: Navigate to the directory where InfluxDB is installed and run the following command to start InfluxDB:\n\n  ``` bash\n  cd C:\\Program Files\\InfluxData\\influxdb\n  influxd\n  ```\n  \n  This starts the InfluxDB service, and it will listen for incoming requests on the default port (8086).\n  You can access the InfluxDB service and dashboard at http://localhost:8086\n\n### 3. Install and Run FastAPI Server\nThe FastAPI server handles the incoming MQTT messages, processes them, and writes the data to InfluxDB.\n\n- Install FastAPI and dependencies: Install FastAPI and the necessary libraries using pip:\n\n``` bash\npip install fastapi paho-mqtt influxdb-client uvicorn\n```\n\n- Set up the FastAPI project: Clone or create a directory for your FastAPI app and save the provided Python code (e.g., app.py) inside the project folder.\n  - Modify the code to match your MQTT and InfluxDB configurations (e.g., MQTT username/password, InfluxDB token).\n\n- Start the FastAPI server: In the FastAPI project directory, run the following command to start the server using Uvicorn:\n\n``` bash\ncd C:\\Users\\...\\MQTT\nuvicorn app:app --reload\n```\n\nThis will start the FastAPI server on http://localhost:8000.\n\n### 4. Install and Run Grafana\nGrafana is used to visualize the sensor data stored in InfluxDB.\n\n- Download and install Grafana:\n  - Download the latest open-source version of Grafana from the official site [Grafana Download](https://grafana.com/grafana/download?pg=oss-graf\u0026plcmt=resources\u0026edition=oss)\n  - Follow the installation instructions for your operating system.\n\n- Start the Grafana service: After installing Grafana, navigate to the installation directory and run the following command:\n\n``` bash\ncd C:\\Program Files\\GrafanaLabs\\grafana\\bin\ngrafana-server.exe\n```\n\nThis will start the Grafana server, and you can access the Grafana dashboard at http://localhost:3000.\n\n- Log in to Grafana: Use the default credentials (admin/admin) to log in to Grafana. You will be prompted to change the password.\n\n- Add InfluxDB as a data source: Once logged in to Grafana, follow these steps to add InfluxDB as a data source:\n  - Go to Configuration \u003e Data Sources \u003e Add data source.\n  - Select InfluxDB from the list.\n  - Set the URL to http://localhost:8086.\n  - Provide your InfluxDB Token, and choose the bucket (Test or the bucket name you created in InfluxDB).\n  - Click Save \u0026 Test to verify the connection.\n\n- Create a dashboard in Grafana: After adding InfluxDB as a data source, you can create a dashboard to visualize your sensor data:\n  - Go to Create \u003e Dashboard and add new panels to display metrics such as temperature, humidity, etc.\n\n### Troubleshooting\n#### Common Issues:\n- MQTT connection issues:\n  - Security may flag 64b install, try 32b.\n  - Verify that your broker IP, username, and password are correctly configured.\n- InfluxDB connection errors: Double-check the InfluxDB URL, token, and bucket settings.\n- FastAPI server errors: Check the logs for details, ensure all dependencies are installed, and make sure the correct IP addresses/ports are configured.\n\n## Results\nYou should have something that starts to resemble this!\n\n![images/pipeline-diagram.png](https://github.com/j-kulp/portfolio/blob/e5d8aaf0b053426ca691c5e1b8806b83e4c09623/images/grafana.png)\n\n\n=====================================================================\n\n=====================================================================\n\n## License\nThis project is licensed under the MIT License - see the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-kulp%2Fiot_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fj-kulp%2Fiot_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fj-kulp%2Fiot_pipeline/lists"}