{"id":21089065,"url":"https://github.com/TouradBaba/sensors_data_streaming","last_synced_at":"2025-05-16T11:32:51.344Z","repository":{"id":266170237,"uuid":"894194700","full_name":"TouradBaba/sensors_data_streaming","owner":"TouradBaba","description":"A real-time data streaming pipeline that collects sensor data, streams it through Kafka, and stores it in InfluxDB and MongoDB for analysis. The system is containerized with Docker and includes Flask APIs for database interaction and Grafana for visualization.","archived":false,"fork":false,"pushed_at":"2024-12-13T23:19:44.000Z","size":3511,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-02T06:44:51.562Z","etag":null,"topics":["api","docker","docker-compose","flask","grafana","influxdb","kafka","mongodb","sensors","websocket"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TouradBaba.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-25T23:26:46.000Z","updated_at":"2025-03-15T13:48:47.000Z","dependencies_parsed_at":"2024-12-14T00:32:53.964Z","dependency_job_id":null,"html_url":"https://github.com/TouradBaba/sensors_data_streaming","commit_stats":null,"previous_names":["touradbaba/iot_sensors_streaming","touradbaba/sensors_data_streaming"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TouradBaba%2Fsensors_data_streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TouradBaba%2Fsensors_data_streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TouradBaba%2Fsensors_data_streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TouradBaba%2Fsensors_data_streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TouradBaba","download_url":"https://codeload.github.com/TouradBaba/sensors_data_streaming/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254522450,"owners_count":22085117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","docker","docker-compose","flask","grafana","influxdb","kafka","mongodb","sensors","websocket"],"created_at":"2024-11-19T21:22:57.211Z","updated_at":"2025-05-16T11:32:51.338Z","avatar_url":"https://github.com/TouradBaba.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sensors Data Streaming\n\n## Overview\n\nThis project is a real-time streaming, storage, and visualization system for sensor data. It leverages **Kafka** for data streaming, **InfluxDB** (time-series database) and **MongoDB** (NoSQL database) for storage, and **Grafana** for visualization. Sensor data is captured using the [SensorServer app](https://github.com/umer0586/SensorServer) and streamed to Kafka. Consumer APIs for each database handle data ingestion and storage, with the system being fully containerized and orchestrated using **Docker Compose**.\n\n---\n\n## System Diagram\n\nThis diagram provides an overview of the system's components and data flow.\n\n![System UML Diagram](assets/UML_Diagram.png)\n\n---\n\n## Features\n\n### 1. **Real-Time Sensor Data Collection**\n- Data is collected from smartphone sensors (Accelerometer, Gyroscope, Magnetic field, Gravity) using the [SensorServer app](https://github.com/umer0586/SensorServer).\n- Sensor data is transmitted via **WebSocket** from the phone to a Kafka producer.\n\n### 2. **Data Streaming and Processing**\n- **Kafka** acts as the message broker, streaming data to designated topics for InfluxDB and MongoDB.\n- The system supports two databases:\n  - **InfluxDB**: Specialized for time-series data storage and includes native visualization capabilities.\n  - **MongoDB**: Stores unstructured sensor data flexibly.\n\n### 3. **Consumer APIs**\n- **Flask**-based APIs consume data from Kafka and store it in respective databases:\n  - **InfluxDB Consumer API**:\n    - Handles data sent to InfluxDB.\n  - **MongoDB Consumer API**:\n    - Handles data sent to MongoDB.\n- **Flask-SocketIO** is used to send log messages to the frontend, such as:\n  - Starting/Stopping the pipeline.\n  - Logs of data written to the database.\n\n### 4. **Interactive Frontend**\n- Both APIs provide a browser-based interface for pipeline management:\n  - Start/stop data pipelines.\n  - Configure database credentials and settings.\n\n### 5. **Visualization**\n- **InfluxDB**'s built-in visualization tools offer quick insights into time-series data.\n- **Grafana** supports advanced dashboards, integrating both MongoDB and InfluxDB for custom visualization.\n\n### 6. **Portability and Scalability**\n- The system is containerized, with all services defined in a `docker-compose.yml` file.\n- APIs are pushed to Docker Hub, allowing deployment across various environments.\n\n---\n\n## Repository Structure\n\n```\nAPIs_Requests/\n  influxdb_consumer_api.py             # Scripts to interact with the InfluxDB consumer API via `requests` library\n  mongodb_consumer_api.py              # Scripts to interact with the MongoDB consumer API via `requests` library\nQuery_DBs/\n  query_influxdb.py                    # Script for querying InfluxDB\n  query_mongodb.py                     # Script for querying MongoDB\nassets/\n  GIF.gif                              # Real-Time Visualization of sensor data\n  UML_Diagram.png                      # System architecture diagram\n  Grafana_Visualization.png            # Grafana dashboard visualization\n  InfluxDB_Visualization.png           # InfluxDB dashboard visualization\n  screenshot_influxdb_consumer_app.png # Screenshot of the InfluxDB Consumer App\n  screenshot_mongodb_consumer_app.png  # Screenshot of the MongoDB Consumer App\ninfluxdb_consumer_api/\n  templates/                           # HTML templates for InfluxDB Consumer API interface\n  Dockerfile                           # Dockerfile for the API\n  app.py                               # Flask API for InfluxDB Consumer\n  producer.py                          # Kafka producer for InfluxDB\n  requirements.txt                     # Dependencies for InfluxDB Consumer API\nmongodb_consumer_api/\n  templates/                           # HTML templates for MongoDB Consumer API interface\n  Dockerfile                           # Dockerfile for the API\n  app.py                               # Flask API for MongoDB Consumer\n  producer.py                          # Kafka producer for MongoDB\n  requirements.txt                     # Dependencies for MongoDB Consumer API\nRealTime_visual_sensors_data.py        # Script for visualizing sensor data in real time\ndocker-compose.yml                     # Docker Compose file for orchestration\nrequirements.txt                       # Project dependencies\n```\n\n---\n\n## Setup and Configuration\n\n### Prerequisites\n1. **SensorServer App**:\n   - Install and configure the [SensorServer app](https://github.com/umer0586/SensorServer) on your smartphone(Android Only) to stream sensor data via WebSocket.\n\n2. **Docker**:\n   - Ensure that Docker is installed and running on your system. Install Docker from [here](https://www.docker.com/products/docker-desktop).\n\n3. **Python**:\n   - This project uses Python version 3.12.6. Install Python from [here](https://www.python.org/downloads/).\n\n---\n\n### Configuration Steps\n\n#### 1. **Clone the Repository**\nClone the repository to your local machine:\n```bash\ngit clone https://github.com/TouradBaba/sensors_data_streaming.git\ncd sensors_data_streaming\n```\n\n#### 2. **Create a Virtual Environment**\nIt's recommended to create a virtual environment to manage the Python dependencies for the project:\n```bash\npython -m venv myenv\n```\nActivate the virtual environment:\n- On Windows:\n  ```bash\n  myvenv\\Scripts\\activate\n  ```\n- On macOS/Linux:\n  ```bash\n  source myenv/bin/activate\n  ```\n\n#### 3. **Install Python Dependencies**\nInstall the required Python dependencies:\n```bash\npip install -r requirements.txt\n```\n\n#### 4. **Start Docker**\nMake sure Docker is installed and running on your system.\n\n#### 5. **Starting the System**\nRun the following command to launch the system:\n```bash\ndocker-compose up --build\n```\n\n#### 6. **InfluxDB**\n- Access the InfluxDB dashboard at [http://localhost:8086](http://localhost:8086) after starting Docker Compose.\n- Default credentials:\n  - Username: `admin`\n  - Password: `admin12345`\n- Create an **API Token**:\n  - Navigate to **InfluxDB CLI → Initialize Client → Copy the Token**.\n  - Use this token in the InfluxDB Consumer API configuration.\n\n#### 7. **MongoDB**\n- MongoDB uses the default database (`sensor_data`) and collection (`sensor_data`).\n- No additional configuration is required.\n\n#### 8. **APIs**\n- APIs are accessible via localhost as configured in the `docker-compose.yml` file.\n  - **InfluxDB Consumer API**:\n    - Access the InfluxDB Consumer App at [http://localhost:5000](http://localhost:5000).\n  - **MongoDB Consumer API**: \n    - Access the MongoDB Consumer App at [http://localhost:5001](http://localhost:5001). \n- You can also interact with the APIs using scripts located in the `APIs_Requests/` directory.\n\n#### 9. **Grafana**\n- Access Grafana at [http://localhost:3000](http://localhost:3000).\n- Add data sources:\n  - **InfluxDB**: Configure it with the credentials and token created earlier.\n  - **MongoDB**: Use the **MongoDB Data Source Plugin**.\n- Create dashboards with custom visualizations.\n\n---\n\n## Workflow\n\n1. **Starting the Pipeline**:\n   - When a pipeline is started in a given consumer API (InfluxDB or MongoDB), the API first calls the **Kafka Producer**.\n   \n2. **Data Collection**:\n   - The **Kafka Producer** retrieves real-time sensor data from the **SensorServer** app, which streams data via **WebSocket**.\n\n3. **Data Streaming**:\n   - The data from the phone's sensors is sent to the corresponding **Kafka topic** (`sensor-data-influxdb` for InfluxDB or `sensor-data-mongodb` for MongoDB).\n\n4. **Data Storage**:\n   - The API then consumes data from the Kafka topic. The data is processed and then stored in the respective database (**InfluxDB** or **MongoDB**).\n\n---\n\n## Querying the Databases and Real-Time Visualization\n\nThe **Query_DBs** and **APIs_Requests** folders contain scripts that enable interaction with the databases and consumer APIs:\n\n- **Query_DBs**:\n  - The scripts `query_influxdb.py` and `query_mongodb.py` allow you to query the data stored in **InfluxDB** and **MongoDB**, respectively.\n  \n- **APIs_Requests**:\n  - The scripts `influxdb_consumer_api.py` and `mongodb_consumer_api.py` enable you to interact with the Flask-based APIs to start/stop the pipeline.\n\nThe **RealTime_visual_sensors_data.py** is for visualizing sensor data in real time. It allows you to directly monitor the sensor data streamed from the SensorServer.\n\n---\n\n## Demonstration\n\n### Real-Time Visualization of Sensor Data\n\n![Real-Time Visualization of Sensor Data](assets/GIF.gif)\n\n### InfluxDB Consumer API Interface\n\n![InfluxDB Consumer API Interface](assets/screenshot_influxdb_consumer_app.png)\n\n### MongoDB Consumer API Interface\n\n![MongoDB Consumer API Interface](assets/screenshot_mongodb_consumer_app.png)\n\n### Grafana Visualization\n\n![Grafana Visualization](assets/Grafana_Visualization.png)\n\n### InfluxDB Visualization\n\n![InfluxDB Visualization](assets/InfluxDB_Visualization.png)\n\n---\n\n## Acknowledgement\n\nI would like to acknowledge **[Umer Farooq](https://github.com/umer0586)** for developing the **SensorServer** app, which was used to collect smartphone sensor data.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTouradBaba%2Fsensors_data_streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTouradBaba%2Fsensors_data_streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTouradBaba%2Fsensors_data_streaming/lists"}