{"id":14982355,"url":"https://github.com/aymane-maghouti/big-data-project","last_synced_at":"2025-10-29T12:31:37.184Z","repository":{"id":233582161,"uuid":"780710789","full_name":"aymane-maghouti/Big-Data-Project","owner":"aymane-maghouti","description":"This project aims to predict smartphone prices using a combination of batch and stream processing techniques in a Big Data environment. The architecture follows the Lambda Architecture pattern, providing both real-time and batch processing capabilities to users.","archived":false,"fork":false,"pushed_at":"2024-04-15T09:08:08.000Z","size":983,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-02T01:31:47.798Z","etag":null,"topics":["apache-airflow","apache-kafka","apache-spark","batch-processing","big-data-projects","hbase","hdfs","ingestion","java","lambda-architecture","machine-learning","postgresql-database","powerbi","pyspark","python","spring-boot","streaming"],"latest_commit_sha":null,"homepage":"https://youtu.be/iClZyC_TZyA","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aymane-maghouti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-02T02:33:48.000Z","updated_at":"2024-12-08T08:19:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"bcec5db7-9d26-4ee5-a8f9-14ddc6643d50","html_url":"https://github.com/aymane-maghouti/Big-Data-Project","commit_stats":{"total_commits":19,"total_committers":2,"mean_commits":9.5,"dds":"0.26315789473684215","last_synced_commit":"25ea72669b8593eb8931f9e90313df25750e0b9e"},"previous_names":["aymane-maghouti/big-data-project"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aymane-maghouti%2FBig-Data-Project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aymane-maghouti%2FBig-Data-Project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aymane-maghouti%2FBig-Data-Project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aymane-maghouti%2FBig-Data-Project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aymane-maghouti","download_url":"https://codeload.github.com/aymane-maghouti/Big-Data-Project/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238825719,"owners_count":19537115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","apache-kafka","apache-spark","batch-processing","big-data-projects","hbase","hdfs","ingestion","java","lambda-architecture","machine-learning","postgresql-database","powerbi","pyspark","python","spring-boot","streaming"],"created_at":"2024-09-24T14:05:15.145Z","updated_at":"2025-10-29T12:31:31.807Z","avatar_url":"https://github.com/aymane-maghouti.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Smartphone Price Prediction in Big Data Environment\n\n## Table of Contents\n1. [Project Overview](#1-project-overview)\n2. [Technologies Used](#2-technologies-used)\n3. [Architecture](#3-architecture)\n4. [Repository Structure](#4-repository-structure)\n5. [Software Requirements for Running the Project](#5-software-requirements-for-running-the-project)\n6. [How to Run](#6-how-to-run)\n7. [Dashboards](#7-dashboards)\n8. [Acknowledgments](#8-acknowledgments)\n9. [Conclusion](#9-conclusion)\n10. [Contacts](#10-contacts)\n\n## 1. Project Overview\nThis project aims to predict smartphone prices using a combination of batch and stream processing techniques in a Big Data environment. The architecture follows the Lambda Architecture pattern, providing both real-time and batch processing capabilities to users.\n\n## 2. Technologies Used\n* **Ingestion Layer:** Apache Kafka (message broker)\n* **Stream Layer:** XGBoost (machine learning model), Apache HBase (real-time View)\n* **Batch Layer:** Apache Spark (data processing framework), Apache Airflow (workflow orchestration), PostgreSQL (data warehouse (Batch View))\n* **Visualization:** Spring Boot (web application framework), Power BI (interactive dashboards)\n\n\n## 3. Architecture\n\n- Here is the architecture :\n ![architecture](images/architecture.png)\n\n\nThe project architecture consists of five main layers: the ingestion layer, the batch layer, the stream layer, the serving layer and the visualization layer.\n\n### Ingestion Layer\n- **Apache Kafka**: Utilized for real-time data ingestion from an API providing smartphone data.\n   - **Consumer**: Collects data from the API and feeds it into the stream and batch layer.\n\n### Stream Layer\n- **Producer**: A machine learning model developed using XGBoost to estimate smartphone prices. This model runs in real-time and stores predictions in a realtime view. (details about the model \u003ca href=\"https://github.com/aymane-maghouti/Sentiment-Analysis-for-Jumia-Reviews-and-Smartphone-Price-Prediction-System\" target=\"_blank\"\u003ehere\u003c/a\u003e )\n\n### Batch Layer\n- **HDFS**: Data from the API is stored in HDFS as part of the data lake solution.\n - **PySpark**: Performs data transformation on stored data using PySpark.\n - **Apache Airflow**: Orchestrates the batch processing workflow.\n### Serving Layer\n- **Realtime View**: Implemented using HBase to provide real-time access to predicted smartphone prices.\n- **Batch View**: Transformed data is stored in PostgreSQL, as the data warehouse solution.\n### Visualization Layer\n- **Spring Boot Web Application**: Provides a user interface to view real-time smartphone prices.\n- **Power BI Dashboard**: Provides batch users with a visualization of processed data.\n\n\n## 4. Repository Structure\nThe repository is organized as follows:\n```batch \nBig-Data-Project:.\n|   README.md\n|\n+---images\n|       architecture.png\n|       dashboard_phone.png\n|       run_web_app.png\n|       spring_boot_web_app.png\n|\n\\---Main\n    |   commands.sh\n    |   Dashboard.pbix\n    |\n    +---.idea\n    |       workspace.xml\n    |\n    +---Lambda\n    |   |   docker-compose.yaml\n    |   |   producer.py\n    |   |   transform.py\n    |   |\n    |   +---.idea\n    |   |   |   .gitignore\n    |   |   |   .name\n    |   |   |   misc.xml\n    |   |   |   modules.xml\n    |   |   |   price prediction (big data envirnment).iml\n    |   |   |   vcs.xml\n    |   |   |   workspace.xml\n    |   |   |\n    |   |   \\---inspectionProfiles\n    |   |           profiles_settings.xml\n    |   |\n    |   +---Batch_layer\n    |   |   |   batch_layer.py\n    |   |   |   batch_pipeline.py\n    |   |   |   HDFS_consumer.py\n    |   |   |   put_data_hdfs.py\n    |   |   |   save_data_postgresql.py\n    |   |   |   spark_tranformation.py\n    |   |   |   __init__.py\n    |   |   |\n    |   |   +---dags\n    |   |   |       syc_with_Airflow.py\n    |   |   |       __init__.py\n    |   |   |\n    |   |   \\---__pycache__\n    |   |           batch_layer.cpython-310.pyc\n    |   |           HDFS_consumer.cpython-310.pyc\n    |   |           put_data_hdfs.cpython-310.pyc\n    |   |           save_data_postgresql.cpython-310.pyc\n    |   |           spark_tranformation.cpython-310.pyc\n    |   |           __init__.cpython-310.pyc\n    |   |\n    |   +---ML_operations\n    |   |   |   xgb_model.pkl\n    |   |   |\n    |   |   \\---__pycache__\n    |   +---real_time_web_app(Flask)\n    |   |   |   app.py\n    |   |   |   get_Data_from_hbase.py\n    |   |   |\n    |   |   +---static\n    |   |   |   +---css\n    |   |   |   |       style.css\n    |   |   |   |\n    |   |   |   \\---js\n    |   |   |           script.js\n    |   |   |\n    |   |   +---templates\n    |   |   |       index.html\n    |   |   |\n    |   |   \\---__pycache__\n    |   |           get_Data_from_hbase.cpython-310.pyc\n    |   |\n    |   +---Stream_data\n    |   |   |   stream_data.csv\n    |   |   |   stream_data.py\n    |   |   |\n    |   |   \\---__pycache__\n    |   +---Stream_layer\n    |   |       insert_data_hbase.py\n    |   |       ML_consumer.py\n    |   |       stream_pipeline.py\n    |   |       __init__.py\n    |   |\n    |   \\---__pycache__\n    |           producer.cpython-310.pyc\n    |           transform.cpython-310.pyc\n    |\n    \\---real_time_app(Spring boot)\n        |   .classpath\n        |   .gitignore\n        |   .project\n        |   HELP.md\n        |   mvnw\n        |   mvnw.cmd\n        |   pom.xml\n        |\n        +---.mvn\n        |   \\---wrapper\n        |           maven-wrapper.jar\n        |           maven-wrapper.properties\n        |\n        +---.settings\n        |       org.eclipse.core.resources.prefs\n        |       org.eclipse.jdt.core.prefs\n        |       org.eclipse.m2e.core.prefs\n        |\n        +---src\n        |   +---main\n        |   |   +---java\n        |   |   |   \\---com\n        |   |   |       \\---example\n        |   |   |           \\---demo\n        |   |   |               |   RealTimeAppApplication.java\n        |   |   |               |\n        |   |   |               +---controller\n        |   |   |               |       IndexController.java\n        |   |   |               |\n        |   |   |               \\---service\n        |   |   |                       HbaseService.java\n        |   |   |\n        |   |   \\---resources\n        |   |       |   application.properties\n        |   |       |\n        |   |       +---static\n        |   |       |   +---css\n        |   |       |   |       style.css\n        |   |       |   |\n        |   |       |   \\---js\n        |   |       |           script.js\n        |   |       |\n        |   |       \\---templates\n        |   |               index.html\n        |   |\n        |   \\---test\n        |       \\---java\n        |           \\---com\n        |               \\---example\n        |                   \\---demo\n        |                           RealTimeAppApplicationTests.java\n        |\n        \\---target\n            +---classes\n            |   |   application.properties\n            |   |\n            |   +---com\n            |   |   \\---example\n            |   |       \\---demo\n            |   |           |   RealTimeAppApplication.class\n            |   |           |\n            |   |           +---controller\n            |   |           |       IndexController.class\n            |   |           |\n            |   |           \\---service\n            |   |                   HbaseService.class\n            |   |\n            |   +---META-INF\n            |   |   |   MANIFEST.MF\n            |   |   |\n            |   |   \\---maven\n            |   |       \\---com.example\n            |   |           \\---real_time_app\n            |   |                   pom.properties\n            |   |                   pom.xml\n            |   |\n            |   +---static\n            |   |   +---css\n            |   |   |       style.css\n            |   |   |\n            |   |   \\---js\n            |   |           script.js\n            |   |\n            |   \\---templates\n            |           index.html\n            |\n            \\---test-classes\n                \\---com\n                    \\---example\n                        \\---demo\n                                RealTimeAppApplicationTests.class\n\n\n```\n\n## 5. Software Requirements for Running the Project\n\nThis project requires the following software to be installed and configured on your system:\n\n**Big Data Stack:**\n\n* **Apache Kafka (version 2.6.0)**\n* **Apache HBase (version 1.2.6)** \n* **Apache Hadoop (version 2.7.0)** \n* **Apache Spark (version 3.3.4)** \n* **PostgreSQL database**\n\n**Programming Languages and Frameworks:**\n\n* **Python (version 3.10.x or later)** \n* **Java 17 (or compatible version)** \n* **Spring Boot** \n\n**Machine Learning Library:**\n\n* **XGBoost** \n\n**Additional Tools:**\n\n* **Apache Airflow** \n* **Power BI Desktop** \n\n\nBy installing and configuring these tools, you will have the necessary environment to run this project and leverage its real-time and batch processing capabilities for smartphone price prediction and analysis.\n\n## 6. How to Run\nTo set up and run the project locally, follow these steps:\n\n  - Clone the repository:\n   ```bash\n   git clone https://github.com/aymane-maghouti/Big-Data-Project\n   ```\n\n\n#### **1. Stream Layer**\n   - Start Apache zookeeper\n\n   ```batch \nzookeeper-server-start.bat C:/kafka_2.13_2.6.0/config/zookeeper.properties\n```\n   - Start Kafka server\n\n   ```batch \nkafka-server-start.bat C:/kafka_2.13_2.6.0/config/server.properties\n```\n   - Create Kafka topic\n\n   ```batch \nkafka-topics.bat --create --topic smartphoneTopic --bootstrap-server localhost:9092\n```\n\n  - Run the kafka producer\n\n   ```batch \nkafka-console-producer.bat --topic smartphoneTopic --bootstrap-server localhost:9092\n```\n\n  - Run the kafka consumer\n\n   ```batch \nkafka-console-consumer.bat --topic smartphoneTopic --from-beginning --bootstrap-server localhost:9092\n```\n\n  - Start HDFS and yarn (start-all or start-dfs and start-yarn)\n\n   ```batch \nstart-all  \n```\n   - Start Hbase\n   ```batch \nstart-hbase  \n```\n   - Run thrift server (for Hbase)\n   ```batch \nhbase thrift start\n```\n\nafter all this run `stream_pipeline.py` script.\n\nand then open the spring boot appliation in your idea and run  it (you can access to the web app locally on  `localhost:8081/`)\n\n---\n\n![spring_boot](images/run_web_app.png)\n\n\nnote that there is another version of the web app developed using Flask micro-framework(watch the demo video for mor details)\n\n#### **2. Batch Layer**\n   - Start the Apache Airflow instance: \n\n   ```batch \ndocker-compose up -d\n```\n   Access the Apache Airflow web UI (localhost:8080) and run the DAG\n   - Start Apache Spark\n\n   ```batch \nspark-shell\n```\n\n   - Start Apache zookeeper\n\n   ```batch \nzookeeper-server-start.bat C:/kafka_2.13_2.6.0/config/zookeeper.properties\n```\n   - Start Kafka server\n\n   ```batch \nkafka-server-start.bat C:/kafka_2.13_2.6.0/config/server.properties\n```\n\n  - Run the kafka producer\n\n   ```batch \nkafka-console-producer.bat --topic smartphoneTopic --bootstrap-server localhost:9092\n```\n\n  - Run the kafka consumer\n\n   ```batch \nkafka-console-consumer.bat --topic smartphoneTopic --from-beginning --bootstrap-server localhost:9092\n```\n\n  - Run HDFS and yarn (start-all or start-dfs and start-yarn)\n\n   ```batch \nstart-all  \n```\n   - Open power BI file `dashboard.pbix` attached with this project \n\nafter all this run `syc_with_Airflow.py` script.\n\n\n## 7. Dashboards\n\nThis project utilizes two dashboards to visualize smartphone price predictions and historical data:\n\n#### **1. Real-Time Dashboard (Spring Boot Application):**\n\n- This dashboard is built using a Spring Boot web application.\n- It displays the **predicted price of smartphones in real-time**.\n- Users can access this dashboard through a web interface. \n\n\nHere is the UI of th Spring Boot web application:\n\n\n![spring_boot_web_ap](images/spring_boot_web_app.png)\n\n\n#### **2. Batch Dashboard (Power BI):**\n\n- This dashboard leverages Power BI for interactive data exploration.\n- It provides insights into **historical smartphone price trends**.\n- This dashboard is designed for batch users interested in historical analysis.\n\n\nHere is the  Dashboard created in Power BI:\n\n![Phone Dashboard](images/dashboard_phone.png)\n\n\n## 8. Acknowledgments\n\n- Special thanks to the open-source communities behind `Python`, `Kafka`, `HDFS` , `Spark`,`Hbase`,`Spring Boot`and `Airflow`\n\n## 9. Conclusion\n\n- This big data architecture effectively predicts smartphone prices in real-time and provides historical analysis capabilities. The Lambda architecture facilitates efficient stream processing for real-time predictions using XGBoost and HBase, while Apache Airflow orchestrates batch processing with Spark to populate the PostgreSQL data warehouse for historical insights. This solution empowers real-time and batch users with valuable price information, enabling data-driven decision-making.\n\nyou can watch the demo video \u003ca href=\"https://youtu.be/iClZyC_TZyA\" target=\"_blank\"\u003ehere\u003c/a\u003e \n\n## 10. Contacts\n\nFor any inquiries or further information, please contact:\n- **Name:** Aymane Maghouti\n- **Email:** aymanemaghouti16@gmail.com\n- **LinkedIn:** \u003ca href=\"https://www.linkedin.com/in/aymane-maghouti/\" target=\"_blank\"\u003eAymane Maghouti\u003c/a\u003e\u003cbr\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faymane-maghouti%2Fbig-data-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faymane-maghouti%2Fbig-data-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faymane-maghouti%2Fbig-data-project/lists"}