{"id":30151700,"url":"https://github.com/dishabasti/real-time-analytics-pipeline","last_synced_at":"2026-05-14T23:07:39.883Z","repository":{"id":306107158,"uuid":"1017459321","full_name":"DishaBasti/real-time-analytics-pipeline","owner":"DishaBasti","description":"Live viewership analytics using Kafka, PySpark, forecasting, and Grafana dashboards","archived":false,"fork":false,"pushed_at":"2025-07-23T16:23:15.000Z","size":21,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-23T18:26:11.354Z","etag":null,"topics":["big-data-analytics","channel-viewership","data-pipeline","grafana-dashboard","kafka","kafka-streams","pyspark","real-time-analytics","spark","streaming-data","user-behavior-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DishaBasti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-10T15:05:21.000Z","updated_at":"2025-07-23T16:23:18.000Z","dependencies_parsed_at":"2025-07-23T18:26:17.724Z","dependency_job_id":null,"html_url":"https://github.com/DishaBasti/real-time-analytics-pipeline","commit_stats":null,"previous_names":["dishabasti/real-time-analytics-pipeline"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/DishaBasti/real-time-analytics-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DishaBasti%2Freal-time-analytics-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DishaBasti%2Freal-time-analytics-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DishaBasti%2Freal-time-analytics-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DishaBasti%2Freal-time-analytics-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DishaBasti","download_url":"https://codeload.github.com/DishaBasti/real-time-analytics-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DishaBasti%2Freal-time-analytics-pipeline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269873158,"owners_count":24488993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data-analytics","channel-viewership","data-pipeline","grafana-dashboard","kafka","kafka-streams","pyspark","real-time-analytics","spark","streaming-data","user-behavior-analysis"],"created_at":"2025-08-11T11:02:25.081Z","updated_at":"2026-05-14T23:07:39.806Z","avatar_url":"https://github.com/DishaBasti.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# 📊 Real-Time Big Data Analytics with Kafka, PySpark \u0026 Grafana  \n\nThis project demonstrates **real-time data analytics** using two approaches:  \n✔ **Kafka + PySpark + Grafana** – A real-time distributed streaming pipeline (PC1 → PC2).  \n✔ **Prometheus + Grafana** – A simulation-based monitoring pipeline.  \n\nIt also includes **offline PySpark analytics** in Google Colab for detailed insights.  \n\n\n## ✅ Table of Contents  \n1. [Project Overview](#project-overview)  \n2. [Architecture](#architecture)  \n3. [Approach 1: Kafka + PySpark + Grafana](#approach-1-kafka--pyspark--grafana)  \n4. [Approach 2: Prometheus + Grafana Simulation](#approach-2-prometheus--grafana-simulation)  \n5. [Offline Analytics in Google Colab](#offline-analytics-in-google-colab)  \n6. [Folder Structure](#folder-structure)  \n7. [Setup Instructions](#setup-instructions)  \n   - [Approach 1: Kafka + PySpark + Grafana (Distributed Setup)](#approach-1-setup-kafka--pyspark--grafana)  \n   - [Approach 2: Prometheus + Grafana Simulation](#approach-2-setup-prometheus--grafana-simulation)  \n8. [Screenshots](#screenshots)  \n9. [Future Enhancements](#future-enhancements)  \n\n\n## 📌 Project Overview  \nThe goal of this project is to **simulate real-time TV channel viewership data** from a CSV and analyze it using:  \n- **Kafka** for distributed event streaming.  \n- **PySpark Structured Streaming** for real-time analytics.  \n- **Prometheus \u0026 Grafana** for monitoring and visualization.  \n- **PostgreSQL or InfluxDB** for optional storage of aggregated results.  \n\n**Dataset Columns**:  \n```\nEvent_ID | Event_Type | User_ID | City | State | User_Type | Timestamp | Channel | Program | Channel_Type | View_Min | Session_Dur | Preferred_Time | Region\n```\n\n\n## Architecture  \n\n### ✅ Approach 1: Kafka + PySpark + Grafana  \n![Kafka Pipeline Architecture](images/kafka_pipeline.png)\n\n```\nPC1 (Producer) → Kafka Broker → PC2 (PySpark Consumer) → Database → Grafana Dashboards\n```\n\n\n### ✅ Approach 2: Prometheus + Grafana Simulation  \n![Prometheus Pipeline Architecture](images/prometheus_pipeline.png)\n\n```\nCSV Dataset → Prometheus Exporter (Python) → Prometheus → Grafana Dashboards\n```\n\n\n## ✅ Approach 1: Kafka + PySpark + Grafana  \n\n### 🔍 How It Works Across Two PCs  \n- **PC 1**: Kafka Producer streams CSV data into **Kafka Topics**.  \n- **PC 2**: PySpark Consumer reads Kafka topics in **real-time**, performs aggregations, and writes to DB or console.  \n- **Grafana**: Connects to DB or Spark output for visualization.  \n\n### Key Features  \n✔ Real-time ingestion \u0026 processing  \n✔ Distributed setup for scalability  \n✔ Visualization in Grafana  \n\n\n## ✅ Approach 2: Prometheus + Grafana Simulation  \nThis approach simulates real-time metrics using **Prometheus exporter** when you don’t have a full Kafka cluster setup.  \n\n- Reads CSV rows sequentially with a time delay.  \n- Exposes metrics at `http://localhost:8000/metrics`.  \n- Grafana pulls data from Prometheus and renders dashboards.  \n\n\n## ✅ Offline Analytics in Google Colab  \nDue to Spark setup constraints locally, detailed analytics were done in **Colab**:  \n\n- **Notebook:** [Analytics \u0026 Visualizations](https://colab.research.google.com/drive/1t2X3r2MHtKUaQ4ilkXLT3vJh5Q8eIaTT?usp=sharing)  \n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1t2X3r2MHtKUaQ4ilkXLT3vJh5Q8eIaTT?usp=sharing)\n\n\n## 📂 Folder Structure  \n\n```\nreal-time-bda-pipeline/\n ┣ 📜 README.md\n ┣ 📜 producer.py             # Kafka Producer (simulated streaming)\n ┣ 📜 subscriber.py           # PySpark Consumer with real-time analytics\n ┣ 📜 prometheus_simulator.py # Prometheus metrics exporter\n ┣ 📜 sample_data.csv         # Example dataset\n ┣ 📂 notebooks\n ┃   ┣ pyspark_analytics.ipynb\n ┣ 📜 requirements.txt\n ┗ 📜 docker-compose.yml      # For Kafka + Zookeeper setup\n```\n\n\n## ✅ Setup Instructions  \n\n### 🔹 **Install Dependencies**  \n```bash\npip install -r requirements.txt\n```\n\n### ✅ Approach 1 Setup: Kafka + PySpark + Grafana  \n\n#### **On PC 1 (Kafka Producer)**  \n1. Start Kafka using Docker Compose:  \n```bash\ndocker-compose up -d\n```\n2. Run the Producer script:  \n```bash\npython producer.py\n```\n3. Producer will stream data from `sample_data.csv` to Kafka topics.  \n\n#### **On PC 2 (PySpark Consumer)**  \n1. Ensure PC 2 can access PC 1's IP and Kafka port (9092).  \n2. Edit `subscriber.py` with PC 1's Kafka IP:  \n```python\nkafka_bootstrap_servers = \"PC1_IP:9092\"\n```\n3. Start PySpark consumer:  \n```bash\nspark-submit subscriber.py\n```\n4. Processed data can be viewed on the console or written to DB.  \n\n#### **Grafana**  \n- Connect Grafana to **PostgreSQL** or **InfluxDB** where processed results are stored.  \n- Import dashboards for visualization.  \n\n### ✅ Approach 2 Setup: Prometheus + Grafana Simulation  \n\n1. Start Prometheus exporter:  \n```bash\npython prometheus_simulator.py\n```\n2. Prometheus scrapes metrics from `http://localhost:8000/metrics`.  \n3. In Grafana:  \n   - Add Prometheus as a data source.  \n   - Build dashboards to visualize real-time metrics.  \n\n## Screenshots  \n✔ **Kafka Console Output** – Top Channels, Regional Trends  \n✔ **Grafana Dashboard for Kafka Pipeline**  \n✔ **Grafana Dashboard for Prometheus Simulation**  \n✔ **Colab Visualizations**  \n\n\n## Future Enhancements  \n- Integrate **forecasting models** (Prophet, ARIMA) into PySpark streaming.  \n- Store real-time processed data in **InfluxDB** for time-series analytics.  \n- Deploy pipeline using **Kubernetes** for scalability.\n\n## Author \n**Disha S Basti**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdishabasti%2Freal-time-analytics-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdishabasti%2Freal-time-analytics-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdishabasti%2Freal-time-analytics-pipeline/lists"}