{"id":25240410,"url":"https://github.com/tashi-2004/apache-flink-spark-data-streaming","last_synced_at":"2026-02-09T23:31:43.379Z","repository":{"id":272444636,"uuid":"916612768","full_name":"tashi-2004/Apache-Flink-Spark-Data-Streaming","owner":"tashi-2004","description":"This project showcases a real-time data streaming pipeline using Apache Flink, Apache Spark, and Grafana. It streams data, stores it in Parquet format, and performs aggregations for insights, with seamless visualization via Grafana dashboards.","archived":false,"fork":false,"pushed_at":"2025-02-10T09:54:49.000Z","size":65636,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-05T20:11:27.137Z","etag":null,"topics":["apache-flink","apache-spark","data-aggregation","data-analysis","data-science","data-streaming","data-visualization","flink","flink-stream-processing","flink-streaming","grafana-dashboard","grafana-plugin","pyflink","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tashi-2004.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-14T12:49:52.000Z","updated_at":"2025-02-10T09:54:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"d278a718-9d8a-4970-ade0-5335235c8a30","html_url":"https://github.com/tashi-2004/Apache-Flink-Spark-Data-Streaming","commit_stats":null,"previous_names":["tashi-2004/data-streaming-flink-spark","tashi-2004/apache-flink-spark-data-streaming"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tashi-2004/Apache-Flink-Spark-Data-Streaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tashi-2004%2FApache-Flink-Spark-Data-Streaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tashi-2004%2FApache-Flink-Spark-Data-Streaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tashi-2004%2FApache-Flink-Spark-Data-Streaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tashi-2004%2FApache-Flink-Spark-Data-Streaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tashi-2004","download_url":"https://codeload.github.com/tashi-2004/Apache-Flink-Spark-Data-Streaming/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tashi-2004%2FApache-Flink-Spark-Data-Streaming/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259995054,"owners_count":22943323,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-flink","apache-spark","data-aggregation","data-analysis","data-science","data-streaming","data-visualization","flink","flink-stream-processing","flink-streaming","grafana-dashboard","grafana-plugin","pyflink","python3"],"created_at":"2025-02-11T19:14:08.706Z","updated_at":"2026-02-09T23:31:43.330Z","avatar_url":"https://github.com/tashi-2004.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data-Streaming-Flink-Spark\nThis repository demonstrates a real-time data streaming and persistence workflow using **Apache Flink**, **Apache Spark**, and **Grafana** for monitoring. The pipeline streams data, persists it in Parquet format, and performs aggregations for analytical insights.\n\n---\n\n## Prerequisites\n\nEnsure the following software is installed on your system:\n\n- **Apache Flink**\n- **Apache Spark**\n- **Python 3.x**\n- **Grafana**\n\nFor installation guidance, refer to tutorials available on YouTube or other online resources.\n\n---\n\n## Setup and Execution\n\nFollow these steps to set up and execute the data streaming pipeline:\n\n### 1. Start the Flink Cluster\n\n1. Navigate to the `bin` folder in Flink's directory.\n2. Run the command:\n   ```bash\n   ./start-cluster.sh\n   ```\n   ![Confirmation_JPS](https://github.com/user-attachments/assets/7c9af80c-fd89-491f-aa1a-ee875113f6b4)\n\n### 2. Stream Data with Flink\n\n1. Execute the following command to start streaming:\n   ```bash\n   python3 flink_streaming.py\n   ```\n\n2. The streaming process will begin, and you can monitor the progress in the terminal.\n![Flink_Stream](https://github.com/user-attachments/assets/58c90e07-b2b3-4c8e-a7f3-cecd1c9fcac1)\n\n### 3. Persist Data with Spark\n\n1. Run the following command to process and persist data:\n   ```bash\n   python3 spark_persist.py\n   ```\n   ![spark_persist](https://github.com/user-attachments/assets/9bf996d4-6306-4df2-b39c-9032c570a1ad)\n2. This step will create **seven Parquet files** in a folder named `spark_persisted_output` located in your home directory.\n   \n   ![persisted_output](https://github.com/user-attachments/assets/6c768f24-c923-48f4-9d2e-5326b772c56a)\n### 4. Perform Aggregations\n\n1. Execute the following command to perform data aggregation:\n   ```bash\n   python3 streaming_aggregates.py\n   ```\n   ![Daily_TotalRevenue](https://github.com/user-attachments/assets/9ca634ea-39ef-42e8-8820-79361a101afb)\n2. The output will be stored in a folder named `aggregated_output` in your home directory.\n   ![Daily_TotalRevenue_OutputFile](https://github.com/user-attachments/assets/eda3d112-e19d-4881-8838-984ce96e609b)\n\n### 5. Set Up Grafana Dashboard\n\n1. Download and install **Grafana**.\n2. Visit Grafana at `http://localhost:3000` (default port).\n3. Create a dashboard and import the provided JSON file to visualize the streaming and aggregated data.\n   \u003cimg width=\"1265\" alt=\"222\" src=\"https://github.com/user-attachments/assets/2cef8d27-4b7c-4d7f-82d9-d9990f60d7f5\" /\u003e\n   \u003cimg width=\"1265\" alt=\"1\" src=\"https://github.com/user-attachments/assets/7ff18f4e-902a-4cd5-9566-0735c3bae975\" /\u003e\n---\n\n## Note\n\n- Update the paths for each Python file (`flink_streaming.py`, `spark_persist.py`, `streaming_aggregates.py`) according to your system setup.\n- Ensure all required dependencies are installed and configured correctly.\n- You can download the original dataset from: [Download](https://mega.nz/file/OJUxVKCB#vWVfFYmnAzAM0PTMBZRSmmrWePcmoN1qIpM0kd4zFRw)\n\n---\n\n## Repository Structure\n\n```\nData-Streaming-Flink-Spark/\n├── flink_streaming.py        # Flink streaming script\n├── spark_persist.py          # Spark persistence script\n├── streaming_aggregates.py   # Data aggregation script\n├── dashboard.json    # Grafana dashboard configuration file\n├── spark_persisted_output/   # Output folder for Spark persisted data\n├── aggregated_output/        # Output folder for aggregated data\n```\n\n---\n\n## Contact\n\nFor queries or contributions, please contact:\n**Tashfeen Abbasi**  \nEmail: abbasitashfeen7@gmail.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftashi-2004%2Fapache-flink-spark-data-streaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftashi-2004%2Fapache-flink-spark-data-streaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftashi-2004%2Fapache-flink-spark-data-streaming/lists"}