{"id":15208851,"url":"https://github.com/mayurasandakalum/ipl-data-engineering-spark-databricks","last_synced_at":"2026-02-09T05:04:56.422Z","repository":{"id":251116359,"uuid":"836426403","full_name":"mayurasandakalum/ipl-data-engineering-spark-databricks","owner":"mayurasandakalum","description":"Comprehensive data engineering and analytics project using IPL dataset with Amazon S3, Apache Spark, Databricks, and SQL. Includes data storage, transformation, analysis, and visualization.","archived":false,"fork":false,"pushed_at":"2024-07-31T21:34:21.000Z","size":2788,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-06T19:43:51.668Z","etag":null,"topics":["amazon-s3","apache-spark","aws","big-data","cricket-analytics","data-analytics","data-engineering","data-visualization","databricks","etl-pipeline","ipl-dataset","machine-learning","python","sql"],"latest_commit_sha":null,"homepage":"https://github.com/mayurasandakalum/ipl-data-engineering-spark-databricks","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mayurasandakalum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-31T20:29:47.000Z","updated_at":"2024-07-31T21:34:24.000Z","dependencies_parsed_at":"2024-08-01T01:33:32.541Z","dependency_job_id":"cd9021f9-71c3-4a0f-8bdd-34a23976239d","html_url":"https://github.com/mayurasandakalum/ipl-data-engineering-spark-databricks","commit_stats":null,"previous_names":["mayurasandakalum/ipl-data-engineering-spark-databricks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mayurasandakalum/ipl-data-engineering-spark-databricks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mayurasandakalum%2Fipl-data-engineering-spark-databricks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mayurasandakalum%2Fipl-data-engineering-spark-databricks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mayurasandakalum%2Fipl-data-engineering-spark-databricks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mayurasandakalum%2Fipl-data-engineering-spark-databricks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mayurasandakalum","download_url":"https://codeload.github.com/mayurasandakalum/ipl-data-engineering-spark-databricks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mayurasandakalum%2Fipl-data-engineering-spark-databricks/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269281180,"owners_count":24390601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-07T02:00:09.698Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-s3","apache-spark","aws","big-data","cricket-analytics","data-analytics","data-engineering","data-visualization","databricks","etl-pipeline","ipl-dataset","machine-learning","python","sql"],"created_at":"2024-09-28T07:02:36.490Z","updated_at":"2026-02-09T05:04:51.380Z","avatar_url":"https://github.com/mayurasandakalum.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# IPL Data Engineering and Analytics with Apache Spark and Databricks 🏏📊🚀\n\nThis project showcases the development and execution of a data engineering and analytics pipeline using the IPL (Indian Premier League) dataset. The primary objective is to demonstrate data storage, transformation, analysis, and visualization using Amazon S3, Apache Spark, Databricks, SQL, and Python.\n\n![Banner](screenshots/banner.png)\n\n## Key Components and Functionalities 🔧\n\n1. **Data Storage 📦**\n   - **Amazon S3**: Data storage solution where the IPL dataset is uploaded and managed.\n   ![AWS S3 Bucket](screenshots/aws-s3-bucket.jpg)\n\n2. **Data Transformation and Analysis 🔄**\n   - **Apache Spark**: Used for data transformation and processing.\n   - **SQL**: Employed for data querying and analysis.\n   ![Architecture Diagram](screenshots/architecture-diagram.jpeg)\n\n3. **Data Visualization 📊**\n   - **Matplotlib and Seaborn**: Used for creating insightful visualizations of the analyzed data.\n\n## Dataset 📚\n\nThe IPL dataset used in this project contains ball-by-ball data of all IPL matches up to the 2017 season. The dataset is sourced from data.world and can be accessed [here](https://data.world/raghu543/ipl-data-till-2017).\n\n![Dataset Source](screenshots/dataset-web.jpg)\n\n## Setup 🛠️\n\n### Prerequisites 📋\n\n- AWS account with access to S3.\n- Databricks account.\n- Python 3.x installed.\n\n### Installation ⚙️\n\n1. **Clone the repository**:\n    ```sh\n    https://github.com/mayurasandakalum/ipl-data-engineering-spark-databricks.git\n    ```\n\n2. **Navigate to the project directory**:\n    ```sh\n    cd ipl-data-engineering-spark-databricks\n    ```\n\n\n## Usage 🚀\n\n### Data Storage on Amazon S3\n\n- Upload the IPL dataset files to your S3 bucket.\n- Ensure the bucket is publicly accessible if required.\n\n### Data Transformation and Analysis\n\n- **Setup Databricks**:\n  - Create a new cluster on Databricks.\n  ![Databricks Cluster](screenshots/cluster.jpg)\n  - Import the notebook files and attach them to your cluster.\n  ![Databricks Notebook](screenshots/databricks-notebook.jpg)\n  - Execute the cells to run the data transformation and analysis steps.\n\n### Data Visualizations\n\n1. **Average Runs Scored by Batsmen in Winning Matches**:\n   ![Average Runs Scored by Batsmen in Winning Matches](screenshots/average-runs-scored-by-batsmen-in-winning-matches.png)\n\n2. **Distribution of Scores by Venue**:\n   ![Distribution of Scores by Venue](screenshots/distribution-of-scores-by-venue.png)\n\n3. **Impact of Winning Toss on Match Outcomes**:\n   ![Impact of Winning Toss on Match Outcomes](screenshots/impact-of-winning-toss-on-match-outcomes.png)\n\n4. **Most Economical Bowlers in Powerplay Overs**:\n   ![Most Economical Bowlers in Powerplay Overs](screenshots/most-economical-bowlers-in-powerplay-overs.png)\n\n5. **Most Frequent Dismissal Types**:\n   ![Most Frequent Dismissal Types](screenshots/most-frequent-dismissal-types.png)\n\n6. **Team Performance After Winning Toss**:\n   ![Team Performance After Winning Toss](screenshots/team-performance-after-winning-toss.png)\n\n### Code Samples 💻\n\n- **Data Transformation Code**:\n  ```python\n  from pyspark.sql import SparkSession\n  from pyspark.sql.functions import col, sum, avg\n\n  spark = SparkSession.builder.appName(\"IPL Data Analysis\").getOrCreate()\n\n  # Read data from S3\n  df = spark.read.csv(\"s3://your-bucket-name/ipl-data.csv\", header=True, inferSchema=True)\n\n  # Data transformation\n  transformed_df = df.groupBy(\"season\").agg(sum(\"runs\").alias(\"total_runs\"), avg(\"runs\").alias(\"avg_runs\"))\n  transformed_df.show()\n  ```\n\n- **SQL Queries**:\n  ```sql\n  SELECT player_name, SUM(runs) as total_runs\n  FROM ball_by_ball\n  GROUP BY player_name\n  ORDER BY total_runs DESC\n  LIMIT 10;\n  ```\n\n## Contributing 🤝\n\nFeel free to fork this repository, make enhancements, and submit pull requests. Your contributions are welcome!\n\n## Acknowledgments 🙏\n\nSpecial thanks to the creators of the datasets and the open-source tools used in this project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmayurasandakalum%2Fipl-data-engineering-spark-databricks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmayurasandakalum%2Fipl-data-engineering-spark-databricks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmayurasandakalum%2Fipl-data-engineering-spark-databricks/lists"}