{"id":23116323,"url":"https://github.com/nabilshadman/pyspark-dataframe-sql-ml-exercises","last_synced_at":"2026-04-16T08:31:24.417Z","repository":{"id":212767873,"uuid":"732268943","full_name":"nabilshadman/pyspark-dataframe-sql-ml-exercises","owner":"nabilshadman","description":"Comprehensive PySpark exercises covering data wrangling, ML pipelines, and AWS deployment from Udacity's Learn Spark course","archived":false,"fork":false,"pushed_at":"2024-12-04T18:46:30.000Z","size":41361,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T01:28:36.840Z","etag":null,"topics":["apache-spark","aws","data-engineering","data-science","machine-learning","pyspark"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nabilshadman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-16T05:30:44.000Z","updated_at":"2024-12-04T18:46:34.000Z","dependencies_parsed_at":"2023-12-16T06:50:40.612Z","dependency_job_id":"96145750-635c-4e40-8fe0-8c86264b33c8","html_url":"https://github.com/nabilshadman/pyspark-dataframe-sql-ml-exercises","commit_stats":null,"previous_names":["nabilshadman/pyspark-dataframe-sql-ml-exercises"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nabilshadman/pyspark-dataframe-sql-ml-exercises","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fpyspark-dataframe-sql-ml-exercises","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fpyspark-dataframe-sql-ml-exercises/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fpyspark-dataframe-sql-ml-exercises/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fpyspark-dataframe-sql-ml-exercises/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nabilshadman","download_url":"https://codeload.github.com/nabilshadman/pyspark-dataframe-sql-ml-exercises/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nabilshadman%2Fpyspark-dataframe-sql-ml-exercises/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31877448,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T07:36:03.521Z","status":"ssl_error","status_checked_at":"2026-04-16T07:35:53.576Z","response_time":69,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","aws","data-engineering","data-science","machine-learning","pyspark"],"created_at":"2024-12-17T04:15:59.726Z","updated_at":"2026-04-16T08:31:24.370Z","avatar_url":"https://github.com/nabilshadman.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PySpark Exercises\n\n![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10-blue.svg?logo=python\u0026logoColor=white)\n![Apache Spark](https://img.shields.io/badge/Apache%20Spark-3.4.0-E25A1C.svg?logo=apache-spark\u0026logoColor=white) \n![Jupyter](https://img.shields.io/badge/Jupyter-Notebook-F37626.svg?logo=jupyter\u0026logoColor=white)\n![AWS](https://img.shields.io/badge/AWS-EMR%20%7C%20S3-FF9900.svg?logo=amazon-aws\u0026logoColor=white)\n![pandas](https://img.shields.io/badge/pandas-2.0.0-150458.svg?logo=pandas\u0026logoColor=white)\n![NumPy](https://img.shields.io/badge/NumPy-1.24.0-013243.svg?logo=numpy\u0026logoColor=white)\n\nA comprehensive collection of exercises and mini-projects using [PySpark](https://spark.apache.org/docs/latest/api/python/index.html) (Python API for Apache Spark). These materials were developed as part of Udacity's [Learn Spark at Udacity](https://www.udacity.com/course/learn-spark-at-udacity--ud2002) course, providing hands-on experience with Apache Spark's core features and advanced capabilities.\n\n## 🛠 Tech Stack\n- Python\n- PySpark\n- NumPy\n- pandas\n- Matplotlib\n- Jupyter Notebook\n- AWS\n- GitHub\n\n## 📂 Repository Structure\n\n```\n.\n├── data_wrangling_with_spark/          # Data processing fundamentals\n│   ├── notebooks covering procedural vs functional programming\n│   ├── Spark operations and lazy evaluation\n│   ├── DataFrame operations and SQL\n│   └── practice datasets\n├── debugging_and_optimization/          # Performance tuning\n│   └── exercises/\n│       ├── data skewness handling\n│       ├── broadcast joins\n│       └── repartitioning strategies\n├── machine_learning_with_spark/         # ML implementations\n│   ├── feature engineering\n│   ├── linear regression\n│   ├── k-means clustering\n│   └── model tuning\n└── setting_up_spark_clusters_with_aws/  # AWS deployment\n    ├── demo_code/\n    └── exercises/\n        ├── EMR cluster creation\n        ├── script submission\n        └── S3 integration\n```\n\n## 📚 Course Content\n\n### 1. The Power of Spark\n- Introduction to Big Data ecosystem\n- MapReduce implementation\n- Fundamental Spark concepts\n\n### 2. Data Wrangling with Spark\n- Functional programming principles\n- DataFrame operations and transformations\n- Spark SQL integration\n- Data input/output operations\n\n### 3. Setting up Spark Clusters with AWS\n- EMR cluster deployment\n- AWS CLI integration\n- S3 data storage\n- Spark job submission\n\n### 4. Debugging and Optimization\n- Data skewness handling\n- Broadcast join optimization\n- Partition management\n- Performance tuning strategies\n\n### 5. Machine Learning with Spark\n- Feature engineering (numeric and text)\n- Linear regression implementation\n- K-means clustering\n- Model tuning and optimization\n- ML pipeline construction\n\n## 🚀 Getting Started\n\n1. **Environment Setup**\n   - Follow PySpark's official [installation guide](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n   - Set up Python environment with required dependencies\n   - Configure AWS credentials (for cluster-related exercises)\n\n2. **Running the Exercises**\n   - Each directory contains Jupyter notebooks and Python scripts\n   - Start with the numbered notebooks in each section\n   - Solutions are provided for self-assessment\n\n## 📝 Notes\n- Exercise solutions are available in corresponding `*_solution` notebooks\n- AWS-related exercises require active AWS credentials\n- Sample datasets are included in respective directories\n\n## 🤝 Contributing\nFeel free to submit issues and enhancement requests!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabilshadman%2Fpyspark-dataframe-sql-ml-exercises","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnabilshadman%2Fpyspark-dataframe-sql-ml-exercises","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnabilshadman%2Fpyspark-dataframe-sql-ml-exercises/lists"}