{"id":50400450,"url":"https://github.com/deypadma2020/pyspark","last_synced_at":"2026-05-30T23:03:20.557Z","repository":{"id":310701770,"uuid":"1040270592","full_name":"deypadma2020/PySpark","owner":"deypadma2020","description":"A beginner-friendly PySpark learning repository that covers essential functions, transformations, and actions to get started with big data processing and distributed computing. Perfect for those new to PySpark and Apache Spark.","archived":false,"fork":false,"pushed_at":"2025-11-08T16:30:47.000Z","size":104,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-08T18:15:43.219Z","etag":null,"topics":["big-data","data-engineering","distributed-computing","etl","mongodb","nosql","pyspark","spark-sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deypadma2020.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-18T18:02:36.000Z","updated_at":"2025-11-08T16:30:51.000Z","dependencies_parsed_at":"2025-09-26T00:20:15.027Z","dependency_job_id":null,"html_url":"https://github.com/deypadma2020/PySpark","commit_stats":null,"previous_names":["deypadma2020/pyspark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/deypadma2020/PySpark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deypadma2020%2FPySpark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deypadma2020%2FPySpark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deypadma2020%2FPySpark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deypadma2020%2FPySpark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deypadma2020","download_url":"https://codeload.github.com/deypadma2020/PySpark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deypadma2020%2FPySpark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33712582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","data-engineering","distributed-computing","etl","mongodb","nosql","pyspark","spark-sql"],"created_at":"2026-05-30T23:03:17.536Z","updated_at":"2026-05-30T23:03:20.543Z","avatar_url":"https://github.com/deypadma2020.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PySpark\n\n---\n\n# ⚡ PySpark Basics for Beginners\n\nA **beginner-friendly PySpark repository** containing essential functions, transformations, and actions to help you quickly learn and apply **Apache Spark** for big data processing and distributed computing.\n\nThis repo is designed as a **hands-on guide** to start learning PySpark from scratch — covering DataFrames, RDDs, SQL, and commonly used operations.\n\n---\n\n## 📌 What You’ll Learn here\n\n* 🔹 Setting up PySpark environment\n* 🔹 Creating RDDs \u0026 DataFrames\n* 🔹 Basic operations: `select`, `filter`, `withColumn`, `groupBy`, `agg`, etc.\n* 🔹 Handling missing values \u0026 data cleaning\n* 🔹 Joins \u0026 SQL queries with SparkSQL\n* 🔹 Common transformations \u0026 actions\n* 🔹 Saving and loading data (CSV, Parquet, JSON)\n\n---\n\n## 📁 Repository Structure\n\n```bash\npyspark-basics/\n│\n├── notebooks/                # Jupyter notebooks for practice\n│   ├── 01_intro.ipynb\n│   ├── 02_dataframes.ipynb\n│   ├── 03_transformations.ipynb\n│   └── 04_spark_sql.ipynb\n│\n├── examples/                 # Python scripts with basic functions\n│   ├── dataframe_examples.py\n│   ├── rdd_examples.py\n│   └── spark_sql_examples.py\n│\n├── requirements.txt          # Dependencies\n└── README.md                 # You are here 🚀\n```\n\n---\n\n## ⚙️ Installation \u0026 Setup\n\n1️⃣ **Install PySpark** (locally):\n\n```bash\npip install pyspark\n```\n\n2️⃣ **Verify Installation**:\n\n```bash\npython -c \"import pyspark; print(pyspark.__version__)\"\n```\n\n3️⃣ **Start Jupyter Notebook** to practice:\n\n```bash\njupyter notebook\n```\n\n---\n\n## 🚀 Quick Start Example\n\n```python\nfrom pyspark.sql import SparkSession\n\n# Create Spark session\nspark = SparkSession.builder.appName(\"BeginnerApp\").getOrCreate()\n\n# Create DataFrame\ndata = [(\"Alice\", 25), (\"Bob\", 30), (\"Charlie\", 35)]\ndf = spark.createDataFrame(data, [\"Name\", \"Age\"])\n\n# Show data\ndf.show()\n\n# Basic operations\ndf.select(\"Name\").show()\ndf.filter(df.Age \u003e 28).show()\n```\n\nOutput:\n\n```\n+-------+---+\n|   Name|Age|\n+-------+---+\n|  Alice| 25|\n|    Bob| 30|\n|Charlie| 35|\n+-------+---+\n```\n\n---\n\n## 📊 Topics Covered\n\n* ✅ DataFrames \u0026 RDDs\n* ✅ Transformations \u0026 Actions\n* ✅ Joins \u0026 Aggregations\n* ✅ SparkSQL queries\n* ✅ File I/O (CSV, JSON, Parquet)\n* ✅ Beginner-friendly examples\n\n---\n\n## 🧑‍💻 Who is this for?\n\n* 🔹 Students learning **Big Data \u0026 Spark**\n* 🔹 Data engineers starting with **ETL pipelines**\n* 🔹 Beginners who want to practice **PySpark functions** hands-on\n\n---\n\n## 📌 Future Enhancements\n\n* Add MLlib (Spark Machine Learning) basics\n* Add streaming examples (Spark Streaming)\n* Add optimization tips for large datasets\n\n---\n\n## 🪪 License\n\nThis project is licensed under the **MIT License** – feel free to use and contribute.\n\n---\n\n✨ **Happy Learning PySpark!** ✨\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeypadma2020%2Fpyspark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeypadma2020%2Fpyspark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeypadma2020%2Fpyspark/lists"}