{"id":14982327,"url":"https://github.com/mahmoudparsian/pyspark-tutorial","last_synced_at":"2025-05-14T15:03:45.102Z","repository":{"id":28537137,"uuid":"32054120","full_name":"mahmoudparsian/pyspark-tutorial","owner":"mahmoudparsian","description":"PySpark-Tutorial provides basic algorithms using PySpark","archived":false,"fork":false,"pushed_at":"2025-01-25T02:17:34.000Z","size":9390,"stargazers_count":1214,"open_issues_count":3,"forks_count":474,"subscribers_count":53,"default_branch":"master","last_synced_at":"2025-04-06T16:02:37.914Z","etag":null,"topics":["big-data","big-data-analytics","data-algorithms","pyspark","spark","spark-dataframes","spark-rdd"],"latest_commit_sha":null,"homepage":"http://mapreduce4hackers.com","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mahmoudparsian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-03-12T03:01:53.000Z","updated_at":"2025-03-30T00:36:08.000Z","dependencies_parsed_at":"2025-02-09T06:10:30.115Z","dependency_job_id":null,"html_url":"https://github.com/mahmoudparsian/pyspark-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudparsian%2Fpyspark-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudparsian%2Fpyspark-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudparsian%2Fpyspark-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahmoudparsian%2Fpyspark-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mahmoudparsian","download_url":"https://codeload.github.com/mahmoudparsian/pyspark-tutorial/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248759373,"owners_count":21157146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","big-data-analytics","data-algorithms","pyspark","spark","spark-dataframes","spark-rdd"],"created_at":"2024-09-24T14:05:12.392Z","updated_at":"2025-04-13T18:25:18.646Z","avatar_url":"https://github.com/mahmoudparsian.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PySpark Tutorial\n\n* PySpark is the Python API for Spark.  \n\n* The purpose of PySpark tutorial is to provide \n  basic distributed algorithms using PySpark. \n\n* PySpark supports two types of Data Abstractions:\n\t* RDDs\n\t* DataFrames\n\n* **PySpark Interactive Mode**: has an interactive shell \n  (`$SPARK_HOME/bin/pyspark`) for basic testing \n  and debugging and is not supposed to be used \n  for production environment.\n\n* **PySpark Batch Mode**: you may use `$SPARK_HOME/bin/spark-submit` \n  command for running PySpark programs (may be used for \n  testing and production environemtns)\n\n------\n\n# [Glossary: big data, MapReduce, Spark](https://github.com/mahmoudparsian/big-data-mapreduce-course/blob/master/slides/glossary/README.md)\n\n------\n\n# [Basics of PySpark with Examples](./howto/README.md) \n\n------\n\n# PySpark Examples and Tutorials\n\n* [PySpark Examples: RDDs](./tutorial/pyspark-examples/rdds/)\n* [PySpark Examples: DataFramess](./tutorial/pyspark-examples/dataframes/)\n* [DNA Base Counting](./tutorial/dna-basecount/README.md)\n* [Classic Word Count](./tutorial/wordcount)\n* [Find Frequency of Bigrams](./tutorial/bigrams)\n* [Join of Two Relations R(K, V\u003csub\u003e1\u003c/sub\u003e), S(K, V\u003csub\u003e2\u003c/sub\u003e)](./tutorial/basic-join)\n* [Basic Mapping of RDD Elements](./tutorial/basic-map)\n* [How to add all RDD elements together](./tutorial/basic-sum)\n* [How to multiply all RDD elements together](./tutorial/basic-multiply)\n* [Find Top-N and Bottom-N](./tutorial/top-N)\n* [Find average by using combineByKey()](./tutorial/combine-by-key)\n* [How to filter RDD elements](./tutorial/basic-filter)\n* [How to find average](./tutorial/basic-average)\n* [Cartesian Product: rdd1.cartesian(rdd2)](./tutorial/cartesian)\n* [Sort By Key: sortByKey() ascending/descending](./tutorial/basic-sort)\n* [How to Add Indices](./tutorial/add-indices)\n* [Map Partitions: mapPartitions() by Examples](./tutorial/map-partitions/README.md)\n* [Monoid: Design Principle](https://github.com/mahmoudparsian/data-algorithms-with-spark/blob/master/wiki-spark/docs/monoid/README.md)\n\n------\n\n# Books\n\n### [Data Algorithms with Spark](https://github.com/mahmoudparsian/data-algorithms-with-spark/) \n\n### [Data Algorithms](https://github.com/mahmoudparsian/data-algorithms-book/) \n\n### [PySpark Algorithms](https://github.com/mahmoudparsian/pyspark-algorithms/) \n\n-----\n\n# Miscellaneous \n\n### [Download, Install Spark and Run PySpark](./howto/download_install_run_spark.md) \n\n### [How to Minimize the Verbosity of Spark](./howto/minimize_verbosity.md) \n\n-------\n\n# PySpark Tutorial and References...\n* [Getting started with PySpark - Part 1](http://www.mccarroll.net/blog/pyspark/)\n* [Getting started with PySpark - Part 2](http://www.mccarroll.net/blog/pyspark2/index.html)\n* [A really really fast introduction to PySpark](http://www.slideshare.net/hkarau/a-really-really-fast-introduction-to-py-spark-lightning-fast-cluster-computing-with-python-1)\n* [PySpark](http://www.slideshare.net/thegiivee/pysaprk?qid=81cf1b31-8b19-4570-89a5-21d03cad6ecd\u0026v=default\u0026b=\u0026from_search=9)\n* [Basic Big Data Manipulation with PySpark](http://bigdatasciencebootcamp.com/posts/Part_3/basic_big_data.html)\n* [Working in Pyspark: Basics of Working with Data and RDDs](http://www.learnbymarketing.com/618/pyspark-rdd-basics-examples/)\n\n-------\n\n# Questions/Comments\n* [View Mahmoud Parsian's profile on LinkedIn](http://www.linkedin.com/in/mahmoudparsian)\n* Please send me an email: mahmoud.parsian@yahoo.com\n* [Twitter: @mahmoudparsian](http://twitter.com/mahmoudparsian) \n\nThank you!\n\n````\nbest regards,\nMahmoud Parsian\n````\n\n-----\n\n\n\u003ca href=\"https://www.oreilly.com/library/view/data-algorithms-with/9781492082378/\"\u003e\n    \u003cimg\n        alt=\"Data Algorithms with Spark\"\n        src=\"images/Data-Algorithms-with-Spark_mech2.png\"\n\u003e\n\n\u003ca href=\"https://www.oreilly.com/library/view/data-algorithms-with/9781492082378/\"\u003e\n    \u003cimg\n        alt=\"Data Algorithms with Spark\"\n        src=\"images/Data_Algorithms_with_Spark_COVER_9781492082385.png\"\n\u003e\n\n\u003ca href=\"https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2\"\u003e\n    \u003cimg\n        alt=\"PySpark Algorithms\"\n        src=\"images/pyspark_algorithms2.jpg\"\n\u003e\n\n\u003ca href=\"http://shop.oreilly.com/product/0636920033950.do\"\u003e\n    \u003cimg\n        alt=\"Data Algorithms\"\n        src=\"images/data_algorithms_image.jpg\"\n\u003e\n\n------\n\n[//]: # (metadata:)\n[//]: # (Spark, PySpark, Python)\n[//]: # (MapReduce, Distributed Algorithms, mappers, reducers, partitioners)\n[//]: # (Transformations, Actions, RDDs, DataFrames, SQL)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoudparsian%2Fpyspark-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahmoudparsian%2Fpyspark-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahmoudparsian%2Fpyspark-tutorial/lists"}