{"id":13935152,"url":"https://github.com/jadianes/spark-py-notebooks","last_synced_at":"2025-05-15T09:06:17.911Z","repository":{"id":31581167,"uuid":"35145949","full_name":"jadianes/spark-py-notebooks","owner":"jadianes","description":"Apache Spark \u0026 Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks","archived":false,"fork":false,"pushed_at":"2024-03-16T21:39:19.000Z","size":2312,"stargazers_count":1646,"open_issues_count":9,"forks_count":917,"subscribers_count":97,"default_branch":"master","last_synced_at":"2025-04-14T15:00:32.909Z","etag":null,"topics":["big-data","bigdata","data-analysis","data-science","ipython","ipython-notebook","machine-learning","mllib","notebook","pyspark","python","spark"],"latest_commit_sha":null,"homepage":"http://jadianes.github.io/spark-py-notebooks","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jadianes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-06T07:45:21.000Z","updated_at":"2025-04-01T07:53:18.000Z","dependencies_parsed_at":"2022-07-15T21:48:09.558Z","dependency_job_id":"36368cd2-505b-4940-9620-797e78bfe4d9","html_url":"https://github.com/jadianes/spark-py-notebooks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadianes%2Fspark-py-notebooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadianes%2Fspark-py-notebooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadianes%2Fspark-py-notebooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jadianes%2Fspark-py-notebooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jadianes","download_url":"https://codeload.github.com/jadianes/spark-py-notebooks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310515,"owners_count":22049469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","bigdata","data-analysis","data-science","ipython","ipython-notebook","machine-learning","mllib","notebook","pyspark","python","spark"],"created_at":"2024-08-07T23:01:25.637Z","updated_at":"2025-05-15T09:06:17.869Z","avatar_url":"https://github.com/jadianes.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","📚 Skill Development \u0026 Career"],"sub_categories":["Curated Jupyter Notebooks"],"readme":"# Spark Python Notebooks  \n\n[![Join the chat at https://gitter.im/jadianes/spark-py-notebooks](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/jadianes/spark-py-notebooks?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\nThis is a collection of [IPython notebook](http://ipython.org/notebook.html)/[Jupyter](https://jupyter.org/) \nnotebooks intended to train the reader on different [Apache Spark](http://spark.apache.org/) concepts, from \nbasic to advanced, by using the **Python** language.  \n\nIf Python is not your language, and it is R, you may want to have a look at our [R on Apache Spark (SparkR) notebooks](https://github.com/jadianes/spark-r-notebooks) instead. Additionally, if your are interested in being introduced to some basic Data Science\nEngineering, you might find [these series of tutorials](https://github.com/jadianes/data-science-your-way)\ninteresting. There we explain different concepts and applications \nusing Python and R.  \n\n## Instructions  \n\nA good way of using these notebooks is by first cloning the repo, and then \nstarting your own [IPython notebook](http://ipython.org/notebook.html)/[Jupyter](https://jupyter.org/) in \n**pySpark mode**. For example, if we have a *standalone* Spark installation\nrunning in our `localhost` with a maximum of 6Gb per node assigned to IPython:  \n\n    MASTER=\"spark://127.0.0.1:7077\" SPARK_EXECUTOR_MEMORY=\"6G\" IPYTHON_OPTS=\"notebook --pylab inline\" ~/spark-1.5.0-bin-hadoop2.6/bin/pyspark\n\nNotice that the path to the `pyspark` command will depend on your specific \ninstallation. So as requirement, you need to have\n[Spark installed](https://spark.apache.org/docs/latest/index.html) in \nthe same machine you are going to start the `IPython notebook` server.     \n\nFor more Spark options see [here](https://spark.apache.org/docs/latest/spark-standalone.html). In general it works the rule of passing options\ndescribed in the form `spark.executor.memory` as `SPARK_EXECUTOR_MEMORY` when\ncalling IPython/pySpark.   \n \n## Datasets  \n\nWe will be using datasets from the [KDD Cup 1999](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html). The results \nof this competition can be found [here](http://cseweb.ucsd.edu/~elkan/clresults.html).  \n\n## References\n\nThe reference book for these and other Spark related topics is:  \n\n- *Learning Spark* by Holden Karau, Andy Konwinski, Patrick Wendell, and Matei Zaharia.  \n\n## Notebooks  \n\nThe following notebooks can be examined individually, although there is a more\nor less linear 'story' when followed in sequence. By using the same dataset\nthey try to solve a related set of tasks with it.  \n \n### [RDD creation](https://github.com/jadianes/spark-py-notebooks/blob/master/nb1-rdd-creation/nb1-rdd-creation.ipynb)  \n\nAbout reading files and parallelize.  \n  \n### [RDDs basics](https://github.com/jadianes/spark-py-notebooks/blob/master/nb2-rdd-basics/nb2-rdd-basics.ipynb)\n\nA look at `map`, `filter`, and `collect`.  \n  \n### [Sampling RDDs](https://github.com/jadianes/spark-py-notebooks/blob/master/nb3-rdd-sampling/nb3-rdd-sampling.ipynb)  \n\nRDD sampling methods explained.    \n  \n### [RDD set operations](https://github.com/jadianes/spark-py-notebooks/blob/master/nb4-rdd-set/nb4-rdd-set.ipynb)    \n\nBrief introduction to some of the RDD pseudo-set operations.  \n\n### [Data aggregations on RDDs](https://github.com/jadianes/spark-py-notebooks/blob/master/nb5-rdd-aggregations/nb5-rdd-aggregations.ipynb)  \n\nRDD actions `reduce`, `fold`, and `aggregate`.   \n\n### [Working with key/value pair RDDs](https://github.com/jadianes/spark-py-notebooks/blob/master/nb6-rdd-key-value/nb6-rdd-key-value.ipynb)    \n\nHow to deal with key/value pairs in order to aggregate and explore data.  \n  \n### [MLlib: Basic Statistics and Exploratory Data Analysis](https://github.com/jadianes/spark-py-notebooks/blob/master/nb7-mllib-statistics/nb7-mllib-statistics.ipynb)    \n\nA notebook introducing Local Vector types, basic statistics \nin MLlib for Exploratory Data Analysis and model selection.  \n  \n### [MLlib: Logistic Regression](https://github.com/jadianes/spark-py-notebooks/blob/master/nb8-mllib-logit/nb8-mllib-logit.ipynb)     \n\nLabeled points and Logistic Regression classification of network attacks in MLlib.\nApplication of model selection techniques using correlation matrix and Hypothesis Testing.    \n\n### [MLlib: Decision Trees](https://github.com/jadianes/spark-py-notebooks/blob/master/nb9-mllib-trees/nb9-mllib-trees.ipynb)  \n\nUse of tree-based methods and how they help explaining models and\n feature selection.  \n\n### [Spark SQL: structured processing for Data Analysis](https://github.com/jadianes/spark-py-notebooks/blob/master/nb10-sql-dataframes/nb10-sql-dataframes.ipynb)  \n\nIn this notebook a schema is inferred for our network interactions dataset. Based on that, we use\nSpark's SQL `DataFrame` abstraction to perform a more structured exploratory data analysis.  \n\n\n## Applications  \n\nBeyond the basics. Close to real-world applications using Spark and other technologies.  \n\n### [Olssen: On-line Spectral Search ENgine for proteomics](https://github.com/jadianes/olssen)  \n\nSame tech stack this time with an AngularJS client app.  \n\n### [An on-line movie recommendation web service](https://github.com/jadianes/spark-movie-lens)  \n\nThis tutorial can be used independently to build a movie recommender model based on the MovieLens dataset. Most of the code in the first part, about how to use ALS with the public MovieLens dataset, comes from my solution to one of the exercises proposed in the [CS100.1x Introduction to Big Data with Apache Spark by Anthony D. Joseph on edX](https://www.edx.org/course/introduction-big-data-apache-spark-uc-berkeleyx-cs100-1x), that is also [**publicly available since 2014 at Spark Summit**](https://databricks-training.s3.amazonaws.com/movie-recommendation-with-mllib.html). \n\nThere I've added with minor modifications to use a larger dataset and also code about how to store and reload the model for later use. On top of that we build a Flask web service so the recommender can be use to provide movie recommendations on-line.  \n\n### [KDD Cup 1999](https://github.com/jadianes/kdd-cup-99-spark)  \n\nMy try using Spark with this classic dataset and Knowledge Discovery competition.  \n\n## Contributing\n\nContributions are welcome!  For bug reports or requests please [submit an issue](https://github.com/jadianes/spark-py-notebooks/issues).\n\n## Contact  \n\nFeel free to contact me to discuss any issues, questions, or comments.\n\n* Twitter: [@ja_dianes](https://twitter.com/ja_dianes)\n* GitHub: [jadianes](https://github.com/jadianes)\n* LinkedIn: [jadianes](https://www.linkedin.com/in/jadianes)\n* Website: [jadianes.me](http://jadianes.me)\n\n## License\n\nThis repository contains a variety of content; some developed by Jose A. Dianes, and some from third-parties.  The third-party content is distributed under the license provided by those parties.\n\nThe content developed by Jose A. Dianes is distributed under the following license:\n\n    Copyright 2016 Jose A Dianes\n\n    Licensed under the Apache License, Version 2.0 (the \"License\");\n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjadianes%2Fspark-py-notebooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjadianes%2Fspark-py-notebooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjadianes%2Fspark-py-notebooks/lists"}