https://github.com/soumyadipta2020/ml-pyspark
Machine Learning in PySpark
https://github.com/soumyadipta2020/ml-pyspark
coding data-science jupyter-notebook machine-learning machine-learning-algorithms modelling notebook notebook-jupyter pyspark python statistical-analysis
Last synced: about 2 months ago
JSON representation
Machine Learning in PySpark
- Host: GitHub
- URL: https://github.com/soumyadipta2020/ml-pyspark
- Owner: Soumyadipta2020
- License: gpl-3.0
- Created: 2024-10-07T18:21:07.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-11-21T15:09:25.000Z (over 1 year ago)
- Last Synced: 2025-09-10T01:32:36.282Z (11 months ago)
- Topics: coding, data-science, jupyter-notebook, machine-learning, machine-learning-algorithms, modelling, notebook, notebook-jupyter, pyspark, python, statistical-analysis
- Language: Jupyter Notebook
- Homepage:
- Size: 3.23 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Machine Learning with PySpark 🤖🧠👾
[](http://hits.dwyl.com/Soumyadipta2020/ML-PySpark)
This repository contains a **Jupyter Notebook** that demonstrates how to perform **Machine Learning** tasks using **PySpark**. It is designed to provide a hands-on, practical introduction to using the Spark MLlib library for scalable machine learning.
## 📚 Contents
- **Notebook**:
- `ML-pyspark.ipynb`: A sample Jupyter Notebook showcasing end-to-end machine learning workflows in PySpark.
- **Sample Dataset**:
- Includes a small dataset for demonstration purposes.
## 🌟 Features Demonstrated
1. **Data Loading and Exploration**
- Loading datasets using Spark's DataFrame API.
- Data cleaning and preprocessing.
2. **Feature Engineering**
- Transforming and scaling features.
- Using `VectorAssembler` and other Spark tools.
3. **Model Training**
- Training ML models using Spark MLlib (e.g., Linear Regression, Random Forest).
- Cross-validation and hyperparameter tuning.
4. **Model Evaluation**
- Using metrics such as RMSE, R2, accuracy, etc., to evaluate models.
5. **Prediction**
- Generating predictions on test datasets.
- Saving and loading trained models.
## 📑 Requirements
- **Python**: 3.8+
- **PySpark**: 3.x
- **Jupyter Notebook**: Installed via Anaconda or pip.
- **Docker** (Optional): For setting up a PySpark environment. (being used here)
## 📢 Getting Started
1. **Clone the Repository**:
```bash
git clone https://github.com/Soumyadipta2020/ML-PySpark.git
cd machine-learning-pyspark
```
2. **Set Up the Environment**:
- Install dependencies:
```bash
pip install -r requirements.txt
```
- Alternatively, set up a Docker container:
```bash
docker pull quay.io/jupyter/all-spark-notebook
docker run -it --name pyspark-jupy --mount type=bind,source=your_location,target=/app/data --rm -p 8888:8888 quay.io/jupyter/all-spark-notebook
```
3. **Launch the Notebook**:
```bash
jupyter notebook ML-pyspark.ipynb
```
4. **Run the Notebook**:
- Follow the step-by-step instructions in the notebook to explore PySpark’s MLlib capabilities.
## ⚡ Usage
This notebook is for:
- **Beginners**: Learn the basics of machine learning with PySpark.
- **Practitioners**: Use it as a template for building scalable ML solutions.
- **Educators**: Teach Spark MLlib concepts interactively.
## 🤝 Contributing
We welcome contributions! If you have additional sample codes or improvements, please:
- Fork this repository.
- Create a feature branch:
```bash
git checkout -b feature/your-feature-name
```
- Commit your changes and push the branch:
```bash
git push origin feature/your-feature-name
```
- Open a Pull Request.
Make sure your code follows the repository's style and is well-documented.
## 🪧 Acknowledgments
- **Apache Spark**: The framework powering this project.
- **Jupyter Project**: For the interactive environment.
- Sample datasets sourced from publicly available resources for demonstration purposes.
---
Happy learning! 🚀