{"id":15208829,"url":"https://github.com/travelxml/apache-spark-pyspark-databricks-machine-learning-mlib","last_synced_at":"2026-01-24T23:33:37.979Z","repository":{"id":252039674,"uuid":"839200865","full_name":"TravelXML/APACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB","owner":"TravelXML","description":"Apache Spark Machine Learning project using MLlib and Linear Regression on Databricks! ","archived":false,"fork":false,"pushed_at":"2024-08-07T08:21:14.000Z","size":42,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-02T01:31:39.699Z","etag":null,"topics":["apache-spark","artificial-intelligence","data-science","data-visualization","databricks","jupyter-notebook","linerregression","machine-learning","ml","mlib","pyspark","python"],"latest_commit_sha":null,"homepage":"https://apige.medium.com/apache-spark-and-pyspark-on-databricks-a-comprehensive-guide-to-ipl-data-analysis-d0d5e02c861c","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TravelXML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-07T06:35:59.000Z","updated_at":"2024-12-06T18:50:47.000Z","dependencies_parsed_at":"2024-08-07T11:15:01.424Z","dependency_job_id":"2fc51c79-af95-474a-96b6-f207e0db1a9f","html_url":"https://github.com/TravelXML/APACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB","commit_stats":null,"previous_names":["travelxml/apache-spark-pyspark-databricks-machine-learning-mlib"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TravelXML%2FAPACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TravelXML%2FAPACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TravelXML%2FAPACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TravelXML%2FAPACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TravelXML","download_url":"https://codeload.github.com/TravelXML/APACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238825705,"owners_count":19537111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","artificial-intelligence","data-science","data-visualization","databricks","jupyter-notebook","linerregression","machine-learning","ml","mlib","pyspark","python"],"created_at":"2024-09-28T07:02:13.340Z","updated_at":"2025-10-29T12:31:30.737Z","avatar_url":"https://github.com/TravelXML.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Spark Machine Learning with MLlib and Linear Regression on Databricks\n\nApache Spark Machine Learning project using MLlib and Linear Regression on Databricks! This project demonstrates the application of machine learning techniques on big data using PySpark, the Python API for Apache Spark. This guide will walk you through the entire process, from setting up your Databricks environment to performing data analysis and building a linear regression model.\n\n## What is Apache Spark and PySpark?\n\n### Apache Spark\n\nApache Spark is an open-source, distributed computing system designed for fast and efficient big data processing. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.\n\n### PySpark\n\nPySpark is the Python API for Apache Spark. It allows Python developers to utilize the powerful distributed computing capabilities of Spark while writing code in Python, a more user-friendly language.\n\n### Differences Between Apache Spark, PySpark, and Pandas\n\n- **Apache Spark**: Best suited for large-scale data processing and analytics across clusters.\n- **PySpark**: Provides the power of Apache Spark with the simplicity of Python.\n- **Pandas**: A data manipulation library ideal for smaller datasets that can be handled on a single machine.\n\n## Packages Used in This Project\n\n- **pyspark**: Python API for Apache Spark, used for data processing and machine learning.\n- **pandas**: Data manipulation library for data transformation and analysis on smaller datasets.\n- **matplotlib**: Visualization library for creating static, animated, and interactive plots in Python.\n- **seaborn**: Statistical data visualization library based on matplotlib.\n\n## Project Setup\n\n### Step 1: Create a Databricks Community Edition Account\n\n1. Visit the [Databricks Community Edition](https://community.cloud.databricks.com/login.html) website.\n2. Click on \"Get Started for Free\".\n3. Fill in your details to create an account.\n4. Verify your email address and log in to Databricks.\n\n### Step 2: Create a New Cluster\n\n1. After logging in, click on \"Clusters\" in the left-hand menu.\n2. Click \"Create Cluster\".\n3. Name your cluster (e.g., \"Spark-ML-Cluster\").\n4. Select the appropriate Databricks runtime version.\n5. Click \"Create Cluster\".\n\n### Step 3: Upload Data to Databricks\n\nYou can either upload the data files directly to Databricks or use S3 for storage.\n\n#### Option A: Upload Data Directly to Databricks\n\n1. Click on \"Data\" in the left-hand menu.\n2. Click \"Add Data\" and select \"Upload File\".\n3. Upload the CSV files containing your data.\n\n#### Option B: Use Amazon S3\n\n1. If you have your data stored in S3, you can access it directly from Databricks.\n2. Ensure you have the necessary AWS credentials configured.\n3. Use the following code snippet to read data from S3:\n   ```python\n   df = spark.read.csv(\"s3a://your-bucket-name/your-file.csv\", header=True, inferSchema=True)\n   ```\n\n### Step 4: Clone the Project Repository\n\n1. In your Databricks workspace, click on \"Repos\" in the left-hand menu.\n2. Click \"Add Repo\" and select \"Clone Existing Repo\".\n3. Enter the URL of the repository: `https://github.com/TravelXML/APACHE-SPARK-PYSPARK-DATABRICKS-MACHINE-LEARNING-MLIB`\n4. Click \"Create Repo\".\n\n### Step 5: Open the Notebooks\n\n1. Navigate to the cloned repository in the \"Repos\" section.\n2. Open the notebook files `PYSPARK - LINER REGRESSION.ipynb` and `PYSPARK ML.ipynb`.\n\n## Running the Analysis\n\n### Step 1: Import Necessary Libraries\n\n```python\nfrom pyspark.sql import SparkSession\nfrom pyspark.ml.feature import VectorAssembler, StringIndexer\nfrom pyspark.ml.regression import LinearRegression\nfrom pyspark.ml.evaluation import RegressionEvaluator\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pandas as pd\n```\n\n### Step 2: Create a Spark Session\n\n```python\nspark = SparkSession.builder.appName('Spark ML Example').getOrCreate()\n```\n\n### Step 3: Load and Prepare Data\n\nReplace `'s3a://your-bucket-name/your-file.csv'` with the actual path to your data file.\n\n```python\n# Load the data\nfile_path = '/FileStore/shared_uploads/astartupcto@gmail.com/test1.csv'\ndf = spark.read.csv(file_path, header=True, inferSchema=True)\n\n# Index categorical columns\nindexer = StringIndexer(inputCols=[\"sex\", \"smoker\", \"day\", \"time\"],\n                        outputCols=[\"sex_indexed\", \"smoker_indexed\", \"day_indexed\", \"time_index\"])\ndf_r = indexer.fit(df).transform(df)\n\n# Assemble features into a vector\nfeatureassembler = VectorAssembler(inputCols=['tip', 'size', 'sex_indexed', 'smoker_indexed', 'day_indexed', 'time_index'],\n                                   outputCol=\"Independent Features\")\nfinalized_data = featureassembler.transform(df_r)\n\n# Select relevant columns\nfinalized_data = finalized_data.select(\"Independent Features\", \"total_bill\")\n```\n\n### Step 4: Split Data into Training and Testing Sets\n\n```python\ntrain_data, test_data = finalized_data.randomSplit([0.75, 0.25])\n```\n\n### Step 5: Train the Linear Regression Model\n\n```python\nregressor = LinearRegression(featuresCol='Independent Features', labelCol='total_bill')\nregressor = regressor.fit(train_data)\n```\n\n### Step 6: Evaluate the Model\n\n```python\n# Make predictions\npredictions = regressor.transform(test_data)\n\n# Evaluate the model\nevaluator = RegressionEvaluator(labelCol=\"total_bill\", predictionCol=\"prediction\", metricName=\"r2\")\nr2 = evaluator.evaluate(predictions)\n\nevaluator = RegressionEvaluator(labelCol=\"total_bill\", predictionCol=\"prediction\", metricName=\"mae\")\nmae = evaluator.evaluate(predictions)\n\nevaluator = RegressionEvaluator(labelCol=\"total_bill\", predictionCol=\"prediction\", metricName=\"mse\")\nmse = evaluator.evaluate(predictions)\n\n# Show predictions\npredictions.select(\"Independent Features\", \"total_bill\", \"prediction\").show()\n\n# Print performance metrics\nprint(f\"R²: {r2}\")\nprint(f\"Mean Absolute Error: {mae}\")\nprint(f\"Mean Squared Error: {mse}\")\n```\n\n\n## Conclusion\n\nCongratulations! You have successfully set up a Databricks environment, uploaded data, and performed machine learning analysis using PySpark. You have learned how to preprocess data, build a linear regression model, and evaluate its performance.\n\nFor more in-depth tutorials and articles on Apache Spark, PySpark, and big data analytics, subscribe to our updates.\n\n## Additional Resources\n\n- [Databricks Documentation](https://docs.databricks.com/)\n- [Apache Spark Documentation](https://spark.apache.org/documentation.html)\n- [PySpark Documentation](https://spark.apache.org/docs/latest/api/python/)\n\nFeel free to reach out if you have any questions or need further assistance. Happy coding!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravelxml%2Fapache-spark-pyspark-databricks-machine-learning-mlib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftravelxml%2Fapache-spark-pyspark-databricks-machine-learning-mlib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravelxml%2Fapache-spark-pyspark-databricks-machine-learning-mlib/lists"}