{"id":14982369,"url":"https://github.com/cartershanklin/pyspark-cheatsheet","last_synced_at":"2025-10-29T12:31:46.033Z","repository":{"id":49356788,"uuid":"268110746","full_name":"cartershanklin/pyspark-cheatsheet","owner":"cartershanklin","description":"PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster","archived":false,"fork":false,"pushed_at":"2024-10-15T18:48:15.000Z","size":17654,"stargazers_count":438,"open_issues_count":1,"forks_count":218,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-01-29T22:34:51.409Z","etag":null,"topics":["apache-spark","big-data","pyspark","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cartershanklin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-30T15:47:52.000Z","updated_at":"2025-01-26T06:41:18.000Z","dependencies_parsed_at":"2022-08-12T20:01:53.344Z","dependency_job_id":null,"html_url":"https://github.com/cartershanklin/pyspark-cheatsheet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cartershanklin%2Fpyspark-cheatsheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cartershanklin%2Fpyspark-cheatsheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cartershanklin%2Fpyspark-cheatsheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cartershanklin%2Fpyspark-cheatsheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cartershanklin","download_url":"https://codeload.github.com/cartershanklin/pyspark-cheatsheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238825750,"owners_count":19537118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","big-data","pyspark","spark"],"created_at":"2024-09-24T14:05:17.481Z","updated_at":"2025-10-29T12:31:44.601Z","avatar_url":"https://github.com/cartershanklin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"PySpark Cheat Sheet\n===================\nThis cheat sheet will help you learn PySpark and write PySpark apps faster. Everything in here is fully functional PySpark code you can run or adapt to your programs.\n\nThese snippets are licensed under the CC0 1.0 Universal License. That means you can freely copy and adapt these code snippets and you don't need to give attribution or include any notices.\n\nThese snippets use DataFrames loaded from various data sources:\n- \"Auto MPG Data Set\" available from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/auto+mpg).\n- customer_spend.csv, a generated time series dataset.\n- date_examples.csv, a generated dataset with various date and time formats.\n- weblog.csv, a cleaned version of this [web log dataset](https://www.kaggle.com/datasets/shawon10/web-log-dataset).\n\nThese snippets were tested against the Spark 3.2.2 API. This page was last updated 2022-09-19 15:31:03.\n\nMake note of these helpful links:\n- [PySpark DataFrame Operations](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html)\n- [Built-in Spark SQL Functions](https://spark.apache.org/docs/latest/api/sql/index.html)\n- [MLlib Main Guide](http://spark.apache.org/docs/latest/ml-guide.html)\n- [Structured Streaming Guide](https://spark.apache.org/docs/latest/api/python/reference/pyspark.ss/index.html)\n- [PySpark SQL Functions Source](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions.html)\n\nTry in a Notebook\n-----------------\nSee the [Notebook How-To](notebook.md) for instructions on running in a Jupyter notebook.\n\nGenerate the Cheatsheet\n-----------------------\nYou can generate the cheatsheet by running `cheatsheet.py` in your PySpark environment as follows:\n- Install dependencies: `pip3 install -r requirements.txt`\n- Generate README.md: `python3 cheatsheet.py`\n- Generate cheatsheet.ipynb: `python3 cheatsheet.py --notebook`\n\n\nTable of contents\n=================\n\n\u003c!--ts--\u003e\n   * [Accessing Data Sources](#accessing-data-sources)\n      * [Load a DataFrame from CSV](#load-a-dataframe-from-csv)\n      * [Load a DataFrame from a Tab Separated Value (TSV) file](#load-a-dataframe-from-a-tab-separated-value-tsv-file)\n      * [Save a DataFrame in CSV format](#save-a-dataframe-in-csv-format)\n      * [Load a DataFrame from Parquet](#load-a-dataframe-from-parquet)\n      * [Save a DataFrame in Parquet format](#save-a-dataframe-in-parquet-format)\n      * [Load a DataFrame from JSON Lines (jsonl) Formatted Data](#load-a-dataframe-from-json-lines-jsonl-formatted-data)\n      * [Save a DataFrame into a Hive catalog table](#save-a-dataframe-into-a-hive-catalog-table)\n      * [Load a Hive catalog table into a DataFrame](#load-a-hive-catalog-table-into-a-dataframe)\n      * [Load a DataFrame from a SQL query](#load-a-dataframe-from-a-sql-query)\n      * [Load a CSV file from Amazon S3](#load-a-csv-file-from-amazon-s3)\n      * [Load a CSV file from Oracle Cloud Infrastructure (OCI) Object Storage](#load-a-csv-file-from-oracle-cloud-infrastructure-oci-object-storage)\n      * [Read an Oracle DB table into a DataFrame using a Wallet](#read-an-oracle-db-table-into-a-dataframe-using-a-wallet)\n      * [Write a DataFrame to an Oracle DB table using a Wallet](#write-a-dataframe-to-an-oracle-db-table-using-a-wallet)\n      * [Write a DataFrame to a Postgres table](#write-a-dataframe-to-a-postgres-table)\n      * [Read a Postgres table into a DataFrame](#read-a-postgres-table-into-a-dataframe)\n   * [Data Handling Options](#data-handling-options)\n      * [Provide the schema when loading a DataFrame from CSV](#provide-the-schema-when-loading-a-dataframe-from-csv)\n      * [Save a DataFrame to CSV, overwriting existing data](#save-a-dataframe-to-csv-overwriting-existing-data)\n      * [Save a DataFrame to CSV with a header](#save-a-dataframe-to-csv-with-a-header)\n      * [Save a DataFrame in a single CSV file](#save-a-dataframe-in-a-single-csv-file)\n      * [Save DataFrame as a dynamic partitioned table](#save-dataframe-as-a-dynamic-partitioned-table)\n      * [Overwrite specific partitions](#overwrite-specific-partitions)\n      * [Load a CSV file with a money column into a DataFrame](#load-a-csv-file-with-a-money-column-into-a-dataframe)\n   * [DataFrame Operations](#dataframe-operations)\n      * [Add a new column to a DataFrame](#add-a-new-column-to-a-dataframe)\n      * [Modify a DataFrame column](#modify-a-dataframe-column)\n      * [Add a column with multiple conditions](#add-a-column-with-multiple-conditions)\n      * [Add a constant column](#add-a-constant-column)\n      * [Concatenate columns](#concatenate-columns)\n      * [Drop a column](#drop-a-column)\n      * [Change a column name](#change-a-column-name)\n      * [Change multiple column names](#change-multiple-column-names)\n      * [Change all column names at once](#change-all-column-names-at-once)\n      * [Convert a DataFrame column to a Python list](#convert-a-dataframe-column-to-a-python-list)\n      * [Convert a scalar query to a Python value](#convert-a-scalar-query-to-a-python-value)\n      * [Consume a DataFrame row-wise as Python dictionaries](#consume-a-dataframe-row-wise-as-python-dictionaries)\n      * [Select particular columns from a DataFrame](#select-particular-columns-from-a-dataframe)\n      * [Create an empty dataframe with a specified schema](#create-an-empty-dataframe-with-a-specified-schema)\n      * [Create a constant dataframe](#create-a-constant-dataframe)\n      * [Convert String to Double](#convert-string-to-double)\n      * [Convert String to Integer](#convert-string-to-integer)\n      * [Get the size of a DataFrame](#get-the-size-of-a-dataframe)\n      * [Get a DataFrame's number of partitions](#get-a-dataframe-s-number-of-partitions)\n      * [Get data types of a DataFrame's columns](#get-data-types-of-a-dataframe-s-columns)\n      * [Convert an RDD to Data Frame](#convert-an-rdd-to-data-frame)\n      * [Print the contents of an RDD](#print-the-contents-of-an-rdd)\n      * [Print the contents of a DataFrame](#print-the-contents-of-a-dataframe)\n      * [Process each row of a DataFrame](#process-each-row-of-a-dataframe)\n      * [DataFrame Map example](#dataframe-map-example)\n      * [DataFrame Flatmap example](#dataframe-flatmap-example)\n      * [Create a custom UDF](#create-a-custom-udf)\n   * [Transforming Data](#transforming-data)\n      * [Run a SparkSQL Statement on a DataFrame](#run-a-sparksql-statement-on-a-dataframe)\n      * [Extract data from a string using a regular expression](#extract-data-from-a-string-using-a-regular-expression)\n      * [Fill NULL values in specific columns](#fill-null-values-in-specific-columns)\n      * [Fill NULL values with column average](#fill-null-values-with-column-average)\n      * [Fill NULL values with group average](#fill-null-values-with-group-average)\n      * [Unpack a DataFrame's JSON column to a new DataFrame](#unpack-a-dataframe-s-json-column-to-a-new-dataframe)\n      * [Query a JSON column](#query-a-json-column)\n   * [Sorting and Searching](#sorting-and-searching)\n      * [Filter a column using a condition](#filter-a-column-using-a-condition)\n      * [Filter based on a specific column value](#filter-based-on-a-specific-column-value)\n      * [Filter based on an IN list](#filter-based-on-an-in-list)\n      * [Filter based on a NOT IN list](#filter-based-on-a-not-in-list)\n      * [Filter values based on keys in another DataFrame](#filter-values-based-on-keys-in-another-dataframe)\n      * [Get Dataframe rows that match a substring](#get-dataframe-rows-that-match-a-substring)\n      * [Filter a Dataframe based on a custom substring search](#filter-a-dataframe-based-on-a-custom-substring-search)\n      * [Filter based on a column's length](#filter-based-on-a-column-s-length)\n      * [Multiple filter conditions](#multiple-filter-conditions)\n      * [Sort DataFrame by a column](#sort-dataframe-by-a-column)\n      * [Take the first N rows of a DataFrame](#take-the-first-n-rows-of-a-dataframe)\n      * [Get distinct values of a column](#get-distinct-values-of-a-column)\n      * [Remove duplicates](#remove-duplicates)\n   * [Grouping](#grouping)\n      * [count(*) on a particular column](#count-on-a-particular-column)\n      * [Group and sort](#group-and-sort)\n      * [Filter groups based on an aggregate value, equivalent to SQL HAVING clause](#filter-groups-based-on-an-aggregate-value-equivalent-to-sql-having-clause)\n      * [Group by multiple columns](#group-by-multiple-columns)\n      * [Aggregate multiple columns](#aggregate-multiple-columns)\n      * [Aggregate multiple columns with custom orderings](#aggregate-multiple-columns-with-custom-orderings)\n      * [Get the maximum of a column](#get-the-maximum-of-a-column)\n      * [Sum a list of columns](#sum-a-list-of-columns)\n      * [Sum a column](#sum-a-column)\n      * [Aggregate all numeric columns](#aggregate-all-numeric-columns)\n      * [Count unique after grouping](#count-unique-after-grouping)\n      * [Count distinct values on all columns](#count-distinct-values-on-all-columns)\n      * [Group by then filter on the count](#group-by-then-filter-on-the-count)\n      * [Find the top N per row group (use N=1 for maximum)](#find-the-top-n-per-row-group-use-n-1-for-maximum)\n      * [Group key/values into a list](#group-key-values-into-a-list)\n      * [Compute a histogram](#compute-a-histogram)\n      * [Compute global percentiles](#compute-global-percentiles)\n      * [Compute percentiles within a partition](#compute-percentiles-within-a-partition)\n      * [Compute percentiles after aggregating](#compute-percentiles-after-aggregating)\n      * [Filter rows with values below a target percentile](#filter-rows-with-values-below-a-target-percentile)\n      * [Aggregate and rollup](#aggregate-and-rollup)\n      * [Aggregate and cube](#aggregate-and-cube)\n   * [Joining DataFrames](#joining-dataframes)\n      * [Join two DataFrames by column name](#join-two-dataframes-by-column-name)\n      * [Join two DataFrames with an expression](#join-two-dataframes-with-an-expression)\n      * [Multiple join conditions](#multiple-join-conditions)\n      * [Various Spark join types](#various-spark-join-types)\n      * [Concatenate two DataFrames](#concatenate-two-dataframes)\n      * [Load multiple files into a single DataFrame](#load-multiple-files-into-a-single-dataframe)\n      * [Subtract DataFrames](#subtract-dataframes)\n   * [File Processing](#file-processing)\n      * [Load Local File Details into a DataFrame](#load-local-file-details-into-a-dataframe)\n      * [Load Files from Oracle Cloud Infrastructure into a DataFrame](#load-files-from-oracle-cloud-infrastructure-into-a-dataframe)\n      * [Transform Many Images using Pillow](#transform-many-images-using-pillow)\n   * [Handling Missing Data](#handling-missing-data)\n      * [Filter rows with None or Null values](#filter-rows-with-none-or-null-values)\n      * [Drop rows with Null values](#drop-rows-with-null-values)\n      * [Count all Null or NaN values in a DataFrame](#count-all-null-or-nan-values-in-a-dataframe)\n   * [Dealing with Dates](#dealing-with-dates)\n      * [Convert an ISO 8601 formatted date string to date type](#convert-an-iso-8601-formatted-date-string-to-date-type)\n      * [Convert a custom formatted date string to date type](#convert-a-custom-formatted-date-string-to-date-type)\n      * [Get the last day of the current month](#get-the-last-day-of-the-current-month)\n      * [Convert UNIX (seconds since epoch) timestamp to date](#convert-unix-seconds-since-epoch-timestamp-to-date)\n      * [Load a CSV file with complex dates into a DataFrame](#load-a-csv-file-with-complex-dates-into-a-dataframe)\n   * [Unstructured Analytics](#unstructured-analytics)\n      * [Flatten top level text fields in a JSONl document](#flatten-top-level-text-fields-in-a-jsonl-document)\n      * [Flatten top level text fields from a JSON column](#flatten-top-level-text-fields-from-a-json-column)\n      * [Unnest an array of complex structures](#unnest-an-array-of-complex-structures)\n   * [Pandas](#pandas)\n      * [Convert Spark DataFrame to Pandas DataFrame](#convert-spark-dataframe-to-pandas-dataframe)\n      * [Convert Pandas DataFrame to Spark DataFrame with Schema Detection](#convert-pandas-dataframe-to-spark-dataframe-with-schema-detection)\n      * [Convert Pandas DataFrame to Spark DataFrame using a Custom Schema](#convert-pandas-dataframe-to-spark-dataframe-using-a-custom-schema)\n      * [Convert N rows from a DataFrame to a Pandas DataFrame](#convert-n-rows-from-a-dataframe-to-a-pandas-dataframe)\n      * [Grouped Aggregation with Pandas](#grouped-aggregation-with-pandas)\n      * [Use a Pandas Grouped Map Function via applyInPandas](#use-a-pandas-grouped-map-function-via-applyinpandas)\n   * [Data Profiling](#data-profiling)\n      * [Compute the number of NULLs across all columns](#compute-the-number-of-nulls-across-all-columns)\n      * [Compute average values of all numeric columns](#compute-average-values-of-all-numeric-columns)\n      * [Compute minimum values of all numeric columns](#compute-minimum-values-of-all-numeric-columns)\n      * [Compute maximum values of all numeric columns](#compute-maximum-values-of-all-numeric-columns)\n      * [Compute median values of all numeric columns](#compute-median-values-of-all-numeric-columns)\n      * [Identify Outliers in a DataFrame](#identify-outliers-in-a-dataframe)\n   * [Data Management](#data-management)\n      * [Save to a Delta Table](#save-to-a-delta-table)\n      * [Update records in a DataFrame using Delta Tables](#update-records-in-a-dataframe-using-delta-tables)\n      * [Merge into a Delta table](#merge-into-a-delta-table)\n      * [Show Table Version History](#show-table-version-history)\n      * [Load a Delta Table by Version ID (Time Travel Query)](#load-a-delta-table-by-version-id-time-travel-query)\n      * [Load a Delta Table by Timestamp (Time Travel Query)](#load-a-delta-table-by-timestamp-time-travel-query)\n      * [Compact a Delta Table](#compact-a-delta-table)\n      * [Add custom metadata to a Delta table write](#add-custom-metadata-to-a-delta-table-write)\n      * [Read custom Delta table metadata](#read-custom-delta-table-metadata)\n   * [Spark Streaming](#spark-streaming)\n      * [Connect to Kafka using SASL PLAIN authentication](#connect-to-kafka-using-sasl-plain-authentication)\n      * [Create a windowed Structured Stream over input CSV files](#create-a-windowed-structured-stream-over-input-csv-files)\n      * [Create an unwindowed Structured Stream over input CSV files](#create-an-unwindowed-structured-stream-over-input-csv-files)\n      * [Add the current timestamp to a DataFrame](#add-the-current-timestamp-to-a-dataframe)\n      * [Session analytics on a DataFrame](#session-analytics-on-a-dataframe)\n      * [Call a UDF only when a threshold is reached](#call-a-udf-only-when-a-threshold-is-reached)\n      * [Streaming Machine Learning](#streaming-machine-learning)\n      * [Control stream processing frequency](#control-stream-processing-frequency)\n      * [Write a streaming DataFrame to a database](#write-a-streaming-dataframe-to-a-database)\n   * [Time Series](#time-series)\n      * [Zero fill missing values in a timeseries](#zero-fill-missing-values-in-a-timeseries)\n      * [First Time an ID is Seen](#first-time-an-id-is-seen)\n      * [Cumulative Sum](#cumulative-sum)\n      * [Cumulative Sum in a Period](#cumulative-sum-in-a-period)\n      * [Cumulative Average](#cumulative-average)\n      * [Cumulative Average in a Period](#cumulative-average-in-a-period)\n   * [Machine Learning](#machine-learning)\n      * [Prepare data for training with a VectorAssembler](#prepare-data-for-training-with-a-vectorassembler)\n      * [A basic Random Forest Regression model](#a-basic-random-forest-regression-model)\n      * [Hyperparameter tuning](#hyperparameter-tuning)\n      * [Encode string variables as numbers](#encode-string-variables-as-numbers)\n      * [One-hot encode a categorical variable](#one-hot-encode-a-categorical-variable)\n      * [Optimize a model after a data preparation pipeline](#optimize-a-model-after-a-data-preparation-pipeline)\n      * [Evaluate Model Performance](#evaluate-model-performance)\n      * [Get feature importances of a trained model](#get-feature-importances-of-a-trained-model)\n      * [Plot Hyperparameter tuning metrics](#plot-hyperparameter-tuning-metrics)\n      * [Compute correlation matrix](#compute-correlation-matrix)\n      * [Save a model](#save-a-model)\n      * [Load a model and use it for transformations](#load-a-model-and-use-it-for-transformations)\n      * [Load a model and use it for predictions](#load-a-model-and-use-it-for-predictions)\n      * [Load a classification model and use it to compute confidences for output labels](#load-a-classification-model-and-use-it-to-compute-confidences-for-output-labels)\n   * [Performance](#performance)\n      * [Get the Spark version](#get-the-spark-version)\n      * [Log messages using Spark's Log4J](#log-messages-using-spark-s-log4j)\n      * [Cache a DataFrame](#cache-a-dataframe)\n      * [Show the execution plan, with costs](#show-the-execution-plan-with-costs)\n      * [Partition by a Column Value](#partition-by-a-column-value)\n      * [Range Partition a DataFrame](#range-partition-a-dataframe)\n      * [Change Number of DataFrame Partitions](#change-number-of-dataframe-partitions)\n      * [Coalesce DataFrame partitions](#coalesce-dataframe-partitions)\n      * [Set the number of shuffle partitions](#set-the-number-of-shuffle-partitions)\n      * [Sample a subset of a DataFrame](#sample-a-subset-of-a-dataframe)\n      * [Run multiple concurrent jobs in different pools](#run-multiple-concurrent-jobs-in-different-pools)\n      * [Print Spark configuration properties](#print-spark-configuration-properties)\n      * [Set Spark configuration properties](#set-spark-configuration-properties)\n      * [Publish Metrics to Graphite](#publish-metrics-to-graphite)\n      * [Increase Spark driver/executor heap space](#increase-spark-driver-executor-heap-space)\n\u003c!--te--\u003e\n    \n\nAccessing Data Sources\n======================\nLoading data stored in filesystems or databases, and saving it.\n\nLoad a DataFrame from CSV\n-------------------------\n\nSee https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameReader.html for a list of supported options.\n\n```python\ndf = spark.read.format(\"csv\").option(\"header\", True).load(\"data/auto-mpg.csv\")\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nLoad a DataFrame from a Tab Separated Value (TSV) file\n------------------------------------------------------\n\nSee https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameReader.html for a list of supported options.\n\n```python\ndf = (\n    spark.read.format(\"csv\")\n    .option(\"header\", True)\n    .option(\"sep\", \"\\t\")\n    .load(\"data/auto-mpg.tsv\")\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nSave a DataFrame in CSV format\n------------------------------\n\nSee https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html for a list of supported options.\n\n```python\nauto_df.write.csv(\"output.csv\")\n```\n\n\nLoad a DataFrame from Parquet\n-----------------------------\n\n```python\ndf = spark.read.format(\"parquet\").load(\"data/auto-mpg.parquet\")\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nSave a DataFrame in Parquet format\n----------------------------------\n\n```python\nauto_df.write.parquet(\"output.parquet\")\n```\n\n\nLoad a DataFrame from JSON Lines (jsonl) Formatted Data\n-------------------------------------------------------\n\nJSON Lines / jsonl format uses one JSON document per line. If you have data with mostly regular structure this is better than nesting it in an array. See [jsonlines.org](https://jsonlines.org/)\n\n```python\ndf = spark.read.json(\"data/weblog.jsonl\")\n```\n```\n# Code snippet result:\n+----------+----------+--------+----------+----------+------+\n|    client|   country| session| timestamp|       uri|  user|\n+----------+----------+--------+----------+----------+------+\n|{false,...|Bangladesh|55fa8213| 869196249|http://...|dde312|\n|{true, ...|      Niue|2fcd4a83|1031238717|http://...|9d00b9|\n|{true, ...|    Rwanda|013b996e| 628683372|http://...|1339d4|\n|{false,...|   Austria|07e8a71a|1043628668|https:/...|966312|\n|{false,...|    Belize|b23d05d8| 192738669|http://...|2af1e1|\n|{false,...|Lao Peo...|d83dfbae|1066490444|http://...|844395|\n|{false,...|French ...|e77dfaa2|1350920869|https:/...|  null|\n|{false,...|Turks a...|56664269| 280986223|http://...|  null|\n|{false,...|  Ethiopia|628d6059| 881914195|https:/...|8ab45a|\n|{false,...|Saint K...|85f9120c|1065114708|https:/...|  null|\n+----------+----------+--------+----------+----------+------+\nonly showing top 10 rows\n```\n\nSave a DataFrame into a Hive catalog table\n------------------------------------------\n\nSave a DataFrame to a Hive-compatible catalog. Use `table` to save in the session's current database or `database.table` to save\nin a specific database.\n\n```python\nauto_df.write.mode(\"overwrite\").saveAsTable(\"autompg\")\n```\n\n\nLoad a Hive catalog table into a DataFrame\n------------------------------------------\nLoad a DataFrame from a particular table. Use `table` to load from the session's current database or `database.table` to load from a specific database.\n\n```python\ndf = spark.table(\"autompg\")\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nLoad a DataFrame from a SQL query\n---------------------------------\n\nThis example shows loading a DataFrame from a query run over the a table in a Hive-compatible catalog.\n\n```python\ndf = sqlContext.sql(\n    \"select carname, mpg, horsepower from autompg where horsepower \u003e 100 and mpg \u003e 25\"\n)\n```\n```\n# Code snippet result:\n+----------+----+----------+\n|   carname| mpg|horsepower|\n+----------+----+----------+\n|  bmw 2002|26.0|     113.0|\n|chevrol...|28.8|     115.0|\n|oldsmob...|26.8|     115.0|\n|dodge colt|27.9|     105.0|\n|datsun ...|32.7|     132.0|\n|oldsmob...|26.6|     105.0|\n+----------+----+----------+\n```\n\nLoad a CSV file from Amazon S3\n------------------------------\n\nThis example shows how to load a CSV file from AWS S3. This example uses a credential pair and the `SimpleAWSCredentialsProvider`. For other authentication options, refer to the [Hadoop-AWS module documentation](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html).\n\n```python\nimport configparser\nimport os\n\nconfig = configparser.ConfigParser()\nconfig.read(os.path.expanduser(\"~/.aws/credentials\"))\naccess_key = config.get(\"default\", \"aws_access_key_id\").replace('\"', \"\")\nsecret_key = config.get(\"default\", \"aws_secret_access_key\").replace('\"', \"\")\n\n# Requires compatible hadoop-aws and aws-java-sdk-bundle JARs.\nspark.conf.set(\n    \"fs.s3a.aws.credentials.provider\",\n    \"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\",\n)\nspark.conf.set(\"fs.s3a.access.key\", access_key)\nspark.conf.set(\"fs.s3a.secret.key\", secret_key)\n\ndf = (\n    spark.read.format(\"csv\")\n    .option(\"header\", True)\n    .load(\"s3a://cheatsheet111/auto-mpg.csv\")\n)\n```\n\n\nLoad a CSV file from Oracle Cloud Infrastructure (OCI) Object Storage\n---------------------------------------------------------------------\n\nThis example shows loading data from Oracle Cloud Infrastructure Object Storage using an API key.\n\n```python\nimport oci\n\noci_config = oci.config.from_file()\nconf = spark.sparkContext.getConf()\nconf.set(\"fs.oci.client.auth.tenantId\", oci_config[\"tenancy\"])\nconf.set(\"fs.oci.client.auth.userId\", oci_config[\"user\"])\nconf.set(\"fs.oci.client.auth.fingerprint\", oci_config[\"fingerprint\"])\nconf.set(\"fs.oci.client.auth.pemfilepath\", oci_config[\"key_file\"])\nconf.set(\n    \"fs.oci.client.hostname\",\n    \"https://objectstorage.{0}.oraclecloud.com\".format(oci_config[\"region\"]),\n)\nPATH = \"oci://\u003cyour_bucket\u003e@\u003cyour_namespace/\u003cyour_path\u003e\"\ndf = spark.read.format(\"csv\").option(\"header\", True).load(PATH)\n```\n\n\nRead an Oracle DB table into a DataFrame using a Wallet\n-------------------------------------------------------\n\nGet the tnsname from tnsnames.ora. The wallet path should point to an extracted wallet file. The wallet files need to be available on all nodes.\n\n```python\npassword = \"my_password\"\ntable = \"source_table\"\ntnsname = \"my_tns_name\"\nuser = \"ADMIN\"\nwallet_path = \"/path/to/your/wallet\"\n\nproperties = {\n    \"driver\": \"oracle.jdbc.driver.OracleDriver\",\n    \"oracle.net.tns_admin\": tnsname,\n    \"password\": password,\n    \"user\": user,\n}\nurl = f\"jdbc:oracle:thin:@{tnsname}?TNS_ADMIN={wallet_path}\"\ndf = spark.read.jdbc(url=url, table=table, properties=properties)\n```\n\n\nWrite a DataFrame to an Oracle DB table using a Wallet\n------------------------------------------------------\n\nGet the tnsname from tnsnames.ora. The wallet path should point to an extracted wallet file. The wallet files need to be available on all nodes.\n\n```python\npassword = \"my_password\"\ntable = \"target_table\"\ntnsname = \"my_tns_name\"\nuser = \"ADMIN\"\nwallet_path = \"/path/to/your/wallet\"\n\nproperties = {\n    \"driver\": \"oracle.jdbc.driver.OracleDriver\",\n    \"oracle.net.tns_admin\": tnsname,\n    \"password\": password,\n    \"user\": user,\n}\nurl = f\"jdbc:oracle:thin:@{tnsname}?TNS_ADMIN={wallet_path}\"\n\n# Possible modes are \"Append\", \"Overwrite\", \"Ignore\", \"Error\"\ndf.write.jdbc(url=url, table=table, mode=\"Append\", properties=properties)\n```\n\n\nWrite a DataFrame to a Postgres table\n-------------------------------------\n\nYou need a Postgres JDBC driver to connect to a Postgres database.\n\nOptions include:\n- Add `org.postgresql:postgresql:\u003cversion\u003e` to `spark.jars.packages`\n- Provide the JDBC driver using `spark-submit --jars`\n- Add the JDBC driver to your Spark runtime (not recommended)\n\nIf you use Delta Lake there is a special procedure for specifying `spark.jars.packages`, see the source code that generates this file for details.\n\n```python\npg_database = os.environ.get(\"PGDATABASE\") or \"postgres\"\npg_host = os.environ.get(\"PGHOST\") or \"localhost\"\npg_password = os.environ.get(\"PGPASSWORD\") or \"password\"\npg_user = os.environ.get(\"PGUSER\") or \"postgres\"\ntable = \"autompg\"\n\nproperties = {\n    \"driver\": \"org.postgresql.Driver\",\n    \"user\": pg_user,\n    \"password\": pg_password,\n}\nurl = f\"jdbc:postgresql://{pg_host}:5432/{pg_database}\"\nauto_df.write.jdbc(url=url, table=table, mode=\"Append\", properties=properties)\n```\n\n\nRead a Postgres table into a DataFrame\n--------------------------------------\n\nYou need a Postgres JDBC driver to connect to a Postgres database.\n\nOptions include:\n- Add `org.postgresql:postgresql:\u003cversion\u003e` to `spark.jars.packages`\n- Provide the JDBC driver using `spark-submit --jars`\n- Add the JDBC driver to your Spark runtime (not recommended)\n\n```python\npg_database = os.environ.get(\"PGDATABASE\") or \"postgres\"\npg_host = os.environ.get(\"PGHOST\") or \"localhost\"\npg_password = os.environ.get(\"PGPASSWORD\") or \"password\"\npg_user = os.environ.get(\"PGUSER\") or \"postgres\"\ntable = \"autompg\"\n\nproperties = {\n    \"driver\": \"org.postgresql.Driver\",\n    \"user\": pg_user,\n    \"password\": pg_password,\n}\nurl = f\"jdbc:postgresql://{pg_host}:5432/{pg_database}\"\ndf = spark.read.jdbc(url=url, table=table, properties=properties)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nData Handling Options\n=====================\nSpecial data handling scenarios.\n\nProvide the schema when loading a DataFrame from CSV\n----------------------------------------------------\n\nSee https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/types.html for a list of types.\n\n```python\nfrom pyspark.sql.types import (\n    DoubleType,\n    IntegerType,\n    StringType,\n    StructField,\n    StructType,\n)\n\nschema = StructType(\n    [\n        StructField(\"mpg\", DoubleType(), True),\n        StructField(\"cylinders\", IntegerType(), True),\n        StructField(\"displacement\", DoubleType(), True),\n        StructField(\"horsepower\", DoubleType(), True),\n        StructField(\"weight\", DoubleType(), True),\n        StructField(\"acceleration\", DoubleType(), True),\n        StructField(\"modelyear\", IntegerType(), True),\n        StructField(\"origin\", IntegerType(), True),\n        StructField(\"carname\", StringType(), True),\n    ]\n)\ndf = (\n    spark.read.format(\"csv\")\n    .option(\"header\", \"true\")\n    .schema(schema)\n    .load(\"data/auto-mpg.csv\")\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0|3504.0|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0|3693.0|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0|3436.0|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0|3433.0|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0|3449.0|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0|4341.0|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0|4354.0|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0|4312.0|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0|4425.0|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0|3850.0|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nSave a DataFrame to CSV, overwriting existing data\n--------------------------------------------------\n\n```python\nauto_df.write.mode(\"overwrite\").csv(\"output.csv\")\n```\n\n\nSave a DataFrame to CSV with a header\n-------------------------------------\n\nSee https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html for a list of supported options.\n\n```python\nauto_df.coalesce(1).write.csv(\"header.csv\", header=\"true\")\n```\n\n\nSave a DataFrame in a single CSV file\n-------------------------------------\n\nThis example outputs CSV data to a single file. The file will be written in a directory called single.csv and have a random name. There is no way to change this behavior.\n\nIf you need to write to a single file with a name you choose, consider converting it to a Pandas dataframe and saving it using Pandas.\n\nEither way all data will be collected on one node before being written so be careful not to run out of memory.\n\n```python\nauto_df.coalesce(1).write.csv(\"single.csv\")\n```\n\n\nSave DataFrame as a dynamic partitioned table\n---------------------------------------------\n\nWhen you write using dynamic partitioning, the output partitions are determined bby the values of a column rather than specified in code.\n\nThe values of the partitions will appear as subdirectories and are not contained in the output files, i.e. they become \"virtual columns\". When you read a partition table these virtual columns will be part of the DataFrame.\n\nDynamic partitioning has the potential to create many small files, this will impact performance negatively. Be sure the partition columns do not have too many distinct values and limit the use of multiple virtual columns.\n\n```python\nspark.conf.set(\"spark.sql.sources.partitionOverwriteMode\", \"dynamic\")\nauto_df.write.mode(\"append\").partitionBy(\"modelyear\").saveAsTable(\n    \"autompg_partitioned\"\n)\n```\n\n\nOverwrite specific partitions\n-----------------------------\n\nEnabling dynamic partitioning lets you add or overwrite partitions based on DataFrame contents. Without dynamic partitioning the overwrite will overwrite the entire table.\n\nWith dynamic partitioning, partitions with keys in the DataFrame are overwritten, but partitions not in the DataFrame are untouched.\n\n```python\nspark.conf.set(\"spark.sql.sources.partitionOverwriteMode\", \"dynamic\")\nyour_dataframe.write.mode(\"overwrite\").insertInto(\"your_table\")\n```\n\n\nLoad a CSV file with a money column into a DataFrame\n----------------------------------------------------\n\nSpark is not that smart when it comes to parsing numbers, not allowing things like commas. If you need to load monetary amounts the safest option is to use a parsing library like `money_parser`.\n\n```python\nfrom pyspark.sql.functions import udf\nfrom pyspark.sql.types import DecimalType\nfrom decimal import Decimal\n\n# Load the text file.\ndf = (\n    spark.read.format(\"csv\")\n    .option(\"header\", True)\n    .load(\"data/customer_spend.csv\")\n)\n\n# Convert with a hardcoded custom UDF.\nmoney_udf = udf(lambda x: Decimal(x[1:].replace(\",\", \"\")), DecimalType(8, 4))\nmoney1 = df.withColumn(\"spend_dollars\", money_udf(df.spend_dollars))\n\n# Convert with the money_parser library (much safer).\nfrom money_parser import price_str\n\nmoney_convert = udf(\n    lambda x: Decimal(price_str(x)) if x is not None else None,\n    DecimalType(8, 4),\n)\ndf = df.withColumn(\"spend_dollars\", money_convert(df.spend_dollars))\n```\n```\n# Code snippet result:\n+----------+-----------+-------------+\n|      date|customer_id|spend_dollars|\n+----------+-----------+-------------+\n|2020-01-31|          0|       0.0700|\n|2020-01-31|          1|       0.9800|\n|2020-01-31|          2|       0.0600|\n|2020-01-31|          3|       0.6500|\n|2020-01-31|          4|       0.5700|\n|2020-02-29|          0|       0.1000|\n|2020-02-29|          2|       4.4000|\n|2020-02-29|          3|       0.3900|\n|2020-02-29|          4|       2.1300|\n|2020-02-29|          5|       0.8200|\n+----------+-----------+-------------+\nonly showing top 10 rows\n```\n\nDataFrame Operations\n====================\nAdding, removing and modifying DataFrame columns.\n\nAdd a new column to a DataFrame\n-------------------------------\n\n`withColumn` returns a new DataFrame with a column added to the source DataFrame. `withColumn` can be chained together multiple times.\n\n```python\nfrom pyspark.sql.functions import upper, lower\n\ndf = auto_df.withColumn(\"upper\", upper(auto_df.carname)).withColumn(\n    \"lower\", lower(auto_df.carname)\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+----------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|     upper|     lower|\n+----+---------+------------+----------+------+------------+---------+------+----------+----------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|CHEVROL...|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|BUICK S...|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|PLYMOUT...|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|AMC REB...|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|FORD TO...|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|FORD GA...|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|CHEVROL...|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|PLYMOUT...|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|PONTIAC...|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|AMC AMB...|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+----------+----------+\nonly showing top 10 rows\n```\n\nModify a DataFrame column\n-------------------------\n\nModify a column in-place using `withColumn`, specifying the output column name to be the same as the existing column name.\n\n```python\nfrom pyspark.sql.functions import col, concat, lit\n\ndf = auto_df.withColumn(\"modelyear\", concat(lit(\"19\"), col(\"modelyear\")))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|     1970|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|     1970|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|     1970|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|     1970|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|     1970|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|     1970|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|     1970|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|     1970|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|     1970|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|     1970|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nAdd a column with multiple conditions\n-------------------------------------\n\nTo set a new column's values when using `withColumn`, use the `when` / `otherwise` idiom. Multiple `when` conditions can be chained together.\n\n```python\nfrom pyspark.sql.functions import col, when\n\ndf = auto_df.withColumn(\n    \"mpg_class\",\n    when(col(\"mpg\") \u003c= 20, \"low\")\n    .when(col(\"mpg\") \u003c= 30, \"mid\")\n    .when(col(\"mpg\") \u003c= 40, \"high\")\n    .otherwise(\"very high\"),\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+---------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|mpg_class|\n+----+---------+------------+----------+------+------------+---------+------+----------+---------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|      low|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|      low|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|      low|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|      low|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|      low|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|      low|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|      low|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|      low|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|      low|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|      low|\n+----+---------+------------+----------+------+------------+---------+------+----------+---------+\nonly showing top 10 rows\n```\n\nAdd a constant column\n---------------------\n\n```python\nfrom pyspark.sql.functions import lit\n\ndf = auto_df.withColumn(\"one\", lit(1))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+---+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|one|\n+----+---------+------------+----------+------+------------+---------+------+----------+---+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|  1|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|  1|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|  1|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|  1|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|  1|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|  1|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|  1|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|  1|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|  1|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|  1|\n+----+---------+------------+----------+------+------------+---------+------+----------+---+\nonly showing top 10 rows\n```\n\nConcatenate columns\n-------------------\nTODO\n```python\nfrom pyspark.sql.functions import concat, col, lit\n\ndf = auto_df.withColumn(\n    \"concatenated\", concat(col(\"cylinders\"), lit(\"_\"), col(\"mpg\"))\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+------------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|concatenated|\n+----+---------+------------+----------+------+------------+---------+------+----------+------------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|      8_18.0|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|      8_15.0|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|      8_18.0|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|      8_16.0|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|      8_17.0|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|      8_15.0|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|      8_14.0|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|      8_14.0|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|      8_14.0|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|      8_15.0|\n+----+---------+------------+----------+------+------------+---------+------+----------+------------+\nonly showing top 10 rows\n```\n\nDrop a column\n-------------\n\n```python\ndf = auto_df.drop(\"horsepower\")\n```\n```\n# Code snippet result:\n+----+---------+------------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+------+------------+---------+------+----------+\n|18.0|        8|       307.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nChange a column name\n--------------------\n\n```python\ndf = auto_df.withColumnRenamed(\"horsepower\", \"horses\")\n```\n```\n# Code snippet result:\n+----+---------+------------+------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horses|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+------+------+------------+---------+------+----------+\n|18.0|        8|       307.0| 130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0| 165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0| 150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0| 150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0| 140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0| 198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0| 220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0| 215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0| 225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0| 190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nChange multiple column names\n----------------------------\n\nIf you need to change multiple column names you can chain `withColumnRenamed` calls together. If you want to change all column names see \"Change all column names at once\".\n\n```python\ndf = auto_df.withColumnRenamed(\"horsepower\", \"horses\").withColumnRenamed(\n    \"modelyear\", \"year\"\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+------+------+------------+----+------+----------+\n| mpg|cylinders|displacement|horses|weight|acceleration|year|origin|   carname|\n+----+---------+------------+------+------+------------+----+------+----------+\n|18.0|        8|       307.0| 130.0| 3504.|        12.0|  70|     1|chevrol...|\n|15.0|        8|       350.0| 165.0| 3693.|        11.5|  70|     1|buick s...|\n|18.0|        8|       318.0| 150.0| 3436.|        11.0|  70|     1|plymout...|\n|16.0|        8|       304.0| 150.0| 3433.|        12.0|  70|     1|amc reb...|\n|17.0|        8|       302.0| 140.0| 3449.|        10.5|  70|     1|ford to...|\n|15.0|        8|       429.0| 198.0| 4341.|        10.0|  70|     1|ford ga...|\n|14.0|        8|       454.0| 220.0| 4354.|         9.0|  70|     1|chevrol...|\n|14.0|        8|       440.0| 215.0| 4312.|         8.5|  70|     1|plymout...|\n|14.0|        8|       455.0| 225.0| 4425.|        10.0|  70|     1|pontiac...|\n|15.0|        8|       390.0| 190.0| 3850.|         8.5|  70|     1|amc amb...|\n+----+---------+------------+------+------+------------+----+------+----------+\nonly showing top 10 rows\n```\n\nChange all column names at once\n-------------------------------\n\nTo rename all columns use toDF with the desired column names in the argument list. This example puts an X in front of all column names.\n\n```python\ndf = auto_df.toDF(*[\"X\" + name for name in auto_df.columns])\n```\n```\n# Code snippet result:\n+----+----------+-------------+-----------+-------+-------------+----------+-------+----------+\n|Xmpg|Xcylinders|Xdisplacement|Xhorsepower|Xweight|Xacceleration|Xmodelyear|Xorigin|  Xcarname|\n+----+----------+-------------+-----------+-------+-------------+----------+-------+----------+\n|18.0|         8|        307.0|      130.0|  3504.|         12.0|        70|      1|chevrol...|\n|15.0|         8|        350.0|      165.0|  3693.|         11.5|        70|      1|buick s...|\n|18.0|         8|        318.0|      150.0|  3436.|         11.0|        70|      1|plymout...|\n|16.0|         8|        304.0|      150.0|  3433.|         12.0|        70|      1|amc reb...|\n|17.0|         8|        302.0|      140.0|  3449.|         10.5|        70|      1|ford to...|\n|15.0|         8|        429.0|      198.0|  4341.|         10.0|        70|      1|ford ga...|\n|14.0|         8|        454.0|      220.0|  4354.|          9.0|        70|      1|chevrol...|\n|14.0|         8|        440.0|      215.0|  4312.|          8.5|        70|      1|plymout...|\n|14.0|         8|        455.0|      225.0|  4425.|         10.0|        70|      1|pontiac...|\n|15.0|         8|        390.0|      190.0|  3850.|          8.5|        70|      1|amc amb...|\n+----+----------+-------------+-----------+-------+-------------+----------+-------+----------+\nonly showing top 10 rows\n```\n\nConvert a DataFrame column to a Python list\n-------------------------------------------\n\nSteps below:\n\n- `select` the target column, this example uses `carname`.\n- Access the DataFrame's rdd using `.rdd`\n- Use `flatMap` to convert the rdd's `Row` objects into simple values.\n- Use `collect` to assemble everything into a list.\n\n```python\nnames = auto_df.select(\"carname\").rdd.flatMap(lambda x: x).collect()\nprint(str(names[:10]))\n```\n```\n# Code snippet result:\n['chevrolet chevelle malibu', 'buick skylark 320', 'plymouth satellite', 'amc rebel sst', 'ford torino', 'ford galaxie 500', 'chevrolet impala', 'plymouth fury iii', 'pontiac catalina', 'amc ambassador dpl']\n```\n\nConvert a scalar query to a Python value\n----------------------------------------\n\nIf you have a `DataFrame` with one row and one column, how do you access its value?\n\nSteps below:\n- Create a DataFrame with one row and one column, this example uses an average but it could be anything.\n- Call the DataFrame's `first` method, this returns the first `Row` of the DataFrame.\n- `Row`s can be accessed like arrays, so we extract the zeroth value of the first `Row` using `first()[0]`.\n\n```python\naverage = auto_df.agg(dict(mpg=\"avg\")).first()[0]\nprint(str(average))\n```\n```\n# Code snippet result:\n23.514572864321615\n```\n\nConsume a DataFrame row-wise as Python dictionaries\n---------------------------------------------------\n\nSteps below:\n\n- `collect` all DataFrame Rows in the driver.\n- Iterate over the Rows.\n- Call the Row's `asDict` method to convert the Row to a Python dictionary.\n\n```python\nfirst_three = auto_df.limit(3)\nfor row in first_three.collect():\n    my_dict = row.asDict()\n    print(my_dict)\n```\n```\n# Code snippet result:\n\n{'mpg': '18.0', 'cylinders': '8', 'displacement': '307.0', 'horsepower': '130.0', 'weight': '3504.', 'acceleration': '12.0', 'modelyear': '70', 'origin': '1', 'carname': 'chevrolet chevelle malibu'}\n{'mpg': '15.0', 'cylinders': '8', 'displacement': '350.0', 'horsepower': '165.0', 'weight': '3693.', 'acceleration': '11.5', 'modelyear': '70', 'origin': '1', 'carname': 'buick skylark 320'}\n{'mpg': '18.0', 'cylinders': '8', 'displacement': '318.0', 'horsepower': '150.0', 'weight': '3436.', 'acceleration': '11.0', 'modelyear': '70', 'origin': '1', 'carname': 'plymouth satellite'}\n```\n\nSelect particular columns from a DataFrame\n------------------------------------------\n\n```python\ndf = auto_df.select([\"mpg\", \"cylinders\", \"displacement\"])\n```\n```\n# Code snippet result:\n+----+---------+------------+\n| mpg|cylinders|displacement|\n+----+---------+------------+\n|18.0|        8|       307.0|\n|15.0|        8|       350.0|\n|18.0|        8|       318.0|\n|16.0|        8|       304.0|\n|17.0|        8|       302.0|\n|15.0|        8|       429.0|\n|14.0|        8|       454.0|\n|14.0|        8|       440.0|\n|14.0|        8|       455.0|\n|15.0|        8|       390.0|\n+----+---------+------------+\nonly showing top 10 rows\n```\n\nCreate an empty dataframe with a specified schema\n-------------------------------------------------\n\nYou can create an empty `DataFrame` the same way you create other in-line `DataFrame`s, but using an empty list.\n\n```python\nfrom pyspark.sql.types import StructField, StructType, LongType, StringType\n\nschema = StructType(\n    [\n        StructField(\"my_id\", LongType(), True),\n        StructField(\"my_string\", StringType(), True),\n    ]\n)\ndf = spark.createDataFrame([], schema)\n```\n```\n# Code snippet result:\n+-----+---------+\n|my_id|my_string|\n+-----+---------+\n+-----+---------+\n```\n\nCreate a constant dataframe\n---------------------------\n\nConstant `DataFrame`s are mostly useful for unit tests.\n\n```python\nimport datetime\nfrom pyspark.sql.types import (\n    StructField,\n    StructType,\n    LongType,\n    StringType,\n    TimestampType,\n)\n\nschema = StructType(\n    [\n        StructField(\"my_id\", LongType(), True),\n        StructField(\"my_string\", StringType(), True),\n        StructField(\"my_timestamp\", TimestampType(), True),\n    ]\n)\ndf = spark.createDataFrame(\n    [\n        (1, \"foo\", datetime.datetime.strptime(\"2021-01-01\", \"%Y-%m-%d\")),\n        (2, \"bar\", datetime.datetime.strptime(\"2021-01-02\", \"%Y-%m-%d\")),\n    ],\n    schema,\n)\n```\n```\n# Code snippet result:\n+-----+---------+------------+\n|my_id|my_string|my_timestamp|\n+-----+---------+------------+\n|    1|      foo|  2021-01...|\n|    2|      bar|  2021-01...|\n+-----+---------+------------+\n```\n\nConvert String to Double\n------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.withColumn(\"horsepower\", col(\"horsepower\").cast(\"double\"))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nConvert String to Integer\n-------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.withColumn(\"horsepower\", col(\"horsepower\").cast(\"int\"))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|       130| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|       165| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|       150| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|       150| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|       140| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|       198| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|       220| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|       215| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|       225| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|       190| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nGet the size of a DataFrame\n---------------------------\n\n```python\nprint(\"{} rows\".format(auto_df.count()))\nprint(\"{} columns\".format(len(auto_df.columns)))\n```\n```\n# Code snippet result:\n398 rows\n9 columns\n```\n\nGet a DataFrame's number of partitions\n--------------------------------------\n\n```python\nprint(\"{} partition(s)\".format(auto_df.rdd.getNumPartitions()))\n```\n```\n# Code snippet result:\n1 partition(s)\n```\n\nGet data types of a DataFrame's columns\n---------------------------------------\n\n```python\nprint(auto_df.dtypes)\n```\n```\n# Code snippet result:\n[('mpg', 'string'), ('cylinders', 'string'), ('displacement', 'string'), ('horsepower', 'string'), ('weight', 'string'), ('acceleration', 'string'), ('modelyear', 'string'), ('origin', 'string'), ('carname', 'string')]\n```\n\nConvert an RDD to Data Frame\n----------------------------\n\nIf you have an `rdd` how do you convert it to a `DataFrame`? The `rdd` method `toDf` can be used, but the `rdd` must be a collection of `Row` objects.\n\nSteps below:\n- Create an `rdd` to be converted to a `DataFrame`.\n- Use the `rdd`'s `map` method:\n  - The example uses a lambda function to convert `rdd` elements to `Row`s.\n  - The `Row` constructor request key/value pairs with the key serving as the \"column name\".\n  - Each `rdd` entry is converted to a dictionary and the dictionary is unpacked to create the `Row`.\n  - `map` creates a new `rdd` containing all the `Row` objects.\n  - This new `rdd` is converted to a `DataFrame` using the `toDF` method.\n\nThe second example is a variation on the first, modifying source `rdd` entries while creating the target `rdd`.\n\n```python\nfrom pyspark.sql import Row\n\n# First, get the RDD from the DataFrame.\nrdd = auto_df.rdd\n\n# This converts it back to an RDD with no changes.\ndf = rdd.map(lambda x: Row(**x.asDict())).toDF()\n\n# This changes the rows before creating the DataFrame.\ndf = rdd.map(\n    lambda x: Row(**{k: v * 2 for (k, v) in x.asDict().items()})\n).toDF()\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|36.0|     16.0|       614.0|     260.0|7008.0|        24.0|      140|     2|chevrol...|\n|30.0|     16.0|       700.0|     330.0|7386.0|        23.0|      140|     2|buick s...|\n|36.0|     16.0|       636.0|     300.0|6872.0|        22.0|      140|     2|plymout...|\n|32.0|     16.0|       608.0|     300.0|6866.0|        24.0|      140|     2|amc reb...|\n|34.0|     16.0|       604.0|     280.0|6898.0|        21.0|      140|     2|ford to...|\n|30.0|     16.0|       858.0|     396.0|8682.0|        20.0|      140|     2|ford ga...|\n|28.0|     16.0|       908.0|     440.0|8708.0|        18.0|      140|     2|chevrol...|\n|28.0|     16.0|       880.0|     430.0|8624.0|        17.0|      140|     2|plymout...|\n|28.0|     16.0|       910.0|     450.0|8850.0|        20.0|      140|     2|pontiac...|\n|30.0|     16.0|       780.0|     380.0|7700.0|        17.0|      140|     2|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nPrint the contents of an RDD\n----------------------------\n\nTo see an `RDD`'s contents, convert the output of the `take` method to a string.\n\n```python\nrdd = auto_df.rdd\nprint(rdd.take(10))\n```\n```\n# Code snippet result:\n[Row(mpg='18.0', cylinders='8', displacement='307.0', horsepower='130.0', weight='3504.', acceleration='12.0', modelyear='70', origin='1', carname='chevrolet chevelle malibu'), Row(mpg='15.0', cylinders='8', displacement='350.0', horsepower='165.0', weight='3693.', acceleration='11.5', modelyear='70', origin='1', carname='buick skylark 320'), Row(mpg='18.0', cylinders='8', displacement='318.0', horsepower='150.0', weight='3436.', acceleration='11.0', modelyear='70', origin='1', carname='plymouth satellite'), Row(mpg='16.0', cylinders='8', displacement='304.0', horsepower='150.0', weight='3433.', acceleration='12.0', modelyear='70', origin='1', carname='amc rebel sst'), Row(mpg='17.0', cylinders='8', displacement='302.0', horsepower='140.0', weight='3449.', acceleration='10.5', modelyear='70', origin='1', carname='ford torino'), Row(mpg='15.0', cylinders='8', displacement='429.0', horsepower='198.0', weight='4341.', acceleration='10.0', modelyear='70', origin='1', carname='ford galaxie 500'), Row(mpg='14.0', cylinders='8', displacement='454.0', horsepower='220.0', weight='4354.', acceleration='9.0', modelyear='70', origin='1', carname='chevrolet impala'), Row(mpg='14.0', cylinders='8', displacement='440.0', horsepower='215.0', weight='4312.', acceleration='8.5', modelyear='70', origin='1', carname='plymouth fury iii'), Row(mpg='14.0', cylinders='8', displacement='455.0', horsepower='225.0', weight='4425.', acceleration='10.0', modelyear='70', origin='1', carname='pontiac catalina'), Row(mpg='15.0', cylinders='8', displacement='390.0', horsepower='190.0', weight='3850.', acceleration='8.5', modelyear='70', origin='1', carname='amc ambassador dpl')]\n```\n\nPrint the contents of a DataFrame\n---------------------------------\n\n```python\nauto_df.show(10)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nProcess each row of a DataFrame\n-------------------------------\n\nUse the `foreach` function to process each row of a `DataFrame` using a Python function. The function will get one argument, a `Row` object. The `Row` will have properties whose names map to the `DataFrame`'s columns.\n\n```python\nimport os\n\ndef foreach_function(row):\n    if row.horsepower is not None:\n        os.system(\"echo \" + row.horsepower)\n\nauto_df.foreach(foreach_function)\n```\n\n\nDataFrame Map example\n---------------------\n\nYou can run `map` on a `DataFrame` by accessing its underlying `RDD`. It is much more common to use `foreach` directly on the `DataFrame` itself. This can be useful if you have code written specifically for `RDD`s that you need to use against a `DataFrame`.\n\n```python\ndef map_function(row):\n    if row.horsepower is not None:\n        return [float(row.horsepower) * 10]\n    else:\n        return [None]\n\ndf = auto_df.rdd.map(map_function).toDF()\n```\n```\n# Code snippet result:\n+------+\n|    _1|\n+------+\n|1300.0|\n|1650.0|\n|1500.0|\n|1500.0|\n|1400.0|\n|1980.0|\n|2200.0|\n|2150.0|\n|2250.0|\n|1900.0|\n+------+\nonly showing top 10 rows\n```\n\nDataFrame Flatmap example\n-------------------------\n\nUse `flatMap` when you have a UDF that produces a list of `Rows` per input `Row`. `flatMap` is an `RDD` operation so we need to access the `DataFrame`'s `RDD`, call `flatMap` and convert the resulting `RDD` back into a `DataFrame`. Spark will handle \"flatting\" arrays into the output `RDD`.\n\nNote also that you can [`yield`](https://docs.python.org/3/reference/expressions.html#yield-expressions) results rather than returning full lists which can simplify code considerably.\n\n```python\nfrom pyspark.sql.types import Row\n\ndef flatmap_function(row):\n    if row.cylinders is not None:\n        return list(range(int(row.cylinders)))\n    else:\n        return [None]\n\nrdd = auto_df.rdd.flatMap(flatmap_function)\nrow = Row(\"val\")\ndf = rdd.map(row).toDF()\n```\n```\n# Code snippet result:\n+---+\n|val|\n+---+\n|  0|\n|  1|\n|  2|\n|  3|\n|  4|\n|  5|\n|  6|\n|  7|\n|  0|\n|  1|\n+---+\nonly showing top 10 rows\n```\n\nCreate a custom UDF\n-------------------\n\nCreate a UDF by providing a function to the udf function. This example shows a [lambda function](https://docs.python.org/3/reference/expressions.html#lambdas-1). You can also use ordinary functions for more complex UDFs.\n\n```python\nfrom pyspark.sql.functions import col, udf\nfrom pyspark.sql.types import StringType\n\nfirst_word_udf = udf(lambda x: x.split()[0], StringType())\ndf = auto_df.withColumn(\"manufacturer\", first_word_udf(col(\"carname\")))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+------------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|manufacturer|\n+----+---------+------------+----------+------+------------+---------+------+----------+------------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|   chevrolet|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|       buick|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|    plymouth|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|         amc|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|        ford|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|        ford|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|   chevrolet|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|    plymouth|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|     pontiac|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|         amc|\n+----+---------+------------+----------+------+------------+---------+------+----------+------------+\nonly showing top 10 rows\n```\n\nTransforming Data\n=================\nData conversions and other modifications.\n\nRun a SparkSQL Statement on a DataFrame\n---------------------------------------\n\nYou can run arbitrary SQL statements on a `DataFrame` provided you:\n\n1. Register the `DataFrame` as a temporary table using `registerTempTable`.\n2. Use `sqlContext.sql` and use the temp table name you specified as the table source.\n\nYou can also join `DataFrames` if you register them. If you're porting complex SQL from another application this can be a lot easier than converting it to use `DataFrame` SQL APIs.\n\n```python\nfrom pyspark.sql.functions import col, regexp_extract\n\nauto_df.registerTempTable(\"auto_df\")\ndf = sqlContext.sql(\n    \"select modelyear, avg(mpg) from auto_df group by modelyear\"\n)\n```\n```\n# Code snippet result:\n+---------+----------+\n|modelyear|  avg(mpg)|\n+---------+----------+\n|       73|      17.1|\n|       71|     21.25|\n|       70|17.6896...|\n|       75|20.2666...|\n|       78|24.0611...|\n|       77|    23.375|\n|       82|31.7096...|\n|       81|30.3344...|\n|       79|25.0931...|\n|       72|18.7142...|\n+---------+----------+\nonly showing top 10 rows\n```\n\nExtract data from a string using a regular expression\n-----------------------------------------------------\n\n```python\nfrom pyspark.sql.functions import col, regexp_extract\n\ngroup = 0\ndf = (\n    auto_df.withColumn(\n        \"identifier\", regexp_extract(col(\"carname\"), \"(\\S?\\d+)\", group)\n    )\n    .drop(\"acceleration\")\n    .drop(\"cylinders\")\n    .drop(\"displacement\")\n    .drop(\"modelyear\")\n    .drop(\"mpg\")\n    .drop(\"origin\")\n    .drop(\"horsepower\")\n    .drop(\"weight\")\n)\n```\n```\n# Code snippet result:\n+----------+----------+\n|   carname|identifier|\n+----------+----------+\n|chevrol...|          |\n|buick s...|       320|\n|plymout...|          |\n|amc reb...|          |\n|ford to...|          |\n|ford ga...|       500|\n|chevrol...|          |\n|plymout...|          |\n|pontiac...|          |\n|amc amb...|          |\n+----------+----------+\nonly showing top 10 rows\n```\n\nFill NULL values in specific columns\n------------------------------------\n\n```python\ndf = auto_df.fillna({\"horsepower\": 0})\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFill NULL values with column average\n------------------------------------\n\n```python\nfrom pyspark.sql.functions import avg\n\ndf = auto_df.fillna({\"horsepower\": auto_df.agg(avg(\"horsepower\")).first()[0]})\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFill NULL values with group average\n-----------------------------------\n\nSometimes NULL values in a column cause problems and it's better to guess at a value than leave it NULL. There are several strategies for doing with this. This example shows replacing NULL values with the average value within that column.\n\n```python\nfrom pyspark.sql.functions import coalesce\n\nunmodified_columns = auto_df.columns\nunmodified_columns.remove(\"horsepower\")\nmanufacturer_avg = auto_df.groupBy(\"cylinders\").agg({\"horsepower\": \"avg\"})\ndf = auto_df.join(manufacturer_avg, \"cylinders\").select(\n    *unmodified_columns,\n    coalesce(\"horsepower\", \"avg(horsepower)\").alias(\"horsepower\"),\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+------+------------+---------+------+----------+----------+\n| mpg|cylinders|displacement|weight|acceleration|modelyear|origin|   carname|horsepower|\n+----+---------+------------+------+------------+---------+------+----------+----------+\n|18.0|        8|       307.0| 3504.|        12.0|       70|     1|chevrol...|     130.0|\n|15.0|        8|       350.0| 3693.|        11.5|       70|     1|buick s...|     165.0|\n|18.0|        8|       318.0| 3436.|        11.0|       70|     1|plymout...|     150.0|\n|16.0|        8|       304.0| 3433.|        12.0|       70|     1|amc reb...|     150.0|\n|17.0|        8|       302.0| 3449.|        10.5|       70|     1|ford to...|     140.0|\n|15.0|        8|       429.0| 4341.|        10.0|       70|     1|ford ga...|     198.0|\n|14.0|        8|       454.0| 4354.|         9.0|       70|     1|chevrol...|     220.0|\n|14.0|        8|       440.0| 4312.|         8.5|       70|     1|plymout...|     215.0|\n|14.0|        8|       455.0| 4425.|        10.0|       70|     1|pontiac...|     225.0|\n|15.0|        8|       390.0| 3850.|         8.5|       70|     1|amc amb...|     190.0|\n+----+---------+------------+------+------------+---------+------+----------+----------+\nonly showing top 10 rows\n```\n\nUnpack a DataFrame's JSON column to a new DataFrame\n---------------------------------------------------\n\n```python\nfrom pyspark.sql.functions import col, json_tuple\n\nsource = spark.sparkContext.parallelize(\n    [[\"1\", '{ \"a\" : 10, \"b\" : 11 }'], [\"2\", '{ \"a\" : 20, \"b\" : 21 }']]\n).toDF([\"id\", \"json\"])\ndf = source.select(\"id\", json_tuple(col(\"json\"), \"a\", \"b\"))\n```\n```\n# Code snippet result:\n+---+---+---+\n| id| c0| c1|\n+---+---+---+\n|  1| 10| 11|\n|  2| 20| 21|\n+---+---+---+\n```\n\nQuery a JSON column\n-------------------\n\nIf you have JSON text data embedded in a String column, `json_tuple` will parse that text and extract fields within the JSON text.\n\n```python\nfrom pyspark.sql.functions import col, json_tuple\n\nsource = spark.sparkContext.parallelize(\n    [[\"1\", '{ \"a\" : 10, \"b\" : 11 }'], [\"2\", '{ \"a\" : 20, \"b\" : 21 }']]\n).toDF([\"id\", \"json\"])\ndf = (\n    source.select(\"id\", json_tuple(col(\"json\"), \"a\", \"b\"))\n    .withColumnRenamed(\"c0\", \"a\")\n    .withColumnRenamed(\"c1\", \"b\")\n    .where(col(\"b\") \u003e 15)\n)\n```\n```\n# Code snippet result:\n+---+---+---+\n| id|  a|  b|\n+---+---+---+\n|  2| 20| 21|\n+---+---+---+\n```\n\nSorting and Searching\n=====================\nFiltering, sorting, removing duplicates and more.\n\nFilter a column using a condition\n---------------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.filter(col(\"mpg\") \u003e \"30\")\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| 9.0|        8|       304.0|     193.0| 4732.|        18.5|       70|     1|  hi 1200d|\n|30.0|        4|       79.00|     70.00| 2074.|        19.5|       71|     2|peugeot...|\n|30.0|        4|       88.00|     76.00| 2065.|        14.5|       71|     2| fiat 124b|\n|31.0|        4|       71.00|     65.00| 1773.|        19.0|       71|     3|toyota ...|\n|35.0|        4|       72.00|     69.00| 1613.|        18.0|       71|     3|datsun ...|\n|31.0|        4|       79.00|     67.00| 1950.|        19.0|       74|     3|datsun ...|\n|32.0|        4|       71.00|     65.00| 1836.|        21.0|       74|     3|toyota ...|\n|31.0|        4|       76.00|     52.00| 1649.|        16.5|       74|     3|toyota ...|\n|32.0|        4|       83.00|     61.00| 2003.|        19.0|       74|     3|datsun 710|\n|31.0|        4|       79.00|     67.00| 2000.|        16.0|       74|     2| fiat x1.9|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFilter based on a specific column value\n---------------------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.where(col(\"cylinders\") == \"8\")\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFilter based on an IN list\n--------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.where(col(\"cylinders\").isin([\"4\", \"6\"]))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|24.0|        4|       113.0|     95.00| 2372.|        15.0|       70|     3|toyota ...|\n|22.0|        6|       198.0|     95.00| 2833.|        15.5|       70|     1|plymout...|\n|18.0|        6|       199.0|     97.00| 2774.|        15.5|       70|     1|amc hornet|\n|21.0|        6|       200.0|     85.00| 2587.|        16.0|       70|     1|ford ma...|\n|27.0|        4|       97.00|     88.00| 2130.|        14.5|       70|     3|datsun ...|\n|26.0|        4|       97.00|     46.00| 1835.|        20.5|       70|     2|volkswa...|\n|25.0|        4|       110.0|     87.00| 2672.|        17.5|       70|     2|peugeot...|\n|24.0|        4|       107.0|     90.00| 2430.|        14.5|       70|     2|audi 10...|\n|25.0|        4|       104.0|     95.00| 2375.|        17.5|       70|     2|  saab 99e|\n|26.0|        4|       121.0|     113.0| 2234.|        12.5|       70|     2|  bmw 2002|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFilter based on a NOT IN list\n-----------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.where(~col(\"cylinders\").isin([\"4\", \"6\"]))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFilter values based on keys in another DataFrame\n------------------------------------------------\n\nIf you have `DataFrame` 1 containing values you want to remove from `DataFrame` 2, join them using the `left_anti` join strategy.\n\n```python\nfrom pyspark.sql.functions import col\n\n# Our DataFrame of keys to exclude.\nexclude_keys = auto_df.select(\n    (col(\"modelyear\") + 1).alias(\"adjusted_year\")\n).distinct()\n\n# The anti join returns only keys with no matches.\nfiltered = auto_df.join(\n    exclude_keys,\n    how=\"left_anti\",\n    on=auto_df.modelyear == exclude_keys.adjusted_year,\n)\n\n# Alternatively we can register a temporary table and use a SQL expression.\nexclude_keys.registerTempTable(\"exclude_keys\")\ndf = auto_df.filter(\n    \"modelyear not in ( select adjusted_year from exclude_keys )\"\n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|      8.0|       307.0|     130.0|3504.0|        12.0|       70|     1|chevrol...|\n|15.0|      8.0|       350.0|     165.0|3693.0|        11.5|       70|     1|buick s...|\n|18.0|      8.0|       318.0|     150.0|3436.0|        11.0|       70|     1|plymout...|\n|16.0|      8.0|       304.0|     150.0|3433.0|        12.0|       70|     1|amc reb...|\n|17.0|      8.0|       302.0|     140.0|3449.0|        10.5|       70|     1|ford to...|\n|15.0|      8.0|       429.0|     198.0|4341.0|        10.0|       70|     1|ford ga...|\n|14.0|      8.0|       454.0|     220.0|4354.0|         9.0|       70|     1|chevrol...|\n|14.0|      8.0|       440.0|     215.0|4312.0|         8.5|       70|     1|plymout...|\n|14.0|      8.0|       455.0|     225.0|4425.0|        10.0|       70|     1|pontiac...|\n|15.0|      8.0|       390.0|     190.0|3850.0|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nGet Dataframe rows that match a substring\n-----------------------------------------\n\n```python\ndf = auto_df.where(auto_df.carname.contains(\"custom\"))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|16.0|        6|       225.0|     105.0| 3439.|        15.5|       71|     1|plymout...|\n|13.0|        8|       350.0|     155.0| 4502.|        13.5|       72|     1|buick l...|\n|14.0|        8|       318.0|     150.0| 4077.|        14.0|       72|     1|plymout...|\n|15.0|        8|       318.0|     150.0| 3777.|        12.5|       73|     1|dodge c...|\n|12.0|        8|       455.0|     225.0| 4951.|        11.0|       73|     1|buick e...|\n|16.0|        6|       250.0|     100.0| 3278.|        18.0|       73|     1|chevrol...|\n|13.0|        8|       360.0|     170.0| 4654.|        13.0|       73|     1|plymout...|\n|15.0|        8|       318.0|     150.0| 3399.|        11.0|       73|     1|dodge d...|\n|14.0|        8|       318.0|     150.0| 4457.|        13.5|       74|     1|dodge c...|\n|19.0|        6|       225.0|     95.00| 3264.|        16.0|       75|     1|plymout...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFilter a Dataframe based on a custom substring search\n-----------------------------------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.where(col(\"carname\").like(\"%custom%\"))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|16.0|        6|       225.0|     105.0| 3439.|        15.5|       71|     1|plymout...|\n|13.0|        8|       350.0|     155.0| 4502.|        13.5|       72|     1|buick l...|\n|14.0|        8|       318.0|     150.0| 4077.|        14.0|       72|     1|plymout...|\n|15.0|        8|       318.0|     150.0| 3777.|        12.5|       73|     1|dodge c...|\n|12.0|        8|       455.0|     225.0| 4951.|        11.0|       73|     1|buick e...|\n|16.0|        6|       250.0|     100.0| 3278.|        18.0|       73|     1|chevrol...|\n|13.0|        8|       360.0|     170.0| 4654.|        13.0|       73|     1|plymout...|\n|15.0|        8|       318.0|     150.0| 3399.|        11.0|       73|     1|dodge d...|\n|14.0|        8|       318.0|     150.0| 4457.|        13.5|       74|     1|dodge c...|\n|19.0|        6|       225.0|     95.00| 3264.|        16.0|       75|     1|plymout...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nFilter based on a column's length\n---------------------------------\n\n```python\nfrom pyspark.sql.functions import col, length\n\ndf = auto_df.where(length(col(\"carname\")) \u003c 12)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|18.0|        6|       199.0|     97.00| 2774.|        15.5|       70|     1|amc hornet|\n|25.0|        4|       110.0|     87.00| 2672.|        17.5|       70|     2|peugeot...|\n|24.0|        4|       107.0|     90.00| 2430.|        14.5|       70|     2|audi 10...|\n|25.0|        4|       104.0|     95.00| 2375.|        17.5|       70|     2|  saab 99e|\n|26.0|        4|       121.0|     113.0| 2234.|        12.5|       70|     2|  bmw 2002|\n|21.0|        6|       199.0|     90.00| 2648.|        15.0|       70|     1|amc gre...|\n|10.0|        8|       360.0|     215.0| 4615.|        14.0|       70|     1| ford f250|\n|10.0|        8|       307.0|     200.0| 4376.|        15.0|       70|     1| chevy c20|\n|11.0|        8|       318.0|     210.0| 4382.|        13.5|       70|     1|dodge d200|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nMultiple filter conditions\n--------------------------\n\nThe key thing to remember if you have multiple filter conditions is that `filter` accepts standard Python expressions. Use bitwise operators to handle and/or conditions.\n\n```python\nfrom pyspark.sql.functions import col\n\n# OR\ndf = auto_df.filter((col(\"mpg\") \u003e \"30\") | (col(\"acceleration\") \u003c \"10\"))\n# AND\ndf = auto_df.filter((col(\"mpg\") \u003e \"30\") \u0026 (col(\"acceleration\") \u003c \"13\"))\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|32.7|        6|       168.0|     132.0| 2910.|        11.4|       80|     3|datsun ...|\n|30.0|        4|       135.0|     84.00| 2385.|        12.9|       81|     1|plymout...|\n|32.0|        4|       135.0|     84.00| 2295.|        11.6|       82|     1|dodge r...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n```\n\nSort DataFrame by a column\n--------------------------\n\n```python\nfrom pyspark.sql.functions import col\n\ndf = auto_df.orderBy(\"carname\")\ndf = auto_df.orderBy(col(\"carname\").desc())\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|31.9|        4|       89.00|     71.00| 1925.|        14.0|       79|     2|vw rabb...|\n|44.3|        4|       90.00|     48.00| 2085.|        21.7|       80|     2|vw rabb...|\n|29.0|        4|       90.00|     70.00| 1937.|        14.2|       76|     2| vw rabbit|\n|41.5|        4|       98.00|     76.00| 2144.|        14.7|       80|     2| vw rabbit|\n|44.0|        4|       97.00|     52.00| 2130.|        24.6|       82|     2| vw pickup|\n|43.4|        4|       90.00|     48.00| 2335.|        23.7|       80|     2|vw dash...|\n|30.7|        6|       145.0|     76.00| 3160.|        19.6|       81|     2|volvo d...|\n|17.0|        6|       163.0|     125.0| 3140.|        13.6|       78|     2|volvo 2...|\n|20.0|        4|       130.0|     102.0| 3150.|        15.7|       76|     2| volvo 245|\n|22.0|        4|       121.0|     98.00| 2945.|        14.5|       75|     2|volvo 2...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nTake the first N rows of a DataFrame\n------------------------------------\n\n```python\nn = 10\ndf = auto_df.limit(n)\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|18.0|        8|       307.0|     130.0| 3504.|        12.0|       70|     1|chevrol...|\n|15.0|        8|       350.0|     165.0| 3693.|        11.5|       70|     1|buick s...|\n|18.0|        8|       318.0|     150.0| 3436.|        11.0|       70|     1|plymout...|\n|16.0|        8|       304.0|     150.0| 3433.|        12.0|       70|     1|amc reb...|\n|17.0|        8|       302.0|     140.0| 3449.|        10.5|       70|     1|ford to...|\n|15.0|        8|       429.0|     198.0| 4341.|        10.0|       70|     1|ford ga...|\n|14.0|        8|       454.0|     220.0| 4354.|         9.0|       70|     1|chevrol...|\n|14.0|        8|       440.0|     215.0| 4312.|         8.5|       70|     1|plymout...|\n|14.0|        8|       455.0|     225.0| 4425.|        10.0|       70|     1|pontiac...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n```\n\nGet distinct values of a column\n-------------------------------\n\n```python\ndf = auto_df.select(\"cylinders\").distinct()\n```\n```\n# Code snippet result:\n+---------+\n|cylinders|\n+---------+\n|        3|\n|        8|\n|        5|\n|        6|\n|        4|\n+---------+\n```\n\nRemove duplicates\n-----------------\n\n```python\ndf = auto_df.dropDuplicates([\"carname\"])\n```\n```\n# Code snippet result:\n+----+---------+------------+----------+------+------------+---------+------+----------+\n| mpg|cylinders|displacement|horsepower|weight|acceleration|modelyear|origin|   carname|\n+----+---------+------------+----------+------+------------+---------+------+----------+\n|13.0|        8|       360.0|     175.0| 3821.|        11.0|       73|     1|amc amb...|\n|15.0|        8|       390.0|     190.0| 3850.|         8.5|       70|     1|amc amb...|\n|17.0|        8|       304.0|     150.0| 3672.|        11.5|       72|     1|amc amb...|\n|19.4|        6|       232.0|     90.00| 3210.|        17.2|       78|     1|amc con...|\n|18.1|        6|       258.0|     120.0| 3410.|        15.1|       78|     1|amc con...|\n|23.0|        4|       151.0|      null| 3035.|        20.5|       82|     1|amc con...|\n|20.2|        6|       232.0|     90.00| 3265.|        18.2|       79|     1|amc con...|\n|21.0|        6|       199.0|     90.00| 2648.|        15.0|       70|     1|amc gre...|\n|18.0|        6|       199.0|     97.00| 2774.|        15.5|       70|     1|amc hornet|\n|18.0|        6|       258.0|     110.0| 2962.|        13.5|       71|     1|amc hor...|\n+----+---------+------------+----------+------+------------+---------+------+----------+\nonly showing top 10 rows\n```\n\nGrouping\n========\nGroup DataFrame data by key to perform aggregates like counting, sums, averages, etc.\n\ncount(*) on a particular column\n-------------------------------\n\n```python\nfrom pyspark.sql.functions import desc\n\n# No sorting.\ndf = auto_df.groupBy(\"cylinders\").count()\n\n# With sorting.\ndf = auto_df.groupBy(\"cylinders\").count().orderBy(desc(\"count\"))\n```\n```\n# Code snippet result:\n+---------+-----+\n|cylinders|count|\n+---------+-----+\n|        4|  204|\n|        8|  103|\n|        6|   84|\n|        3|    4|\n|        5|    3|\n+---------+-----+\n```\n\nGroup and sort\n--------------\n\n```python\nfrom pyspark.sql.functions import avg, desc\n\ndf = (\n    auto_df.groupBy(\"cylinders\")\n    .agg(avg(\"horsepower\").alias(\"avg_horsepower\"))\n    .orderBy(desc(\"avg_horsepower\"))\n)\n```\n```\n# Code snippet result:\n+---------+--------------+\n|cylinders|avg_horsepower|\n+---------+--------------+\n|        8|    158.300...|\n|        6|    101.506...|\n|        3|         99.25|\n|        5|    82.3333...|\n|        4|    78.2814...|\n+---------+--------------+\n```\n\nFilter groups based on an aggregate value, equivalent to SQL HAVING clause\n--------------------------------------------------------------------------\n\nTo filter values after an aggregation simply use `.filter` on the `DataFrame` after the aggregate, using the column name the aggregate generates.\n\n```python\nfrom pyspark.sql.functions import col, desc\n\ndf = (\n    auto_df.groupBy(\"cylinders\")\n    .count()\n    .orderBy(desc(\"count\"))\n    .filter(col(\"coun","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcartershanklin%2Fpyspark-cheatsheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcartershanklin%2Fpyspark-cheatsheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcartershanklin%2Fpyspark-cheatsheet/lists"}