{"id":13486924,"url":"https://github.com/kevinschaich/pyspark-cheatsheet","last_synced_at":"2025-04-10T22:36:22.531Z","repository":{"id":42680756,"uuid":"174259884","full_name":"kevinschaich/pyspark-cheatsheet","owner":"kevinschaich","description":"🐍 Quick reference guide to common patterns \u0026 functions in PySpark.","archived":false,"fork":false,"pushed_at":"2023-02-21T13:19:32.000Z","size":51,"stargazers_count":509,"open_issues_count":2,"forks_count":164,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-24T19:22:16.577Z","etag":null,"topics":["cheat","cheatsheet","cheatsheets","data","data-science","docs","documentation","guide","guides","pyspark","pyspark-tutorial","quickstart","reference","references","spark","spark-sql"],"latest_commit_sha":null,"homepage":"https://spark.apache.org/docs/latest/api/python/pyspark.sql.html","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevinschaich.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-03-07T02:52:32.000Z","updated_at":"2025-03-24T13:35:29.000Z","dependencies_parsed_at":"2024-01-03T02:30:09.431Z","dependency_job_id":"53e253e4-b3c2-4722-ab8c-c56d2d9fd3f0","html_url":"https://github.com/kevinschaich/pyspark-cheatsheet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinschaich%2Fpyspark-cheatsheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinschaich%2Fpyspark-cheatsheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinschaich%2Fpyspark-cheatsheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinschaich%2Fpyspark-cheatsheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevinschaich","download_url":"https://codeload.github.com/kevinschaich/pyspark-cheatsheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248311668,"owners_count":21082630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cheat","cheatsheet","cheatsheets","data","data-science","docs","documentation","guide","guides","pyspark","pyspark-tutorial","quickstart","reference","references","spark","spark-sql"],"created_at":"2024-07-31T18:00:53.109Z","updated_at":"2025-10-14T12:03:33.055Z","avatar_url":"https://github.com/kevinschaich.png","language":null,"funding_links":[],"categories":["Training Resources","Tutorials","📋 Cheatsheets"],"sub_categories":["Tutorials","Data Science \u0026 Machine Learning"],"readme":"# 🐍 📄 PySpark Cheat Sheet\n\nA quick reference guide to the most commonly used patterns and functions in PySpark SQL.\n\n#### Table of Contents\n- [Quickstart](#quickstart)\n- [Basics](#basics)\n- [Common Patterns](#common-patterns)\n    - [Importing Functions \u0026 Types](#importing-functions--types)\n    - [Filtering](#filtering)\n    - [Joins](#joins)\n    - [Column Operations](#column-operations)\n    - [Casting \u0026 Coalescing Null Values \u0026 Duplicates](#casting--coalescing-null-values--duplicates)\n- [String Operations](#string-operations)\n    - [String Filters](#string-filters)\n    - [String Functions](#string-functions)\n- [Number Operations](#number-operations)\n- [Date \u0026 Timestamp Operations](#date--timestamp-operations)\n- [Array Operations](#array-operations)\n- [Struct Operations](#struct-operations)\n- [Aggregation Operations](#aggregation-operations)\n- [Advanced Operations](#advanced-operations)\n    - [Repartitioning](#repartitioning)\n    - [UDFs (User Defined Functions](#udfs-user-defined-functions)\n- [Useful Functions / Tranformations](#useful-functions--transformations)\n\nIf you can't find what you're looking for, check out the [PySpark Official Documentation](https://spark.apache.org/docs/latest/api/python/pyspark.sql.html) and add it here!\n\n## Quickstart\n\nInstall on macOS:\n\n```bash\nbrew install apache-spark \u0026\u0026 pip install pyspark\n```\n\nCreate your first DataFrame:\n\n```python\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder.getOrCreate()\n\n# I/O options: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/io.html\ndf = spark.read.csv('/path/to/your/input/file')\n```\n\n## Basics\n\n```python\n# Show a preview\ndf.show()\n\n# Show preview of first / last n rows\ndf.head(5)\ndf.tail(5)\n\n# Show preview as JSON (WARNING: in-memory)\ndf = df.limit(10) # optional\nprint(json.dumps([row.asDict(recursive=True) for row in df.collect()], indent=2))\n\n# Limit actual DataFrame to n rows (non-deterministic)\ndf = df.limit(5)\n\n# Get columns\ndf.columns\n\n# Get columns + column types\ndf.dtypes\n\n# Get schema\ndf.schema\n\n# Get row count\ndf.count()\n\n# Get column count\nlen(df.columns)\n\n# Write output to disk\ndf.write.csv('/path/to/your/output/file')\n\n# Get results (WARNING: in-memory) as list of PySpark Rows\ndf = df.collect()\n\n# Get results (WARNING: in-memory) as list of Python dicts\ndicts = [row.asDict(recursive=True) for row in df.collect()]\n\n# Convert (WARNING: in-memory) to Pandas DataFrame\ndf = df.toPandas()\n```\n\n## Common Patterns\n\n#### Importing Functions \u0026 Types\n\n```python\n# Easily reference these as F.my_function() and T.my_type() below\nfrom pyspark.sql import functions as F, types as T\n```\n\n#### Filtering\n\n```python\n# Filter on equals condition\ndf = df.filter(df.is_adult == 'Y')\n\n# Filter on \u003e, \u003c, \u003e=, \u003c= condition\ndf = df.filter(df.age \u003e 25)\n\n# Multiple conditions require parentheses around each condition\ndf = df.filter((df.age \u003e 25) \u0026 (df.is_adult == 'Y'))\n\n# Compare against a list of allowed values\ndf = df.filter(col('first_name').isin([3, 4, 7]))\n\n# Sort results\ndf = df.orderBy(df.age.asc()))\ndf = df.orderBy(df.age.desc()))\n```\n\n#### Joins\n\n```python\n# Left join in another dataset\ndf = df.join(person_lookup_table, 'person_id', 'left')\n\n# Match on different columns in left \u0026 right datasets\ndf = df.join(other_table, df.id == other_table.person_id, 'left')\n\n# Match on multiple columns\ndf = df.join(other_table, ['first_name', 'last_name'], 'left')\n```\n\n#### Column Operations\n\n```python\n# Add a new static column\ndf = df.withColumn('status', F.lit('PASS'))\n\n# Construct a new dynamic column\ndf = df.withColumn('full_name', F.when(\n    (df.fname.isNotNull() \u0026 df.lname.isNotNull()), F.concat(df.fname, df.lname)\n).otherwise(F.lit('N/A'))\n\n# Pick which columns to keep, optionally rename some\ndf = df.select(\n    'name',\n    'age',\n    F.col('dob').alias('date_of_birth'),\n)\n\n# Remove columns\ndf = df.drop('mod_dt', 'mod_username')\n\n# Rename a column\ndf = df.withColumnRenamed('dob', 'date_of_birth')\n\n# Keep all the columns which also occur in another dataset\ndf = df.select(*(F.col(c) for c in df2.columns))\n\n# Batch Rename/Clean Columns\nfor col in df.columns:\n    df = df.withColumnRenamed(col, col.lower().replace(' ', '_').replace('-', '_'))\n```\n\n#### Casting \u0026 Coalescing Null Values \u0026 Duplicates\n\n```python\n# Cast a column to a different type\ndf = df.withColumn('price', df.price.cast(T.DoubleType()))\n\n# Replace all nulls with a specific value\ndf = df.fillna({\n    'first_name': 'Tom',\n    'age': 0,\n})\n\n# Take the first value that is not null\ndf = df.withColumn('last_name', F.coalesce(df.last_name, df.surname, F.lit('N/A')))\n\n# Drop duplicate rows in a dataset (distinct)\ndf = df.dropDuplicates() # or\ndf = df.distinct()\n\n# Drop duplicate rows, but consider only specific columns\ndf = df.dropDuplicates(['name', 'height'])\n\n# Replace empty strings with null (leave out subset keyword arg to replace in all columns)\ndf = df.replace({\"\": None}, subset=[\"name\"])\n\n# Convert Python/PySpark/NumPy NaN operator to null\ndf = df.replace(float(\"nan\"), None)\n```\n\n## String Operations\n\n#### String Filters\n\n```python\n# Contains - col.contains(string)\ndf = df.filter(df.name.contains('o'))\n\n# Starts With - col.startswith(string)\ndf = df.filter(df.name.startswith('Al'))\n\n# Ends With - col.endswith(string)\ndf = df.filter(df.name.endswith('ice'))\n\n# Is Null - col.isNull()\ndf = df.filter(df.is_adult.isNull())\n\n# Is Not Null - col.isNotNull()\ndf = df.filter(df.first_name.isNotNull())\n\n# Like - col.like(string_with_sql_wildcards)\ndf = df.filter(df.name.like('Al%'))\n\n# Regex Like - col.rlike(regex)\ndf = df.filter(df.name.rlike('[A-Z]*ice$'))\n\n# Is In List - col.isin(*cols)\ndf = df.filter(df.name.isin('Bob', 'Mike'))\n```\n\n#### String Functions\n\n```python\n# Substring - col.substr(startPos, length)\ndf = df.withColumn('short_id', df.id.substr(0, 10))\n\n# Trim - F.trim(col)\ndf = df.withColumn('name', F.trim(df.name))\n\n# Left Pad - F.lpad(col, len, pad)\n# Right Pad - F.rpad(col, len, pad)\ndf = df.withColumn('id', F.lpad('id', 4, '0'))\n\n# Left Trim - F.ltrim(col)\n# Right Trim - F.rtrim(col)\ndf = df.withColumn('id', F.ltrim('id'))\n\n# Concatenate - F.concat(*cols)\ndf = df.withColumn('full_name', F.concat('fname', F.lit(' '), 'lname'))\n\n# Concatenate with Separator/Delimiter - F.concat_ws(delimiter, *cols)\ndf = df.withColumn('full_name', F.concat_ws('-', 'fname', 'lname'))\n\n# Regex Replace - F.regexp_replace(str, pattern, replacement)[source]\ndf = df.withColumn('id', F.regexp_replace(id, '0F1(.*)', '1F1-$1'))\n\n# Regex Extract - F.regexp_extract(str, pattern, idx)\ndf = df.withColumn('id', F.regexp_extract(id, '[0-9]*', 0))\n```\n\n## Number Operations\n\n```python\n# Round - F.round(col, scale=0)\ndf = df.withColumn('price', F.round('price', 0))\n\n# Floor - F.floor(col)\ndf = df.withColumn('price', F.floor('price'))\n\n# Ceiling - F.ceil(col)\ndf = df.withColumn('price', F.ceil('price'))\n\n# Absolute Value - F.abs(col)\ndf = df.withColumn('price', F.abs('price'))\n\n# X raised to power Y – F.pow(x, y)\ndf = df.withColumn('exponential_growth', F.pow('x', 'y'))\n\n# Select smallest value out of multiple columns – F.least(*cols)\ndf = df.withColumn('least', F.least('subtotal', 'total'))\n\n# Select largest value out of multiple columns – F.greatest(*cols)\ndf = df.withColumn('greatest', F.greatest('subtotal', 'total'))\n```\n\n## Date \u0026 Timestamp Operations\n\n```python\n# Add a column with the current date\ndf = df.withColumn('current_date', F.current_date())\n\n# Convert a string of known format to a date (excludes time information)\ndf = df.withColumn('date_of_birth', F.to_date('date_of_birth', 'yyyy-MM-dd'))\n\n# Convert a string of known format to a timestamp (includes time information)\ndf = df.withColumn('time_of_birth', F.to_timestamp('time_of_birth', 'yyyy-MM-dd HH:mm:ss'))\n\n# Get year from date:       F.year(col)\n# Get month from date:      F.month(col)\n# Get day from date:        F.dayofmonth(col)\n# Get hour from date:       F.hour(col)\n# Get minute from date:     F.minute(col)\n# Get second from date:     F.second(col)\ndf = df.filter(F.year('date_of_birth') == F.lit('2017'))\n\n# Add \u0026 subtract days\ndf = df.withColumn('three_days_after', F.date_add('date_of_birth', 3))\ndf = df.withColumn('three_days_before', F.date_sub('date_of_birth', 3))\n\n# Add \u0026 Subtract months\ndf = df.withColumn('next_month', F.add_month('date_of_birth', 1))\n\n# Get number of days between two dates\ndf = df.withColumn('days_between', F.datediff('start', 'end'))\n\n# Get number of months between two dates\ndf = df.withColumn('months_between', F.months_between('start', 'end'))\n\n# Keep only rows where date_of_birth is between 2017-05-10 and 2018-07-21\ndf = df.filter(\n    (F.col('date_of_birth') \u003e= F.lit('2017-05-10')) \u0026\n    (F.col('date_of_birth') \u003c= F.lit('2018-07-21'))\n)\n```\n\n## Array Operations\n\n```python\n# Column Array - F.array(*cols)\ndf = df.withColumn('full_name', F.array('fname', 'lname'))\n\n# Empty Array - F.array(*cols)\ndf = df.withColumn('empty_array_column', F.array([]))\n\n# Get element at index – col.getItem(n)\ndf = df.withColumn('first_element', F.col(\"my_array\").getItem(0))\n\n# Array Size/Length – F.size(col)\ndf = df.withColumn('array_length', F.size('my_array'))\n\n# Flatten Array – F.flatten(col)\ndf = df.withColumn('flattened', F.flatten('my_array'))\n\n# Unique/Distinct Elements – F.array_distinct(col)\ndf = df.withColumn('unique_elements', F.array_distinct('my_array'))\n\n# Map over \u0026 transform array elements – F.transform(col, func: col -\u003e col)\ndf = df.withColumn('elem_ids', F.transform(F.col('my_array'), lambda x: x.getField('id')))\n\n# Return a row per array element – F.explode(col)\ndf = df.select(F.explode('my_array'))\n```\n\n## Struct Operations\n\n```python\n# Make a new Struct column (similar to Python's `dict()`) – F.struct(*cols)\ndf = df.withColumn('my_struct', F.struct(F.col('col_a'), F.col('col_b')))\n\n# Get item from struct by key – col.getField(str)\ndf = df.withColumn('col_a', F.col('my_struct').getField('col_a'))\n```\n\n\n## Aggregation Operations\n\n```python\n# Row Count:                F.count()\n# Sum of Rows in Group:     F.sum(*cols)\n# Mean of Rows in Group:    F.mean(*cols)\n# Max of Rows in Group:     F.max(*cols)\n# Min of Rows in Group:     F.min(*cols)\n# First Row in Group:       F.alias(*cols)\ndf = df.groupBy('gender').agg(F.max('age').alias('max_age_by_gender'))\n\n# Collect a Set of all Rows in Group:       F.collect_set(col)\n# Collect a List of all Rows in Group:      F.collect_list(col)\ndf = df.groupBy('age').agg(F.collect_set('name').alias('person_names'))\n\n# Just take the lastest row for each combination (Window Functions)\nfrom pyspark.sql import Window as W\n\nwindow = W.partitionBy(\"first_name\", \"last_name\").orderBy(F.desc(\"date\"))\ndf = df.withColumn(\"row_number\", F.row_number().over(window))\ndf = df.filter(F.col(\"row_number\") == 1)\ndf = df.drop(\"row_number\")\n```\n\n## Advanced Operations\n\n#### Repartitioning\n\n```python\n# Repartition – df.repartition(num_output_partitions)\ndf = df.repartition(1)\n```\n\n#### UDFs (User Defined Functions\n\n```python\n# Multiply each row's age column by two\ntimes_two_udf = F.udf(lambda x: x * 2)\ndf = df.withColumn('age', times_two_udf(df.age))\n\n# Randomly choose a value to use as a row's name\nimport random\n\nrandom_name_udf = F.udf(lambda: random.choice(['Bob', 'Tom', 'Amy', 'Jenna']))\ndf = df.withColumn('name', random_name_udf())\n```\n\n## Useful Functions / Transformations\n\n```python\ndef flatten(df: DataFrame, delimiter=\"_\") -\u003e DataFrame:\n    '''\n    Flatten nested struct columns in `df` by one level separated by `delimiter`, i.e.:\n\n    df = [ {'a': {'b': 1, 'c': 2} } ]\n    df = flatten(df, '_')\n    -\u003e [ {'a_b': 1, 'a_c': 2} ]\n    '''\n    flat_cols = [name for name, type in df.dtypes if not type.startswith(\"struct\")]\n    nested_cols = [name for name, type in df.dtypes if type.startswith(\"struct\")]\n\n    flat_df = df.select(\n        flat_cols\n        + [F.col(nc + \".\" + c).alias(nc + delimiter + c) for nc in nested_cols for c in df.select(nc + \".*\").columns]\n    )\n    return flat_df\n\n\ndef lookup_and_replace(df1, df2, df1_key, df2_key, df2_value):\n    '''\n    Replace every value in `df1`'s `df1_key` column with the corresponding value\n    `df2_value` from `df2` where `df1_key` matches `df2_key`\n\n    df = lookup_and_replace(people, pay_codes, id, pay_code_id, pay_code_desc)\n    '''\n    return (\n        df1\n        .join(df2[[df2_key, df2_value]], df1[df1_key] == df2[df2_key], 'left')\n        .withColumn(df1_key, F.coalesce(F.col(df2_value), F.col(df1_key)))\n        .drop(df2_key)\n        .drop(df2_value)\n    )\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinschaich%2Fpyspark-cheatsheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevinschaich%2Fpyspark-cheatsheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinschaich%2Fpyspark-cheatsheet/lists"}