{"id":13516122,"url":"https://github.com/mrpowers/chispa","last_synced_at":"2026-04-02T12:02:45.702Z","repository":{"id":41565929,"uuid":"176548161","full_name":"MrPowers/chispa","owner":"MrPowers","description":"PySpark test helper methods with beautiful error messages","archived":false,"fork":false,"pushed_at":"2026-03-19T22:02:40.000Z","size":3863,"stargazers_count":755,"open_issues_count":43,"forks_count":77,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-03-20T13:13:35.845Z","etag":null,"topics":["pyspark","testing"],"latest_commit_sha":null,"homepage":"https://mrpowers.github.io/chispa/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MrPowers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2019-03-19T15:52:21.000Z","updated_at":"2026-03-19T22:01:14.000Z","dependencies_parsed_at":"2024-01-14T15:22:02.182Z","dependency_job_id":"04263728-30a3-4efd-a83b-1a0315e88739","html_url":"https://github.com/MrPowers/chispa","commit_stats":{"total_commits":170,"total_committers":19,"mean_commits":8.947368421052632,"dds":"0.48235294117647054","last_synced_commit":"202ab2aa8205b4d14520df8d904ff96c1f0839df"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/MrPowers/chispa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrPowers%2Fchispa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrPowers%2Fchispa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrPowers%2Fchispa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrPowers%2Fchispa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MrPowers","download_url":"https://codeload.github.com/MrPowers/chispa/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrPowers%2Fchispa/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31305971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T09:48:21.550Z","status":"ssl_error","status_checked_at":"2026-04-02T09:48:19.196Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pyspark","testing"],"created_at":"2024-08-01T05:01:19.335Z","updated_at":"2026-04-02T12:02:45.690Z","avatar_url":"https://github.com/MrPowers.png","language":"Python","funding_links":[],"categories":["External Resources"],"sub_categories":["Repos"],"readme":"# chispa\n\n\n[![Release](https://img.shields.io/github/v/release/MrPowers/chispa)](https://pypi.org/project/chispa/)\n[![Build status](https://github.com/MrPowers/chispa/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/MrPowers/chispa/actions/workflows/ci.yml)\n[![Supported Python versions](https://img.shields.io/pypi/pyversions/chispa)](https://pypi.org/project/chispa/)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/chispa)](https://pypistats.org/packages/chispa)\n[![License](https://img.shields.io/github/license/MrPowers/chispa)](https://img.shields.io/github/license/MrPowers/chispa)\n\n\n*chispa* provides fast PySpark test helper methods that output descriptive error messages.\n\nThis library makes it easy to write high quality PySpark code.\n\nFun fact: \"chispa\" means Spark in Spanish ;)\n\n## Installation\n\nInstall the latest version with\n\n```sh\npip install chispa\n```\n\nOr if you use Poetry add this library as a development dependency with\n\n```sh\npoetry add chispa --group dev\n```\n\n## Column equality\n\nSuppose you have a function that removes the non-word characters in a string.\n\n```python\ndef remove_non_word_characters(col):\n    return F.regexp_replace(col, \"[^\\\\w\\\\s]+\", \"\")\n```\n\nCreate a `SparkSession` so you can create DataFrames.\n\n```python\nfrom pyspark.sql import SparkSession\n\nspark = (SparkSession.builder\n  .master(\"local\")\n  .appName(\"chispa\")\n  .getOrCreate())\n```\n\nCreate a DataFrame with a column that contains strings with non-word characters, run the `remove_non_word_characters` function, and check that all these characters are removed with the chispa `assert_column_equality` method.\n\n```python\nimport pytest\n\nfrom chispa import assert_column_equality\nimport pyspark.sql.functions as F\n\ndef test_remove_non_word_characters_short():\n    data = [\n        (\"jo\u0026\u0026se\", \"jose\"),\n        (\"**li**\", \"li\"),\n        (\"#::luisa\", \"luisa\"),\n        (None, None)\n    ]\n    df = (spark.createDataFrame(data, [\"name\", \"expected_name\"])\n        .withColumn(\"clean_name\", remove_non_word_characters(F.col(\"name\"))))\n    assert_column_equality(df, \"clean_name\", \"expected_name\")\n```\n\nLet's write another test that'll fail to see how the descriptive error message lets you easily debug the underlying issue.\n\nHere's the failing test:\n\n```python\ndef test_remove_non_word_characters_nice_error():\n    data = [\n        (\"matt7\", \"matt\"),\n        (\"bill\u0026\", \"bill\"),\n        (\"isabela*\", \"isabela\"),\n        (None, None)\n    ]\n    df = (spark.createDataFrame(data, [\"name\", \"expected_name\"])\n        .withColumn(\"clean_name\", remove_non_word_characters(F.col(\"name\"))))\n    assert_column_equality(df, \"clean_name\", \"expected_name\")\n```\n\nHere's the nicely formatted error message:\n\n![ColumnsNotEqualError](https://raw.githubusercontent.com/MrPowers/chispa/main/images/columns_not_equal_error.png)\n\nYou can see the `matt7` / `matt` row of data is what's causing the error (note it's highlighted in red).  The other rows are colored blue because they're equal.\n\n## DataFrame equality\n\nWe can also test the `remove_non_word_characters` method by creating two DataFrames and verifying that they're equal.\n\nCreating two DataFrames is slower and requires more code, but comparing entire DataFrames is necessary for some tests.\n\n```python\nfrom chispa import assert_df_equality\n\ndef test_remove_non_word_characters_long():\n    source_data = [\n        (\"jo\u0026\u0026se\",),\n        (\"**li**\",),\n        (\"#::luisa\",),\n        (None,)\n    ]\n    source_df = spark.createDataFrame(source_data, [\"name\"])\n\n    actual_df = source_df.withColumn(\n        \"clean_name\",\n        remove_non_word_characters(F.col(\"name\"))\n    )\n\n    expected_data = [\n        (\"jo\u0026\u0026se\", \"jose\"),\n        (\"**li**\", \"li\"),\n        (\"#::luisa\", \"luisa\"),\n        (None, None)\n    ]\n    expected_df = spark.createDataFrame(expected_data, [\"name\", \"clean_name\"])\n\n    assert_df_equality(actual_df, expected_df)\n```\n\nLet's write another test that'll return an error, so you can see the descriptive error message.\n\n```python\ndef test_remove_non_word_characters_long_error():\n    source_data = [\n        (\"matt7\",),\n        (\"bill\u0026\",),\n        (\"isabela*\",),\n        (None,)\n    ]\n    source_df = spark.createDataFrame(source_data, [\"name\"])\n\n    actual_df = source_df.withColumn(\n        \"clean_name\",\n        remove_non_word_characters(F.col(\"name\"))\n    )\n\n    expected_data = [\n        (\"matt7\", \"matt\"),\n        (\"bill\u0026\", \"bill\"),\n        (\"isabela*\", \"isabela\"),\n        (None, None)\n    ]\n    expected_df = spark.createDataFrame(expected_data, [\"name\", \"clean_name\"])\n\n    assert_df_equality(actual_df, expected_df)\n```\n\nHere's the nicely formatted error message:\n\n![DataFramesNotEqualError](https://raw.githubusercontent.com/MrPowers/chispa/main/images/dfs_not_equal_error.png)\n\n### Ignore row order\n\nYou can easily compare DataFrames, ignoring the order of the rows.  The content of the DataFrames is usually what matters, not the order of the rows.\n\nHere are the contents of `df1`:\n\n```\n+--------+\n|some_num|\n+--------+\n|       1|\n|       2|\n|       3|\n+--------+\n```\n\nHere are the contents of `df2`:\n\n```\n+--------+\n|some_num|\n+--------+\n|       2|\n|       1|\n|       3|\n+--------+\n```\n\nHere's how to confirm `df1` and `df2` are equal when the row order is ignored.\n\n```python\nassert_df_equality(df1, df2, ignore_row_order=True)\n```\n\nIf you don't specify to `ignore_row_order` then the test will error out with this message:\n\n![ignore_row_order_false](https://raw.githubusercontent.com/MrPowers/chispa/main/images/ignore_row_order_false.png)\n\nThe rows aren't ordered by default because sorting slows down the function.\n\n### Ignore column order\n\nThis section explains how to compare DataFrames, ignoring the order of the columns.\n\nSuppose you have the following `df1`:\n\n```\n+----+----+\n|num1|num2|\n+----+----+\n|   1|   7|\n|   2|   8|\n|   3|   9|\n+----+----+\n```\n\nHere are the contents of `df2`:\n\n```\n+----+----+\n|num2|num1|\n+----+----+\n|   7|   1|\n|   8|   2|\n|   9|   3|\n+----+----+\n```\n\nHere's how to compare the equality of `df1` and `df2`, ignoring the column order:\n\n```python\nassert_df_equality(df1, df2, ignore_column_order=True)\n```\n\nHere's the error message you'll see if you run `assert_df_equality(df1, df2)`, without ignoring the column order.\n\n![ignore_column_order_false](https://raw.githubusercontent.com/MrPowers/chispa/main/images/ignore_column_order_false.png)\n\n### Ignore specific columns\n\nThis section explains how to compare DataFrames, ignoring specific columns.\n\nSuppose you have the following `df1`:\n\n```\n+------------+-------------+\n|    name    | clean_name  |\n+------------+-------------+\n| \"matt7\"    |   \"matt7\"    |\n| \"bill\u0026\"    |   \"bill\"    |\n| \"isabela*\" |   \"isabela\" |\n| \"None\"     |   \"None\"    |\n+------------+-------------+\n```\n\nHere are the contents of `df2`:\n\n```\n+------------+-------------+\n|    name    | clean_name  |\n+------------+-------------+\n| \"matt7\"    |   \"matt\"    |\n| \"bill\u0026\"    |   \"bill\"    |\n| \"isabela*\" |   \"isabela\" |\n| \"None\"     |   \"None\"    |\n+------------+-------------+\n```\n\nHere's how to compare the equality of `df1` and `df2`, ignoring the column `clean_name`:\n\n```python\nassert_df_equality(df1, df2, ignore_columns=[\"clean_name\"])\n```\n\nHere's the error message you'll see if you run `assert_df_equality(df1, df2)`, without ignoring the column `clean_name`.\n\n![ignore_columns_none](https://raw.githubusercontent.com/MrPowers/chispa/main/images/dfs_not_equal_error.png)\n\n### Ignore nullability\n\nEach column in a schema has three properties: a name, data type, and nullable property.  The column can accept null values if `nullable` is set to true.\n\nYou'll sometimes want to ignore the nullable property when making DataFrame comparisons.\n\nSuppose you have the following `df1`:\n\n```\n+-----+---+\n| name|age|\n+-----+---+\n| juan|  7|\n|bruna|  8|\n+-----+---+\n```\n\nAnd this `df2`:\n\n```\n+-----+---+\n| name|age|\n+-----+---+\n| juan|  7|\n|bruna|  8|\n+-----+---+\n```\n\nYou might be surprised to find that in this example, `df1` and `df2` are not equal and will error out with this message:\n\n![nullable_off_error](https://raw.githubusercontent.com/MrPowers/chispa/main/images/nullable_off_error.png)\n\nExamine the code in this contrived example to better understand the error:\n\n```python\ndef ignore_nullable_property():\n    s1 = StructType([\n       StructField(\"name\", StringType(), True),\n       StructField(\"age\", IntegerType(), True)])\n    df1 = spark.createDataFrame([(\"juan\", 7), (\"bruna\", 8)], s1)\n    s2 = StructType([\n       StructField(\"name\", StringType(), True),\n       StructField(\"age\", IntegerType(), False)])\n    df2 = spark.createDataFrame([(\"juan\", 7), (\"bruna\", 8)], s2)\n    assert_df_equality(df1, df2)\n```\n\nYou can ignore the nullable property when assessing equality by adding a flag:\n\n```python\nassert_df_equality(df1, df2, ignore_nullable=True)\n```\n\n### Other public DataFrame comparison options\n\n`assert_df_equality` also supports additional public options that are useful in real-world test suites:\n\n- `ignore_metadata=True` ignores schema metadata differences when data and field types are otherwise equivalent.\n- `transforms=[...]` applies the same transform pipeline to both DataFrames before comparison.\n- `underline_cells=True` highlights mismatched cells in error output for faster debugging.\n\nExample:\n\n```python\nassert_df_equality(\n    actual_df,\n    expected_df,\n    transforms=[lambda df: df.orderBy(\"id\")],\n    ignore_metadata=True,\n    underline_cells=True,\n)\n```\n\nElements contained within an `ArrayType()` also have a nullable property, in addition to the nullable property of the column schema. These are also ignored when passing `ignore_nullable=True`.\n\nAgain, examine the following code to understand the error that `ignore_nullable=True` bypasses:\n\n```python\ndef ignore_nullable_property_array():\n    s1 = StructType([\n        StructField(\"name\", StringType(), True),\n        StructField(\"coords\", ArrayType(DoubleType(), True), True),])\n    df1 = spark.createDataFrame([(\"juan\", [1.42, 3.5]), (\"bruna\", [2.76, 3.2])], s1)\n    s2 = StructType([\n        StructField(\"name\", StringType(), True),\n        StructField(\"coords\", ArrayType(DoubleType(), False), True),])\n    df2 = spark.createDataFrame([(\"juan\", [1.42, 3.5]), (\"bruna\", [2.76, 3.2])], s2)\n    assert_df_equality(df1, df2)\n```\n\n### Allow NaN equality\n\nPython has NaN (not a number) values and two NaN values are not considered equal by default. Create two NaN values, compare them, and confirm they're not considered equal by default.\n\n```python\nnan1 = float('nan')\nnan2 = float('nan')\nnan1 == nan2 # False\n```\n\npandas considers NaN values to be equal by default, but this library requires you to set a flag to consider two NaN values to be equal.\n\n```python\n# default behavior remains strict (NaN != NaN)\nassert_df_equality(df1, df2)\n\n# opt in to treat NaN values as equal\nassert_df_equality(df1, df2, allow_nan_equality=True)\n```\n\n## Customize formatting\n\nYou can specify custom formats for the printed error messages as follows:\n\n```python\nfrom chispa import FormattingConfig\n\nformats = FormattingConfig(\n        mismatched_rows={\"color\": \"light_yellow\"},\n        matched_rows={\"color\": \"cyan\", \"style\": \"bold\"},\n        mismatched_cells={\"color\": \"purple\"},\n        matched_cells={\"color\": \"blue\"},\n    )\n\nassert_basic_rows_equality(df1.collect(), df2.collect(), formats=formats)\n```\n\nor similarly:\n\n```python\nfrom chispa import FormattingConfig, Color, Style\n\nformats = FormattingConfig(\n        mismatched_rows={\"color\": Color.LIGHT_YELLOW},\n        matched_rows={\"color\": Color.CYAN, \"style\": Style.BOLD},\n        mismatched_cells={\"color\": Color.PURPLE},\n        matched_cells={\"color\": Color.BLUE},\n    )\n\nassert_basic_rows_equality(df1.collect(), df2.collect(), formats=formats)\n```\n\nYou can also define these formats in `conftest.py` and inject them via a fixture:\n\n```python\n@pytest.fixture()\ndef chispa_formats():\n    return FormattingConfig(\n        mismatched_rows={\"color\": \"light_yellow\"},\n        matched_rows={\"color\": \"cyan\", \"style\": \"bold\"},\n        mismatched_cells={\"color\": \"purple\"},\n        matched_cells={\"color\": \"blue\"},\n    )\n\ndef test_shows_assert_basic_rows_equality(chispa_formats):\n  ...\n  assert_basic_rows_equality(df1.collect(), df2.collect(), formats=chispa_formats)\n```\n\n![custom_formats](https://raw.githubusercontent.com/MrPowers/chispa/main/images/custom_formats.png)\n\n## Approximate column equality\n\nWe can check if columns are approximately equal, which is especially useful for floating number comparisons.\n\nHere's a test that creates a DataFrame with two floating point columns and verifies that the columns are approximately equal.  In this example, values are considered approximately equal if the difference is less than 0.1.\n\n```python\ndef test_approx_col_equality_same():\n    data = [\n        (1.1, 1.1),\n        (2.2, 2.15),\n        (3.3, 3.37),\n        (None, None)\n    ]\n    df = spark.createDataFrame(data, [\"num1\", \"num2\"])\n    assert_approx_column_equality(df, \"num1\", \"num2\", 0.1)\n```\n\nHere's an example of a test with columns that are not approximately equal.\n\n```python\ndef test_approx_col_equality_different():\n    data = [\n        (1.1, 1.1),\n        (2.2, 2.15),\n        (3.3, 5.0),\n        (None, None)\n    ]\n    df = spark.createDataFrame(data, [\"num1\", \"num2\"])\n    assert_approx_column_equality(df, \"num1\", \"num2\", 0.1)\n```\n\nThis failing test will output a readable error message so the issue is easy to debug.\n\n![ColumnsNotEqualError](https://raw.githubusercontent.com/MrPowers/chispa/main/images/columns_not_approx_equal.png)\n\n## Approximate DataFrame equality\n\nLet's create two DataFrames and confirm they're approximately equal.\n\n```python\ndef test_approx_df_equality_same():\n    data1 = [\n        (1.1, \"a\"),\n        (2.2, \"b\"),\n        (3.3, \"c\"),\n        (None, None)\n    ]\n    df1 = spark.createDataFrame(data1, [\"num\", \"letter\"])\n\n    data2 = [\n        (1.05, \"a\"),\n        (2.13, \"b\"),\n        (3.3, \"c\"),\n        (None, None)\n    ]\n    df2 = spark.createDataFrame(data2, [\"num\", \"letter\"])\n\n    assert_approx_df_equality(df1, df2, 0.1)\n```\n\nThe `assert_approx_df_equality` method is smart and will only perform approximate equality operations for floating point numbers in DataFrames.  It'll perform regular equality for strings and other types.\n\nLet's perform an approximate equality comparison for two DataFrames that are not equal.\n\n```python\ndef test_approx_df_equality_different():\n    data1 = [\n        (1.1, \"a\"),\n        (2.2, \"b\"),\n        (3.3, \"c\"),\n        (None, None)\n    ]\n    df1 = spark.createDataFrame(data1, [\"num\", \"letter\"])\n\n    data2 = [\n        (1.1, \"a\"),\n        (5.0, \"b\"),\n        (3.3, \"z\"),\n        (None, None)\n    ]\n    df2 = spark.createDataFrame(data2, [\"num\", \"letter\"])\n\n    assert_approx_df_equality(df1, df2, 0.1)\n```\n\nHere's the pretty error message that's outputted:\n\n![DataFramesNotEqualError](https://raw.githubusercontent.com/MrPowers/chispa/main/images/dfs_not_approx_equal.png)\n\n## Schema mismatch messages\n\nDataFrame equality messages peform schema comparisons before analyzing the actual content of the DataFrames.  DataFrames that don't have the same schemas should error out as fast as possible.\n\nLet's compare a DataFrame that has a string column an integer column with a DataFrame that has two integer columns to observe the schema mismatch message.\n\n```python\ndef test_schema_mismatch_message():\n    data1 = [\n        (1, \"a\"),\n        (2, \"b\"),\n        (3, \"c\"),\n        (None, None)\n    ]\n    df1 = spark.createDataFrame(data1, [\"num\", \"letter\"])\n\n    data2 = [\n        (1, 6),\n        (2, 7),\n        (3, 8),\n        (None, None)\n    ]\n    df2 = spark.createDataFrame(data2, [\"num\", \"num2\"])\n\n    assert_df_equality(df1, df2)\n```\n\nHere's the error message:\n\n![SchemasNotEqualError](https://raw.githubusercontent.com/MrPowers/chispa/main/images/schemas_not_approx_equal.png)\n\n## Supported PySpark / Python versions\n\nchispa requires Python `\u003e=3.10,\u003c4.0`.\n\nThe CI matrix currently tests chispa against:\n\n- Python 3.10, 3.11, and 3.12\n- PySpark 3.5.x, 4.0.x, and 4.1.x\n\n## Developing chispa on your local machine\n\nYou are encouraged to clone and/or fork this repo.\n\nThis project uses [Poetry](https://python-poetry.org/) for packaging and dependency management.\n\n* Setup the virtual environment with `poetry install`\n* Run the tests with `poetry run pytest tests`\n\nStudying the codebase is a great way to learn about PySpark!\n\n## Contributing\n\nAnyone is encouraged to submit a pull request, open an issue, or submit a bug report.\n\nWe're happy to promote folks to be library maintainers if they make good contributions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrpowers%2Fchispa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrpowers%2Fchispa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrpowers%2Fchispa/lists"}