{"id":29878804,"url":"https://github.com/jafeerr/spark-data-test","last_synced_at":"2026-04-20T13:39:06.955Z","repository":{"id":296117312,"uuid":"909034282","full_name":"jafeerr/spark-data-test","owner":"jafeerr","description":"Spark Data Test - A PySpark-based automation testing utility to compare Spark DataFrames","archived":false,"fork":false,"pushed_at":"2025-10-17T10:26:43.000Z","size":35,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-18T13:44:13.184Z","etag":null,"topics":["apache-spark","data-testing","dataframe","pyspark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jafeerr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-27T15:33:00.000Z","updated_at":"2025-10-17T10:26:47.000Z","dependencies_parsed_at":"2025-05-29T03:32:13.661Z","dependency_job_id":"bfd1a5cd-27c1-4b6f-8963-0fa6d41edcc4","html_url":"https://github.com/jafeerr/spark-data-test","commit_stats":null,"previous_names":["jafeerr/spark-data-test"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jafeerr/spark-data-test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafeerr%2Fspark-data-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafeerr%2Fspark-data-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafeerr%2Fspark-data-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafeerr%2Fspark-data-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jafeerr","download_url":"https://codeload.github.com/jafeerr/spark-data-test/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jafeerr%2Fspark-data-test/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32049371,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","data-testing","dataframe","pyspark"],"created_at":"2025-07-31T08:01:23.922Z","updated_at":"2026-04-20T13:39:06.948Z","avatar_url":"https://github.com/jafeerr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Data Test\n[![Coverage Status](https://coveralls.io/repos/github/jafeerr/spark-data-test/badge.svg?branch=main)](https://coveralls.io/github/jafeerr/spark-data-test?branch=main)\n[![License](https://img.shields.io/pypi/l/spark-data-test.svg)](https://pypi.python.org/pypi/spark-data-test/)\n[![Version](https://img.shields.io/pypi/v/spark-data-test.svg)](https://pypi.python.org/pypi/spark-data-test/)\n[![PyPI Downloads](https://static.pepy.tech/personalized-badge/spark-data-test?period=total\u0026units=INTERNATIONAL_SYSTEM\u0026left_color=BLACK\u0026right_color=GREEN\u0026left_text=downloads)](https://pepy.tech/projects/spark-data-test)\n## Overview\n\n`spark-data-test` provides utilities to compare two Spark DataFrames or datasets, generating detailed reports on matches, mismatches, and missing records. It is designed for data validation, ETL testing, and regression testing in Spark pipelines.\n\n## Installation\n\nTo install, simply use `pip`:\n\n```\n$ pip install spark-data-test\n```\n\n## Requirements\n\nMinimum Python version supported by `spark-data-test` is 3.7.\n\n## Usage\n\n### 1. Compare DataFrames Directly\n\nUse `run_comparison_job_from_dfs` to compare two Spark DataFrames directly.\n\n#### Function Signature\n\n```python\nrun_comparison_job_from_dfs(\n    spark: SparkSession,\n    job_name: str,\n    source_df: DataFrame,\n    target_df: DataFrame,\n    params: DatasetParams|dict,\n    output_config: OutputConfig|=dict\n)\n```\n\n#### Parameters\n\n- `spark`: The active `SparkSession`.\n- `job_name`: Name for the comparison job (used in output paths).\n- `source_df`: Source DataFrame.\n- `target_df`: Target DataFrame.\n- `params`: An instance of `DatasetParams` specifying dataset name, primary keys, columns to select/drop, etc.\n- `output_config`: An instance of [`OutputConfig`](#outputconfig) specifying output directory, file format, Spark write options, etc.\n\n#### Example\n\n```python\nfrom spark_data_test.jobs.comparison_job import run_comparison_job_from_dfs\nfrom spark_data_test.entities.config import DatasetParams, OutputConfig\n\nparams = DatasetParams(\n    dataset_name=\"my_table\",\n    primary_keys=[\"id\"]\n)\noutput_config = OutputConfig(\n    output_dir=\"/tmp/comparison_results\"\n)\n\nrun_comparison_job_from_dfs(spark, \"my_job\", df1, df2, params, output_config)\n```\n\n---\n\n### 2. Compare Using Config (dict/dataclasses)\n\nUse `run_comparison_job` to compare multiple datasets using a configuration dictionary or object.\n\n#### Function Signature\n\n```python\nrun_comparison_job(\n    spark: SparkSession,\n    config: ComparisonJobConfig | dict\n)\n```\n\n#### Parameters\n\n- `spark`: The active `SparkSession`.\n- `config`: A dictionary or [`ComparisonJobConfig`](#comparisonjobconfig) instance describing one or more datasets to compare, their source/target configs, and output config.\n\n#### Example\n\n```python\nfrom spark_data_test.jobs.comparison_job import run_comparison_job\n\nconfig = {\n    \"job_name\": \"multi_dataset_job\",\n    \"dataset_configs\": [\n        {\n            \"params\": {\n                \"dataset_name\": \"table1\",\n                \"primary_keys\": [\"id\"]\n            },\n            \"source_config\": {\n                \"path\": \"/data/source/table1\",\n                \"file_format\": \"parquet\"\n            },\n            \"target_config\": {\n                \"path\": \"/data/target/table1\",\n                \"file_format\": \"parquet\"\n            }\n        }\n    ],\n    \"output_config\": {\n        \"output_dir\": \"/tmp/comparison_results\"\n    }\n}\n\nrun_comparison_job(spark, config)\n```\n\n---\n\n## Example Configuration (Python dict)\n\nBelow is an example of how to create a configuration dictionary for `run_comparison_job` using the dataclass structure:\n\n```python\nconfig = {\n    \"job_name\": \"sample_comparison_job\",\n    \"dataset_configs\": [\n        {\n            \"params\": {\n                \"dataset_name\": \"table1\",\n                \"primary_keys\": [\"id\"],\n                \"test_params\": {\"difference_tolerance\": 0.1},\n                \"select_cols\": [\"id\", \"name\", \"value\"],\n                \"drop_cols\": []\n            },\n            \"source_config\": {\n                \"path\": \"/data/source/table1\",\n                \"file_format\": \"parquet\",\n                \"spark_options\": {}\n            },\n            \"target_config\": {\n                \"path\": \"/data/target/table1\",\n                \"file_format\": \"parquet\",\n                \"spark_options\": {}\n            }\n        },\n        {\n            \"params\": {\n                \"dataset_name\": \"table2\",\n                \"primary_keys\": [\"key\"],\n                \"test_params\": {\"difference_tolerance\": 0.0},\n                \"select_cols\": [\"key\", \"amount\"],\n                \"drop_cols\": [\"extra_col\"]\n            },\n            \"source_config\": {\n                \"path\": \"/data/source/table2\",\n                \"file_format\": \"csv\",\n                \"spark_options\": {\"header\": \"true\"}\n            },\n            \"target_config\": {\n                \"path\": \"/data/target/table2\",\n                \"file_format\": \"csv\",\n                \"spark_options\": {\"header\": \"true\"}\n            }\n        }\n    ],\n    \"output_config\": {\n        \"output_dir\": \"/tmp/comparison_results\",\n        \"output_file_format\": \"parquet\",\n        \"spark_options\": {},\n        \"no_of_partitions\": -1\n    }\n}\n```\n\nYou can pass this config directly to `run_comparison_job(spark, config)`.\n\n---\n\n## Configuration Dataclasses\n\nBelow are the main dataclasses used for configuration in `spark-data-test`. You can use these directly in Python or as a reference for your JSON configs.\n\n### DatasetParams\nDefines parameters for a single dataset comparison.\n```python\n@dataclass\nclass TestParams:\n    difference_margin: float = 0.0  # Allowed numeric difference for matching numeric columns.\n```\n```python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass DatasetParams:\n    dataset_name: str  # Name of the dataset/table\n    primary_keys: list # List of primary key column names\n    test_params: TestParams # Testing parameters for dataset (Optional)\n    select_cols: list # Columns to select (default: all) (Optional)\n    drop_cols: list # Columns to drop (default: none) (Optional)\n```\n\n### DataframeConfig\n\nDefines how to read a DataFrame from storage.\n\n```python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass DataframeConfig:\n    path: str        # Path to the data (e.g., file or table)\n    file_format: str # File format (parquet, csv, etc.) (default:parquet) (Optional)\n    spark_options: dict # Spark read options (e.g., {\"header\": \"true\"}) (Optional)\n```\n\n### OutputConfig\n\nDefines output options for writing comparison results.\n\n```python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass OutputConfig:\n    output_dir: str         # Directory to write output files\n    output_file_format: str # Output file format (default:parquet) (Optional)\n    spark_options: dict  # Spark write options (Optional)\n    no_of_partitions: int = -1 # Number of partitions for output (-1 for default partitions) (Optional)\n```\n\n### DatasetConfig\n\nGroups together the configs for a single dataset comparison.\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass DatasetConfig:\n    params: DatasetParams              # Dataset parameters\n    source_config: DataframeConfig     # Source DataFrame config\n    target_config: DataframeConfig     # Target DataFrame config\n```\n\n### ComparisonJobConfig\n\nTop-level config for a comparison job (can include multiple datasets).\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass ComparisonJobConfig:\n    job_name: str                      # Name of the comparison job\n    dataset_configs: list[DatasetConfig] # List of dataset configs to compare\n    output_config: OutputConfig        # Output config for all results\n```\n\n---\n\n## Output Files\n\nAfter running a comparison job, the following files/directories are generated under the specified `output_dir` and `job_name`:\n\n### **overall_test_report**\n\nSummary DataFrame with row counts, matched counts, duplicate counts, missing rows, and test status for each dataset. Output will generate under `\u003coutput_dir\u003e/\u003cjob_name\u003e/overall_test_report`\n\n| dataset_name | count                | matched_count | duplicate_count         | missing_rows           | test_status |\n|--------------|----------------------|---------------|------------------------|------------------------|-------------|\n| table1       | {\"source\": 100, \"target\": 98} | 97            | {\"source\": 0, \"target\": 1} | {\"source\": 1, \"target\": 3} | FAILED      |\n\n---\n\n### **col_lvl_test_report**\n\nColumn-level report showing the count of unmatched values for each non-key column. Output will generate under `\u003coutput_dir\u003e/\u003cjob_name\u003e/col_lvl_test_report`\n\n| dataset_name | column_name | unmatched_rows_count |\n|--------------|-------------|---------------------|\n| table1       | colA        | 2                   |\n| table1       | colB        | 0                   |\n\n---\n\n### **row_lvl_test_report**\n\nRow-level report with primary keys, duplicate count, missing row status, and match status for each row. Output will generate under `\u003coutput_dir\u003e/\u003cjob_name\u003e/row_lvl_test_report`\n\n| dataset_name | id | duplicate_count | missing_row_status   | all_rows_matched |\n|--------------|----|----------------|----------------------|------------------|\n| table1       | 1  | 0              | PRESENT_IN_BOTH      | true             |\n| table1       | 2  | 0              | MISSING_AT_TARGET    | false            |\n\n---\n\n### **unmatched_rows/**\n\nDirectory containing one file per column with all rows where that column did not match between source and target. Output will generate under `\u003coutput_dir\u003e/\u003cjob_name\u003e/unmatched_rows/\u003cdataset_name\u003e/\u003ccolumn_name\u003e`\n\nExample for `unmatched_rows/colA`:\n\n| dataset_name | id | colA_src | colA_target |\n|--------------|----|----------|-------------|\n| table1       | 5  | foo      | bar         |\n| table1       | 8  | baz      | qux         |\n\nAll outputs are written in the format specified by `output_file_format` (default: parquet).\n\n---\n\n## Notes\n\n- The package requires PySpark and is intended for use in Spark environments.\n- For more details on configuration options, see the `entities/config.py` dataclasses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjafeerr%2Fspark-data-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjafeerr%2Fspark-data-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjafeerr%2Fspark-data-test/lists"}