{"id":19870695,"url":"https://github.com/malexer/pytest-spark","last_synced_at":"2025-05-16T05:07:29.074Z","repository":{"id":60775465,"uuid":"77562657","full_name":"malexer/pytest-spark","owner":"malexer","description":"pytest plugin to run the tests with support of pyspark","archived":false,"fork":false,"pushed_at":"2025-03-21T03:27:17.000Z","size":53,"stargazers_count":86,"open_issues_count":6,"forks_count":30,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-12T20:45:08.641Z","etag":null,"topics":["pytest","python","spark","unit-test","unittest"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/malexer.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-12-28T21:32:56.000Z","updated_at":"2025-03-23T15:05:19.000Z","dependencies_parsed_at":"2024-06-18T17:02:20.869Z","dependency_job_id":"952e300b-6b5a-4286-8d9b-f76eadb4b5fc","html_url":"https://github.com/malexer/pytest-spark","commit_stats":{"total_commits":79,"total_committers":7,"mean_commits":"11.285714285714286","dds":0.5822784810126582,"last_synced_commit":"1125be3451b72d6890f6cce6849baa6effa1bc58"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malexer%2Fpytest-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malexer%2Fpytest-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malexer%2Fpytest-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malexer%2Fpytest-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/malexer","download_url":"https://codeload.github.com/malexer/pytest-spark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254471060,"owners_count":22076585,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pytest","python","spark","unit-test","unittest"],"created_at":"2024-11-12T16:09:49.860Z","updated_at":"2025-05-16T05:07:29.014Z","avatar_url":"https://github.com/malexer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"pytest-spark\n############\n\npytest_ plugin to run the tests with support of pyspark (`Apache Spark`_).\n\nThis plugin will allow to specify SPARK_HOME directory in ``pytest.ini``\nand thus to make \"pyspark\" importable in your tests which are executed\nby pytest.\n\nYou can also define \"spark_options\" in ``pytest.ini`` to customize pyspark,\nincluding \"spark.jars.packages\" option which allows to load external\nlibraries (e.g. \"com.databricks:spark-xml\").\n\npytest-spark provides session scope fixtures ``spark_context`` and\n``spark_session`` which can be used in your tests.\n\n**Note:** no need to define SPARK_HOME if you've installed pyspark using\npip (e.g. ``pip install pyspark``) - it should be already importable. In\nthis case just don't define SPARK_HOME neither in pytest\n(pytest.ini / --spark_home) nor as environment variable.\n\n\nInstall\n=======\n\n.. code-block:: shell\n\n    $ pip install pytest-spark\n\n\nUsage\n=====\n\nSet Spark location\n------------------\n\nTo run tests with required spark_home location you need to define it by\nusing one of the following methods:\n\n1. Specify command line option \"--spark_home\"::\n\n    $ pytest --spark_home=/opt/spark\n\n2. Add \"spark_home\" value to ``pytest.ini`` in your project directory::\n\n    [pytest]\n    spark_home = /opt/spark\n\n3. Set the \"SPARK_HOME\" environment variable.\n\npytest-spark will try to import ``pyspark`` from provided location.\n\n\n.. note::\n    \"spark_home\" will be read in the specified order. i.e. you can\n    override ``pytest.ini`` value by command line option.\n\n\nCustomize spark_options\n-----------------------\n\nJust define \"spark_options\" in your ``pytest.ini``, e.g.::\n\n    [pytest]\n    spark_home = /opt/spark\n    spark_options =\n        spark.app.name: my-pytest-spark-tests\n        spark.executor.instances: 1\n        spark.jars.packages: com.databricks:spark-xml_2.12:0.5.0\n\n\nUsing the ``spark_context`` fixture\n-----------------------------------\n\nUse fixture ``spark_context`` in your tests as a regular pyspark fixture.\nSparkContext instance will be created once and reused for the whole test\nsession.\n\nExample::\n\n    def test_my_case(spark_context):\n        test_rdd = spark_context.parallelize([1, 2, 3, 4])\n        # ...\n\n\nUsing the ``spark_session`` fixture (Spark 2.0 and above)\n---------------------------------------------------------\n\nUse fixture ``spark_session`` in your tests as a regular pyspark fixture.\nA SparkSession instance with Hive support enabled will be created once and reused for the whole test\nsession.\n\nExample::\n\n    def test_spark_session_dataframe(spark_session):\n        test_df = spark_session.createDataFrame([[1,3],[2,4]], \"a: int, b: int\")\n        # ...\n\nOverriding default parameters of the ``spark_session`` fixture\n--------------------------------------------------------------\nBy default ``spark_session`` will be loaded with the following configurations :\n\nExample::\n\n    {\n        'spark.app.name': 'pytest-spark',\n        'spark.default.parallelism': 1,\n        'spark.dynamicAllocation.enabled': 'false',\n        'spark.executor.cores': 1,\n        'spark.executor.instances': 1,\n        'spark.io.compression.codec': 'lz4',\n        'spark.rdd.compress': 'false',\n        'spark.sql.shuffle.partitions': 1,\n        'spark.shuffle.compress': 'false',\n        'spark.sql.catalogImplementation': 'hive',\n    }\n\nYou can override some of these parameters in your ``pytest.ini``.\nFor example, removing Hive Support for the spark session :\n\nExample::\n\n    [pytest]\n    spark_home = /opt/spark\n    spark_options =\n        spark.sql.catalogImplementation: in-memory\n\nDevelopment\n===========\n\nTests\n-----\n\nRun tests locally::\n\n    $ docker-compose up --build\n\n\n.. _pytest: http://pytest.org/\n.. _Apache Spark: https://spark.apache.org/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalexer%2Fpytest-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmalexer%2Fpytest-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalexer%2Fpytest-spark/lists"}