{"id":13474545,"url":"https://github.com/zero323/pyspark-stubs","last_synced_at":"2025-10-03T01:31:08.581Z","repository":{"id":45384341,"uuid":"80480364","full_name":"zero323/pyspark-stubs","owner":"zero323","description":"Apache (Py)Spark type annotations  (stub files).","archived":true,"fork":false,"pushed_at":"2022-08-17T08:48:09.000Z","size":1362,"stargazers_count":115,"open_issues_count":0,"forks_count":37,"subscribers_count":6,"default_branch":"branch-3.0","last_synced_at":"2025-01-17T06:11:52.585Z","etag":null,"topics":["apache-spark","mypy","pep484","pyspark","python","python-3","stub-files","type-annotations"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zero323.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://archive.org/donate","https://supporters.eff.org/donate","https://www.msf.org/donate"]}},"created_at":"2017-01-31T01:13:15.000Z","updated_at":"2024-09-12T19:34:21.000Z","dependencies_parsed_at":"2022-09-19T09:31:38.998Z","dependency_job_id":null,"html_url":"https://github.com/zero323/pyspark-stubs","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zero323%2Fpyspark-stubs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zero323%2Fpyspark-stubs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zero323%2Fpyspark-stubs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zero323%2Fpyspark-stubs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zero323","download_url":"https://codeload.github.com/zero323/pyspark-stubs/tar.gz/refs/heads/branch-3.0","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235059234,"owners_count":18929279,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","mypy","pep484","pyspark","python","python-3","stub-files","type-annotations"],"created_at":"2024-07-31T16:01:13.075Z","updated_at":"2025-10-03T01:31:08.217Z","avatar_url":"https://github.com/zero323.png","language":"Python","funding_links":["https://archive.org/donate","https://supporters.eff.org/donate","https://www.msf.org/donate"],"categories":["Python"],"sub_categories":[],"readme":"PySpark Stubs\n=============\n\n|Build Status| |PyPI version| |Conda Forge version|\n\nA collection of the Apache Spark `stub\nfiles \u003chttps://www.python.org/dev/peps/pep-0484/#stub-files\u003e`__. These\nfiles were generated by\n`stubgen \u003chttps://github.com/python/mypy/blob/master/mypy/stubgen.py\u003e`__\nand manually edited to include accurate type hints.\n\nTests and configuration files have been originally contributed to the\n`Typeshed project \u003chttps://github.com/python/typeshed/\u003e`__. Please refer\nto its `contributors\nlist \u003chttps://github.com/python/typeshed/graphs/contributors\u003e`__ and\n`license \u003chttps://github.com/python/typeshed/blob/master/LICENSE\u003e`__ for\ndetails.\n\nImportant\n----------\n\nThis project `has been merged \u003chttps://github.com/apache/spark/commit/31a16fbb405a19dc3eb732347e0e1f873b16971d#diff-23eeeb4347bdd26bfc6b7ee9a3b755dd\u003e`_  with the main Apache Spark repository (`SPARK-32714 \u003chttps://issues.apache.org/jira/browse/SPARK-32714\u003e`_). All further development for Spark 3.1 and onwards will be continued there.\n\nFor Spark 2.4 and 3.0, development of this package will be continued, until their official deprecation.\n\n- If your problem is specific to Spark 2.3 and 3.0 feel free to create an issue or open pull requests here.\n- Otherwise, please check `the official Spark JIRA \u003chttps://issues.apache.org/jira/projects/SPARK/issues/\u003e`_ and `contributing guidelines \u003chttps://spark.apache.org/contributing.html\u003e`_. If you create a JIRA ticket or Spark PR related to type hints, please ping me with `[~zero323] \u003chttps://issues.apache.org/jira/secure/ViewProfile.jspa?name=zero323\u003e`_ or `@zero323 \u003chttps://github.com/zero323\u003e`_ respectively. Thanks in advance.\n\nMotivation\n----------\n\n-  Static error detection (see\n   `SPARK-20631 \u003chttps://issues.apache.org/jira/browse/SPARK-20631\u003e`__)\n\n   |SPARK-20631|\n\n-  Improved autocompletion.\n\n   |Syntax completion|\n\nInstallation and usage\n----------------------\n\nPlease note that the guidelines for distribution of type information is\nstill work in progress (`PEP 561 - Distributing and Packaging Type\nInformation \u003chttps://www.python.org/dev/peps/pep-0561/\u003e`__). Currently\ninstallation script overlays existing Spark installations (``pyi`` stub\nfiles are copied next to their ``py`` counterparts in the PySpark\ninstallation directory). If this approach is not acceptable you can add stub\nfiles to the search path manually.\n\nAccording to `PEP\n484 \u003chttps://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files\u003e`__:\n\n    Third-party stub packages can use any location for stub storage.\n    Type checkers should search for them using PYTHONPATH.\n\nMoreover:\n\n    Third-party stub packages can use any location for stub storage.\n    Type checkers should search for them using PYTHONPATH. A default\n    fallback directory that is always checked is\n    shared/typehints/python3.5/ (or 3.6, etc.)\n\nPlease check usage before proceeding.\n\nThe package is available on `PYPI \u003chttps://pypi.org/project/pyspark-stubs/\u003e`__:\n\n.. code:: bash\n\n    pip install pyspark-stubs\n\nand `conda-forge \u003chttps://anaconda.org/conda-forge/pyspark-stubs\u003e`__:\n\n.. code:: bash\n\n    conda install -c conda-forge pyspark-stubs\n\nDepending on your environment you might also need a type checker, like `Mypy`_\nor `Pytype`_ [#f1]_, and autocompletion tool, like `Jedi`_.\n\n\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n| Editor                                           |  Type checking      | Autocompletion     | Notes                               |\n+==================================================+=====================+====================+=====================================+\n|  `Atom`_                                         | ✔ [#f2]_            | ✔ [#f3]_           | Through plugins.                    |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n|  `IPython`_ / `Jupyter Notebook`_                | ✘ [#f4]_            | ✔                  |                                     |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n| `PyCharm`_                                       | ✔                   | ✔                  |                                     |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n|  `PyDev`_                                        | ✔ [#f5]_            | ?                  |                                     |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n| `VIM`_ / `Neovim`_                               | ✔ [#f6]_            | ✔ [#f7]_           | Through plugins.                    |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n| `Visual Studio Code`_                            | ✔ [#f8]_            | ✔ [#f9]_           | Completion with plugin              |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n| Environment independent / other editors          | ✔ [#f10]_           | ✔ [#f11]_          | Through `Mypy`_ and `Jedi`_.        |\n+--------------------------------------------------+---------------------+--------------------+-------------------------------------+\n\n\n\n\nThis package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.\n\nPySpark Version Compatibility\n-----------------------------\n\nPackage versions follow PySpark versions with exception to maintenance releases - i.e. `pyspark-stubs==2.3.0` should be compatible with `pyspark\u003e=2.3.0,\u003c2.4.0`.\nMaintenance releases (`post1`, `post2`, ..., `postN`) are reserved for internal annotations updates.\n\nAPI Coverage:\n-------------\n\nAs of release 2.4.0 most of the public API is covered. For details please check `API coverage document \u003chttps://github.com/zero323/pyspark-stubs/blob/master/doc/api-coverage.rst\u003e`__.\n\nSee also\n--------\n\n- `SPARK-17333 \u003chttps://issues.apache.org/jira/browse/SPARK-17333\u003e`__ - *Make pyspark interface friendly with static analysis*.\n- `PySpark typing hints \u003chttp://apache-spark-developers-list.1001551.n3.nabble.com/PYTHON-PySpark-typing-hints-td21560.html\u003e`__ and `Revisiting PySpark type annotations \u003chttp://apache-spark-developers-list.1001551.n3.nabble.com/Re-PySpark-Revisiting-PySpark-type-annotations-td26232.html\u003e`__ on `Apache Spark Developers List \u003chttp://apache-spark-developers-list.1001551.n3.nabble.com/\u003e`__.\n\n\nDisclaimer\n----------\n\nApache Spark, Spark, PySpark, Apache, and the Spark logo are `trademarks \u003chttps://www.apache.org/foundation/marks/\u003e`__ of `The\nApache Software Foundation \u003chttp://www.apache.org/\u003e`__. This project is not owned, endorsed, or\nsponsored by The Apache Software Foundation.\n\nFootnotes\n---------\n\n.. [#f1] Not supported or tested.\n.. [#f2] Requires `atom-mypy \u003chttps://atom.io/packages/atom-mypy\u003e`__ or equivalent.\n.. [#f3] Requires `autocomplete-python-jedi \u003chttps://atom.io/packages/autocomplete-python-jedi\u003e`__ or equivalent.\n.. [#f4] `It is possible \u003chttps://web.archive.org/web/20190126155957/http://journalpanic.com/post/spice-up-thy-jupyter-notebooks-with-mypy/\u003e`__\n         to use magics to type check directly in the notebook. In general though, you'll have to export whole notebook to `.py` file and run\n         type checker on the result.\n.. [#f5] Requires PyDev 7.0.3 or later.\n.. [#f6] TODO Using `vim-mypy \u003chttps://github.com/Integralist/vim-mypy\u003e`__, `syntastic \u003chttps://github.com/vim-syntastic/syntastic\u003e`__ or `Neomake \u003chttps://github.com/neomake/neomake\u003e`__.\n.. [#f7] With `jedi-vim \u003chttps://github.com/davidhalter/jedi-vim\u003e`__.\n.. [#f8] With `Mypy linter \u003chttps://code.visualstudio.com/docs/python/linting#_specific-linters\u003e`__.\n.. [#f9] With `Python extension for Visual Studio Code \u003chttps://marketplace.visualstudio.com/items?itemName=ms-python.python\u003e`__.\n.. [#f10] Just use your favorite checker directly, optionally combined with tool like `entr \u003chttp://eradman.com/entrproject/\u003e`__.\n.. [#f11] See `Jedi editor plugins list \u003chttps://jedi.readthedocs.io/en/latest/docs/usage.html#editor-plugins\u003e`__.\n\n\n.. |Build Status| image:: https://github.com/zero323/pyspark-stubs/actions/workflows/test.yml/badge.svg\n   :target: https://github.com/zero323/pyspark-stubs/actions/workflows/test.yml\n.. |PyPI version| image:: https://img.shields.io/pypi/v/pyspark-stubs.svg\n   :target: https://pypi.org/project/pyspark-stubs/\n.. |Conda Forge version| image:: https://img.shields.io/conda/vn/conda-forge/pyspark-stubs.svg\n   :target: https://anaconda.org/conda-forge/pyspark-stubs\n.. |SPARK-20631| image:: https://i.imgur.com/GfDCGjv.gif\n     :alt: SPARK-20631\n.. |Syntax completion| image:: https://i.imgur.com/qvkLTAp.gif\n     :alt: Syntax completion\n\n.. _Atom: https://atom.io/\n.. _IPython: https://ipython.org/\n.. _Jedi: https://github.com/davidhalter/jedi\n.. _Jupyter Notebook: https://jupyter.org/\n.. _Mypy: http://mypy-lang.org/\n.. _Neovim : https://neovim.io/\n.. _PyCharm: https://www.jetbrains.com/pycharm/\n.. _PyDev: https://www.pydev.org/\n.. _Pytype: https://github.com/google/pytype\n.. _VIM: https://www.vim.org/\n.. _Visual Studio Code: https://code.visualstudio.com/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzero323%2Fpyspark-stubs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzero323%2Fpyspark-stubs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzero323%2Fpyspark-stubs/lists"}