Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zero323/pyspark-stubs
Apache (Py)Spark type annotations (stub files).
https://github.com/zero323/pyspark-stubs
apache-spark mypy pep484 pyspark python python-3 stub-files type-annotations
Last synced: about 2 months ago
JSON representation
Apache (Py)Spark type annotations (stub files).
- Host: GitHub
- URL: https://github.com/zero323/pyspark-stubs
- Owner: zero323
- License: apache-2.0
- Archived: true
- Created: 2017-01-31T01:13:15.000Z (almost 8 years ago)
- Default Branch: branch-3.0
- Last Pushed: 2022-08-17T08:48:09.000Z (about 2 years ago)
- Last Synced: 2024-09-24T09:02:16.820Z (about 2 months ago)
- Topics: apache-spark, mypy, pep484, pyspark, python, python-3, stub-files, type-annotations
- Language: Python
- Homepage:
- Size: 1.3 MB
- Stars: 115
- Watchers: 6
- Forks: 37
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
PySpark Stubs
=============
|Build Status| |PyPI version| |Conda Forge version|
A collection of the Apache Spark `stub
files `__. These
files were generated by
`stubgen `__
and manually edited to include accurate type hints.
Tests and configuration files have been originally contributed to the
`Typeshed project `__. Please refer
to its `contributors
list `__ and
`license `__ for
details.
Important
----------
This project `has been merged `_ with the main Apache Spark repository (`SPARK-32714 `_). All further development for Spark 3.1 and onwards will be continued there.
For Spark 2.4 and 3.0, development of this package will be continued, until their official deprecation.
- If your problem is specific to Spark 2.3 and 3.0 feel free to create an issue or open pull requests here.
- Otherwise, please check `the official Spark JIRA `_ and `contributing guidelines `_. If you create a JIRA ticket or Spark PR related to type hints, please ping me with `[~zero323] `_ or `@zero323 `_ respectively. Thanks in advance.
Motivation
----------
- Static error detection (see
`SPARK-20631 `__)
|SPARK-20631|
- Improved autocompletion.
|Syntax completion|
Installation and usage
----------------------
Please note that the guidelines for distribution of type information is
still work in progress (`PEP 561 - Distributing and Packaging Type
Information `__). Currently
installation script overlays existing Spark installations (``pyi`` stub
files are copied next to their ``py`` counterparts in the PySpark
installation directory). If this approach is not acceptable you can add stub
files to the search path manually.
According to `PEP
484 `__:
Third-party stub packages can use any location for stub storage.
Type checkers should search for them using PYTHONPATH.
Moreover:
Third-party stub packages can use any location for stub storage.
Type checkers should search for them using PYTHONPATH. A default
fallback directory that is always checked is
shared/typehints/python3.5/ (or 3.6, etc.)
Please check usage before proceeding.
The package is available on `PYPI `__:
.. code:: bash
pip install pyspark-stubs
and `conda-forge `__:
.. code:: bash
conda install -c conda-forge pyspark-stubs
Depending on your environment you might also need a type checker, like `Mypy`_
or `Pytype`_ [#f1]_, and autocompletion tool, like `Jedi`_.
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| Editor | Type checking | Autocompletion | Notes |
+==================================================+=====================+====================+=====================================+
| `Atom`_ | ✔ [#f2]_ | ✔ [#f3]_ | Through plugins. |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| `IPython`_ / `Jupyter Notebook`_ | ✘ [#f4]_ | ✔ | |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| `PyCharm`_ | ✔ | ✔ | |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| `PyDev`_ | ✔ [#f5]_ | ? | |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| `VIM`_ / `Neovim`_ | ✔ [#f6]_ | ✔ [#f7]_ | Through plugins. |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| `Visual Studio Code`_ | ✔ [#f8]_ | ✔ [#f9]_ | Completion with plugin |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
| Environment independent / other editors | ✔ [#f10]_ | ✔ [#f11]_ | Through `Mypy`_ and `Jedi`_. |
+--------------------------------------------------+---------------------+--------------------+-------------------------------------+
This package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.
PySpark Version Compatibility
-----------------------------
Package versions follow PySpark versions with exception to maintenance releases - i.e. `pyspark-stubs==2.3.0` should be compatible with `pyspark>=2.3.0,<2.4.0`.
Maintenance releases (`post1`, `post2`, ..., `postN`) are reserved for internal annotations updates.
API Coverage:
-------------
As of release 2.4.0 most of the public API is covered. For details please check `API coverage document `__.
See also
--------
- `SPARK-17333 `__ - *Make pyspark interface friendly with static analysis*.
- `PySpark typing hints `__ and `Revisiting PySpark type annotations `__ on `Apache Spark Developers List `__.
Disclaimer
----------
Apache Spark, Spark, PySpark, Apache, and the Spark logo are `trademarks `__ of `The
Apache Software Foundation `__. This project is not owned, endorsed, or
sponsored by The Apache Software Foundation.
Footnotes
---------
.. [#f1] Not supported or tested.
.. [#f2] Requires `atom-mypy `__ or equivalent.
.. [#f3] Requires `autocomplete-python-jedi `__ or equivalent.
.. [#f4] `It is possible `__
to use magics to type check directly in the notebook. In general though, you'll have to export whole notebook to `.py` file and run
type checker on the result.
.. [#f5] Requires PyDev 7.0.3 or later.
.. [#f6] TODO Using `vim-mypy `__, `syntastic `__ or `Neomake `__.
.. [#f7] With `jedi-vim `__.
.. [#f8] With `Mypy linter `__.
.. [#f9] With `Python extension for Visual Studio Code `__.
.. [#f10] Just use your favorite checker directly, optionally combined with tool like `entr `__.
.. [#f11] See `Jedi editor plugins list `__.
.. |Build Status| image:: https://github.com/zero323/pyspark-stubs/actions/workflows/test.yml/badge.svg
:target: https://github.com/zero323/pyspark-stubs/actions/workflows/test.yml
.. |PyPI version| image:: https://img.shields.io/pypi/v/pyspark-stubs.svg
:target: https://pypi.org/project/pyspark-stubs/
.. |Conda Forge version| image:: https://img.shields.io/conda/vn/conda-forge/pyspark-stubs.svg
:target: https://anaconda.org/conda-forge/pyspark-stubs
.. |SPARK-20631| image:: https://i.imgur.com/GfDCGjv.gif
:alt: SPARK-20631
.. |Syntax completion| image:: https://i.imgur.com/qvkLTAp.gif
:alt: Syntax completion
.. _Atom: https://atom.io/
.. _IPython: https://ipython.org/
.. _Jedi: https://github.com/davidhalter/jedi
.. _Jupyter Notebook: https://jupyter.org/
.. _Mypy: http://mypy-lang.org/
.. _Neovim : https://neovim.io/
.. _PyCharm: https://www.jetbrains.com/pycharm/
.. _PyDev: https://www.pydev.org/
.. _Pytype: https://github.com/google/pytype
.. _VIM: https://www.vim.org/
.. _Visual Studio Code: https://code.visualstudio.com/