Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/tonywu6/issues-repro-pyright-pybind11


https://github.com/tonywu6/issues-repro-pyright-pybind11

pybind11 pyright python-typing

Last synced: 5 days ago
JSON representation

Awesome Lists containing this project

README 

        
# Pyright & pybind11



Run [`./demo.sh`](./demo.sh) to execute this example from start to finish.



---



Trying to figure out how to package type information together with [pybind11]

modules.



[pybind11]: http://pybind11.readthedocs.io



This example builds and packages a `calculator` module using pybind. The package

has the following structure:



- _module_ `calculator`

  - `add(a: int, b: int) -> int`

  - _submodule_ `subtract`

    - `sub(a: int, b: int) -> int`



Tested on macOS 13.5.1 (Apple Silicon) and in a [Dev Container] with Debian

bullseye.



[Dev Container]: ./.devcontainer/devcontainer.json



## Prerequisites



- Conda, to manage Python environments and dependencies.

- Your system should also meet the minimum requirements for building C++

  (`make`; `gcc` or `clang`, etc.)



## Building the bindings



Create a new conda environment in `.venv` using

[`environment.yml`](./environment.yml), and then activate it.



```bash

conda env create --file environment.yml --prefix .venv \

  && conda activate $(realpath .venv)

```



The environment uses Python 3.8.17, and installs the following packages:



- pybind11

- [CMake], for generating the build system

- [MyPy], for generating type stubs



[CMake]: https://pypi.org/project/cmake/

[MyPy]: http://mypy.readthedocs.io



Conda also installs the necessary Python/pybind11 headers as well as pybind's

CMake input files, which CMake will need.



Generate the build system:



```bash

cmake .

```



CMake will use [`CMakelists.txt`](./CMakeLists.txt). Your command output should

look similar to the following (paths and compiler versions may vary):



> ```

> -- The CXX compiler identification is AppleClang 14.0.3.14030022

> -- Detecting CXX compiler ABI info

> -- Detecting CXX compiler ABI info - done

> -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped

> -- Detecting CXX compile features

> -- Detecting CXX compile features - done

> -- Found PythonInterp: ... (found suitable version "3.8.17", minimum required is "3.6")

> -- Found PythonLibs: .../.venv/lib/libpython3.8.dylib

> -- Found pybind11: .../.venv/lib/python3.8/site-packages/pybind11/include (found version "2.11.1")

> -- Configuring done (0.5s)

> -- Generating done (0.0s)

> -- Build files have been written to: ...

> ```



Build the bindings:



```bash

make

```



This creates a dynamic library `calculator.*.so` in the current directory. Your

command output should look similar to the following:



> ```

> [ 50%] Building CXX object CMakeFiles/calculator.dir/calculator.cpp.o

> [100%] Linking CXX shared module calculator.cpython-38-darwin.so

> [100%] Built target calculator

> ```



It is now possible to import the module from Python:



```py

>>> import calculator

>>> calculator.add(1, 2)

... 3

```



## Generating type stubs



Generate type stubs from the built bindings using MyPy's [stubgen] command:



[stubgen]: https://mypy.readthedocs.io/en/stable/stubgen.html



```bash

stubgen --package calculator --output typings

```



The stubs will be written to `typings/calculator`.



> ```

> Processed 2 modules

> Generated files under typings/calculator/

> ```



## Installing & testing



Create a new conda environment inside `example/.venv` using

[`example/environment.yml`](./example/environment.yml), and then activate it:



```bash

conda env create --file example/environment.yml --prefix example/.venv \

  && conda activate $(realpath example/.venv)

```



This time, the environment installs [MyPy], [Pyright], and the `calculator`

package we just built.



[Pyright]: https://microsoft.github.io/pyright/



See what files were installed:



```

pip show -f calculator

```



> ```

> Name: calculator

> ...

> Files:

>   ...

>   calculator.*.so

>   calculator/__init__.pyi

>   calculator/py.typed

>   calculator/subtract.pyi

> ```



Run the example script:



```bash

python example/main.py

```



> ```

> 41 + 1 = 42

> 43 - 1 = 42

> ```



## Comparing MyPy and Pyright



The example script contains a typing error:



```py

# def add(a: int, b: int) -> int:

add(41, "1")

```



Type-check with MyPy:



```bash

mypy example/main.py

```



> ```

> example/main.py:10: error: Argument 2 to "sub" has incompatible type "str"; expected "int"  [arg-type]

> Found 1 error in 1 file (checked 1 source file)

> ```



Type-check with Pyright:



```bash

pyright example/main.py

```



> ```

> .../example/main.py

>   .../example/main.py:3:6 - warning: Import "calculator.subtract" could not be resolved from source (reportMissingModuleSource)

>   .../example/main.py:10:37 - error: Argument of type "Literal['1']" cannot be assigned to parameter "arg1" of type "int" in function "sub"

>     "Literal['1']" is incompatible with "int" (reportGeneralTypeIssues)

> 1 error, 1 warning, 0 informations

> ```



`calculator.subtract` is a submodule generated by pybind11 which has no

corresponding source on the filesystem (the entire `calculator` package is in a

single `.so` file).



Here, Pyright warns about not being able to resolve it from source, but MyPy

does not. Notice also that Pyright still reports the type error correctly thanks

to our type stubs.