https://github.com/karellen/geventmp
GeventMP - Gevent Multiprocessing Extension
https://github.com/karellen/geventmp
gevent monkey-patching multiprocessing non-blocking-io non-blocking-sockets python python-multiprocessing
Last synced: 5 months ago
JSON representation
GeventMP - Gevent Multiprocessing Extension
- Host: GitHub
- URL: https://github.com/karellen/geventmp
- Owner: karellen
- License: apache-2.0
- Created: 2019-08-13T20:25:05.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-10-29T00:51:52.000Z (7 months ago)
- Last Synced: 2024-11-28T22:19:15.435Z (6 months ago)
- Topics: gevent, monkey-patching, multiprocessing, non-blocking-io, non-blocking-sockets, python, python-multiprocessing
- Language: Python
- Homepage: https://github.com/karellen/geventmp
- Size: 104 KB
- Stars: 18
- Watchers: 6
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
===============================================
GeventMP_ - Gevent_ Multiprocessing Extension
===============================================
.. image:: https://img.shields.io/gitter/room/karellen/Lobby?logo=gitter
:target: https://app.gitter.im/#/room/#karellen_Lobby:gitter.im
:alt: Gitter
.. image:: https://img.shields.io/github/actions/workflow/status/karellen/geventmp/build.yml?branch=master
:target: https://github.com/karellen/geventmp/actions/workflows/build.yml
:alt: Build Status
.. image:: https://img.shields.io/coveralls/github/karellen/geventmp/master?logo=coveralls
:target: https://coveralls.io/r/karellen/geventmp?branch=master
:alt: Coverage Status
|
.. image:: https://img.shields.io/pypi/v/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Version
.. image:: https://img.shields.io/pypi/pyversions/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Python Versions
.. image:: https://img.shields.io/pypi/dd/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Downloads Per Day
.. image:: https://img.shields.io/pypi/dw/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Downloads Per Week
.. image:: https://img.shields.io/pypi/dm/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Downloads Per Month
|
.. warning::
HIC SUNT DRACONES!!!
This code is experimental (beta). There is some testing, but a lots of things are in flux, and
some platforms don't work at all.
You MAY try to use this in production with the understanding that this is a beta-quality software.
That said, this code may crash your server, bankrupt your company, burn your house down and be mean
to your puppy.
You've been warned.
Problem
=======
Due to internal implementation, `multiprocessing` (`MP`) is unsafe to use with Gevent_ even when `monkey-patched`__.
Namely, the use of OS semaphore primitives and inter-process IO in `MP` will cause the main
loop to stall/deadlock/block (specific issue depends on the version of CPython).
__ monkey_
Solution
========
GeventMP_ (`Gee-vent Em-Pee`) is a gevent_ multiprocessing extension plugin for the `monkey-patching`__ subsystem.
As with the rest of the monkey patch subsystem the process is fairly clear:
__ monkey_
1. Identify all places where blocking occurs and where it may stall the loop.
2. If blocking occurs on a file descriptor (`FD`), try to convert the file descriptor from blocking to non-blocking
(sockets/pipes/fifos, sometimes even files where, rarely, appropriate) and replace blocking IO functions with their
gevent_ non-blocking equivalents.
3. If blocking occurs in a Python/OS primitive that does not support non-blocking access and thus cannot be geventized,
wrap all blocking access to that primitive with native thread-pool-based wrappers and call it a day (while fully
understanding that primitive access latency will increase and raw performance may suffer as a result).
4. If you are really brave and have lots of free time on your hands, completely replace a standard blocking Python
non-`FD`-based primitive with implementation based on an `FD`-based OS primitive (e.g. POSIX semaphore =>
Linux `eventfd-based semaphore for kernels > 2.6.30`__).
5. Due to launching of separate processes in `MP`, figure out how, when, and whether to `monkey-patch`__ spawned/forked
children and grandchildren.
__ eventfd_
__ monkey_
Installation
============
The package is hosted on PyPi_.
For stable version:
.. code-block:: bash
pip install geventmp
For unstable version:
.. code-block:: bash
pip install --pre geventmp
Once installed, `GeventMP`_ will activate by default in the below stanza.
.. code-block:: python
from gevent.monkey import patch_all
patch_all()
If you would like `GeventMP`_ to not activate by default, either do not install it or explicitly disable it:
.. code-block:: python
from gevent.monkey import patch_all
patch_all(geventmp=False)
That's it - there are no other flags, settings, properties or config values so far.
Supported Platforms
===================
.. note::
All claims of support may not be real at all. You're welcome to experiment. See warnings on top.
* Linux and Darwin.
* CPython 3.9, 3.10, 3.11, 3.12, 3.13
* PyPy 3.9, 3.10
Known Issues
============
* Multiprocessing `forkserver` works in GeventMP_, but the spawned child isn't green.
TODO
====
1. Monkey patch Windows to the extent possible.
2. Lots of applications use `Billiard `_ for multiprocessing instead of stock Python
package. Consider monkey patching Billiard if detected.
Contact Us
==========
Post feedback and issues on the `Bug Tracker`_, `Gitter`_,
and `Twitter (@karelleninc)`_.
.. _Gevent: https://github.com/gevent/gevent/
.. _geventmp: https://github.com/karellen/geventmp
.. _bug tracker: https://github.com/karellen/geventmp/issues
.. _gitter: https://gitter.im/karellen/Lobby
.. _twitter (@karelleninc): https://twitter.com/karelleninc
.. _monkey: https://en.wikipedia.org/wiki/Monkey_patch
.. _eventfd: https://linux.die.net/man/2/eventfd
.. _pypi: https://pypi.org/project/geventmp/