https://github.com/rohanpm/more-executors
Library of composable Python executors
https://github.com/rohanpm/more-executors
executors futures python python2 python3
Last synced: 8 months ago
JSON representation
Library of composable Python executors
- Host: GitHub
- URL: https://github.com/rohanpm/more-executors
- Owner: rohanpm
- License: gpl-3.0
- Created: 2018-03-09T06:04:39.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2024-04-07T20:24:51.000Z (almost 2 years ago)
- Last Synced: 2024-04-07T21:28:10.014Z (almost 2 years ago)
- Topics: executors, futures, python, python2, python3
- Language: Python
- Size: 25.6 MB
- Stars: 23
- Watchers: 2
- Forks: 5
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# more-executors
A library of composable Python executors and futures.
[](https://circleci.com/gh/rohanpm/more-executors/tree/master)
[](https://codeclimate.com/github/rohanpm/more-executors/maintainability)
[](https://codeclimate.com/github/rohanpm/more-executors/test_coverage)
- [API documentation](https://rohanpm.github.io/more-executors/)
- [Source](https://github.com/rohanpm/more-executors)
- [PyPI](https://pypi.python.org/pypi/more-executors)
This library is intended for use with the
[`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html)
module. It includes a collection of `Executor` implementations in order to
extend the behavior of `Future` objects.
## Features
- Futures with implicit retry
- Futures with implicit cancel on executor shutdown
- Futures with implicit cancel after timeout
- Futures with transformed output values (sync & async)
- Futures resolved by a caller-provided polling function
- Throttle the number of futures running at once
- Synchronous executor
- Bridge `concurrent.futures` with `asyncio`
- Convenience API for creating executors
- Instrumented with [Prometheus](https://prometheus.io/)
See the [API documentation](https://rohanpm.github.io/more-executors/) for detailed information on usage.
## Example
This example combines the map and retry executors to create futures for
HTTP requests running concurrently, decoding JSON responses within the
future and retrying on error.
```python
import requests
from concurrent.futures import as_completed
from more_executors import Executors
def get_json(response):
response.raise_for_status()
return (response.url, response.json())
def fetch_urls(urls):
# Configure an executor:
# - run up to 4 requests concurrently, in separate threads
# - run get_json on each response
# - retry up to several minutes on any errors
executor = Executors.\
thread_pool(max_workers=4).\
with_map(get_json).\
with_retry()
# Submit requests for each given URL
futures = [executor.submit(requests.get, url)
for url in urls]
# Futures API works as normal; we can block on the completed
# futures and map/retry happens implicitly
for future in as_completed(futures):
(url, data) = future.result()
do_something(url, data)
```
## Development
`virtualenv` and `pip` may be used to locally install this project from
source:
```
virtualenv ~/dev/python
. ~/dev/python/bin/activate
git clone https://github.com/rohanpm/more-executors
cd more-executors
pip install --editable .
```
Autotests may be run with pytest:
```
pip install -r test-requirements.txt
py.test
```
Submit pull requests against https://github.com/rohanpm/more-executors.
## License
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.