https://github.com/agoose77/parallels

An asyncio API that mimics concurrent.futures, with support for task-graph executors
https://github.com/agoose77/parallels

asyncio concurrent-futures dask delayed executors jupyter python ray

Last synced: about 2 months ago
JSON representation

An asyncio API that mimics concurrent.futures, with support for task-graph executors

• Host: GitHub
• URL: https://github.com/agoose77/parallels
• Owner: agoose77
• License: bsd-3-clause
• Created: 2021-05-05T11:42:17.000Z (about 5 years ago)
• Default Branch: main
• Last Pushed: 2021-05-21T13:53:18.000Z (about 5 years ago)
• Last Synced: 2025-12-09T11:53:06.847Z (6 months ago)
• Topics: asyncio, concurrent-futures, dask, delayed, executors, jupyter, python, ray
• Language: Jupyter Notebook
• Homepage:
• Size: 82 KB
• Stars: 0
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# parallels

[![pypi-badge][]][pypi] 

[pypi-badge]: https://img.shields.io/pypi/v/parallels

[pypi]: https://pypi.org/project/parallels

Parallels is a [literary](https://github.com/agoose77/literary) powered library. 

It provides an async API that mimics concurrent.futures, with support for task-graph executors.

These notebooks can be viewed using [nbviewer](https://nbviewer.jupyter.org/github/agoose77/parallels/tree/main/) until the documentation generator is complete.

## What?

In Python there are several standard APIs for interacting with `executors`. AsyncIO has the `run_in_executor` API, `concurrent.futures` has the `Executor` API, and other

libraries like Dask and Ray have equivalent approaches. `concurrent.futures` is often available within other libraries, but its reduced features-set prevents the underlying

library from implementing useful optimisations like Dask's deferred computation or task graph building.

Parallels implements a standard `Executor` interface which defines a synchronous `submit` method, and an asynchronous `retrieve` method. 

These methods operate upon value-less `asyncio.Future` handles which yield `True` upon task success, and raise an `Exception` otherwise. 

The Dask and Ray implementations accept these handles as arguments to future `submit()` calls, which can be used to build task graphs

and avoid copying data to the local machine.