https://github.com/macrat/parallelline
A Framework for buffering heavy calculation (or I/O) with multiprocessing for Python.
https://github.com/macrat/parallelline
Last synced: 9 months ago
JSON representation
A Framework for buffering heavy calculation (or I/O) with multiprocessing for Python.
- Host: GitHub
- URL: https://github.com/macrat/parallelline
- Owner: macrat
- License: mit
- Created: 2017-09-18T11:53:58.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2017-09-20T07:40:11.000Z (almost 9 years ago)
- Last Synced: 2025-07-14T00:59:49.442Z (12 months ago)
- Language: Python
- Size: 4.88 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Parallel Line
A Framework for buffering heavy calculation (or I/O) with multiprocessing for Python.
## Demo
``` shell
$ git clone http://github.com/macrat/parallelline
$ cd parallelline
$ python3 parallelline.py
```
## Usage
Make something Task. The Task is a function that takes one Pipe instance.
``` python
>>> def load_task(pipe: Pipe) -> None:
... for i in range(10):
... with open('{}.dat', 'rb') as f:
... pipe.put(f.read())
```
The Task function can replace with a generator. Please use `@generator_task` decorator.
The pipe in arguments is Pipe instance but can use as an iterator.
Of course, can use Pipe as an iterator in the normal Task function.
``` python
>>> @generator_task
... def something_process(pipe: Pipe) -> typing.Iterator:
... for data in pipe:
... yield something_heavy_process(data)
```
You can use `@task` decorator if Task hasn't to hold state.
Please make a function that takes one object and return one object, and allies `@task` decorator. In this example, show function will return None but work well.
``` python
>>> @task
... def show(data: typing.Any) -> typing.Any:
... print(data)
```
At last, concatenate all tasks by Pipeline class, and start processes.
The Pipeline will do coloring all messages that wrote into stdio by tasks.
``` python
>>> Pipeline(load_task, something_heavy_process, show).start()
```
## License
MIT License