https://github.com/gmega/sparmap
Simple implementation of parallel map for Python.
https://github.com/gmega/sparmap
Last synced: about 1 year ago
JSON representation
Simple implementation of parallel map for Python.
- Host: GitHub
- URL: https://github.com/gmega/sparmap
- Owner: gmega
- License: apache-2.0
- Created: 2014-08-22T14:51:12.000Z (almost 12 years ago)
- Default Branch: master
- Last Pushed: 2015-07-09T12:18:29.000Z (almost 11 years ago)
- Last Synced: 2025-05-08T21:58:04.411Z (about 1 year ago)
- Language: Python
- Size: 211 KB
- Stars: 4
- Watchers: 0
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
sparmap [](https://travis-ci.org/gmega/parmap)
======
A **S**imple **PAR**allel **MAP** implementation for Python based on the
multiprocessing package.
Usage
-----
As an example, to sum 1 to all elements in the array `[1, 2, 3, 4, 5]` using two processes in parallel,
and printing the output on the terminal, you'd write:
```python
from sparmap import parmap
for result in parmap([1, 2, 3, 4, 5], fun=lambda x: x + 1, workers=2):
print result
```
Streaming
---------
*sparmap* uses a bounded queue internally, and streams results immediately as they
become available. This means that in a situation like:
```python
for result in parmap(very_long_list, fun=expensive_computation, workers=8):
print result
```
you will start seeing results as soon as the first computation inside any of the
workers completes, without having to worry about running out of memory (unless you
have really big records in the input list, or returned as output from your map
function).