https://github.com/mynameisvinn/sprite
serverless mapreduce
https://github.com/mynameisvinn/sprite
mapreduce python serverless
Last synced: 5 months ago
JSON representation
serverless mapreduce
- Host: GitHub
- URL: https://github.com/mynameisvinn/sprite
- Owner: mynameisvinn
- Created: 2018-03-31T11:56:52.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2018-04-28T16:52:34.000Z (almost 8 years ago)
- Last Synced: 2024-12-27T14:25:47.210Z (over 1 year ago)
- Topics: mapreduce, python, serverless
- Language: Python
- Size: 368 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# sprite
sprite is an mapreduce implementation with lambda.
with sprite, you can write single threaded code and still get distributed systems performance, with zero provisioning or cluster management.
## why serverless?
classic mapreduce with hadoop/spark/dask means touching infrastructure code. with Sprite, we dont have to think about server provisioning, cluster management, resource management, scheduling. in fact, we wont need to know *anything* about infrastructure.
## example
```python
v = Sprite(bucket="mynameisvinn", access="a", secret="s")
v.map(np.sum, np.arange(100), n_chunks=10)
# returns [45, 145, 245, 345, 445, 545, 645, 745, 845, 945]
```