Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/lloydzhou/mapred.io
mapreduce based on socket.io
https://github.com/lloydzhou/mapred.io
Last synced: 7 days ago
JSON representation
mapreduce based on socket.io
- Host: GitHub
- URL: https://github.com/lloydzhou/mapred.io
- Owner: lloydzhou
- Created: 2013-05-28T14:49:35.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2013-11-29T08:23:24.000Z (about 11 years ago)
- Last Synced: 2024-12-06T21:58:25.450Z (27 days ago)
- Language: JavaScript
- Size: 669 KB
- Stars: 3
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[mapred.io](http://lloydzhou.github.io/mapred.io/)
=========
mapreduce based on socket.io
You can use this module, use the browser to easily build mapreduce programming model based on parallel computing platform. On this platform, you can submit the page using javascript mapreduce tasks.
##install
```js
npm install mapred.io
```
## Server
```js
var app = require('http').createServer(handler)
, fs = require('fs')
, io = require('mapred.io').listen(app)
, parse = require('url').parse;
app.listen(80);
io.set('log level', 2);
function handler (req, res) {
var info = parse(req.url, true);
var hander = function (err, data) {
if (err) {
res.writeHead(500);
return res.end('Error loading file');
}
res.writeHead(200);
res.end(data);
}
if (info.pathname.match(/\/public/))
fs.readFile(__dirname + info.pathname, hander )
else if (info.pathname.match(/\/client/))
fs.readFile(__dirname + '/client.html', hander )
else fs.readFile(__dirname + '/job.html', hander )
}
```
## Client
```html
var socket = io.connect();
var MapredClient = new MapredClient(socket);
```
## Submit Job
```html
var input = [
['frase primera', 'primer trozo de informacion para procesado primer trozo'],
['segunda frase', 'segundo trozo de informacion trozo de'],
['cacho 3', 'otro trozo para ser procesado otro otro otro trozo'],
['cuarta frase', 'primer trozo de informacion para procesado primer trozo'],
['frase 5', 'segundo trozo de informacion trozo de']
], inputs = input, i;
//for ( i = 0 ; i < 20000 ; i ++ ) inputs = inputs.concat(input);
socket.emit('job', { map: (function(key, value){
var list = [], aux = {};
value = value.split(' ');
value.forEach(function(w){
aux[w] = (aux[w] || 0) + 1;
});
for(var k in aux){
list.push([k, aux[k]]);
}
return list;
}).toString(), reduce: (function(key, values){
var sum = 0;
values.forEach(function(e){
sum += e;
});
return sum;
}).toString(), inputs:inputs)
```
## change log
2013-11-29 update mapred.io-client.js can be using in browser and command line, add client.js can run as a node.
## To do
1. Performance Test
2. Client program (the standalone client not on the browser).
3. Stored the datas (the server do not have enough memory to handle large data, Need a fast persistent storage framework. Just using the store of socket.io default is "MemoryStore", you can using "RedisStore")
update the screeshot