https://github.com/discoproject/disco

a Map/Reduce framework for distributed computing
https://github.com/discoproject/disco

Last synced: about 2 months ago
JSON representation

a Map/Reduce framework for distributed computing

• Host: GitHub
• URL: https://github.com/discoproject/disco
• Owner: discoproject
• License: bsd-3-clause
• Created: 2008-07-24T08:46:39.000Z (about 16 years ago)
• Default Branch: develop
• Last Pushed: 2018-01-30T20:55:22.000Z (over 6 years ago)
• Last Synced: 2024-04-13T21:01:54.346Z (5 months ago)
• Language: Erlang
• Homepage: http://discoproject.org
• Size: 9.51 MB
• Stars: 1,633
• Watchers: 85
• Forks: 244
• Open Issues: 140
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
Disco - Massive data, Minimal code

==================================

![Disco Logo](./master/www/images/logo.png)

Disco is a distributed map-reduce and big-data framework. Like

the original framework, which was publicized by Google, Disco supports

parallel computations over large data sets on an unreliable cluster of

computers. This makes it a perfect tool for analyzing and processing large

datasets without having to bother about difficult technical questions

related to distributed computing, such as communication protocols, load

balancing, locking, job scheduling or fault tolerance, all of which are taken

care by Disco.

Writing a Disco job is very simple. For example, the following job counts the number of words in a document:

    from disco.core import Job, result_iterator

    def map(line, params):

        for word in line.split():

            yield word, 1

    def reduce(iter, params):

        from disco.util import kvgroup

        for word, counts in kvgroup(sorted(iter)):

            yield word, sum(counts)

    if __name__ == '__main__':

        input = ["http://discoproject.org/media/text/chekhov.txt"]

        job = Job().run(input=input, map=map, reduce=reduce)

        for word, count in result_iterator(job.wait()):

            print word, count

Note: For installing Disco, you cannot use the zip or tar.gz packages generated by github, instead you should clone this repository.

The develop branch contains the newest features and is not recommended for use

in production.  The master branch is the latest stable release and is tested in

production.  Important bug fixes will be first merged into the develop branch

and then backported into the master branch.

Disco integrates with a lot of different tools. The following screenshot,

for example, shows using ipython notebook to write a Disco job and using

matplotlib to plot the results:

![ipython example](https://raw.githubusercontent.com/discoproject/disco/develop/examples/ipython/ipython.png)

To learn more about the Disco Ecosystem see [Disco Integrations](https://github.com/discoproject/disco/wiki/Disco-Integration-With-Other-Softwares). For some other resources, check out the [Talks on Disco](https://github.com/discoproject/disco/wiki/Talks-about-Disco). Visit [discoproject.org] (http://discoproject.org) for more information.

Build Status: [Travis-CI](http://travis-ci.org/discoproject/disco) :: ![Travis-CI](https://secure.travis-ci.org/discoproject/disco.png)