https://github.com/wangkuiyi/mapreduce-lite

A C++ implementaton of MapReduce without distributed filesystem
https://github.com/wangkuiyi/mapreduce-lite

Last synced: 17 days ago
JSON representation

A C++ implementaton of MapReduce without distributed filesystem

• Host: GitHub
• URL: https://github.com/wangkuiyi/mapreduce-lite
• Owner: wangkuiyi
• License: other
• Created: 2013-11-30T05:35:31.000Z (almost 11 years ago)
• Default Branch: master
• Last Pushed: 2017-01-22T09:37:18.000Z (almost 8 years ago)
• Last Synced: 2024-07-31T22:46:36.007Z (3 months ago)
• Language: C++
• Size: 311 KB
• Stars: 266
• Watchers: 37
• Forks: 86
• Open Issues: 8
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
MapReduce Lite is a C++ implementation of the MapReduce programming paradigm.

## Pros

First of all, MapReduce Lite is Lite!

  * It does not rely on a distributed filesystem -- it can simply use local filesystem;

  * It does not have a dynamic task scheduling system -- the map/reduce tasks were scheduled before the parallel job is started;

  * There is zero deployment / configuration cost -- just link your program against the MapReduce Lite library statically and run it.

In addition to the functions described in Google's famous MapReduce paper, known as batch reduction mode in MapReduce Lite, there is also an incremental reduction mode, doing the shuffling phase in memory and without disk access. In this mode, MapReduce Lite programs run much faster than rigid implementations like Hadoop.

## Cons

As a lite implementation, MapReduce Lite does not support fault recovery, which, however, is arguably not too difficult to achieve if we do not require backup workers or global counters and can use a distributed filesystem (DFS).

## Applications

In Tencent, we have been using MapReduce Lite with a Tencent's DFS to run jobs like search engine log processing, search and ads click model training, and distributed language model training.

## A Sample

    using mapreduce_lite::Mapper;

    using mapreduce_lite::BatchReducer;

    using mapreduce_lite::ReduceInputIterator;

    class WordCountMapper : public Mapper {

     public:

      void Map(const std::string& key, const std::string& value) {

        std::vector words;

        SplitStringUsing(value, " ", &words);

        for (int i = 0; i < words.size(); ++i) {

          Output(words[i], "1");

        }

      }

    };

    REGISTER_MAPPER(WordCountMapper);

    class WordCountBatchReducer : public BatchReducer {

     public:

      void Reduce(const string& key, ReduceInputIterator* values) {

        int sum = 0;

        LOG(INFO) << "key:[" << key << "]";

        for (; !values->Done(); values->Next()) {

          //LOG(INFO) << "value:[" << values->value() << "]";

          istringstream parser(values->value());

          int count = 0;

          parser >> count;

          sum += count;

        }

        ostringstream formater;

        formater << key << " " << sum;

        Output(key, formater.str());

      }

    };

    REGISTER_BATCH_REDUCER(WordCountBatchReducer);

## Install

Please refer to the [HowToInstall](doc/install.md) document.

## Updates

  1. 2013-10-4: MapReduce Lite supports Mac OS X and FreeBSD in addition to Linux. You can build your MapReduce Lite programs using GCC or Clang.