https://github.com/whitfin/efflux

Easy Hadoop Streaming and MapReduce interfaces in Rust
https://github.com/whitfin/efflux

hadoop mapreduce processing

Last synced: 3 days ago
JSON representation

Easy Hadoop Streaming and MapReduce interfaces in Rust

• Host: GitHub
• URL: https://github.com/whitfin/efflux
• Owner: whitfin
• License: mit
• Created: 2018-09-02T20:51:21.000Z (about 6 years ago)
• Default Branch: main
• Last Pushed: 2024-04-13T01:04:31.000Z (7 months ago)
• Last Synced: 2024-04-14T15:23:40.680Z (7 months ago)
• Topics: hadoop, mapreduce, processing
• Language: Rust
• Size: 51.8 KB
• Stars: 34
• Watchers: 6
• Forks: 8
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-rust-cn - whitfin/efflux
• awesome-rust - whitfin/efflux
• awesome-rust-zh - whitfin/efflux - Rust 中的 Easy Hadoop Streaming 和 MapReduce 接口。 (库 / 分布式系统)
• awesome-rust - whitfin/efflux

README

        
# Efflux

[![Crates.io](https://img.shields.io/crates/v/efflux.svg)](https://crates.io/crates/efflux) [![Build Status](https://img.shields.io/github/actions/workflow/status/whitfin/efflux/ci.yml)](https://github.com/whitfin/efflux/actions)

Efflux is a set of Rust interfaces for MapReduce and Hadoop Streaming. It enables Rust developers to run batch jobs on Hadoop infrastructure whilst staying with the efficiency and safety they're used to.

Initially written to scratch a personal itch, this crate offers simple traits to mask the internals of working with Hadoop Streaming which lend themselves well to writing jobs quickly. Functionality is handed off to macros where possible to provide compile time guarantees, and any other functionality is kept simple to avoid overhead wherever possible.

## Installation

Efflux is available on [crates.io](https://crates.io/crates/efflux) as a library crate, so you only need to add it as a dependency:

```toml

[dependencies]

efflux = "2.0"

```

You can then gain access to everything relevant using the `prelude` module of Efflux:

```rust

use efflux::prelude::*;

```

## Usage

Efflux comes with a handy template to help generate new projects, using the [kickstart](https://github.com/Keats/kickstart) tool. You can simply use the commands below and follow the prompt to generate a new project skeleton:

```shell

# install kickstart

$ cargo install kickstart

# create a project from the template

$ kickstart -s examples/template https://github.com/whitfin/efflux

```

If you'd rather not use the templating tool, you can always work from the examples found in this repository. A good place to start is the traditional [wordcount](examples/wordcount) example.

## Testing

Testing your binaries is actually fairly simple, as you can simulate the Hadoop phases using a basic UNIX pipeline. The following example replicates the Hadoop job flow and generates output that matches a job executed with Hadoop itself:

```shell

# example Hadoop task invocation

$ hadoop jar hadoop-streaming-2.8.2.jar \

    -input  \

    -output  \

    -mapper  \

    -reducer 

# example simulation run via UNIX utilities

$ cat  |  | sort -k1,1 |  > 

```

This can be tested using the [wordcount](examples/wordcount) example to confirm that the outputs are indeed the same. There may be some cases where output differs, but it should be sufficient for many cases.