An open API service indexing awesome lists of open source software.

https://github.com/soreana/hadoop-testbench


https://github.com/soreana/hadoop-testbench

Last synced: 3 months ago
JSON representation

Awesome Lists containing this project

README

        

# create git repo
initial setup:
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-git-server-on-a-vps

on server side run 'git init --bare hadoop-yarn-server-resourcemanager.git' to create empty repository
on client side run 'git remote set-url pip git@pip:hadoop-yarn-server-resourcemanager.git' or 'git init && git remote add pip git@pip:hadoop-yarn-server-resourcemanager.git' to set remote repository.

build with no cache:
docker build -t hadoop --build-arg CACHEBUST=$(date +%s) .

## to run map-reduce test after build
1. docker run -it hadoop
2. cd /home/
3. hadoop com.sun.tools.javac.Main WordCount.java
4. jar cf wc.jar WordCount*.class
5. hadoop jar wc.jar WordCount /user/sina/data /user/sina/output

## start in multi user mode
1. build in same way
2. docker run -it -e HADOOP_HOSTS="10.0.0.1 master,10.0.0.2 slave1,10.0.0.3 slave2" -e MY_ROLE="master" hadoop
`HADOOP_HOSTS` shows clusters machines (default = "127.0.0.1 master") and `MY_ROLE` shows machines role in cluster that could be master,slave (default for multi node) or master-slave (default for single mode). Based on number of machines in HADOOP_HOSTS container whould run in single mode or multi node.

## setup network
1. create network : docker network create --subnet=172.18.0.0/16 hadoop-cluster
2. attach to network : docker run --net hadoop-cluster --ip 172.18.0.22 -it ubuntu bash

## run containernet
1. sudo python cluster.py

## useful
1. hadoop fs -stat %r /user/sina/data/mahdiz.big