https://github.com/soreana/hadoop-testbench
https://github.com/soreana/hadoop-testbench
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/soreana/hadoop-testbench
- Owner: soreana
- Created: 2017-11-04T15:49:55.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-12-28T15:53:29.000Z (over 6 years ago)
- Last Synced: 2025-01-25T23:16:44.606Z (5 months ago)
- Language: Dockerfile
- Size: 268 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# create git repo
initial setup:
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-git-server-on-a-vps
on server side run 'git init --bare hadoop-yarn-server-resourcemanager.git' to create empty repository
on client side run 'git remote set-url pip git@pip:hadoop-yarn-server-resourcemanager.git' or 'git init && git remote add pip git@pip:hadoop-yarn-server-resourcemanager.git' to set remote repository.
build with no cache:
docker build -t hadoop --build-arg CACHEBUST=$(date +%s) .
## to run map-reduce test after build
1. docker run -it hadoop
2. cd /home/
3. hadoop com.sun.tools.javac.Main WordCount.java
4. jar cf wc.jar WordCount*.class
5. hadoop jar wc.jar WordCount /user/sina/data /user/sina/output
## start in multi user mode
1. build in same way
2. docker run -it -e HADOOP_HOSTS="10.0.0.1 master,10.0.0.2 slave1,10.0.0.3 slave2" -e MY_ROLE="master" hadoop
`HADOOP_HOSTS` shows clusters machines (default = "127.0.0.1 master") and `MY_ROLE` shows machines role in cluster that could be master,slave (default for multi node) or master-slave (default for single mode). Based on number of machines in HADOOP_HOSTS container whould run in single mode or multi node.
## setup network
1. create network : docker network create --subnet=172.18.0.0/16 hadoop-cluster
2. attach to network : docker run --net hadoop-cluster --ip 172.18.0.22 -it ubuntu bash
## run containernet
1. sudo python cluster.py
## useful
1. hadoop fs -stat %r /user/sina/data/mahdiz.big