https://github.com/aleskandro/r-hadoop-madreduce-examples
A lot of examples about using R with hadoop for MapReduce with and without libraries as rhadoop/rhipe - DIEEI@unict.it - Advanced Programming Languages
https://github.com/aleskandro/r-hadoop-madreduce-examples
data-analysis hadoop mapreduce r
Last synced: 8 months ago
JSON representation
A lot of examples about using R with hadoop for MapReduce with and without libraries as rhadoop/rhipe - DIEEI@unict.it - Advanced Programming Languages
- Host: GitHub
- URL: https://github.com/aleskandro/r-hadoop-madreduce-examples
- Owner: aleskandro
- Created: 2018-01-05T14:38:03.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2018-01-18T14:11:29.000Z (over 8 years ago)
- Last Synced: 2024-12-27T09:42:38.283Z (over 1 year ago)
- Topics: data-analysis, hadoop, mapreduce, r
- Language: R
- Homepage:
- Size: 1.11 MB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.en.md
Awesome Lists containing this project
README
## R & Hadoop for data analysis with MapReduce
### Dependencies
* Hadoop
* devtools (R)
* klmr/modules (R)
* roxygen2 (R)
* linux-coreutils: (cat, less, sort... )
## TCPDump Logs
### Streaming
To run the code without hadoop:
```
$ cd src/streaming
$ cat myTcpDump.log | ./mapper.r | ./reducer.r
```
#### Hadoop (and Docker)
The code was tested in a Docker container derived from 'sequenceiq/hadoop-docker' hub.docker.com, using a single node hadoop server.
To build and run the container (take care of permissions and SELinux):
```
$ docker build -t rhadoop .
$ docker run -v /ABSPATH/TO/THIS/REPO/src/:/code -it rhadoop /etc/bootstrap.sh -bash
```
In the container, to run the analyzer:
```
# cd /code/
# hdfs dfs -copyFromLocal $myTcpDumpLogFile $myTcpDumpLogFile
# hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.0.jar \
-file ./utils.r
-file ./mapper.r -mapper ./mapper.r \
-file ./reducer.r -reducer ./reducer.r \
-input $MyTCPDumpLogFile -output log.log '
# hdfs dfs -cat log.log/part* | less
```
hdfs dfs -rm log.log will delete the log directory if you need to restart the map-reduce as above.
WARNING: Verify the path of the hadoop-streaming-\*.jar file
`src/streaming/script.sh` provide a simple way to be ran inside the docker container avoiding the handwritten commands above
## Using RHadoop
```
$ docker build -t rhadoop .
$ docker run -v /ABSPATH/TO/THIS/REPO/src/:/code -it rhadoop /etc/bootstrap.sh -bash
# cd src/rhadoop
# hdfs dfs -copyFromLocal ../../samples/tcpdump.log
# ./rhadoop.r
```
## Using RHipe
Not working on Hadoop 2.7: https://github.com/delta-rho/RHIPE/issues/45