{"id":18640061,"url":"https://github.com/aleskandro/r-hadoop-madreduce-examples","last_synced_at":"2025-11-04T19:30:38.736Z","repository":{"id":90194287,"uuid":"116394177","full_name":"aleskandro/r-hadoop-madreduce-examples","owner":"aleskandro","description":"A lot of examples about using R with hadoop for MapReduce with and without libraries as rhadoop/rhipe - DIEEI@unict.it - Advanced Programming Languages","archived":false,"fork":false,"pushed_at":"2018-01-18T14:11:29.000Z","size":1169,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-27T09:42:38.283Z","etag":null,"topics":["data-analysis","hadoop","mapreduce","r"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aleskandro.png","metadata":{"files":{"readme":"README.en.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-05T14:38:03.000Z","updated_at":"2018-01-05T16:19:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"5773ed9c-7db2-4a7a-b70b-575e0ed36c19","html_url":"https://github.com/aleskandro/r-hadoop-madreduce-examples","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleskandro%2Fr-hadoop-madreduce-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleskandro%2Fr-hadoop-madreduce-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleskandro%2Fr-hadoop-madreduce-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aleskandro%2Fr-hadoop-madreduce-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aleskandro","download_url":"https://codeload.github.com/aleskandro/r-hadoop-madreduce-examples/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239441656,"owners_count":19639121,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","hadoop","mapreduce","r"],"created_at":"2024-11-07T05:53:05.651Z","updated_at":"2025-11-04T19:30:38.702Z","avatar_url":"https://github.com/aleskandro.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"## R \u0026 Hadoop for data analysis with MapReduce\n\n### Dependencies\n\n* Hadoop\n* devtools (R)\n* klmr/modules (R)\n* roxygen2 (R)\n* linux-coreutils: (cat, less, sort... )\n\n## TCPDump Logs\n\n### Streaming\n\nTo run the code without hadoop:\n\n```\n    $ cd src/streaming\n    $ cat myTcpDump.log | ./mapper.r | ./reducer.r\n\n```\n\n\n#### Hadoop (and Docker)\n\nThe code was tested in a Docker container derived from 'sequenceiq/hadoop-docker' hub.docker.com, using a single node hadoop server.\n\nTo build and run the container (take care of permissions and SELinux):\n```\n    $ docker build -t rhadoop .\n    $ docker run -v /ABSPATH/TO/THIS/REPO/src/:/code -it rhadoop /etc/bootstrap.sh -bash\n```\n\nIn the container, to run the analyzer:\n```\n    # cd /code/\n    # hdfs dfs -copyFromLocal $myTcpDumpLogFile $myTcpDumpLogFile\n    # hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.0.jar \\\n        -file ./utils.r\n        -file ./mapper.r    -mapper ./mapper.r \\\n        -file ./reducer.r   -reducer ./reducer.r \\\n        -input $MyTCPDumpLogFile -output log.log ' \n    # hdfs dfs -cat log.log/part* | less\n```\n\nhdfs dfs -rm log.log will delete the log directory if you need to restart the map-reduce as above.\n\nWARNING: Verify the path of the hadoop-streaming-\\*.jar file \n\n`src/streaming/script.sh` provide a simple way to be ran inside the docker container avoiding the handwritten commands above\n\n## Using RHadoop\n\n```\n    $ docker build -t rhadoop .\n    $ docker run -v /ABSPATH/TO/THIS/REPO/src/:/code -it rhadoop /etc/bootstrap.sh -bash\n    # cd src/rhadoop\n    # hdfs dfs -copyFromLocal ../../samples/tcpdump.log \n    # ./rhadoop.r\n```\n\n## Using RHipe\n\nNot working on Hadoop 2.7: https://github.com/delta-rho/RHIPE/issues/45\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleskandro%2Fr-hadoop-madreduce-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleskandro%2Fr-hadoop-madreduce-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleskandro%2Fr-hadoop-madreduce-examples/lists"}