{"id":20384584,"url":"https://github.com/hibuz/hadoop-example","last_synced_at":"2026-04-15T20:31:33.600Z","repository":{"id":47137376,"uuid":"400998474","full_name":"hibuz/hadoop-example","owner":"hibuz","description":null,"archived":false,"fork":false,"pushed_at":"2022-02-27T13:43:36.000Z","size":63,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-04T08:57:15.808Z","etag":null,"topics":["docker","hadoop","wordcount"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hibuz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-29T09:14:20.000Z","updated_at":"2021-12-04T06:37:24.000Z","dependencies_parsed_at":"2022-09-05T17:10:31.077Z","dependency_job_id":null,"html_url":"https://github.com/hibuz/hadoop-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hibuz/hadoop-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibuz%2Fhadoop-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibuz%2Fhadoop-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibuz%2Fhadoop-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibuz%2Fhadoop-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hibuz","download_url":"https://codeload.github.com/hibuz/hadoop-example/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hibuz%2Fhadoop-example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31859202,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","hadoop","wordcount"],"created_at":"2024-11-15T02:28:35.539Z","updated_at":"2026-04-15T20:31:33.582Z","avatar_url":"https://github.com/hibuz.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hadoop maven examples\n\n## Build\n``` bash\ngit clone https://github.com/hibuz/hadoop-example\n\ncd hadoop-example\n\n# Run and Execute bash on docker container\n./docker_up.sh\n\n# Build in the docker container\n./mvnw package\n```\n\n## Prepare input files into the distributed filesystem\n``` bash\n# Make the HDFS directories\nhdfs dfs -mkdir -p /user/hadoop/wordcount/input\n\n# Create the input files\necho \"Hello World Bye World\" | hdfs dfs -put - wordcount/input/file01\necho \"Hello Hadoop Goodbye Hadoop\" | hdfs dfs -put - wordcount/input/file02\n```\n\n## Run some of the examples provided:\n``` bash\n# WordCount Example\nhadoop jar target/examples-0.0.1-SNAPSHOT.jar wordcount wordcount/input wordcount/output\n\n# View the output files on the distributed filesystem:\nhdfs dfs -cat wordcount/output/*\n\n# Result of the output files \nBye\t1\nGoodbye\t1\nHadoop\t2\nHello\t2\nWorld\t2\n\n# Grep Example\nhadoop jar target/examples-0.0.1-SNAPSHOT.jar grep wordcount/input output '([G-H])\\w+'\n\n# View the output files on the distributed filesystem:\nhdfs dfs -cat output/*\n\n# Result of the output files \n2\tHello\n2\tHadoop\n1\tGoodbye\n```\n\n## Modify source to double count:\n```java\n// WordCount.java(reduce func)\n...\n    for (IntWritable val : values) {\n        sum += val.get() * 2;  // modify this line\n    }\n...\n```\n\n```bash\n#build and run using shell script\n./run_example.sh\n\n...\n[INFO] BUILD SUCCESS\n...\n2022-02-27 07:43:00,199 INFO mapreduce.Job: Job job_1645946721365_0006 completed successfully\n...\nBye     2\nGoodbye 2\nHadoop  4\nHello   4\nWorld   4\n```\n\n## Stops containers and removes containers, networks, and volumes\n``` bash\nexit\n\n./docker_down.sh -v\n\n[+] Running 3/3\n ⠿ Container hadoop                  Removed\n ⠿ Volume hadoop-example_hadoop-vol  Removed\n ⠿ Network hadoop-example_default    Removed\n ```\n\n # Reference\n- Docker env: https://github.com/hibuz/ubuntu-docker/tree/main/hadoop\n- Reference: https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v1.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhibuz%2Fhadoop-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhibuz%2Fhadoop-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhibuz%2Fhadoop-example/lists"}