{"id":21358168,"url":"https://github.com/raycad/hadoop","last_synced_at":"2025-07-30T11:04:32.957Z","repository":{"id":141974696,"uuid":"168642077","full_name":"raycad/hadoop","owner":"raycad","description":"Apache Hadoop cluster docker","archived":false,"fork":false,"pushed_at":"2019-02-04T10:38:08.000Z","size":25,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-16T06:16:27.618Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raycad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-01T04:28:12.000Z","updated_at":"2019-03-12T13:25:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"c7af4eb9-7515-4622-a688-58d2998e33a2","html_url":"https://github.com/raycad/hadoop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/raycad/hadoop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fhadoop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fhadoop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fhadoop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fhadoop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raycad","download_url":"https://codeload.github.com/raycad/hadoop/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raycad%2Fhadoop/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267857755,"owners_count":24155914,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T05:14:53.470Z","updated_at":"2025-07-30T11:04:32.882Z","avatar_url":"https://github.com/raycad.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"### A. Create Docker Image\n##### 1. Login to Docker Hub\n```\n$ docker login --username=yourhubusername --password=yourpassword\n```\n\n##### 2. Build docker image\n```\n$ ./build_image.sh\n```\n\n##### 3. Push the image to the docker hub\n```\n$ sudo docker push $DOCKER_ACC/$DOCKER_REPO:$IMG_TAG\n\ne.g\n$ sudo docker push seedotech/hadoop:2.9.2\n```\n\n### B. Pull and Start Hadoop cluster\n##### 1. Pull the image\n```\n$ sudo docker pull seedotech/hadoop:2.9.2\n```\n\n##### 2. Create a hadoop network\n```\n$ sudo docker network create --driver=bridge hadoop\n```\n\n##### 3. Start hadoop containers\n```\n# The 1st argument is set to YES if you want to create a master, the default is YES\n# The 2nd argument is the number of slaves you want to create, the default is 2\n# Start as the default will create a cluster with 3 nodes included 1 master and 2 slaves\n$ sudo ./start_containers.sh\n\nOutput:\nStart hadoop-master container...\nStart hadoop-slave1 container...\nStart hadoop-slave2 container...\nroot@hadoop-master:~#\n\n# Create a cluster has 4 nodes included 1 master and 3 slaves\n$ sudo ./start_containers.sh YES 3\n\n# Create a cluster has 3 nodes included 3 slaves (no hadoop master)\n$ sudo ./start_containers.sh NO 3\n```\n\n##### 4. Verify all the Hadoop services/daemons\n```\n$ docker exec hadoop-master sh -c \"jps\"\n\nOutput:\n161 NameNode\n841 Jps\n378 SecondaryNameNode\n555 ResourceManager\n```\n\n##### 5. Run Wordcount in the docker container\n```\n# Get into the container\n$ sudo docker exec -it hadoop-master bash\n$ ./run_wordcount.sh\n```\n\nOutput\n```\nInput file1.txt:\nHello Docker\nInput file2.txt:\nHello Hadoop\n\nWordcount output:\nDocker  1\nHadoop  1\nHello   2\n```\n\n##### 6. Browse the HDFS system\n```\nhttp://localhost:50070/explorer.html#\nhttp://localhost:8088/cluster\n\nCheck datanode information\n(To avoid port conflict in the same host machine, the 1st hadoop slave port is mapped to 50075. From the other slaves the port is mapped to 2007$i, e.g 20072, 20073,...)\n\nhadoop-slave1\nhttp://localhost:50075\n\nhadoop-slave2\nhttp://localhost:20072\n```\n\n**NOTE**\n\nYou might not upload files to the hadoop cluster via HDFS Web browser. It's due to the cluster will call to http://hadoop-slave1:50075 to process while your machine could not recorgnize the \"hadoop-slave1\" address. To fix this you have to register the Hadoop Slave address to the hosts file:\n```\n$ sudo nano /etc/hosts\n\nThen add the following lines:\n# Set host for Hadoop cluster\n192.168.1.8 hadoop-master\n192.168.1.6 hadoop-slave1\n192.168.1.5 hadoop-slave2\n```\n\n### C. References\n```\nhttp://odewahn.github.io/docker-jumpstart/building-images-with-dockerfiles.html\n\nhttps://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html\nhttps://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraycad%2Fhadoop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraycad%2Fhadoop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraycad%2Fhadoop/lists"}