{"id":23402583,"url":"https://github.com/martincastroalvarez/hadoop-hdfs-map-reduce-docker","last_synced_at":"2025-04-08T22:22:50.623Z","repository":{"id":218297558,"uuid":"622400598","full_name":"MartinCastroAlvarez/hadoop-hdfs-map-reduce-docker","owner":"MartinCastroAlvarez","description":"Running Map Reduce in Hadoop using Docker","archived":false,"fork":false,"pushed_at":"2024-01-21T02:32:24.000Z","size":3613,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-14T17:31:07.404Z","etag":null,"topics":["big-data","bigdata","hadoop","hdfs","map-reduce"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MartinCastroAlvarez.png","metadata":{"files":{"readme":"README.md","changelog":"history.png","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-04-02T01:52:31.000Z","updated_at":"2024-05-06T19:09:31.000Z","dependencies_parsed_at":"2024-01-21T03:25:07.686Z","dependency_job_id":"0dfa2b21-bef6-41c6-974f-ff50c4816d84","html_url":"https://github.com/MartinCastroAlvarez/hadoop-hdfs-map-reduce-docker","commit_stats":null,"previous_names":["martincastroalvarez/hadoop-hdfs-map-reduce-docker"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinCastroAlvarez%2Fhadoop-hdfs-map-reduce-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinCastroAlvarez%2Fhadoop-hdfs-map-reduce-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinCastroAlvarez%2Fhadoop-hdfs-map-reduce-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MartinCastroAlvarez%2Fhadoop-hdfs-map-reduce-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MartinCastroAlvarez","download_url":"https://codeload.github.com/MartinCastroAlvarez/hadoop-hdfs-map-reduce-docker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247935649,"owners_count":21020862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","bigdata","hadoop","hdfs","map-reduce"],"created_at":"2024-12-22T12:29:35.061Z","updated_at":"2025-04-08T22:22:50.606Z","avatar_url":"https://github.com/MartinCastroAlvarez.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hadoop Map Reduce Docker\nRunning Map Reduce jobs using Docker\n\n![img](./wallpaper.jpg)\n\n## Overview\n\n#### HDFS\n\nHDFS, or Hadoop Distributed File System, is a distributed file system designed to store and\nprocess large datasets using commodity hardware. It is part of the Apache Hadoop ecosystem\nand is widely used in big data processing. HDFS uses a master-slave architecture with one\nNameNode and multiple DataNodes. The NameNode manages the file system metadata, while the\nDataNodes store the actual data. This allows for scalable and fault-tolerant data storage\nand processing. HDFS is optimized for batch processing and sequential reads, making it\nwell-suited for applications like log analysis, data warehousing, and machine learning.\nHowever, it is not well suited for random writes and low-latency data access. HDFS is a\ncritical component of the Hadoop ecosystem and is used by many big data applications.\nIts scalable and fault-tolerant design makes it a reliable choice for storing and\nprocessing large datasets. Overall, HDFS plays a crucial role in the world of big\ndata and is an essential tool for data engineers and analysts.\n\n![hadoop.png](hadoop.png)\n\n#### Map Reduce\n\nIn the context of Hadoop, MapReduce is a programming model and framework for processing\nlarge datasets in a distributed computing environment. Hadoop is an open-source software\nframework that is widely used for distributed storage and processing of big data.\n\nHadoop MapReduce is the processing component of the Hadoop framework, and it provides a\nway to process large datasets in parallel across a cluster of nodes. The MapReduce\nframework is based on the MapReduce programming model, which divides the data\nprocessing into two main stages: the \"map\" stage and the \"reduce\" stage.\n\nIn Hadoop MapReduce, the map stage reads the input data from Hadoop Distributed File\nSystem (HDFS) and processes it in parallel across multiple nodes using mapper functions.\nThe output of the map stage is then shuffled and sorted before being sent to the reduce stage.\n\nIn the reduce stage, the output of the map stage is processed further using reducer functions,\nwhich perform operations such as aggregation, summarization, and filtering. The output of the\n reduce stage is then written to the HDFS.\n\nThe Hadoop MapReduce framework provides a scalable and fault-tolerant way to process large\ndatasets in a distributed computing environment. It can be used for a wide range of data\nprocessing tasks, including data mining, machine learning, and analytics.\n\nHadoop MapReduce is one of the core components of the Hadoop ecosystem, and it is widely\nused in industries such as finance, healthcare, and e-commerce for processing and analyzing\nbig data.\n\n#### Hadoop Streaming\n\nHadoop streaming is a utility that comes with the Hadoop distribution.\nThe utility allows you to create and run Map/Reduce jobs with any executable or script as\nthe mapper and/or the reducer. For example:\n\n## Software Architecture\n\n|File|Purpose|\n|---|---|\n|[docker-compose.yml](docker-compose.yml)|Docker compose with the infrastructure required to run the Hadoop cluster.|\n|[requirements.txt](tests/requirements.txt)|Python requirements file.|\n|[app/test_hdfs.py](tests/test_hdfs.py)|Python script that tests writing data into HDFS.|\n|[app/mapper.sh](tests/mapper.sh)|Mapper job.|\n|[app/reducer.sh](tests/reducer.sh)|Reducer job.|\n\n## References\n\n- [Docker Hadoop](https://github.com/big-data-europe/docker-hadoop)\n- [HDFS Simple Docker Installation Guide for Data Science Workflow](https://towardsdatascience.com/hdfs-simple-docker-installation-guide-for-data-science-workflow-b3ca764fc94b)\n- [Set Up Containerize and Test a Single Hadoop Cluster using Docker and Docker compose](https://www.section.io/engineering-education/set-up-containerize-and-test-a-single-hadoop-cluster-using-docker-and-docker-compose/)=\n- [Spark Docker](https://github.com/big-data-europe/docker-spark)\n- [Hadoop Namenode](https://hub.docker.com/r/bde2020/hadoop-namenode)\n- [Apache ZooKeeper](https://zookeeper.apache.org/)\n- [Word Counter using Map Reduce on Hadoop](https://medium.com/analytics-vidhya/word-count-using-mapreduce-on-hadoop-6eaefe127502)\n- [Hadoop Streaming](https://hadoop.apache.org/docs/r1.2.1/streaming.html)\n\n## Instructions\n\n#### Starting the Hadoop ecosystem\n```bash\ndocker rm -f $(docker ps -a -q)\ndocker volume rm $(docker volume ls -q)\ndocker-compose up\n```\n\n#### Validating the status of the Hadoop cluster\n```bash\ndocker ps\n```\n```bash\nCONTAINER ID        IMAGE                                                    COMMAND                  CREATED             STATUS                    PORTS                                            NAMES\n0f87a832960b        bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8   \"/entrypoint.sh /r...\"   12 hours ago        Up 54 seconds             0.0.0.0:8088-\u003e8088/tcp                           yarn\n51da2508f5b8        bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8     \"/entrypoint.sh /r...\"   12 hours ago        Up 55 seconds (healthy)   0.0.0.0:8188-\u003e8188/tcp                           historyserver\nec544695c49a        bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8       \"/entrypoint.sh /r...\"   12 hours ago        Up 56 seconds (healthy)   0.0.0.0:8042-\u003e8042/tcp                           nodemanager\n810f87434b2f        bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8          \"/entrypoint.sh /r...\"   12 hours ago        Up 56 seconds (healthy)   0.0.0.0:9864-\u003e9864/tcp                           datenode1\nca5186635150        bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8          \"/entrypoint.sh /r...\"   12 hours ago        Up 56 seconds (healthy)   0.0.0.0:9000-\u003e9000/tcp, 0.0.0.0:9870-\u003e9870/tcp   namenode\nbeed8502828c        bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8          \"/entrypoint.sh /r...\"   12 hours ago        Up 55 seconds (healthy)   0.0.0.0:9865-\u003e9864/tcp                           datenode2\n[...]\n```\n\n#### Testing HDFS using raw HTTP requests.\nThe `-L` flag allows redirections. By default, the namenode redirects the request to any of the datanodes.\n````bash\ndocker exec -it namenode /bin/bash\ncurl -L -i -X PUT \"http://127.0.0.1:9870/webhdfs/v1/data/martin/lorem-ipsum.txt?op=CREATE\" -d 'testing'\n````\n````bash\nHTTP/1.1 307 Temporary Redirect\nDate: Thu, 30 Mar 2023 00:40:44 GMT\nCache-Control: no-cache\nExpires: Thu, 30 Mar 2023 00:40:44 GMT\nDate: Thu, 30 Mar 2023 00:40:44 GMT\nPragma: no-cache\nX-Content-Type-Options: nosniff\nX-FRAME-OPTIONS: SAMEORIGIN\nX-XSS-Protection: 1; mode=block\nLocation: http://datanode2.martincastroalvarez.com:9864/webhdfs/v1/data/martin/lorem-ipsum.txt?op=CREATE\u0026namenoderpcaddress=namenode:9000\u0026createflag=\u0026createparent=true\u0026overwrite=false\nContent-Type: application/octet-stream\nContent-Length: 0\n\nHTTP/1.1 100 Continue\n\nHTTP/1.1 201 Created\nLocation: hdfs://namenode:9000/data/martin/lorem-ipsum.txt\nContent-Length: 0\nAccess-Control-Allow-Origin: *\nConnection: close\n````\n\n#### Listing the content of the root directory\n```bash\ndocker exec -it namenode /bin/bash\nhdfs dfs -ls /\n```\n```bash\nFound 1 items\ndrwxr-xr-x   - root supergroup          0 2023-03-03 14:15 /rmstate\n```\n\n#### Creating a new directory in HDFS\n```bash\ndocker exec -it namenode /bin/bash\nhdfs dfs -mkdir -p /user/root\nhdfs dfs -ls /\n```\n```bash\nFound 2 items\ndrwxr-xr-x   - root supergroup          0 2023-03-03 14:15 /rmstate\ndrwxr-xr-x   - root supergroup          0 2023-03-03 14:17 /user\n```\n\n#### Adding a file to HDFS\n```bash\ndocker exec -it namenode /bin/bash\necho \"lorem\" \u003e /tmp/hadoop.txt \nhdfs dfs -put ./input/* input\nhdfs dfs -ls /user/\n```\n```bash\nFound 2 items\n-rw-r--r--   3 root supergroup          6 2023-03-03 14:20 /user/hadoop.txt\ndrwxr-xr-x   - root supergroup          0 2023-03-03 14:17 /user/root\n```\n\n#### Printing the content of a file in HDFS\n```bash\ndocker exec -it namenode /bin/bash\nhdfs dfs -cat /user/hadoop.txt \n```\n```bash\nlorem\n```\n\n#### Checking the status of the NameNode at [http://127.0.0.1:9870/dfshealth.html](http://127.0.0.1:9870/dfshealth.html)\n\n![status1.png](status1.png)\n![status2.png](status2.png)\n\n#### Testing HDFS using Python\n\n```python3\nvirtualenv -p python3 .env\nsource .env/bin/activate\npip install -r requirements.txt\npython3 app/test_hdfs.py\n```\n```bash\n[...]\nWritten: 684 files 336846 words 1852059 chars\n```\n\n#### Test the mapper job\n```bash\necho \"asdf asdf asdkfjh 99asdf asdf\" | app/mapper.sh\n```\n```bash\nasdf\t1\nasdf\t1\nasdkfjh\t1\n99asdf\t1\nasdf\t1\n```\n\n#### Test the reducer job\n```bash\necho \"asdf 1 1 2 3\" | app/reducer.sh \n```\n```bash\nasdf\t7\n```\n\n#### Uploading the mapper and reducer jobs to the Docker container.\n```bash\ndocker cp ./app/mapper.sh namenode:/tmp/mapper.sh\ndocker cp ./app/reducer.sh namenode:/tmp/reducer.sh\n```\n\n#### Entering into the namenode\n```bash\ndocker exec -it namenode /bin/bash\n```\n\n#### Creating the directory for the Rap Reduce job.\n```bash\nhadoop fs -mkdir /jobs\n```\n\n#### Uploading the map and reduce job to HDFS.\n```bash\nhadoop fs -rm /jobs/mapper.sh\nhadoop fs -rm /jobs/reducer.sh\nhadoop fs -put /tmp/mapper.sh /jobs/mapper.sh\nhadoop fs -put /tmp/reducer.sh /jobs/reducer.sh\nhadoop fs -chmod 555 /jobs/mapper.sh\nhadoop fs -chmod 555 /jobs/reducer.sh\nhadoop fs -ls /jobs/\n```\n\n#### Running a Map Reduce job\n```bash\nhadoop fs -rmdir /jobs/output\nhadoop jar \\\n    /opt/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-streaming-3.2.1.jar \\\n    -files \"/tmp/mapper.sh,/tmp/reducer.sh\" \\\n    -input /user/martin/*.txt \\\n    -output /jobs/output \\\n    -mapper \"mapper.sh\" \\\n    -reducer \"reducer.sh\"\n```\n```bash\n[...]\n2023-03-31 06:52:25,803 INFO mapred.FileInputFormat: Total input files to process : 1164\n[...]\n2023-03-31 07:41:45,021 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1680237794579_0008\n[...]\n2023-03-31 07:41:45,627 INFO mapreduce.Job: Running job: job_1680237794579_0008\n[...]\n\tShuffle Errors\n\t\tBAD_ID=0\n\t\tCONNECTION=0\n\t\tIO_ERROR=0\n\t\tWRONG_LENGTH=0\n\t\tWRONG_MAP=0\n\t\tWRONG_REDUCE=0\n\tFile Input Format Counters \n\t\tBytes Read=3776682\n\tFile Output Format Counters \n\t\tBytes Written=4940135\n2023-03-31 10:36:45,036 INFO streaming.StreamJob: Output directory: /jobs/output\n```\n\n#### Troubleshooting in case of failure\n```bash\nhadoop fs -ls /app-logs/root/logs-tfile/\nhadoop fs -cat $(hadoop fs -ls /app-logs/root/logs-tfile/ | tail -1 | awk '{print $8}')/*\n```\n```bash\nFound 7 items\ndrwxrwx---   - root root          0 2023-03-31 06:53 /app-logs/root/logs-tfile/application_1680237794579_0002\ndrwxrwx---   - root root          0 2023-03-31 07:39 /app-logs/root/logs-tfile/application_1680237794579_0003\ndrwxrwx---   - root root          0 2023-03-31 07:42 /app-logs/root/logs-tfile/application_1680237794579_0004\ndrwxrwx---   - root root          0 2023-03-31 07:46 /app-logs/root/logs-tfile/application_1680237794579_0005\ndrwxrwx---   - root root          0 2023-03-31 07:49 /app-logs/root/logs-tfile/application_1680237794579_0006\ndrwxrwx---   - root root          0 2023-03-31 07:51 /app-logs/root/logs-tfile/application_1680237794579_0007\ndrwxrwx---   - root root          0 2023-03-31 07:52 /app-logs/root/logs-tfile/application_1680237794579_0008\n[...]\n```\n\n#### Open the History Server web interface at[http://127.0.0.1:8188/applicationhistory](http://127.0.0.1:8188/applicationhistory)\n\n![history.png](history.png)\n![success.png](success.png)\n\n#### Verifying the results of the map reduce job\n```bash\nhadoop fs -ls /jobs/output/\n```\n```bash\nFound 2 items\n-rw-r--r--   3 root supergroup          0 2023-03-31 10:36 /jobs/output/_SUCCESS\n-rw-r--r--   3 root supergroup    4940135 2023-03-31 10:36 /jobs/output/part-00000\n```\n\n#### Printing the output of the Map Reduce job\n```bash\nhadoop fs -cat /jobs/output/part-00000\n```\n```bash\n[...]\n1DXpWP\t    12\n1DYnnz\t    44\n1DZLd3\t    3\n1DZnOPEuK\t20\n1DZzz\t    4\n1Da\t        10\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartincastroalvarez%2Fhadoop-hdfs-map-reduce-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmartincastroalvarez%2Fhadoop-hdfs-map-reduce-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartincastroalvarez%2Fhadoop-hdfs-map-reduce-docker/lists"}