{"id":21075437,"url":"https://github.com/shirshadatta/hadoop-cheatsheet","last_synced_at":"2025-03-14T03:41:34.831Z","repository":{"id":112513515,"uuid":"302682317","full_name":"ShirshaDatta/Hadoop-CheatSheet","owner":"ShirshaDatta","description":"Your go-to-cheatsheet to learn apache-Hadoop.","archived":false,"fork":false,"pushed_at":"2021-01-25T20:23:46.000Z","size":3203,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-20T22:51:27.506Z","etag":null,"topics":["dfs","hadoop","hadoop-cheatcheet","hdfs-client","hdfs-cluster","jdk-","masternode","multitier-architecture","redhat-enterprise-linux","slave-nodes"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShirshaDatta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-09T15:31:09.000Z","updated_at":"2021-01-25T20:23:55.000Z","dependencies_parsed_at":"2023-05-15T12:00:24.436Z","dependency_job_id":null,"html_url":"https://github.com/ShirshaDatta/Hadoop-CheatSheet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShirshaDatta%2FHadoop-CheatSheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShirshaDatta%2FHadoop-CheatSheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShirshaDatta%2FHadoop-CheatSheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShirshaDatta%2FHadoop-CheatSheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShirshaDatta","download_url":"https://codeload.github.com/ShirshaDatta/Hadoop-CheatSheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243521185,"owners_count":20304183,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dfs","hadoop","hadoop-cheatcheet","hdfs-client","hdfs-cluster","jdk-","masternode","multitier-architecture","redhat-enterprise-linux","slave-nodes"],"created_at":"2024-11-19T19:22:52.084Z","updated_at":"2025-03-14T03:41:34.801Z","avatar_url":"https://github.com/ShirshaDatta.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hadoop-CheatSheet 🐘\n\nA cheatsheet to get you started with Hadoop\n\nBut the question is why should we learn Hadoop? How will it make our life easier?\n\nRead till the end to know more. \n\nHappy learning 👩‍🎓\n\n\n## Index Of Contents\n1. [Introduction](#Introduction)\n2. [Installation](#Installation)\n3. [Configuration](#Configuration)\u003cbr\u003e\n    i) [NameNode](#NameNode)\u003cbr\u003e\n    ii) [DataNode](#DataNode)\u003cbr\u003e\n    iii) [ClientNode](#ClientNode)\n4. [GUI](#GUI)\n5. [Frequently Asked Questions](#FAQs)\n6. [Testing](#Testing)\n7. [Contributing](#Contributions)\u003cbr\u003e\n    i)[Contribution Practices](#Contribution-Practices)\u003cbr\u003e\n    ii)[Pull Request Process](#Pull-Request-Process)\u003cbr\u003e\n    iii)[Branch Policy](#Branch-Policy)\n8. [Cool Links to Check out](#Cool-Links-To-Checkout)\n7. [License](#License)\n8. [Contact](#Contact)\n\u003c!--9. [Download Free PDF](#Download-free-PDF)--\u003e\n\n## Introduction\n\nSimple answer to the the above question is to store data. Again the question, when there is Database as well as Drive storage why should we use Hadoop?\n\n\u003cb\u003eTO STORE BIG DATA\u003c/b\u003e\u003cbr\u003e\n\nNow the question, What is Big Data?\nAn example of big data might be petabytes (1,024 terabytes) or exabytes (1,024 petabytes) of data consisting of billions to trillions of records of millions of people—all from different sources (e.g. Web, sales, customer contact center, social media, mobile data and so on).\n\nTo store so much of data we use the concept of \u003cb\u003eDISTRIBUTED STORAGE CLUSTER\u003c/b\u003e. To implement these concepts we use Apache Hadoop.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://miro.medium.com/max/1050/1*H4_yv5YskknPaJ4lWJpzXA.png\"\u003e\n\u003c/p\u003e\n\n## Installation \n(For 1 master and multi slave and multi client nodes)\n**For Master,Slave and Client Nodes** \n```\nThis is for RedHat\n    - Install Java JDK as Hadoop depends on it\n        wget https://www.oracle.com/webapps/redirect/signon?nexturl=https://download.oracle.com/otn/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.rpm\n        rpm -i -v -h jdk-8u171-linux-x64.rpm\n    - Install apache hadoop\n        wget https://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1-1.x86_64.rpm\n        rpm -i -v -h hadoop-1.2.1-1.x86_64.rpm --force\n    - Verify if it is correctly installed with\n        java -version\n        hadoop version\n\n```\n\n![Preview Image](./assets/installing.PNG)\n\n\n## Configuration\n## NameNode\n(NameNode is also called Master Node)\n```\n    mkdir /nn\n    vim /etc/hadoop/core-site.xml\n        \u003cconfiguration\u003e\n            \u003cproperty\u003e\n                \u003cname\u003efs.default.name\u003c/name\u003e\n                \u003cvalue\u003ehdfs://MasterIP:PortNo\u003c/value\u003e\n            \u003c/property\u003e\n        \u003c/configuration\u003e\n\n    vim /etc/hadoop/hdfs-site.xml\n        \u003cconfiguration\u003e\n            \u003cproperty\u003e\n                \u003cname\u003edfs.name.dir\u003c/name\u003e\n                \u003cvalue\u003e/nn\u003c/value\u003e\n            \u003c/property\u003e\n        \u003c/configuration\u003e\n\n```\n\nThe configured files:\n\u003cimg src=\"assets/master files.PNG\" alt=\"Logo\"\u003e\n#Check if the port number you assigned is free, if not then change the port number in the core-site.xml\n\nThen we will have to format the /nn folder of the namenode.\n``` hadoop namenode -format ```\n\u003cbr\u003e\n\u003cimg src=\"assets/master format.PNG\" alt=\"Logo\"\u003e\n```\n    jps \n    netstat -tnlp \n```\nWe see that the process has not yet started and the assigned port is free \n\n\n\n\u003cimg src=\"assets/master before.png\" alt=\"Logo\"\u003e\n\n\u003cb\u003e Then we will have to start the service:\u003c/b\u003e\n```\nhadoop-daemon.sh start namenode\njps\nnetstat -tnlp\n```\nWe see that the process has started and the port is assigned\n\u003cimg src=\"assets/master successful.PNG\" alt=\"Logo\"\u003e\n\nTo view the no of slave nodes connected\n```hadoop dfsadmin -report``` \n\n\u003cimg src=\"assets/dfsadmin master.PNG\" alt=\"Logo\"\u003e\n\n### DataNode\n(DataNode is also called Slave Node)\n\n```\n    vim /etc/hadoop/core-site.xml\n        \u003cconfiguration\u003e\n            \u003cproperty\u003e\n                \u003cname\u003efs.default.name\u003c/name\u003e\n                \u003cvalue\u003ehdfs://MasterIP:PortNo\u003c/value\u003e\n            \u003c/property\u003e\n        \u003c/configuration\u003e\n    mkdir /dn1\n    vim /etc/hadoop/hdfs-site.xml\n        \u003cconfiguration\u003e\n            \u003cproperty\u003e\n                \u003cname\u003edfs.name.dir\u003c/name\u003e\n                \u003cvalue\u003e/dn1\u003c/value\u003e\n            \u003c/property\u003e\n        \u003c/configuration\u003e\n```\nThe Configured files:\n\u003cimg src=\"assets/slave files.PNG\" alt=\"Logo\"\u003e\n\n\u003cb\u003eThen we will have to start the service\u003c/b\u003e\nMake sure that if you doing the setup locally using VM's , then the firewall should be stopped in the master node.\nTo check so:\n```\n    systemctl status firewalld\n   - If it is active then stop or disable(if you don't want to start after system reboot)\n        systemctl stop firewalld\n        systemctl disable firewalld\n```\n\u003cimg src=\"assets/firewalld master.PNG\" alt=\"Logo\"\u003e\n\n```\nhadoop-daemon.sh start datanode\njps\n```\nWe see that the process has started.\n\u003cimg src=\"assets/slave node done.PNG\" alt=\"Logo\"\u003e\n\nTo view the no of slave nodes connected\n\n```hadoop dfsadmin -report```\n\u003cimg src=\"assets/slave connected\" alt=\"Logo\"\u003e\n\n### ClientNode \n\n```\n    vim /etc/hadoop/core-site.xml\n        \u003cconfiguration\u003e\n            \u003cproperty\u003e\n                \u003cname\u003efs.default.name\u003c/name\u003e\n                \u003cvalue\u003ehdfs://MasterIP:PortNo\u003c/value\u003e\n            \u003c/property\u003e\n        \u003c/configuration\u003e\n\n    - To see how many files we have in their storage\n        hadoop fs -ls /\n    - To add a file\n        cat \u003e /file1.txt\n        Hi I am the first file\n        Ctrl+C\n        hadoop fs - put /file1.txt /\n    - To read the contents of the file\n        hadoop fs -cat /file1.txt\n    - To check the size of the file\n        hadoop fs -count /file1.txt\n    - To create a directory\n        hadoop fs -mkdir /textfiles\n    -To upload a blank file on the fly\n        hadoop fs -touchz /my.txt\n    -To move a file (source➡destination)\n        hadoop fs -mv /lw.txt /textfiles\n    - To copy a file (source➡destination)\n        hadoop fs -cp /file1.txt /textfiles\n    - To remove a file\n        hadoop fs -rm  /file1.txt\n    - To checkout and explore all the available options\n        hadoop fs \n```\nThe attached screenshots of the above mentioned commands are :\n\u003cimg src=\"assets/cmds client1.PNG\" alt=\"Logo\"\u003e\n\u003cimg src=\"assets/cmds client2.PNG\" alt=\"Logo\"\u003e\n\u003cimg src=\"assets/cmds client3.png\" alt=\"Logo\"\u003e\n\n## GUI\nWe can also visualize using GUI\n    Namenode : MasterIP:50070\n    Datanode : SlaveIP:50075\n\u003cimg src=\"assets/gui.PNG\" alt=\"Logo\"\u003e\nWe can visualize the uploaded files\n\u003cimg src=\"assets/gui1.PNG\" alt=\"Logo\"\u003e\n\nWe see that if the file is small it is broken in only 1 block\n\u003cimg src=\"assets/gui filetxt.PNG\" alt=\"Logo\"\u003e\nWe can check the size of the name.txt file like:\n```\n    -To see the permissions as well as the size of the block in bytes\n        ls -l name.txt\n    -To see the permissions as well as the size of the block \n        ls -l -h name.txt\n```\n\u003cimg src=\"assets/size of nametxt.PNG\" alt=\"Logo\"\u003e\nThe default DFS block size is 32768 , and therefore it is divided into blocks before storing. \n\u003cimg src=\"assets/gui nametxt.PNG\" alt=\"Logo\"\u003e\n\n## FAQs\nWill come up soon, stay tuned :)\n\n## Testing \nThese commands are even checked in AWS cloud.\n\n## Contributions\nContributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\n### Contribution Guidelines\n\nWhen contributing to this repository, please first discuss the change you wish to make via issue,\nemail, or any other method with the owners of this repository before making a change. \n\n\n### Contribution Practices\n* Write clear and meaningful commit messages.\n* If you report a bug please provide steps to reproduce the bug.\n* In case of changing the backend routes please submit an updated routes documentation for the same.\n* If there is an UI related change it would be great if you could attach a screenshot \n with the resultant changes so it is easier to review for the maintainers \n\n### Pull Request Process\n1. Ensure any install or build dependencies are removed before the end of the layer when doing a \n   build.\n2. Update the README.md with details of changes to the interface, this includes new environment \n   variables, exposed ports, useful file locations and container parameters.\n3. Only send your pull requests to the development branch where once we reach a stable point \n    it will be merged with the   master branch \n4. Associate each Pull Request with the required issue number \n\n### Branch Policy \n* development: If you are making a contribution make sure to send your Pull Request to this branch . All\n            developments goes in this branch.\n* master: After significant features/bug-fixes are accumulated in development branch we merge it with the master branch.\n\n\n## Cool Links to Checkout \n\n- [How Facebook stores so much data and its statistics](https://shirshadatta2000.medium.com/how-facebook-stores-so-much-data-and-its-statistics-bd0911ad39a1)\n\n- [Facebook and Hadoop](https://www.facebook.com/notes/facebook-engineering/hadoop/16121578919/)\n\n- [How Google stores massive amounts of data](https://medium.com/@avantikadasgupta/how-google-stores-massive-amounts-of-data-bigtable-d67f49bfc40e)\n\n- [Apache Hadoop Ecosystem](https://www.cloudera.com/products/open-source/apache-hadoop.html)\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n\n## Contact\n\n- My Name - Shirsha Datta\n \n- You can contact me at shirshadatta2000@gmail.com\n\n- Connect with me on [LinkedIn](https://www.linkedin.com/in/shirsha-datta-30335a178/)\n\n\u003c!--\n## Download free PDF\n\n\n[Claim your free PDF here]()\n\n\n** \nAdd the line in Contribution : if you are facing in any issue in configuration contact, Open ISSUE\n![before](./assets/master before.png)\nDynamically increase volume\n💯21-10 topics\n💯Add that these are tested in AWS too\n💯Some intro to big data nd apache\n💯heart beat and stuffs\n💯How to change block size\n** Make it more readable\n** safemode and tcpdump\n💯** Who is the one uploading the file?\n✴️ Answer: Client gets the IP from Master and uploads the file to DataNode.\n\n💯**Does client go to master and then read the file on slave via Master or Does Client go to slave directly and read the data?\n✴️ Answer: Client goes to slave directly and reads the data stored on slave.\n--\u003e\n\n\u003c!-- ADD IF NECESSARY💯\n# Contributing\nContributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n--\u003e\n\u003c!-- make changes by referring the drive link of arth --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshirshadatta%2Fhadoop-cheatsheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshirshadatta%2Fhadoop-cheatsheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshirshadatta%2Fhadoop-cheatsheet/lists"}