{"id":22911880,"url":"https://github.com/valtri/demo-hadoop","last_synced_at":"2025-04-01T10:47:34.061Z","repository":{"id":69640207,"uuid":"88770720","full_name":"valtri/demo-hadoop","owner":"valtri","description":"Hadoop Demo for EGI Conference 2017","archived":false,"fork":false,"pushed_at":"2017-04-23T21:25:19.000Z","size":20,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-07T06:14:58.023Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/valtri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-19T17:13:08.000Z","updated_at":"2017-04-20T07:12:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"5228ba12-4f02-4a58-85e5-64953a3ad905","html_url":"https://github.com/valtri/demo-hadoop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valtri%2Fdemo-hadoop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valtri%2Fdemo-hadoop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valtri%2Fdemo-hadoop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/valtri%2Fdemo-hadoop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/valtri","download_url":"https://codeload.github.com/valtri/demo-hadoop/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246628418,"owners_count":20808106,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T04:18:45.385Z","updated_at":"2025-04-01T10:47:34.034Z","avatar_url":"https://github.com/valtri.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Demo Hadoop script\n\nThe script will install unsecured Hadoop cluster, which is expected to be run inside private network. Hue web interface has enabled https, so certificates for the public endpoint are required.\n\nEnabled Hadoop parts and add-ons:\n\n* Hadoop HDFS\n* Hadoop YARN\n* Hive\n* Hue (plus required add-ons Oozie, httpfs)\n* Impala\n* Pig\n* Spark\n\nSupported platforms (2017):\n\n* Debian 7/wheezy (except master, [Bug #828836](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=828836), requires pre-installed puppet \u003e= 3)\n* Debian 8/jessie\n* Ubuntu 14/trusty\n* Ubuntu 16/xenial\n\nNetwork requirements (for all nodes):\n\n* all hostnames in DNS or host files\n* when enabling HBase, only one interface may be available, or use NAT PREROUTING rules\n\nCESNET puppet modules are used are used for installation and setup.\n\n# Deployment\n\nReplace *$MASTER\\_HOSTNAME*, *$MASTER\\_IP*, and *$DEVICE* variables by proper values. For example:\n\n    MASTER_HOSTNAME='hadoop-master'\n    MASTER_IP='192.168.0.1'\n    DEVICE='eth0'\n\nDevice is the private interface to run Hadoop cluster.\n\nOptionaly network topology script can be installed as */usr/local/sbin/topology*.\n\nNote, the master has installation in two-steps.\n\n## Master - preparation\n\nStore certificates in */etc/grid-security*:\n\n* *hostcert.pem*\n* *hostkey.pem*\n* *ca-chain.pem*\n\nPreparation:\n\n    ./demo-hadoop.sh $MASTER_HOSTNAME $MASTER_IP $DEVICE master false\n\n## Workers\n\n    ./demo-hadoop.sh $MASTER_HOSTNAME $MASTER_IP $DEVICE\n\n## Master\n\n    ./demo-hadoop.sh $MASTER_HOSTNAME $MASTER_IP $DEVICE master true\n\n# Result\n\n* http://$MASTER\\_HOSTNAME:50070 - HDFS service status page\n* https://$MASTER\\_HOSTNAME:8888 - Hue web interface\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvaltri%2Fdemo-hadoop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvaltri%2Fdemo-hadoop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvaltri%2Fdemo-hadoop/lists"}