{"id":16703282,"url":"https://github.com/phstudy/vagrant-hadoop-etu","last_synced_at":"2025-04-10T04:58:19.304Z","repository":{"id":16230291,"uuid":"18977777","full_name":"phstudy/vagrant-hadoop-etu","owner":"phstudy","description":null,"archived":false,"fork":false,"pushed_at":"2014-05-17T02:54:38.000Z","size":128088,"stargazers_count":6,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T06:11:46.828Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://hackpad.com/-Hadoop-VkP678YX21n","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phstudy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-21T01:08:14.000Z","updated_at":"2018-03-14T21:53:33.000Z","dependencies_parsed_at":"2022-09-24T11:41:02.507Z","dependency_job_id":null,"html_url":"https://github.com/phstudy/vagrant-hadoop-etu","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phstudy%2Fvagrant-hadoop-etu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phstudy%2Fvagrant-hadoop-etu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phstudy%2Fvagrant-hadoop-etu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phstudy%2Fvagrant-hadoop-etu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phstudy","download_url":"https://codeload.github.com/phstudy/vagrant-hadoop-etu/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248161264,"owners_count":21057554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T19:07:41.193Z","updated_at":"2025-04-10T04:58:19.286Z","avatar_url":"https://github.com/phstudy.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vagrant-hadoop\n\nsome vagrant examples for different hadoop cluster deployment model\n\n## Requirement\n\n * Internet Access\n * OS: tested with [Ubuntu](http://www.ubuntu.com) Desktop 12.04 LTS 64 bit -- [Download ISO](http://www.ubuntu.com/start-download?distro=desktop\u0026bits=64\u0026release=lts)\n * [git](http://git-scm.com/) : tested with 1.7.9.5-1\n * [VirtualBox](http://www.virtualbox.org) : tested with 4.3.10\n * [Vagrant](http://www.vagrantup.com) : tested with 1.4.3+\n * [vagrant-aws plugin](https://github.com/mitchellh/vagrant-aws) : tested with 0.4.1\n * Amazon EC2 API Tools - [ec2-api-tools](http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/command-reference.html) : tested with 1.5.0.0-0ubuntu1  *(Optional)*\n\n-----\n### GNU/Linux - Ubuntu 12.04\nThese scripts are tested on Ubuntu Desktop 12.04 LTS **64 bit** GNU/Linux.\nWhy **64 bit**? Because [Apache BigTop](http://bigtop.apache.org) only provides 64 bit deb/rpm packages.\nBesides, to run 64 bit VirtualBox images, you also need a 64 bit host OS.\n```\n### How to check your OS version\n~$ uname -a\nLinux EA-dev 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux\n~$ lsb_release -a\nNo LSB modules are available.\nDistributor ID:\tUbuntu\nDescription:  Ubuntu 12.04.4 LTS\nRelease:  12.04\nCodename: precise\n```\n### git\nTo checkout the github repository, you need to install `git`.\n```\n### Install git on Ubuntu 12.04\n~$ sudo apt-get -y install git\n### How to check your git version\n~$ git --version\ngit version 1.7.9.5\n```\n### VirtualBox\nSome scripts are tested on local virtualbox VMs, you need to install virtualbox before running these scripts. \nDue to PXE boot issue of some NICs, we suggest to run official virtualbox instead of ubuntu package (4.1.12_Ubuntur77245).\nPlease install official [virtualbox rpm/deb/exe](https://www.virtualbox.org/wiki/Downloads) for your environment.\n```\n### Install VirtualBox on Ubuntu 12.04\n~$ echo \"deb http://download.virtualbox.org/virtualbox/debian precise contrib\" \u003e virtualbox.list\n~$ sudo mv virtualbox.list /etc/apt/sources.list.d/\n~$ wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -\n~$ sudo apt-get update; sudo apt-get -y install virtualbox-4.3\n### How to check your virtualbox version\n~$ vboxmanage --version\n4.3.10r93012\n```\n### Vagrant\nThese scripts are based on [Vagrant 2 syntex](http://docs.vagrantup.com/v2/vagrantfile/version.html), so you can not use the default ubuntu package (1.0.1). \nPlease install [official vagrant rpm/deb/exe](http://www.vagrantup.com/downloads.html) for your environment.\n```\n### Install official vagrant package on Ubuntu\n~$ wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.5.2_x86_64.deb\n~$ sudo dpkg -i vagrant_1.5.2_x86_64.deb \n### How to check your vagrant version\n~$ vagrant --version\nVagrant 1.5.2\n```\n### Amazon EC2 Access Key\nTo run virtual machines on Amazon EC2, you need to [register AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html). We will use [Amazon EC2 API Tools](http://aws.amazon.com/developertools/351) to simplify some web operations. Please reference Ubuntu community Wiki [\"EC2 Starter's Guide\" ](https://help.ubuntu.com/community/EC2StartersGuide) to register account, intall ec2-api-tools and configure environment variables.\n```\n### Install ec2-api-tools on Ubuntu\n~$ sudo apt-get -y install ec2-api-tools\n### How to check your ec2-api-tools\n~$ dpkg -l ec2-api-tools \nDesired=Unknown/Install/Remove/Purge/Hold\n| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend\n|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)\n||/ Name                           Version                        Description\n+++-==============================-==============================-============================================================================\nii  ec2-api-tools                  1.5.0.0-0ubuntu1               Amazon EC2 API tools\n```\nset up your Amazon API credentials. Go to [Account -\u003e Security Credentials](https://console.aws.amazon.com/iam/home?#security_credential)\n- click \"Access Keys (Access Key ID and Secret Access Key)\" tab\n- Create a new Certificate\n- Download the root key and save it in ~/.ec2/rootkey.csv\n- Make your credential files private: chmod go-rwx ~/.ec2/*.csv \n```\n~$ mkdir -p ~/.ec2\n### download Access Key Certificates and private key into ~/.ec2\n~$ chmod go-rwx ~/.ec2/*.csv\n### generate a shell script for furthur \n~$ source ~/.ec2/rootkey.csv\n~$ cat \u003e ~/.ec2/ec2_keys \u003c\u003c EOF\nexport EC2_URL=https://ec2.ap-southeast-1.amazonaws.com\nexport AWS_ACCESS_KEY=$AWSAccessKeyId\nexport AWS_SECRET_KEY=$AWSSecretKey\nexport AWS_KEYPAIR_NAME=vagrant\nexport AWS_KEYPAIR_PATH=~/.ec2/vagrant.pem\nexport AWS_SEC_GROUPS=default\nEOF\n~$ source ~/.ec2/ec2_keys\n### generate keypair named by 'vagrant' and store the private key in ~/.ec2/vagrant.pem\n~$ ec2-create-keypair vagrant \u003e ~/.ec2/vagrant.pem\n```\n### \"vagrant-aws\" plugin\nTo run some example vagrant script for AWS EC2, please install vagrant-aws plugin.\n```\n### Install vagrant-aws plugin\n~$ vagrant plugin install vagrant-aws\n### How to check installed plugins\n~$ vagrant plugin list\nvagrant-aws (0.4.1)\nvagrant-login (1.0.1, system)\nvagrant-share (1.0.1, system)\n### Add 'dummy' box for 'vagrant-aws' plugin\n~$ vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box\n```\n-----\n## Get the source code\nYou can check out the source code with following commands\n```\n~$ git clone https://github.com/jazzwang/vagrant-hadoop\n```\n## Example 1 : run bigtop example on local virtualbox\n\nThe first example is to run BigTop single node Hadoop on AWS EC2.\n```\n~$ cd vagrant-hadoop/bigtop-aws/ubuntu\n~/vagrant-hadoop/bigtop-aws/ubuntu$ vagrant up bigtop1\n```\nYou could login local virtualbox instance with `vagrant ssh`\n```\n~/vagrant-hadoop/bigtop-aws/ubuntu$ vagrant ssh bigtop1\n```\n## Example 2 : run bigtop example on remote EC2\n```\n~$ cd vagrant-hadoop/bigtop-aws/ubuntu\n~/vagrant-hadoop/bigtop-aws/ubuntu$ source ~/.ec2/ec2_keys\n~/vagrant-hadoop/bigtop-aws/ubuntu$ vagrant up --provider=aws\n```\nYou could get the public ip of your EC2 instances with `ec2-describe-instance-status` command.\n```\n~/vagrant-hadoop/bigtop-aws/ubuntu$ ec2-describe-instance-status\n```\nYou could login remote EC2 instance with `vagrant ssh`\n```\n~/vagrant-hadoop/bigtop-aws/ubuntu$ vagrant ssh bigtop2\n```\n**Note:**\nPlease shutdown your EC2 instance after testing with command `vagrant halt`, or it will take USD 0.077 in Singapore region.\n```\n~/vagrant-hadoop/bigtop-aws/ubuntu$ vagrant halt bigtop2\n```\nTo complete terminate your EC2 instance, you could use the command `vagrant destroy -f`.\n```\n~/vagrant-hadoop/bigtop-aws/ubuntu$ vagrant destroy -f bigtop2\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphstudy%2Fvagrant-hadoop-etu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphstudy%2Fvagrant-hadoop-etu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphstudy%2Fvagrant-hadoop-etu/lists"}