{"id":15336691,"url":"https://github.com/jixuan1989/init_ec2","last_synced_at":"2025-04-15T03:12:03.401Z","repository":{"id":109784693,"uuid":"52188746","full_name":"jixuan1989/init_ec2","owner":"jixuan1989","description":"init EC2 cluster, for free-password-login(ubuntu and root). for hostname, for hosts file.  ","archived":false,"fork":false,"pushed_at":"2018-12-12T07:02:47.000Z","size":71,"stargazers_count":11,"open_issues_count":3,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-04-15T03:11:44.017Z","etag":null,"topics":["cluster","deployment"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jixuan1989.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-21T03:25:24.000Z","updated_at":"2023-10-15T17:23:38.000Z","dependencies_parsed_at":"2023-05-25T07:15:46.137Z","dependency_job_id":null,"html_url":"https://github.com/jixuan1989/init_ec2","commit_stats":{"total_commits":101,"total_committers":3,"mean_commits":"33.666666666666664","dds":0.04950495049504955,"last_synced_commit":"1dedf57607e337a04f6452fafc39cd9651e16fa4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jixuan1989%2Finit_ec2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jixuan1989%2Finit_ec2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jixuan1989%2Finit_ec2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jixuan1989%2Finit_ec2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jixuan1989","download_url":"https://codeload.github.com/jixuan1989/init_ec2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248997079,"owners_count":21195799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cluster","deployment"],"created_at":"2024-10-01T10:17:02.372Z","updated_at":"2025-04-15T03:12:03.393Z","avatar_url":"https://github.com/jixuan1989.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# init_ec2\nThis project helps users to initialize a new cluster.\n\nFor example, you can watch a use case vedio on Youtube: https://youtu.be/OvVaib5xllo\n\nIt is suitable for:\n \n * a real cluster, which consists of PCs, small servers or other  physical machines you own.\n * cloud environment, such as Amazon EC2, aliyun. \n\nThe only conditions you should have are:\n\n- you have installed ubuntu (tested on 14.04) OS on all the nodes. (it is always eligibly in cloud environments, e.g., EC2, aliyun)\n- you  know or have set the ip address on each node. \n- **python 2.7** has been installed on your laptop/PC. (unless you use a Windows OS, otherwise you are eligibly. on Windows, you need to install Python yourself)\n- **fabric** (version \u003e=1.10, but not compatible with 2.0 version) has been installed on your laptop/PC\n\t- it seems a little harder than above steps. But believe me, you deserve it! \n\t- on Mac, install easy_install and then fabric:\n\t\t- download file https://pypi.python.org/pypi/ez_setup \n\t\t- sudo python ez_setup.py\n\t\t- sudo easy_install \"fabric == 1.10.5\"\n\t\t\n\t- on Linux (Ubuntu), the same way (notice, do not use pip to install fabric, you will be disappointed; besides, apt-get install fabric is ok, but the version is 1.8 now (2016.4))\n\t\t- sudo apt-get install python-setuptools python-dev build-essential\n\t\t- sudo easy_install \"six==1.6.0\" # I am not sure that whether it is necessary,\n\t\t- sudo easy_install \"fabric == 1.10.5\"\n\t\t- notice that, if there is something wrong, google it: e.g., lack of some .h file, just use apt-get install\n\t- on Windows,... Good luck, guy. I have not tried how to install fabric on Windows. But it should not be a probelm.  \ninit EC2 cluster, for free-password-login(ubuntu and root). for hostname, for hosts file.  \n\n##how to use\n1. modify passwd.ini\n \n    this is an example\n\t\t\n\t\t[default]\n\t\t#which section is actived\n\t\tactiveSession=\n\t\t#the sudo user, only used for sudo tasks\n\t\troot =\n\t\t#passwd of sudoer\n\t\tpasswd =\n\t\t\n\t\t#example of section\n\t\t[server]\n\t\t#uesr name, if use createUser task, this user name will be the new user\n\t\tnewuser =\n\t\t#password of newuser\n\t\tpasswd =\n\t\t#cluster ips, use comma to split\n\t\thosts=\n\t\t#hostnames of cluster , the number of hostnames must equal with hosts. the hostnames is used for changeHostname tass\n\t\thostnames=\n\t\t#locally jdk file. the related path is ./\n\t\tjdk_source_file=\n\t\t# when the jdk.tar.gz is unziped, the folder in the tar file.\n\t\tjdk_folder=\n\t\t# the ntp server address you  want to sync\n\t\tntp_server=\n\t\t# the allowed net that can sync with your ntp server\n\t\tntp_net=\n\t\t# the allowed net mask that can sync with your ntp server\n\t\tntp_net_mask=\n\t\t\n\tthis is an instance:\n\t\n\t\t[default]\n\t\tactiveSession=server\n\t\troot = fit\n\t\tpasswd =111\n\t\t\n\t\t[server]\n\t\tnewuser = hxd\n\t\tpasswd = 1112\n\t\thosts=192.168.130.3, 192.168.130.5\t\thostnames=s3, s5, \n\t\tjdk_source_file=files/jdk1.8.77.tar.gz\n\t\tjdk_folder=jdk1.8.0_77\n\t\tntp_server=192.168.130.2\n\t\t[client]\n\t\tnewuser = hxd\n\t\tpasswd = 1112\n\t\thosts = 192.168.130.2\n\t\thostnames = s2\n\t\tjdk_source_file=files/jdk1.8.77.tar.gz\n\t\tjdk_folder=jdk1.8.0_77\n\t\tntp_net=192.168.130.0\n\t\tntp_net_mask=255.255.255.0\n\t\tntp_server=192.168.130.2\n\t\t[all]\n\t\tnewuser = hxd\n\t\tpasswd = 1112\n\t\thosts=192.168.130.2,192.168.130.3, 192.168.130.5\n\t\thostnames=s2,s3, s5\n\t\tjdk_source_file=files/jdk1.8.77.tar.gz\n\t\tjdk_folder=jdk1.8.0_77\n2. you can use this script like this \n   \n   `fab task_name` \n   \n   you can use `fab help` to show how many tasks supportted.\n   \n   current supportted tasks:\n   \n    * createUser\n        - create a new user (username is 'newuser' in the configurations, password is 'passwd' in the active configurations).\n    * removeUser\n    \t- delete the user we created\n    * addIntoHostFile\n    \t- add all the \u003cip,hostname\u003e into /etc/hosts\n    * changeHostname\n    \t- change all the hostnames as the values in the actived conf.\n    * installnptserver\n    \t- install ntp server on nodes. We suggest you only install ntp server on one node in the cluster, instead of all the nodes.(how to do: enable a configruation which has only one ip in 'hosts')\n    * distributeJDK\n    \t- after you manual download a jdk file, you can use this task to push it to all the nodes and modify the $JAVA_HOME and $PATH \n    * setnpttaks\n    \t- add a scheduled task every day for ntpdate time with the ntp server\n    * password-free ssh\n    \t- setting password-free ssh. how to use: \n    \t\t- fab ssh1  (this can generate pem files)\n    \t\t- fab ssh2  (this finish the password-free)\n    \t\t- fab ssh3  (cleanup)\n    * restart\n    \t- reboot the cluster\n    * correct_bashrc\n    \t- normally, we use `ssh hxd@node1 \"jps\"`, the terminal says: jps command not found, though we have set $JAVA_HOME and $PATH in .bashrc file. The reason is that .bashrc is ignored when we run a non-interactive login command. To solve that, use this task to repair the bashrc file.\n    * modifyAtpSource\n    \t- modify the apt sources.list as files/sources.list.\n    * addDisk\n    \t- when you add a new disk, you should format it and then mount it. this task help you. use `fab addDisk:device,location`. e.g., `fab addDisk:/dev/sdb,/datab`\n    * TODO: we will add new tasks to help user add new nodes in an existed cluster    \t\n\n3. some bigdata system and monitor system. you can use it like `fab -f fab_bigdata taskname`\n\n    current supported task:\n\n    * installCollectd\n        - you need add a section in passwd.ini\n\n            [collectd]\n            \\#server ip tell the node that where to send the collected data\n            \\#the receiver should have a receive process. e.g. influxdb and others\n            \\#the most easy receiver is collected itself. In this way, you should modify the receiver's collectd configuration file yourself: /etc/collectd/collectd.conf\n            \\#in network plugin: replace \"Server\" by \"Listen\"\n            server_ip=\n            server_port=\n            \\#collect interval unit: second\n            interval=5\n    * runCollectd\n        - `fab -f fab_bigdata runCollectd:stop` or `fab -f fab_bigdata runCollectd:start`\n    * installGangliaClient\n        - ganglia is a master-slave architecture. now you can only install the client, ganglia-monitor. The master processes, gmetad  and ganglia-webfrontend, are not included.\n        - if you want to install the master processes, see https://www.digitalocean.com/community/tutorials/introduction-to-ganglia-on-ubuntu-14-04.\n        - to install the client process, you need a gmond.conf in files folder. Notice, modify the ip address in the file as your master ip, and modify the cluster name as the same as your master configuration.\n\n    * installCouch\n        - you can install CouchDB.\n        - normally, couchdb is a single-node. therefore, your active section.hosts should have only one ip. But if you exactly want to install couchdb on all the nodes, use `fab -f fab_bigdata installCouch:y`\n        - in current version, we do not modify the bind-address of CouchDB, the default is  localhost.\n    * runCouch\n        - you can start or stop CouchDB\n        - how to use: `fab -f fab_bigdata runCouch:'start'` or `fab -f fab_bigdata runCouch:'stop'` or `fab -f fab_bigdata runCouch:'start','y'` or  `fab -f fab_bigdata runCouch:'stop','y'`. 'y' means you have  couchdb on each node\n\n\n##Amazon Ec2 (or other pem based ssh)\nin Amazon Ec2, we use a pem file to login instead of password.\n\nTo use the script on Amazon EC2. you need modify the script now. But do not worry, just a little thing you need to do:\n\n* comment \"env.password=...\" in rootUser() \n* add env.key_filename=['filepath'] in rootUser()\n* Cheers! \n\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjixuan1989%2Finit_ec2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjixuan1989%2Finit_ec2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjixuan1989%2Finit_ec2/lists"}