{"id":18597026,"url":"https://github.com/focusaurus/clouddial","last_synced_at":"2025-05-16T13:33:10.288Z","repository":{"id":141309068,"uuid":"2229007","full_name":"focusaurus/clouddial","owner":"focusaurus","description":"Exercise in cloud server provisioning and management (ReportGrid challenge)","archived":false,"fork":false,"pushed_at":"2015-10-07T21:29:26.000Z","size":151,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T18:02:54.673Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/focusaurus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-08-18T16:48:35.000Z","updated_at":"2015-10-07T21:29:27.000Z","dependencies_parsed_at":"2023-03-13T11:37:48.277Z","dependency_job_id":null,"html_url":"https://github.com/focusaurus/clouddial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/focusaurus%2Fclouddial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/focusaurus%2Fclouddial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/focusaurus%2Fclouddial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/focusaurus%2Fclouddial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/focusaurus","download_url":"https://codeload.github.com/focusaurus/clouddial/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254538476,"owners_count":22087869,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T01:26:42.348Z","updated_at":"2025-05-16T13:33:10.268Z","avatar_url":"https://github.com/focusaurus.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"#Cloud Dial Server Provisioning Project\n(ReportGrid interview challenge problem)\n\n##Challenge Problem Text with Commentary\n\nHere is the original challenge specification with my comments.\n\n\u003e Here is your challenge problem:\n\u003e Your challenge consists of building a script for very simple EC2 server\n\u003e management. You are welcome to use the command-line tools, \n\u003e third-party EC2 libraries, or the EC2 service APIs directly to complete this\n\u003e task.\n\n* I used 3rd party stuff:\n    * The knife-ec2 plugin\n    * Opscode Chef\n    * knife-ec2 is built on top of the Fog cloud management library\n\n\u003e Here's what you script needs to do:\n\u003e Given a number n, provision that number of EC2 micro Linux instances. It\n\u003e should be possible to provide a simple configuration file (preferably in\n\u003e something like JSON, YAML, or an internal Ruby/Python/whatever DSL) that \n\u003e specifies any additional packages that need to be installed as part of the\n\u003e configuration process. These packages can be either standard packages\n\u003e for the distro (we prefer you focus on dpkg and apt - our servers are\n\u003e Ubuntu-based, but handling RPM-based distros would be good too) or .tar.gz \n\u003e zipped binaries following the standard Unix layout conventions \n\u003e (/bin, /lib, /src, etc...). If a binary package of this sort is specified,\n\u003e the script should be able to download and install the package from an Amazon\n\u003e S3 bucket.\n\n* run.sh takes the number of instances as the first argument\n* The configuration file in JSON format at data-bags/clouddial_conf.json\n* Note that the configuration file MUST be up to date in Chef, so use `./task.rb\n  upload` to upload it to Chef before running run.sh\n* I tested only ubuntu packages, but I believe the code in the `default.rb`\n  recipe will also work unmodified for RPMs on RPM-based systems\n* S3 bucket needs to be publically accessible for now.  Adding the\n  authentication token should\n  be straightforward but I didn't tackle that at this time\n\n\u003e Once each server is fully configured, run a process (which should also be\n\u003e specified in the configuration file) on each system and collect the output.\n\u003e The script should be able to handle a relatively long-running process that will\n\u003e need to be polled for termination, although for the purpose of this exercise it\n\u003e can be something as simple as using curl to retrieve some information from a\n\u003e remote service - search Google for ReportGrid or something.\n\n* I used curl to download the ReportGrid home page.  Chef handles the polling\n  for us automatically.\n* Output is created in a directory structure under `results` with a subdirectory\n  for each target machine named with the machine's FQDN.  Any file(s) put into\n`/tmp/rg_results` on the target instance will be transfered to the local results\ndirectory for that instance.\n\n\u003e Once the process is complete on all servers (though you can do this\n\u003e incrementally) collect the output from all the processes and return it to the\n\u003e client. Once all the output is retrieved, terminate the instances.\n\n* Both the Chef nodes and the EC2 instances are deleted at the end. For\n  troubleshooting, this can be disabled by adding a `return 0` as the top line\nof the `deleteInstance` function in `run.sh`\n\n\u003e Please publish your solution to a public git repository on GitHub and send us\n\u003e the link. Take as long as you need, but please do keep track of how long it\n\u003e takes you to write the solution and let us know when you've completed it - the\n\u003e faster the better! You are welcome to use your language(s) of choice, and don't\n\u003e feel restricted to use just a single language; if parts need to be in shell,\n\u003e parts in Ruby, parts in Python, etc so be it.\n\n\u003c/div\u003e\n\n* I worked on this over about 4 total sessions between Thursday and Sunday for a\n  total of about 17 hours\n* It's in ruby and shell\n\n\n\n# Prerequisites\n* A Unixy client OS (built on OS X. Probably works fine on Linux)\n* Bash\n* OpenSSH\n* [Opscode Chef](http://wiki.opscode.com/display/chef/Support)\n    * rvm (recommended but not strictly required)\n    * ruby 1.9\n    * Chef 0.9 or newer\n* knife-ec2 gem\n\n#Setting up your environment\n* Make sure you have ssh installed (everyone does...)\n* [Install RVM](http://beginrescueend.com/rvm/install/)\n* Use rvm to install ruby and activate it\n    rvm install ruby-1.9.2\n    rvm use ruby-1.9.2\n* Install chef and knife-ec2\n    gem install chef knife-ec2 --no-rdoc --no-ri\n* Your AWS configuration must have a firewall rule allowing SSH\n  connections. [Here is an article with detailed explanations](http://www.agileweboperations.com/amazon-ec2-instances-with-opscode-chef-using-knife)\n\n#Third Party Accounts Required\n* Set up an account with the Opscode platform and follow their\n  quickstart guide. [http://www.opscode.com/chef/]()\n* Amazon Web Services (AWS) Elastic Compute Cloud (EC2)\n\n#User Guide\n\n##run.sh\n\n`run.sh` is the main entry point.  It takes 2 optional positional\narguments.\n\n* COUNT = how many servers to create. Defaults to 1.\n* PREFIX = Prefix to use when naming the nodes in Chef. Will be followed\n  by a simple integer serial number. (Web-1, Web-2, Web-3, etc).\nDefaults to \"rgtest\".\n\nAll servers will build and process in parallel.  However, this means the\noutput from the build processes are all intermingled (as of now), so\nthey are hard to read.  The script can be easily modified to run in\nserial instead as needed (See the Troubleshooting section below).\n\n##task.rb\n\n`task.rb` is a little utility script not unlike a Makefile or Rakefile.  It just\nautomates most things you might have to type on the command line while working\nin this project. Run it with `./task.rb COMMAND` where COMMAND is one of the\nfollowing.\n\n* real_creds\n    * We don't want to store real credentials for S3 in git, so I use this to\n      quickly enable a different knife.rb file that has the real info\n* sample_creds\n    * opposite of real_creds\n* upload\n    * Builds the chef cookbook and uploads it to chef\n    * Uploads the clouddial_conf.json file as a chef data bag\n    * This command MUST be run at least once before run.sh will work\n* clean\n    * remove the `results` directory\n* list\n    * list all EC2 instances and chef nodes\n* delete\n   * Delete a chef node or EC2 instance by name\n   * If the name passed on the command line starts with \"i-\", it is assumed to\n     be an EC2 instance ID as opposed to a chef node name\n\n#Troubleshooting\n\nThe server jobs can trivially be switched from parallel execution to\nserial execution by commenting out the \u0026 sign from the createInstances\nfunction in run.sh (It is clearly marked with a comment to this effect).\n\nYou may want to disable deletion of the EC2 instances and/or the chef nodes for\ntroubleshooting failed installed.  This can be done in run.sh\n\n#Known Limitations\n* Error handling! There is slim to none thus far.  If your JSON config\n  file is imperfect in any way, stack traces are coming your way. Most failures\nare not yet properly detected and handled.\n* The spec asked for EC2 Micro instances. Fog doesn't support that\n  (yet).  The code we'd have to fix is\n[here](https://github.com/geemus/fog/blob/b1655ab847c2a822c35198c496affed30f78a71a/lib/fog/aws/models/ec2/flavors.rb)\nWe created `m1.small` instances instead.\n* Not everything is cleanly separated out into a config file yet.\n  Specifically the region \"us-west-1\" is still hard coded.  If you don't\nspecify the right region, most EC2 operations return nothing, which can\nbe confusing.\n\n#Open Issues\n* If you create enough instances to end up with a re-using of the same EC2 FQDN, ssh\n  can complain about incorrect host key.  Delete stale entries from your\n`~/.ssh/known_hosts` for EC2 FQDNs to work around this.\n\n#Resolved Issues\n* knife bootstrap process is not highly reliable.  Sometimes fails.\n    * It hangs at \"INFO: Bootstrapping Chef on\" and my local workstation\n      CPU starts to thrash\n    * [Here's a\n      link](http://help.opscode.com/discussions/problems/233-ec2-instances-sometimes-do-not-bootstrap-with-knife-due-to-authentication-failure)\nto others having this problem\n    * Applying that monkey patch to knife.rb DOES seem to have fixed the\n      issue\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffocusaurus%2Fclouddial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffocusaurus%2Fclouddial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffocusaurus%2Fclouddial/lists"}