{"id":13936212,"url":"https://github.com/InsightDataScience/aws-ops-insight","last_synced_at":"2025-07-19T21:32:02.968Z","repository":{"id":77923443,"uuid":"129343772","full_name":"InsightDataScience/aws-ops-insight","owner":"InsightDataScience","description":"Collection of Terraform and Ansible scripts for easy AWS operations","archived":false,"fork":false,"pushed_at":"2020-01-10T15:14:29.000Z","size":209,"stargazers_count":14,"open_issues_count":2,"forks_count":14,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-06-03T21:05:01.126Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/InsightDataScience.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-13T03:41:06.000Z","updated_at":"2021-12-29T21:46:47.000Z","dependencies_parsed_at":"2023-06-10T05:45:16.775Z","dependency_job_id":null,"html_url":"https://github.com/InsightDataScience/aws-ops-insight","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/InsightDataScience/aws-ops-insight","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InsightDataScience%2Faws-ops-insight","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InsightDataScience%2Faws-ops-insight/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InsightDataScience%2Faws-ops-insight/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InsightDataScience%2Faws-ops-insight/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/InsightDataScience","download_url":"https://codeload.github.com/InsightDataScience/aws-ops-insight/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InsightDataScience%2Faws-ops-insight/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266019657,"owners_count":23864916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T23:02:28.570Z","updated_at":"2025-07-19T21:31:57.961Z","avatar_url":"https://github.com/InsightDataScience.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# AWS Ops for Insight\nCollection of Terraform (0.12.x) and Ansible scripts for easy AWS operations.\n\n## Clone this repo into the home directory to your local machine\n\nThis repo is pre-installed with Terraform and Ansible, and is designed to allow you to spin up the necessary infrastructure with minimal setup. However, you still need to change to the home directory (abbreviated `~` in bash) and download the latest scripts from this repo:\n    \n    cd ~\n    git clone https://github.com/InsightDataScience/aws-ops-insight.git\n\n## AWS credentials for your IAM user (not the root account)\nYour Linux user has a `.profile` file in your home directory where you can configure your local machine. The AWS credentials for your IAM user should be placed in the `.profile` using your editor of choice (e.g. with a command like `nano ~/.profile` or `vim ~/.profile`). Of course, your credentials will be different, but you should have something like this in your `.profile`:\n\n    export AWS_ACCESS_KEY_ID=ABCDE1F2G3HIJKLMNOP  \n    export AWS_SECRET_ACCESS_KEY=1abc2d34e/f5ghJKlmnopqSr678stUV/WXYZa12\n\n**WARNING: DO NOT COMMIT YOUR AWS CREDENTIALS TO GITHUB!** AWS and bots are constantly searching Github for these credentials, and you will either have your account hacked, or your credentials revoked by AWS.\n\nWhenever you change your `.profile`, don't forget to source it with the command (note the `.` at the beginning of the command):\n\n    . ~/.profile\n    \n# Setting up your AWS Environment\n\nWe'll start by using Terraform to \"provision\" resources on your AWS account. Terraform is an industry-standard open source technology for provisioning hardware, whether on any popular cloud providers (e.g. AWS, GCP, Azure), or in-house data centers. Terraform is written in Go, and is designed to quickly and easily create and destroy infrastructure of hundreds of resources in parallel. \n\nTerraform also has a great community of open source modules available in the [Terraform Registry](https://registry.terraform.io/). We'll be using several of the pre-built AWS modules now.\n\n## Setting up your Virtual Private Cloud (VPC)\nIn past sessions, someone gets hacked and Bitcoin miners go crazy burning through AWS resources. To ensure that simple mistakes don’t cost you tremendously, you'll set up guardrails with a network that can only contain a fixed number of nodes. If you need more instances later, we can help you expand your network. \n\nAWS uses software-defined network to offer a small network that is secure from others called a Virtual Private Cloud (VPC). We'll use Terraform to set up a simple and small \"sandbox VPC\" where you can build your infrastructure safely.\n\nMove into the `terraform` directory of the repo you just cloned:\n\n    cd aws-ops-insight/terraform\n\nEdit the variables.tf file. Navigate to the variable \"aws_region\", input applicable AWS region.\n\nvariable \"aws_region\" {\n  description = \"AWS region to launch servers. For NY,BOS,VA use us-east-1. For SF use us-west-2\"\n  default     = \"Insert AWS Region here\"\n\nNavigate to the variable \"amis\", remove the region amazon machine image (AMI) that isn't applicable.\n\nvariable \"amis\" {\n type = map (string)\n  default = {\n    \"us-east-1\" = \"ami-0b6b1f8f449568786\"\n    \"us-west-2\" = \"ami-02c8040256f30fb45\"\n\nSave and exit the variables.tf file.\n    \nThen initialize Terraform and apply the configuration we've set up in the `.tf` files within that directory:\n\n    terraform init\n    terraform apply\n    \nTerraform will ask your name (enter whatever you want) and IAM keypair (enter the exact name of your key, **without the `.pem` extension**). Then it will show you it's plan to create, modify, or destroy resources to get to the correct configuration you specified. After saying `yes` to the prompt, and waiting a few moments (the NAT gateways can take a minute or two), you should see a successful message like this: \n\n    Apply complete! Resources: 18 added, 0 changed, 0 destroyed.\n\n    Outputs:\n\n    cluster_size = 4\n\nTerraform is designed to be idempotent, so you can always run the `terraform apply` command multiple times, without any issues. It's also smart about only changing what it absolutely needs to change. For example if you ran the apply command again, but use a different name, it will only rename a few resources rather than tearing down everything and spinning up more.\n\nIf all went well, you have the following resources added:\n\n- VPC sandbox with all the necessary networking\n- Security Group with inbound SSH connectivity from your local machine.\n- 4 node cluster, with 1 \"master\" and 3 \"workers\"\n\n### Destroying your infrastructure\n\n**Don't** destroy your infra now, but if you ever want to tear down your infrastructure, you can always do that with the command:\n\n    terraform destroy\n\nIf you want to destroy a subset of your infrastructure, simply change your `main.tf` file accordingly and re-run the `tf apply` to apply these changes. Terraform tracks the current state in the `terraform.tfstate` file (or a remote backend like S3 or Consul), but if you manually delete resources on the console, this can mess up your Terraform state. As a result, **we don't recommend mixing a Terraform and manual setup** - try to do everything within Terraform if possible.\n\n### Setting Terraform Variables\nYou probably don't want to enter your name everytime you run the apply command (and you can't automate that), so let's set the variable. You could set the variable from the command line with something like:\n\n    terraform apply -var 'fellow_name=Insert Name Here'\n    \nor by setting an environment variable that starts with `TF_VAR_` like:\n\n    export TF_VAR_fellow_name=david\n    \n(but don't forget to source your `.profile` again). Note that Terraform treats your AWS credentials specially - they don't need the `TF_VAR` prefix to be detected.\n\nFinally, you can set variables in the file `terraform.tfvars`. Go into the file and uncomment the lines with variables (but use your name and key pair, of course):\n\n    # name of the Fellow (swap for your name)\n    fellow_name=\"Insert Name Here\"\n\n    # name of your key pair (already created in AWS)\n    keypair_name=\"Insert Name Here-IAM-keypair\"\n\nFor security, **YOU SHOULD NEVER PUT YOUR AWS CREDENTIALS IN A FILE THAT COULD BE COMMITTED TO GITHUB**, so you can use environment variables for your credentials. For other types of variables, you should use the method that's most convenient (e.g. files are easy to share with others, but the command line could be easier when prototyping).\n\n# Configuring Technologies with Ansible\n\n## Setup Ansible to connect with AWS\nWith all our resources provisioned, we'll now use the \"configuration management\" tool, Ansible, to actually install and start technologies like Hadoop, Spark, etc. Ansible is also open source and popular at many startups for it's ease of use, though many larger companies use alternative tools like Puppet and Chef. \n\nWe'll start configuring machines with the scripts in the `ansible` directory. Change to it with:\n\n    cd ../ansible\n\nIn order for your control machine to SSH into your nodes via Ansible, it will need your PEM key stored on it. You can add it with a `scp` command like the following (**which should be ran from your local machine, not the control machine**):\n\n    scp -i ~/.ssh/control.pem ~/.ssh/david-IAM-keypair.pem david-d@bos.insightdata.com:~\n    \nor if you've set up your `.ssh/config` file as described above, it would be something like (but with **your name and key**):\n\n    scp ~/.ssh/david-IAM-keypair.pem dd-control:~\n\n**Back on the control machine**, copy your keypair to the `.ssh` folder on your control machine, with a command similar to:\n\n    sudo mv ~/david-IAM-keypair.pem ~/.ssh\n\nNext, you'll need to configure the `ansible.cfg` file to reflect your key pair name and location on the control machine. This sets global configuration This also assumes that this repo is cloned into your home directory. The relevant lines of the `ansible.cfg` are:\n\n    [defaults]\n    host_key_checking = False\n    private_key_file = /home/david-d/.ssh/david-IAM-keypair.pem\n    ansible_user = david-d\n    log_path = ~/ansible.log\n    roles_path = ~/aws-ops-insight/ansible/roles\n    ....\n\nNext, install the AWS Software Development Kit (SDK) for Python, which is called `boto`. Ansible is written in Python, so this is how Ansible connects with AWS. Unfortunately, Ansible is in the middle of migrating from Python 2 to Python 3, so you'll need both the `boto` and `boto3` libraries to use all the modules.\n\n    pip install boto boto3\n    \nNext, add the following additonal environment variables to your control machine's `.profile` for Ansible:\n\n    export AWS_REGION=us-west-2\n    export EC2_INI_PATH=~/aws-ops-insight/ansible/ec2.ini\n    export ANSIBLE_INVENTORY=~/aws-ops-insight/ansible/ec2.py\n\nThe first sets the region to Oregon (**you should set your region to `us-east-1` if you're on the East coast**). The other  two lines are necessary to initialize and use Ansible's \"Dynamic Inventory\" feature. Ansible keeps an \"inventory\" of the host machines you're installing technologies onto, and the `ec2.py` script collects this information dynamically. This is how Ansible knows about the instances that Terraform just launched.\n\nThe `ec2.ini` is simply some initializations that we're using. For example, it ignores regions and services that most Fellows don't use (e.g. the Beijing data cente `cn-north-1`, or the AWS DNS service, Route53).\n\nTo use this, make sure that the `ec2.py` script has permission to execute (e.g. `+x`) with the `chmod` command:\n\n    chmod +x ec2.py\n\nFinally, you can test this by running the Ansible playbook `get-cluster-list.yml`:\n\n    ansible-playbook get-cluster-list.yml\n    \nwhich should display the \"facts\" for any instances in AWS with the tag `Cluster` set to `hadoop` **(if you don't have this tag set by Terraform, go back and add it to `main.tf`, apply it now, and re-run the playbook)**. There should be a lot of information displayed with a summary at the end like:\n\n    PLAY RECAP ****************************************************************\n    localhost                  : ok=3    changed=0    unreachable=0    failed=0\n\n  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FInsightDataScience%2Faws-ops-insight","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FInsightDataScience%2Faws-ops-insight","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FInsightDataScience%2Faws-ops-insight/lists"}