{"id":24725611,"url":"https://github.com/markcallen/aws-cld","last_synced_at":"2026-04-10T01:14:19.337Z","repository":{"id":44146487,"uuid":"511771648","full_name":"markcallen/aws-cld","owner":"markcallen","description":"Common setup for a AWS global cluster with kubernetes and databases","archived":false,"fork":false,"pushed_at":"2024-10-11T19:37:10.000Z","size":196,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-27T13:20:46.020Z","etag":null,"topics":["aws","kubernetes","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/markcallen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-08T05:19:49.000Z","updated_at":"2024-10-11T19:36:51.000Z","dependencies_parsed_at":"2023-01-23T08:46:00.280Z","dependency_job_id":"98d707df-a9b0-4b65-82de-fcb5a053bf8a","html_url":"https://github.com/markcallen/aws-cld","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markcallen%2Faws-cld","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markcallen%2Faws-cld/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markcallen%2Faws-cld/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markcallen%2Faws-cld/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/markcallen","download_url":"https://codeload.github.com/markcallen/aws-cld/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244966524,"owners_count":20539797,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","kubernetes","terraform"],"created_at":"2025-01-27T13:20:25.683Z","updated_at":"2025-12-30T23:46:56.586Z","avatar_url":"https://github.com/markcallen.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aws-cld\n\nThis is the 2.x branch.  There are mutliple breaking changes in how modules are managed mainly with infra/iam and env/vpc to solve some of the major\nproblems with using this on small projects (no longer needing VPC peering) and managing IAM user credentials.\n\nCommon setup for building out AWS infrastructure for dev, stage and prod.\nIncludes kubernetes (EKS), databases (Aurora), caches (ElasticCache) and ec2 instances.\n\n## Concepts\n\nBootstrap creates the s3 bucket that all state will be stored.  It will generate the backend.tf and tfvar files.\n\nInfra stores configurations that are for the entire project such as IAM users and roles.\n\nEnv includes configurations for different environments.  By default they are dev, stage and prod.\n - requires using terraform workspaces to keep everything seperate in the state files\n\n## Architecture\n\nCreate diagram\n\n## Setup\n\n### AWS\n\nLog into AWS with the root account and create a new user and attach the AdministratorAccess policy.  Turn on MFA.  Create a secure password.  Create access key and secret.\n - this user will not be used in the iam setup, just for running these scripts.\n\n### Cloudflare\n\nLog into the cloudflare dashboard using the main account and invite a user into the account.  Login with that user and create an API token with Edit:DNS.\n\n### Environment\n\nSetup the following environment variables.  The cloudflare is only required to modify DNS.\n\nAWS user should be an administrator.\n\n```\nexport AWS_ACCESS_KEY_ID=\nexport AWS_SECRET_ACCESS_KEY=\nexport AWS_DEFAULT_REGION=us-east-1\nexport CLOUDFLARE_EMAIL=\nexport CLOUDFLARE_API_TOKEN=\n```\n\nInstall [keybase](https://keybase.io/download)\n\nSetup a pgp key, and have other users setup their key as well so you can distribute IAM passwords and AWS credentials securly.\n\nInstall tfenv\n\n```\nbrew install tfenv\n```\n\nInstall terraform 1.9.4\n\n```\ntfenv install 1.9.4\n```\n\nUse terraform 1.9.4\n\n```\ntfenv use 1.9.4\nterraform --version\n\nTerraform v1.9.4\n```\n\nUse [tflint](https://github.com/terraform-linters/tflint)\n\n```\nbrew install tflint\n```\n\nSetup alias for terraform\n```\nalias tf=terraform\n```\n\n## Migration from 1.x to 2.x\n\nNeed to use terraform version 1.9.4 or newer. This can require deleting the .terraform.lock.hcl file.\n\n### Infra\n\nModules\n\n| module | 1.x | 2.x | notes |\n| ------ | --- | --- | ----- |\n| iam | :heavy_check_mark:  | :heavy_check_mark: | requires using a moved blocks for module.iam.aws_iam_user.user, module.iam.aws_iam_user_login_profile.login_profile, module.iam.aws_iam_user_policy_attachment.access_keys_attach |\n\n\n## Configure\n\ncreate bootstrap, infra and env directories\n\n```\nmkdir bootstrap infra env\n```\n\n### Bootstrap\n\n```\ncd bootstrap\n```\n\nCall the bootstrap module to create the storage s3 bucket\n\nbootstrap/main.tf\n\n```\nterraform {\n  required_providers {\n    aws = {\n      source  = \"hashicorp/aws\"\n      version = \"~\u003e 4.0\"\n    }\n  }\n  required_version = \"\u003e= 0.13\"\n}\n\nmodule \"bootstrap\" {\n  source = \"github.com/markcallen/aws-cld//bootstrap/\"\n  project = \"myproject\"\n}\n\noutput \"project\" {\n  value = module.bootstrap.project_name\n}\n```\n\nif you don't supply a project variable in the bootstrap module then a project name will be created for you\n\n```\nterraform init\nterraform plan\nterraform apply\n```\n\nto get the name of the project\n\n```\nterraform output\n```\n\nmake sure you encrypt the terraform.tfstate file and add it to git\n\n\n### Infra\n\n```\ncd infra\n```\n\nif using iam\n```\ngpg --export your.email@address.com | base64 \u003e public-key.gpg\n```\n\ninfra/main.tf\n```\nvariable \"project\" {\n}\n\nmodule \"iam\" {\n  source = \"github.com/markcallen/aws-cld//infra/iam\"\n\n  project             = var.project\n  public_key_filename = \"./public-key.gpg\"\n  iam_users           = [\"test1_eng\", \"test1_ops\"]\n  eng_users           = [\"test1_eng\"]\n  ops_users           = [\"test1_ops\"]\n}\n\nmodule \"ecr\" {\n  source = \"github.com/markcallen/aws-cld//infra/ecr\"\n\n  project          = var.project\n  ecr_repositories = [\"example1\", \"example2\"]\n}\n```\n\nIf you already have users created in IAM and just want to add them to the project then\nleave the iam_users array empty and include the users you want in this project to\neng_users and ops_users.\n\n\n```\nterraform init\nterraform plan -var-file=default.tfvars\nterraform apply -var-file=default.tfvars\n```\n\n### Env\n\nCreate workspaces for each environment\n\n```\nterraform init\nterraform workspace new dev\nterraform workspace new prod\n```\n\nvpc \u0026\u0026 vpc-peering\n\nAdd to variables.tf\n\n```\nvariable \"cidr\" {\n  type = string\n}\nvariable \"region_us_east\" {\n  type = string\n}\nvariable \"region_us_west\" {\n  type = string\n}\nvariable \"subnet_count\" {\n  type = map(any)\n}\n```\n\nCreate main.tf\n\n```\nmodule \"vpc\" {\n  source = \"github.com/markcallen/aws-cld//env/vpc\"\n\n  project        = var.project\n  environment    = var.environment\n  region_us_east = var.region_us_east\n  region_us_west = var.region_us_west\n  cidr           = var.cidr\n  subnet_count   = var.subnet_count\n}\n\nmodule \"vpc_peering\" {\n  source = \"github.com/markcallen/aws-cld//env/vpc-peering\"\n\n  project        = var.project\n  environment    = var.environment\n  region_us_east = var.region_us_east\n  region_us_west = var.region_us_west\n  vpc_id_us_east = module.vpc.us_east_vpc.vpc_id\n  vpc_id_us_west = module.vpc.us_west_vpc.vpc_id\n}\n```\n\nadd to dev.tfvars\n```\ncidr = {\n  us_east = \"10.111.0.0/16\"\n  us_west = \"10.112.0.0/16\"\n}\nregion_us_east = \"us-east-1\"\nregion_us_west = \"us-west-1\"\nsubnet_count   = 2\n```\n\nNeed to target the module.vpc before setting up the vpc-peering\n\nMake sure you are in the correct workspace\n\n```\nterraform workspace select dev\nterraform init\nterraform plan -var-file=dev.tfvars -target module.vpc\nterraform apply -var-file=dev.tfvars -target module.vpc\n```\n\n```\nterraform plan -var-file=dev.tfvars\nterraform apply -var-file=dev.tfvars\n```\n\nRoute53\n\nAdding a set of environment domains\n\nadd to variables.tf\n\n```\nvariable \"domain\" {\n  type = string\n}\nvariable \"environment_dns_name\" {\n  type = string\n}\n```\n\nadd to main.tf\n\n```\nmodule \"route53\" {\n  source = \"github.com/markcallen/aws-cld//env/route53\"\n\n  project             = var.project\n  environment         = var.environment\n  environment_name    = var.environment_dns_name\n  domain              = var.domain\n}\n```\n\nadd to dev.tfvars\n\n```\ndomain = \"mydomain.com\"\nenvironment_dns_name = \"d\"\n```\n\nAdding environment domains to a route53 root domain\n\n```\nmodule \"route53ns\" {\n  source = \"github.com/markcallen/aws-cld//env/route53-ns\"\n\n  project          = var.project\n  environment      = var.environment\n  environment_name = var.environment_dns_name\n  domain           = var.domain\n  name_servers     = module.route53.environment.name_servers\n\n  depends_on = [\n    module.route53.environment\n  ]\n}\n```\n\nACM\n\n```\nmodule \"acm\" {\n  source = \"github.com/markcallen/aws-cld//env/acm\"\n\n  project          = var.project\n  environment      = var.environment\n  environment_name = var.environment_dns_name\n  domain           = var.domain\n  region_us_east   = var.region_us_east\n  region_us_west   = var.region_us_west\n\n\n  subject_alternative_names = [\n    \"*.${var.environment_dns_name}.${var.domain}\",\n  ]\n}\n\n```\n\nsubject_alternative_names can included specific domain names if a wildcard is needed\n\nEKS\n\nAdd to variables.tf\n\n```\nvariable \"app_desired_count\" {\n}\nvariable \"app_max_count\" {\n}\nvariable \"app_min_count\" {\n}\nvariable \"enable_us_east\" {\n}\nvariable \"enable_us_west\" {\n}\n\n```\n\nAdd to main.tf\n\n```\nmodule \"eks\" {\n  source = \"github.com/markcallen/aws-cld//env/eks\"\n\n  project           = var.project\n  environment       = var.environment\n  app_desired_count = var.app_desired_count\n  app_min_count     = var.app_min_count\n  app_max_count     = var.app_max_count\n\n  enable_us_east = var.enable_us_east\n  enable_us_west = var.enable_us_west\n}\n\n```\n\nAdd to dev.tfvars\n```\napp_desired_count = 1\napp_max_count     = 5\napp_min_count     = 1\nenable_us_east    = 1\nenable_us_west    = 0\n```\n\nAdd to outputs.tf\n\n```\noutput \"cluster_id\" {\n  value = module.eks.cluster_id\n}\n```\n\nEnable us_east and us_west clusters using enable_us_east or enable_us_west to a number greater than 0.\n - TODO: this is not working yet\n\nConfigure kubectl\n\nGet the cluster name\n\n```\nterraform output cluster_id\n```\n\n```\naws eks update-kubeconfig --region \u003cregion\u003e --name \u003ccluster_id\u003e\n```\n\nEC2\n\n```\nmodule \"ec2\" {\n  source = \"github.com/markcallen/aws-cld//env/ec2\"\n\n  project        = var.project\n  environment    = var.environment\n  region_us_east = var.region_us_east\n  region_us_west = var.region_us_west\n\n  route53_zone_id_us_east = module.route53.us_east.zone_id\n  route53_zone_id_us_west = module.route53.us_west.zone_id\n\n  ssh_keys = [\"ssh-rsa AA... my@key\"]\n\n  instance_count_us_east = 1\n  instance_count_us_west = 0\n\n  extra_disk_size  = 100\n  extra_disk_count = 1\n\n  architecture = \"x86_64\"\n}\n\n```\n\nCloudflare\n\nAdding environment domains to a cloudflare root domain\n\n```\nmodule \"cloudflarens\" {\n  source = \"github.com/markcallen/aws-cld//env/cloudflare-ns\"\n\n  project      = var.project\n  environment  = var.environment\n  name_servers = module.route53.environment.name_servers\n  zone_id      = var.zone_id\n\n  depends_on = [\n    module.route53.environment\n  ]\n}\n```\n\n\nAnsible\n\nTo create an inventory file for ansible add the following:\n\n```\nresource \"local_file\" \"inventory\" {\n  content         = format(\"%s\", module.ec2.inventory)\n  file_permission = \"0644\"\n  filename        = \"${path.module}/ansible/${var.environment}\"\n}\n```\n\nansible/requirements.yaml\n```\n- name: geerlingguy.docker\n- name: deekayen.awscli2\n```\n\nansible/docker.yaml\n```\n- hosts: all\n  vars:\n    docker_install_compose: true\n    docker_users:\n      - ubuntu\n  become: true\n  roles:\n    - geerlingguy.docker\n    - deekayen.awscli2\n```\n\nansible/data.yaml\n```\n- hosts: all\n  become: yes\n  become_method: sudo\n  roles:\n    - role: aws-volumes\n      vars:\n        ebs_volumes:\n          data:\n            device_name: /dev/xvdh\n            mount_point: \"/data\"\n            fs: ext4\n            mount_opts: noatime,nodiratime\n            owner: ubuntu\n            group: ubuntu\n```\n\nrun\n\n```\ncd ansible\nansible-galaxy role install -r requirements.yaml\nansible-playbook -i dev -u ubuntu docker.yaml\nansible-playbook -i dev -u ubuntu data.yaml\n```\n\n## Develop\n\nInstall terraform v1.1.7 using tfenv\n\n```\ntfenv install v1.1.7\ntfenv use v1.1.7\n```\nInstall other tools for devops tasks.  See [pre-commit-terraform](https://github.com/antonbabenko/pre-com\nmit-terraform?search=1)\n\n```\nbrew install pre-commit terraform-docs tflint tfsec checkov terrascan infracost tfupdate minamijoyo/hcled\nit/hcledit jq\n```\n\nInstall pre-commit\n```\npre-commit install\n```\n\nRun pre-commit manually\n\n```\npre-commit run --all-files\n```\n\n## Pull Request\n\nTo update the version of this module add the patch, minor or major tags to the PR.\n\n## License\n\nDistributed under the Apache-2.0 License. See `LICENSE` for more information.\n\n## Contact\n\nYour Name - [@markcallen](https://www.linkedin.com/in/markcallen/)\n\nProject Link: [https://github.com/markcallen/aws-cld](https://github.com/markcallen/aws-cld)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkcallen%2Faws-cld","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkcallen%2Faws-cld","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkcallen%2Faws-cld/lists"}