{"id":13575347,"url":"https://github.com/cogini/multi-env-deploy","last_synced_at":"2025-04-08T09:08:59.371Z","repository":{"id":41094145,"uuid":"233200463","full_name":"cogini/multi-env-deploy","owner":"cogini","description":"Complete example of deploying complex web apps to AWS using Terraform, Ansible, and Packer","archived":false,"fork":false,"pushed_at":"2024-12-18T05:38:29.000Z","size":634,"stargazers_count":370,"open_issues_count":15,"forks_count":121,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-01T08:30:43.379Z","etag":null,"topics":["ansible","aws","aws-codebuild","aws-codedeploy","aws-codepipeline","aws-ec2","aws-ecs","packer","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cogini.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-11T08:30:37.000Z","updated_at":"2025-03-23T22:03:05.000Z","dependencies_parsed_at":"2024-12-28T16:20:53.720Z","dependency_job_id":null,"html_url":"https://github.com/cogini/multi-env-deploy","commit_stats":{"total_commits":242,"total_committers":5,"mean_commits":48.4,"dds":0.03305785123966942,"last_synced_commit":"e4d95407913849b949a4cad8b128e2dc0df6d203"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cogini%2Fmulti-env-deploy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cogini%2Fmulti-env-deploy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cogini%2Fmulti-env-deploy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cogini%2Fmulti-env-deploy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cogini","download_url":"https://codeload.github.com/cogini/multi-env-deploy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247809963,"owners_count":20999816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","aws","aws-codebuild","aws-codedeploy","aws-codepipeline","aws-ec2","aws-ecs","packer","terraform"],"created_at":"2024-08-01T15:01:00.221Z","updated_at":"2025-04-08T09:08:59.350Z","avatar_url":"https://github.com/cogini.png","language":"HCL","funding_links":[],"categories":["HCL"],"sub_categories":[],"readme":"This is an example of how to deploy a real-world complex web app to AWS.\n\nFull-featured apps in languages like Ruby on Rails have multiple components,\ne.g. web front end, background job handler, periodic jobs, maybe a separate\nserver to handle API traffic or web sockets. They use a relational database,\nRedis or Memcached, Elasticsearch, CDN for static assets, SSL, S3 buckets,\nencryption.  They need logging, metrics, and alerting.\n\nThey run in an autoscaling group or ECS and use a CI/CD pipeline to handle\nblue/green deployment. They need to run in multiple environments: dev, staging,\nprod, demo, with slight differences for each. They have some weird things to\nintegrate with partners.\n\nThis framework handles all that :-)\n\nIt's built in in a modular way using Terraform, Ansible and Packer. We have\nused it to deploy multiple complex apps, so it handles many things that\nyou will need, but it's also flexible enough to be tweaked when necessary for\nspecial requirements. It represents months of work.\n\nThe blog post\n[Deploying complex apps to AWS with Terraform, Ansible, and Packer](https://www.cogini.com/blog/deploying-complex-apps-to-aws-with-terraform-ansible-and-packer/)\ngives an example.\n\n# Scenarios\n\nThese modules cover the following scenarios:\n\n## EC2 + RDS\n\n* Virtual private cloud (VPC) with public, private and database subnets\n* App runs in EC2 instance(s) in the public subnet\n* RDS database\n* App data stored in S3\n* Route53 DNS with health checks directs traffic to app instances\n\nThis is good for a simple app, and is also a stepping stone when deploying\nmore complex apps. EC2 instances can be used for development or as a canary.\nSee the [AWS docs](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html)\nfor overview.\n\n## ECS with Fargate\n\nThis runs ECS applications with the same infrastructure as above, but running\nin containers in Fargate. It supports x86 and Arm.\n\nAn example app is here: https://github.com/cogini/phoenix_container_example\n\n## CloudFront for assets\n\nStore app assets like JavaScript and CSS in CloudFront for performance\n\n## CodePipeline for CI/CD\n\nWhenever code changes, pull from git, build in CodeBuild, run tests and deploy\nautomatically using CodeDeploy. Run tests against resources such\nas RDS or Redis. Supports both GitHub and CodeCommit.\n\n## Auto Scaling Group and Load Balancer\n\n* App runs in an ASG in the private VPC subnet\n* Blue/Green deployment\n* SSL using Amazon Certificate Manager\n* Spot instances to reduce cost\n* Multiple deploy targets\n* Manual approval process\n* Notifications\n\n## Containers running in ECS\n\n* App is built in CodePipeline\n* Deployed to ECS using CodeDeploy Blue/Green deployment\n\n## Worker ASG\n\nWorker runs background tasks in an ASG, with its own build and deploy pipeline.\n\n## Multiple front end apps\n\nRoute traffic between web apps using the Load Balancer, e.g. separate\nservers for API, customer admin, back end admin.\n\n## S3 buckets\n\n* Share data between apps using S3 buckets with access control\n* Use signed URLs to handle protected user content\n\n## Static website\n\nBuild the public website using a static site generator in CodeBuild, deploying\nto CloudFront CDN. Use Lambda@Edge to rewrite URLs.\n\n## Elasticache\n\nAdd Elasticache Redis or Memcached for app caching.\n\n## Elasticsearch\n\nAdd Elasticsearch for the app.\n\n## DevOps\n\nAdd a DevOps instance to handle deployment and management tasks.\n\n## Bastion host\n\nAdd Bastion host to control access to servers in the private subnet.\nOr use with AWS SSM Sessions.\n\n## Prometheus metrics\n\nAdd Prometheus for application metrics and monitoring\n\n## SES\n\nUse SES for email.\n\n# How it works\n\nIt uses [Terraform](https://www.terraform.io/) to create the infrastructure,\n[Ansible](https://www.ansible.com/) and [Packer](https://www.packer.io/) to set\nup instances and AMIs. It uses AWS CodePipeline/CodeBuild/CodeDeploy to build\nand deploy code, running the app components in one or more autoscaling groups\nrunning EC2 instances.\n\nThe base of the system is Terraform and [Terragrunt](https://github.com/gruntwork-io/terragrunt).\nCommon Terraform modules can be enabled according to the specific application\nrequirements. Similarly, it uses common Ansible playbooks which can be modified\nfor specific applications. If an app needs something special, we can easily add a\ncustom module for it.\n\nWe use the following terminology:\n\n* Apps are under an `org`, or organization, e.g. a company. `org_unique` is\n  a globally unique identifier, used to name e.g. S3 buckets\n\n* An `env` is an environment, e.g. dev, stage, or prod. Each gets its own\n  AWS account\n\n* An `app` is a single shared set of data, potentially accessed by multiple\n  front end interfaces and back end workers. Each app gets it's own VPC.\n  A separate VPC, generally one per environment, handles logging and monitoring\n  using ELK and Prometheus\n\n* A `comp` is an application component\n\nWe have three standard types of components: web app, worker and cron.\n\n**Web apps** process external client requests. Simple apps consist of only a single\nweb app, but complex apps may have more, e.g. an API server, admin interface or\ninstance per customer.\n\n**Workers** handle asynchronous background processing driven by a job queue\nsuch as Sidekiq, SQS or a Kafka stream. They make the front end more responsive\nby offloading long running tasks. The number of worker instances in the ASG\ndepends on the load.\n\n**Cron** servers handle timed batch workloads, e.g. periodic jobs. From a\nprovisioning perspective, there is not much difference between a worker and a\ncron instance, except that cron instances are expected to always be running so\nthat they can schedule jobs.  Generally speaking, we prefer to move periodic\ntasks to Lambda functions where possible.\n\nWe normally run application components in an auto scaling group, allowing\nthem to start and stop according to load. This also provides high availability,\nas the ASG will start instances in a different availability zone if they die.\nThis makes it useful even if we normally only have one instance running.\n\nRunning in an ASG requires that instances start from a \"template\" image AMI and\nbe stateless, storing their data in S3 or RDS. We can also run components in\nstandalone EC2 instances, useful for development and earlier in the process of\nmigrating the app to the cloud.\n\nWe can also deploy the app to containers via ECS as part of the same system.\nEverything is tied together with a common ALB, so it's just a question of\nrouting traffic.\n\nWhen possible, we utilize managed AWS services such as RDS, ElastiCache, and\nElasticsearch. When managed services lack functionality, are immature or are\nexpensive at high load, we can run our own.\n\nThe system makes use of CloudFront to host application assets as well as static\ncontent websites or \"JAM stack\" apps using tools like\n[Gatsby](https://www.gatsbyjs.org/).\n\nWe deploy the application using AWS CodeDeploy using a blue/green deployment\nstrategy. The CodeDeploy releases can be built using CodePipeline or a DevOps\nEC2 instance.\n\nBy default we use Route53 for DNS and ACM for certificates, though it can\nwork with external DNS, certs and other CDNs like CloudFlare.\n\n## Getting started\n\n* Install tools and libraries, see [doc/install.md](doc/install.md)\n* Configure the system, see [doc/config.md](doc/config.md)\n* Create infrastructure with Terraform, see [terraform/README.md](terraform/README.md)\n  and [terraform/building.md](terraform/building.md)\n* Create configuration with Ansible, see [ansible/README.md](ansible/README.md)\n* Create AMIs with Ansible and Packer, see [packer/README.md](packer/README.md)\n* Have fun!\n\n[Contact Us](https://www.cogini.com/contact/) if you would like help deploying\nyour complex app.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcogini%2Fmulti-env-deploy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcogini%2Fmulti-env-deploy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcogini%2Fmulti-env-deploy/lists"}