https://github.com/hanchiang/url-shortener-infra

Infrastructure as code for URL shortener. Backend: https://github.com/hanchiang/url-shortener-backend
https://github.com/hanchiang/url-shortener-infra

aws iaac packer terraform

Last synced: 7 months ago
JSON representation

Infrastructure as code for URL shortener. Backend: https://github.com/hanchiang/url-shortener-backend

• Host: GitHub
• URL: https://github.com/hanchiang/url-shortener-infra
• Owner: hanchiang
• Created: 2022-06-26T14:24:48.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2023-08-16T15:27:47.000Z (about 2 years ago)
• Last Synced: 2023-08-16T17:50:46.732Z (about 2 years ago)
• Topics: aws, iaac, packer, terraform
• Language: Shell
• Homepage: https://www.yaphc.com/automating-infrastructure-provisioning-configuration-and-application-deployment
• Size: 629 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Introduction

![example workflow](https://github.com/hanchiang/url-shortener-infra/actions/workflows/health_check.yml/badge.svg)

![example workflow](https://github.com/hanchiang/url-shortener-infra/actions/workflows/start_url_shortener.yml/badge.svg)

![example workflow](https://github.com/hanchiang/url-shortener-infra/actions/workflows/stop_url_shortener.yml/badge.svg)

This project is the infrastructure as code management for [URL shortener backend](https://github.com/hanchiang/url-shortener-backend) using AWS.

# Structure

* `images/`: Packer files for building AMI

    * `image.pkr.hcl`: Main packer script

    * `scripts/`: Scripts to be run when provisioning AMI

* `instances/`: Terraform files to provision EC2 in VPC

    * `main.tf`: Main terraform script

    * `ansible/`: Ansible scripts to run post-provisioning tasks such as mounting EBS volume, set up file system, copy postgres data, setup SSL for nginx 

    * `scripts/`: Scripts to automate(everything after step 2 of the workflow) start and stop of EC2, DNS, and deployment of [URL shortener backend](https://github.com/hanchiang/url-shortener-backend). Calls ansible scripts

# Workflow

## 1. Provision EC2 AMI using packer

Provisions a EBS-backed EC2 AMI, and install the necessary softwares for [URL shortener backend](https://github.com/hanchiang/url-shortener-backend), i.e. postgres, redis, as well as nginx

cd into `images/`

Define variables that are declared in `image.pkr.hcl` in a new file `variables.auto.pkrvars.hcl`

Build image: `packer build -machine-readable -var-file variables.auto.pkrvars.hcl image.pkr.hcl | tee build.log`

## 2. Provision EC2 in a VPC using terraform

cd into `instances/`

Copy the AMI ID from packer build, update it in `variables.tf`

Provision infra: `terraform apply`

Everything from here onwards is handled in `instances/scripts/start.sh`

## 3. Run ansible script

Run post-provisioning configurations such as mounting EBS volume, setting PostGreSQL data directory, nginx SSL, grafana

## 4. Deploy application

Rerun the latest deploy job in github action

## Diagram

**Traffic flow**

![](diagrams/traffic-flow.drawio.png)

**Deployment pipeline**

![](diagrams/deployment-pipeline.drawio.png)

## TODO:

* Extract common functions into a reusable python module

* Use ansible roles to define reusable configurations

* Use terraform modules to define reusable configurations

* IAM user and policies for system admin

* container image scan

* Use terraform vault to store secrets

* Create postgres roles and user: app, grafana

* Grafana: Traces, alerts on infra & app, monitor grafana itself

* Use AWS ECS for multi container deployment

* Update architecture diagram

* Dockerise nginx

# Learnings:

* Messed up PostGreSQL WAL by `rsync`ing `/var/lib/postgresql/13/main/` to its new data directory. Don't do it.