{"id":15955298,"url":"https://github.com/vsilverman/insight","last_synced_at":"2026-05-10T16:38:45.057Z","repository":{"id":148907009,"uuid":"208107905","full_name":"vsilverman/insight","owner":"vsilverman","description":"This is my project @ DevOps Fellows Program","archived":false,"fork":false,"pushed_at":"2019-11-18T00:08:21.000Z","size":17668,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-21T09:56:07.699Z","etag":null,"topics":["consul","docker","packer","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vsilverman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-12T17:34:11.000Z","updated_at":"2019-11-18T00:08:23.000Z","dependencies_parsed_at":"2023-07-24T09:00:08.673Z","dependency_job_id":null,"html_url":"https://github.com/vsilverman/insight","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vsilverman/insight","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsilverman%2Finsight","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsilverman%2Finsight/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsilverman%2Finsight/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsilverman%2Finsight/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vsilverman","download_url":"https://codeload.github.com/vsilverman/insight/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsilverman%2Finsight/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32864082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"ssl_error","status_checked_at":"2026-05-10T13:40:02.145Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","docker","packer","terraform"],"created_at":"2024-10-07T13:22:18.753Z","updated_at":"2026-05-10T16:38:45.019Z","avatar_url":"https://github.com/vsilverman.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hashi Service Mesh\n\nThis repository contains the sources for Vlad Silverman's 2019 \nInsight DevOps project. Initial presentation of the project may be viewed \n[here](https://github.com/vsilverman/insight/blob/master/Vlad_Silverman_Hashi_Service_Mesh.pdf).\n\n## Table of Contents\n\n1. [Introduction](README.md#introduction)\n2. [DevOps Pipeline](README.md#devops-pipeline)\n    * Overview\n    * Terraform\n    * Packer\n    * Docker\n    * Consul\n3. [Build Instructions](README.md#build-instructions)\n\t* Prerequisites\n    * Deploy using Terraform and Packer\n    * Manage services with Docker and Consul\n6. [Conclusion](README.md#conclusion)\n7. [Future Work](README.md#future-work)\n    * CI/CD with Jenkins\n    * Docker + ECS Containerization for web service\n    * AWS RDS for Postgres\n\n## Introduction\n\nThe goal of this project is to automate the deployment of an \napplication onto AWS by writing infrastructure as code (IaC), \nand manage container communication by using service mesh \napproach. The DevOps pipeline will use Terraform and Packer \nfor automatic deployment, and Git for version control.\n\n## DevOps Pipeline\n\n### Overview\n\nThe DevOps pipeline will write infrastructure as code (IaC) \nusing Terraform and Packer and version control the application \nand IaC using Git.\n\nThe proposed DevOps pipeline is an example of an immutable \ninfrastructure where once an instance is launched, it is never \nchanged, only replaced. The benefits of an immutable infrastructure \ninclude more consistency and reliability in addition to a simpler, \nmore predictable deployment process.\n\n### Terraform\n\nTerraform is used to setup the virtual private cloud (VPC) and \nother security group settings.\n\nThere should be two subnets: public and private. \nWeb server uses the public subnet which is connected to the \ninternet through the internet gateway. \nThe remaining data pipeline components (i.e., Proxy and PostgreSQL \nservice) reside in the private subnet since the outside internet \nshould not have access to these components. \nIn addition to setting up the VPC, Terraform also sets up the \nsecurity groups which limit communication between components to \nspecific ports. Terraform is also used to spin up the amazon \nmachine images (AMIs) created by Packer and configures them accordingly.\n\n### Packer\n\nPacker is used to create the Amazon machine images (AMI) for each \nof the components (i.e., Httpd, Consul and PostgreSQL) of the \ndata engineering pipeline. The AMIs use a base Ubuntu image and \ninstalls the required software.\n\nExample, showing how above tools were used in this project to \nbuild, do sanity tests and deploy consul-ubuntu image is \npublished in the following draft  \n[video](https://www.youtube.com/watch?v=qYGJg2jEsDs).\n\n## Build Instructions\n\n### Prerequisites\n\nThe following software must be installed into your local environment:\n\n* Terraform\n* Packer\n* Docker\n* Consul\n* AWS command line interface (CLI)\n\nClone the repository:\n\n`git clone https://github.com/vsilverman/insight.git`\n\n### Build Infrastructure using Terraform and Packer\n\n* `cd insight`\n* `vi build.sh` and change the user inputs as needed.\n* `./build.sh --packer y --terraform y`\n\nRunning `build.sh` performs the following:\n\n* Calls Packer to build the Postgres, Consul and Httpd AMIs.\n* Calls Terraform to spin up the cluster with Httpd, Consul and \nPostgresql.\n\n\n## Conclusion\n\nIn this project, we have automated the deployment of an application \nonto AWS using a high-reliability infrastructure. We used Terraform \nand Packer to automate deployment and showed how communication \nbetween containers may be organized using mesh services approach.\n\n## Future Work\n\n### CI/CD with Jenkins\n\nIntegration of above steps into CI/CD pipeline may be done similar \nto the following \n[Continuous Integration project](https://vsilverman.github.io/jenkins-ci/)\n\nThe developer-to-customer pipeline is summarized below:\n1. Developer\n2. Build\n3. Test\n4. Release\n5. Provision and Deploy\n6. Customer\n\nTerraform and Packer handles steps 4 and 5. However, we still need \na CI/CD tool (e.g., Jenkins) to handle steps 2 and 3, and to \nautomatically trigger Terraform and Packer to perform steps 4 and 5. \nThe implementation of CI/CD using Jenkins is summarized below:\n* Developer pushes code into source Git repository.\n* Jenkins detects the change and automatically triggers:\n    + Packer to build the AMIs in the staging environment.\n    + Terraform to spin up the AMIs in the staging environment.\n* Jenkins performs unit tests.\n* If build is not green, developers are notified.\n* If green, we can either automatically deploy into the production environment (continuous deployment) or wait for manual approval (continuous delivery).\n\nBelow are some specific work items to incorporate Jenkins for CI/CD:\n* Create separate staging and production environments that both use the same Terraform modules.\n* Use Terraform to spin up an additional instance to run Jenkins.\n* Create a `Jenkinsfile` as follows:\n    + Monitors for changes to the AirAware Git repository.\n    + Create a `build` stage which triggers Packer to build new AMIs in the staging environment.\n    + Create a `deploy-to-staging` stage which triggers Terraform to spin up the new AMIs in the staging environment.\n    + Create a `testing` stage to perform unit tests on the staging environment.\n    + Create a `deploy-to-production` stage which either automatically deploys to production or waits for manual approval if `testing` stage passes.\n\nAs Jenkins alternative CI/CD tool CircleCI may be used for smaller projects, which nicely integrates with GitHub.\nAnother CI/CD alternative may be GitHub Actions, newly released directly from GitHub \n\n### Docker + ECS Containerization for Httpd\n\nContainers are MB instead of GB in size compared to VMs, and take seconds rather than minutes to spin up. Httpd is a good candidate for containerization as it requires low OS overhead, and needs to be quickly spun up or down based on user-demand. We can use Docker to containerize Httpd service. This is done by creating a `DockerFile` that performs many of the same tasks as Packer in order to create a TBD Docker image. Docker can be used in conjunction with AWS elastic container service (ECS) for container orchestration. A container orchestration tool:\n* Defines the relationship between containers.\n* Sets up container auto-scaling.\n* Defines how containers connect with the internet.\n\nNote that ECS clusters can be built in Terraform using the `aws_ecs_cluser` resources respectively.\n\n### AWS RDS for Postgres\n\nAmazon relational database service (RDS) supports Postgres, and performs the following tasks:\n* Scales database storage with little to no downtime.\n* Performs backups.\n* Patches software.\n* Manages synchronous data replication across availability zones.\n\nNote that RDS can be built in Terraform using the `aws_rds_cluster` resource.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsilverman%2Finsight","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvsilverman%2Finsight","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsilverman%2Finsight/lists"}