{"id":21202515,"url":"https://github.com/infrablocks/end-to-end-concourse-ci","last_synced_at":"2025-07-11T15:04:52.112Z","repository":{"id":39866902,"uuid":"127753974","full_name":"infrablocks/end-to-end-concourse-ci","owner":"infrablocks","description":"An end-to-end example of InfraBlocks, by provisioning Concourse CI.","archived":false,"fork":false,"pushed_at":"2024-10-28T18:12:32.000Z","size":139,"stargazers_count":5,"open_issues_count":2,"forks_count":1,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-05T11:34:33.820Z","etag":null,"topics":["concourse","concourse-ci","end-to-end","example","infrablocks","infrastructure","infrastucture-as-code","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/infrablocks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-04-02T12:35:42.000Z","updated_at":"2025-02-12T10:04:32.000Z","dependencies_parsed_at":"2023-02-10T17:31:16.396Z","dependency_job_id":"7f97afc4-08f3-4434-b6e9-d0c756d9adcd","html_url":"https://github.com/infrablocks/end-to-end-concourse-ci","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/infrablocks/end-to-end-concourse-ci","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fend-to-end-concourse-ci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fend-to-end-concourse-ci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fend-to-end-concourse-ci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fend-to-end-concourse-ci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/infrablocks","download_url":"https://codeload.github.com/infrablocks/end-to-end-concourse-ci/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infrablocks%2Fend-to-end-concourse-ci/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264538585,"owners_count":23624436,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concourse","concourse-ci","end-to-end","example","infrablocks","infrastructure","infrastucture-as-code","terraform"],"created_at":"2024-11-20T20:16:40.103Z","updated_at":"2025-07-10T06:33:00.890Z","avatar_url":"https://github.com/infrablocks.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# End to End Example - Concourse CI\n\nThis is an end-to-end deployment of a single infrastructure component, in this\ncase it's a ConcourseCI instance. We combine several tools to make this work:\n\n* Rake - for task management\n* Terraform - for infrastructure provisioning, heavily leaning on InfraBlocks\n* Confidante - for configuration management\n\n## Concepts\n\nInfraBlocks modules, and our configuration, contain some terms which need\nexplanation:\n\n### Component\n\nA collection of infrastructure, which together provides value. For example, a\ntypical micro-service to serve customer information, with a database and an ECS\nService, would all come under the component `customer-service`. For this \ncodebase, we've just used `concourse-example`, but this could easily be \n`ci-server`.\n\n### Role\n\nThe individual bits that make up a component. Examples of roles include the \n`database`, `log-group`, and the `service`. You can see how we layer together\nroles in the `config/infra/roles` directory.\n\n### Deployment Identifier\n\nA label so you can differentiate between multiple deployments of the same \ncomponent. This could be tied to an environment, e.g. `development` or \n`production`, or something more clever. We've used a mixture of environment and\nbuild flavours, so we could run A/B tests of services, e.g. `production-blue`\nand `production-green`.\n\n## Deployment\n\nThis code requires terraform 0.12 or greater.\n\n### Setting up your machine (optional)\n\nWe use the `go` script to automate pre-install steps like installing Gems. To \nget `go` onto the PATH, we use direnv. If you want to skip this step, use `rake` \ninstead of `go` in all the commands below.\n\n```bash\n$ brew install direnv\n$ direnv allow\n```\n\n### Unlocking secrets\n\nIt's not recommended, but for this example we keep secrets in the repository.\n\nWe keep them locked up using `git-crypt`, but we've provided the key for you to\nunlock them.\n\n**If you want to deploy this for real, roll these secrets!**\n\n```bash\n$ brew install git-crypt\n$ git-crypt unlock ./git-crypt-key\n```\n\n### Choose a deployment identifier\n\nBecause S3 buckets are global, if you deployed this all as-is you'd likely bump\ninto others (including us!) for things like S3 buckets. So you need to change\nthe deployment identifier.\n\nIt can be anything you want. :-)\n\n``` bash\n$ export DEPLOYMENT_IDENTIFIER=example\n```\n\n### Provision the state bucket\n\nWe need to store remote terraform state, so the first thing we do is build an S3\nbucket to keep it all in.\n\n```bash\n$ go \"bootstrap:provision[$DEPLOYMENT_IDENTIFIER]\"\n```\n\nThe state for this bucket is stored in the `state` folder in this repository.\n\nIf you want to use this repository as part of a team environment, you need to go\ninto the `.gitignore` file and delete the following:\n\n```\n# State bucket state - remove this\nstate/*\n```\n\n### Provision the DNS zone\n\nIn this example, we stand up a public and private zone so we can refer to our\nCI by name rather than by IP address.\n\n```bash\n$ go \"domain:provision[$DEPLOYMENT_IDENTIFIER,example.com]\"\n```\n\n### Setup a TLS Certificate\n\nWe need a valid certificate so we can access the Concourse dashboard over \nHTTPS.\n\n```bash\n$ go \"certificate:provision[$DEPLOYMENT_IDENTIFIER]\"\n```\n\n### Provision the network\n\nWe need to build a network to put our services into. At the moment it just takes\nup `10.0.0.0/16`.\n\n```bash\n$ go \"network:provision[$DEPLOYMENT_IDENTIFIER]\"\n```\n\n### Publishing the Docker images\n\nWe want to deploy Concourse on ECS, so we need somewhere to put our Concourse\nDocker images, and then we need to deploy them.\n\n#### Web Interface\n\n```\n$ go \"web_image_repository:provision[$DEPLOYMENT_IDENTIFIER]\"\n$ go \"web_image:publish[$DEPLOYMENT_IDENTIFIER]\"\n```\n\n#### Workers\n\n```\n$ go \"worker_image_repository:provision[$DEPLOYMENT_IDENTIFIER]\"\n$ go \"worker_image:publish[$DEPLOYMENT_IDENTIFIER]\"\n```\n\n### Provision the Cluster\n\nWe need to provision some machines to run our ECS cluster on. In this example\nwe spin up a single `t2.medium` box per availability zone. In this case, it's\nthree.\n\n```\n$ go \"cluster:provision[$DEPLOYMENT_IDENTIFIER]\"\n```\n\n### Provision the Database\n\nConcourse needs some kind of SQL database to store build information in, so we\nprovision a Postgres instance using RDS.\n\n```\n$ go \"database:provision[$DEPLOYMENT_IDENTIFIER]\"\n```\n\n### Provision the Services\n\nOnce we have everything we need, now we just need to tell ECS to deploy the\nservices. This will give us some ECS services, as well as a load balancer.\n\nNote: In this example, we've opened up the CI to `0.0.0.0/0`.\n\n```\n$ go \"services:provision[$DEPLOYMENT_IDENTIFIER]\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfrablocks%2Fend-to-end-concourse-ci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfrablocks%2Fend-to-end-concourse-ci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfrablocks%2Fend-to-end-concourse-ci/lists"}