https://github.com/infrablocks/end-to-end-concourse-ci
An end-to-end example of InfraBlocks, by provisioning Concourse CI.
https://github.com/infrablocks/end-to-end-concourse-ci
concourse concourse-ci end-to-end example infrablocks infrastructure infrastucture-as-code terraform
Last synced: 5 months ago
JSON representation
An end-to-end example of InfraBlocks, by provisioning Concourse CI.
- Host: GitHub
- URL: https://github.com/infrablocks/end-to-end-concourse-ci
- Owner: infrablocks
- License: mit
- Created: 2018-04-02T12:35:42.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2024-07-30T15:43:15.000Z (9 months ago)
- Last Synced: 2024-07-31T17:52:36.377Z (9 months ago)
- Topics: concourse, concourse-ci, end-to-end, example, infrablocks, infrastructure, infrastucture-as-code, terraform
- Language: HCL
- Homepage:
- Size: 133 KB
- Stars: 4
- Watchers: 12
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# End to End Example - Concourse CI
This is an end-to-end deployment of a single infrastructure component, in this
case it's a ConcourseCI instance. We combine several tools to make this work:
* Rake - for task management
* Terraform - for infrastructure provisioning, heavily leaning on InfraBlocks
* Confidante - for configuration management
## Concepts
InfraBlocks modules, and our configuration, contain some terms which need
explanation:
### Component
A collection of infrastructure, which together provides value. For example, a
typical micro-service to serve customer information, with a database and an ECS
Service, would all come under the component `customer-service`. For this
codebase, we've just used `concourse-example`, but this could easily be
`ci-server`.
### Role
The individual bits that make up a component. Examples of roles include the
`database`, `log-group`, and the `service`. You can see how we layer together
roles in the `config/infra/roles` directory.
### Deployment Identifier
A label so you can differentiate between multiple deployments of the same
component. This could be tied to an environment, e.g. `development` or
`production`, or something more clever. We've used a mixture of environment and
build flavours, so we could run A/B tests of services, e.g. `production-blue`
and `production-green`.
## Deployment
This code requires terraform 0.12 or greater.
### Setting up your machine (optional)
We use the `go` script to automate pre-install steps like installing Gems. To
get `go` onto the PATH, we use direnv. If you want to skip this step, use `rake`
instead of `go` in all the commands below.
```bash
$ brew install direnv
$ direnv allow
```
### Unlocking secrets
It's not recommended, but for this example we keep secrets in the repository.
We keep them locked up using `git-crypt`, but we've provided the key for you to
unlock them.
**If you want to deploy this for real, roll these secrets!**
```bash
$ brew install git-crypt
$ git-crypt unlock ./git-crypt-key
```
### Choose a deployment identifier
Because S3 buckets are global, if you deployed this all as-is you'd likely bump
into others (including us!) for things like S3 buckets. So you need to change
the deployment identifier.
It can be anything you want. :-)
``` bash
$ export DEPLOYMENT_IDENTIFIER=example
```
### Provision the state bucket
We need to store remote terraform state, so the first thing we do is build an S3
bucket to keep it all in.
```bash
$ go "bootstrap:provision[$DEPLOYMENT_IDENTIFIER]"
```
The state for this bucket is stored in the `state` folder in this repository.
If you want to use this repository as part of a team environment, you need to go
into the `.gitignore` file and delete the following:
```
# State bucket state - remove this
state/*
```
### Provision the DNS zone
In this example, we stand up a public and private zone so we can refer to our
CI by name rather than by IP address.
```bash
$ go "domain:provision[$DEPLOYMENT_IDENTIFIER,example.com]"
```
### Setup a TLS Certificate
We need a valid certificate so we can access the Concourse dashboard over
HTTPS.
```bash
$ go "certificate:provision[$DEPLOYMENT_IDENTIFIER]"
```
### Provision the network
We need to build a network to put our services into. At the moment it just takes
up `10.0.0.0/16`.
```bash
$ go "network:provision[$DEPLOYMENT_IDENTIFIER]"
```
### Publishing the Docker images
We want to deploy Concourse on ECS, so we need somewhere to put our Concourse
Docker images, and then we need to deploy them.
#### Web Interface
```
$ go "web_image_repository:provision[$DEPLOYMENT_IDENTIFIER]"
$ go "web_image:publish[$DEPLOYMENT_IDENTIFIER]"
```
#### Workers
```
$ go "worker_image_repository:provision[$DEPLOYMENT_IDENTIFIER]"
$ go "worker_image:publish[$DEPLOYMENT_IDENTIFIER]"
```
### Provision the Cluster
We need to provision some machines to run our ECS cluster on. In this example
we spin up a single `t2.medium` box per availability zone. In this case, it's
three.
```
$ go "cluster:provision[$DEPLOYMENT_IDENTIFIER]"
```
### Provision the Database
Concourse needs some kind of SQL database to store build information in, so we
provision a Postgres instance using RDS.
```
$ go "database:provision[$DEPLOYMENT_IDENTIFIER]"
```
### Provision the Services
Once we have everything we need, now we just need to tell ECS to deploy the
services. This will give us some ECS services, as well as a load balancer.
Note: In this example, we've opened up the CI to `0.0.0.0/0`.
```
$ go "services:provision[$DEPLOYMENT_IDENTIFIER]"
```