{"id":23954369,"url":"https://github.com/cloud-gov/terraform-provision","last_synced_at":"2026-01-17T18:26:15.435Z","repository":{"id":37067321,"uuid":"58405565","full_name":"cloud-gov/terraform-provision","owner":"cloud-gov","description":"cloud.gov infrastructure provisioning and deployment","archived":false,"fork":false,"pushed_at":"2026-01-12T21:48:06.000Z","size":5211,"stargazers_count":69,"open_issues_count":21,"forks_count":24,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-01-12T22:20:33.966Z","etag":null,"topics":["cloud-gov"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloud-gov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-05-09T20:22:26.000Z","updated_at":"2026-01-12T21:48:09.000Z","dependencies_parsed_at":"2023-09-26T06:20:01.551Z","dependency_job_id":"835cab89-dfd1-47f7-96e8-9bd74e8522b3","html_url":"https://github.com/cloud-gov/terraform-provision","commit_stats":null,"previous_names":["cloud-gov/terraform-provision","cloud-gov/cg-provision"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cloud-gov/terraform-provision","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud-gov%2Fterraform-provision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud-gov%2Fterraform-provision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud-gov%2Fterraform-provision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud-gov%2Fterraform-provision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloud-gov","download_url":"https://codeload.github.com/cloud-gov/terraform-provision/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloud-gov%2Fterraform-provision/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28515466,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T17:57:59.192Z","status":"ssl_error","status_checked_at":"2026-01-17T17:57:52.527Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-gov"],"created_at":"2025-01-06T15:00:48.561Z","updated_at":"2026-01-17T18:26:15.405Z","avatar_url":"https://github.com/cloud-gov.png","language":"HCL","funding_links":[],"categories":["HCL"],"sub_categories":[],"readme":"# Cloud.gov Provisioning System\n\nThis repository holds the terraform configuration (and BOSH vars and ops-files)\nto bootstrap our infrastructure.\n\nBe sure to read the internal developer documentation (\"cg-provision\") for\nnon-public information about using this repository.\n\n## Local development\n\n### Git hooks\n\nThis project uses [`pre-commit`](https://pre-commit.com/) to manage git hooks. To make sure your code is formatted and checked automatically, [install pre-commit](https://pre-commit.com/index.html#installation) then run `pre-commit install` on this repo.\n\n_n.b., pre-commit recommends `pip install pre-commit`to install, which will install it on your system python.\nYou probably don't want this, and should instead `pipx install pre-commit`, which will automagically install it into its own virtual environment._\n\n## Layout\n\n### Terraform\n\nOur Terraform code is organized by two concepts, with two corresponding directories.\n\n* **Modules** are reusable units of terraform code. Each module describes a useful concept in our infrastructure. A module can be reused in different contexts by declaring variables that the caller must pass in. Modules are located in `terraform/modules`.\n  * Two important modules are `modules/stack/base` and `modules/stack/spoke`.\n  * Read more about Terraform modules: https://developer.hashicorp.com/terraform/language/modules\n* **Stacks** combine and configure modules for use in an environment. They are also parameterized by variables, with values such as the environment name. Stacks are located in `terraform/stacks`.\n\nAs an example, if we wanted to write terraform code to deploy several CloudFront distributions in front of three load balancers in an environment, we could:\n\n1. Create a `cloudfront` module that declares a CloudFront distribution, a Shield Advanced resource to protect it, and an Access Control List (ACL) association between the distribution and an ACL. (The ACL itself is not declared in the module.) It could take an origin, a list of domains, and an ACL ARN as variables.\n2. Create a `cloudfront` stack uses the `cloudfront` module three times, once for each load balancer in the environment. It would pass an external domain and load balancer domain to each module. It could also declare a single ACL for the environment and pass its ARN to each `cloudfront` module. The stack could take an environment name as a variable.\n3. Add a job to Concourse for each environment. Each job would deploy the `cloudfront` stack and pass the environment name as a variable.\n\nIn the future, we would like to add a third concept: An entire **runtime environment**. An environment would combine multiple stacks to represent the entire cloud.gov runtime stack. This collection of resources could be deployed as a single unit to a new AWS region or multiple times in the same region.\n\n#### Environments\n\nThe `main` stack is a template that is used to provision the production,\nstaging, and development \"environments.\"\n\nThe `regionalmasterbosh` stack contains our masterbosh for a given region, which deploys the tooling BOSH for that region.\nThe tooling BOSH then deploys the BOSH directors in the main stacks across all accounts in that region.\n\nThe `tooling` stack is the same as the `regionalmasterbosh` stack, but has some extras from before we started going multi-region\nand multi-account:\n  - concourse and staging concourse\n  - buckets that we need only one of across all accounts and regions\n  - some things that really should be in child environment accounts\n  - nessus\nIn the future, we should work towards disentagling these pieces out, so the old tooling is deployed as a regionalmasterbosh and the\n_other_ stuff is its own stack(s)\n\nThe `external` and `dns` stacks are both outside of GovCloud (commercial AWS).\n\n#### Wiring up users\n\nAs mentioned above, we have four categories of environment:\n- `main` - this is the thing we're actually after. It's the pieces that directly\n  support the platform components. There should be several of these across multiple\n  AWS accounts\n- `tooling` - this is used to support the things in the `main` platform - our CI\n  system, managment tools such as Nessus, etc.\n- `external` - this manages some things that don't (or historically didn't) exist\n  in govcloud (really just cloudfront and the users, etc, to support it). There's\n  one of these per `main` environment\n- `dns` - this manages route53. There's exactly one of these, although we really\n  should split it out to one per `main` + one for `tooling`\n\nTo allow the `tooling` environment to manage the `main` environment, there's a\n`tooling-terraform` role associated with each `main` environment, which has an\nassumerole policy allowing access by concourse workers in the `tooling` account.\n\nTo add a new `main` environment, see the [README here](./scripts/add_environment/README.md)\n\n### BOSH\n\nThe `bosh` directory contains vars and opsfiles for use by the BOSH directors.\n\n## How Concourse Creates AWS Resources\n\nThe Concourse worker VMs must have AWS access to create and apply Terraform plans. How they are given that access depends on the partition being changed.\n\nYou can determine how a failing Concourse container is configured by hijacking it. Connect to the container (see `fly hijack --help`) and run `aws configure list` to see the current configuration.\n\n### GovCloud\n\nThe Concourse worker VMs are associated with an IAM role with read-write access to GovCloud resources. The AWS SDK in the Concourse containers is automatically configured to fetch credentials from the [Instance metadata service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html). No further configuration is necessary - note that no access keys are passed to GovCloud jobs in [pipeline.yml](./ci/pipeline.yml).\n\nAWS IAM roles documentation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html\n\n### Commercial Stacks\n\nEach Concourse job that manages AWS Commercial resources must override the Concourse worker's IAM role. The jobs set the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_DEFAULT_REGION` environment variables to do this. Environment variables [have higher precedence](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-precedence) in the AWS SDK, so they are used instead of the IAM role. No further configuration in Terraform is necessary.\n\n### DNS and CloudFront Stacks\n\nThe DNS stack is a special case because it must read state from GovCloud but read and write resources and state to Commercial. AWS IAM users [cannot have cross-partition permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html), so the job must use two separate AWS accounts (one for each partition).\n\nTo achieve this, the Concourse jobs pass an access key to a Commercial IAM user as a TF_VAR instead of using the standard `AWS_*` environment variables. (Setting `AWS_` variables would make the AWS SDK use them by default, and we want it to continue using the GovCloud IAM role by default.)\n\nThe IAM role and `TF_VAR_` credentials are used as follows:\n\n* The `terraform init` command is run with the Commercial credentials using [this script](https://github.com/cloud-gov/cg-pipeline-tasks/blob/ca4120f9ca5c56cb16b8550de16d5b097190e466/terraform-apply.sh#L40-L48). This configures the s3 backend for the DNS stack to be set up in the Commercial account.\n* The terraform provider is configured with the Commercial credentials, ensuring that all resources will be created in the Commercial account.\n* The `terraform_remote_state` data blocks for each GovCloud s3 state object are configured with the GovCloud region. Because they are accessed using Terraform's initialization process, but separately from the initial `terraform init`, they are not passed the Commercial credentials. Without any credentials set explicitly, the AWS SDK uses the GovCloud IAM role.\n\n## Deployment Workflow\n\nSince IaaS is a shared resource (we don't have the money or time to provision\nentire stacks for each developer), we never apply this configuration manually.\nInstead, all execution is done through the Concourse pipeline, which is\nconfigured to first run `terraform plan`, and then wait for manual triggering\nbefore running `terraform apply`.\n\nIf you want to make infrastructure changes:\n\n1. Create a branch and pull-request with your changes and ask for review and\n   merge from a teammate.\n1. Once the teammate :thumbsup: the changes, head over to the Concourse\n   pipeline and review the resultant Terraform plan output.\n1. If the plan looks like what you intended, then manually trigger the\n   appropriate apply jobs.\n\n## Other Points of Note\n\nYou may see `access_key_id_prev` and `aws_key_id_prev` as outputs for our `iam`\nmodules. [These are used for cred\nrotation](https://cloud.gov/docs/ops/runbook/rotating-iam-users/#rotating-iam-user-access-key-ids-and-secret-access-keys)\n\n`modules/stack/spoke` composes `modules/stack/base` and some of the VPC\nmodules.  It's not entirely clear why, and why the VPC modules weren't simply\nincluded in `base` (removing `spoke` altogether).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloud-gov%2Fterraform-provision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloud-gov%2Fterraform-provision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloud-gov%2Fterraform-provision/lists"}