{"id":27998167,"url":"https://github.com/alphagov/paas-cf","last_synced_at":"2025-05-08T22:50:05.156Z","repository":{"id":37502541,"uuid":"46716359","full_name":"alphagov/paas-cf","owner":"alphagov","description":"GOV.UK PaaS - Cloud Foundry","archived":false,"fork":false,"pushed_at":"2025-05-08T18:21:33.000Z","size":90424,"stargazers_count":84,"open_issues_count":62,"forks_count":28,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-05-08T22:49:54.800Z","etag":null,"topics":["cloud-foundry","concourse","paas","reliability-engineering"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alphagov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-11-23T11:36:57.000Z","updated_at":"2025-05-08T09:41:03.000Z","dependencies_parsed_at":"2023-10-14T16:46:05.472Z","dependency_job_id":"403a4ab5-520e-4ef6-886c-3da5faad9cc7","html_url":"https://github.com/alphagov/paas-cf","commit_stats":{"total_commits":6873,"total_committers":86,"mean_commits":79.9186046511628,"dds":0.8753091808526117,"last_synced_commit":"58321ae6a1ca3dbee1465fe986144aee505beee3"},"previous_names":[],"tags_count":3391,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphagov%2Fpaas-cf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphagov%2Fpaas-cf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphagov%2Fpaas-cf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphagov%2Fpaas-cf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alphagov","download_url":"https://codeload.github.com/alphagov/paas-cf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253160827,"owners_count":21863624,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-foundry","concourse","paas","reliability-engineering"],"created_at":"2025-05-08T22:50:04.404Z","updated_at":"2025-05-08T22:50:05.147Z","avatar_url":"https://github.com/alphagov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# paas-cf\n\n⚠️\nWhen merging pull requests, use the [gds-cli](https://github.com/alphagov/gds-cli): `gds git merge-sign alphagov/paas-cf PR_NUMBER`\n⚠️\n\nGOV.UK Platform as a Service (PaaS) CF creates a deployment of [Cloud Foundry](https://www.cloudfoundry.org/) (CF) on VMs for GOV.UK PaaS. It builds upon the foundations laid out in [`paas-bootstrap`](https://github.com/alphagov/paas-bootstrap) and it handles the following non-exhaustive list of duties:\n\n+ [Deploying CF using Concourse](https://github.com/alphagov/paas-cf/blob/main/concourse/pipelines/create-cloudfoundry.yml)\n+ [Configuring CF](https://github.com/alphagov/paas-cf/tree/main/manifests/cf-manifest) based on [`cf-deployment`](https://github.com/cloudfoundry/cf-deployment)\n+ [Provisioning AWS resources using Terraform](https://github.com/alphagov/paas-cf/tree/main/terraform/cloudfoundry). This includes\n   + load balancers\n   + databases\n   + IAM roles and policies\n   + DNS records\n   + networking\n   + S3 buckets\n+ [Configuring Prometheus](https://github.com/alphagov/paas-cf/tree/main/manifests/prometheus) based on [`prometheus-boshrelease`](https://github.com/bosh-prometheus/prometheus-boshrelease)\n+ Running continuous [platform-level tests](https://github.com/alphagov/paas-cf/tree/main/platform-tests)\n+ Deploying and configuring our different service brokers (for example, the [RDS broker](https://github.com/alphagov/paas-cf/blob/main/manifests/cf-manifest/operations.d/710-rds-broker.yml) and [Aiven broker](https://github.com/alphagov/paas-cf/blob/main/manifests/cf-manifest/operations.d/741-aiven-broker.yml))\n\nIt does not include the AWS IAM roles which are assumed by different system components. Those are created in the account wide terraform (private repository).\n\n## Contents\n1. [What does `paas-cf` contain?](#what-does-paas-cf-contain)\n1. [Deploying a new environment](#deploying-a-new-environment)\n1. [Cloud Foundry deployment configuration options](#cloud-foundry-deployment-configuration-options)\n1. [Accessing Concourse](#accessing-concourse)\n1. [Finding configuration](#finding-configuration)\n1. [Utility Scripts](#utility-scripts)\n\n## What does `paas-cf` contain?\n`paas-cf` separates the responsibility for configuring, deploying, running, and monitoring Cloud Foundry, from those responsibilities held by [`paas-bootstrap`](https://github.com/alphagov/paas-bootstrap).\n\nThis repository does not itself contain the code that runs in an environment (for the most part), but instead serves to compose the different pieces into a cohesive whole. As a result, it contains a variety of pieces that tell only part of the story. The table under the heading [Finding configuration](#Finding-configuration) outlines some key directories and their purposes.\n\n## Deploying a new environment\nAt a very high level, the [`create-cloudfoundry` Concourse pipeline](https://github.com/alphagov/paas-cf/blob/main/concourse/pipelines/create-cloudfoundry.yml) generates a [Bosh manifest](https://bosh.io/docs/manifest-v2/) which describes the virtual machines and their networking which make up the Cloud Foundry deployment, as well as the software which runs on each machine. The manifest is then [submitted](https://github.com/alphagov/paas-cf/blob/main/concourse/pipelines/create-cloudfoundry.yml#L2899) to the Bosh director configured in `paas-bootstrap`.\n\n### Pre-requisites\nBefore you can get a Cloud Foundry deployment up and running, you will need the following available\n\n+ [ ] A running [`deployer-concourse` instance from `paas-bootstrap`](https://github.com/alphagov/paas-bootstrap)\n+ [ ] Make\n+ [ ] Ruby \u003e= 2.7\n+ [ ] [GDS CLI](https://github.com/alphagov/gds-cli)\n+ [ ] Access to `paas-credentials` (private repository) and tools installed\n+ [ ] Connection to GDS VPN\n+ [ ] Permission to assume the `Admin` role of the relevant AWS account (dev, ci, staging, production)\n+ [ ] `AWS_DEFAULT_REGION` environment set the desired region for the environment\n+ [ ] [cf CLI](https://docs.cloudfoundry.org/cf-cli/install-go-cli.html) \u003e=7\n\n### Deploy Cloud Foundry\n\nThese instructions contain placeholders where the exact command may vary. The below table explains the purpose of those placeholders:\n\n| Placeholder   | Purpose                                                                                                                                                                                                           |\n| ------------- | ------------------------------------------|\n| `$ACCOUNT`    | The AWS account being targeted (for example, `dev`, `staging`)|\n| `$ENV` | The name of the environment being targeted. In the case of short lived development environments, this should have a value of `dev`, and the specific environment is set by the `DEPLOY_ENV` environment variable (max 8 chars)|\n\n1. Log in to [CredHub](https://docs.cloudfoundry.org/credhub/) in the environment by running this and following the instructions on screen. This will take you into a new shell session.\n   ```shell\n   gds aws paas-$ACCOUNT-admin -- make $ENV credhub\n   ```\n\n1. Upload the secrets to CredHub from the CredHub shell session.\n   ```shell\n   make $ENV upload-all-secrets\n   ```\n\n   Note: you do not need to use GDS CLI here because the CredHub shell session contains the AWS credentials in environment variables\n\n1. Exit the CredHub shell session\n\n1. Deploy the pipeline configurations using `make`. This will upload or update the pipelines. Select the target based on which AWS account you want to work with:\n\n   ```shell\n   gds aws paas-$ACCOUNT-admin -- make $ENV pipelines\n   ```\n1. Log in to Concourse. See the [Accessing Concourse](#accessing-concourse).\n\n1. Tun the `generate-paas-admin-git-keys`and `generate-paas-aiven-broker-git-keys` jobs in the job group `operator`. This will generate and store some SSH keys needed by other jobs.\n\n1. Run the `create-cloudfoundry` pipeline, starting from the left-hand `pipeline-lock` job. This will configure and deploy Cloud Foundry. It might take a couple of hours to complete.\n\n## Cloud Foundry deployment configuration options\n\nThere are a handful of configuration options which can change a Cloud Foundry deployment which can only be set at pipeline level. Each of the properties in the below table should be set as [Make variables](https://www.gnu.org/software/make/manual/make.html#Environment) when setting the pipelines:\n\n```\ngds aws paas-$ACCOUNT-admin -- make $ENV pipelines VAR=value\n```\n\n| Property (VAR) | Type | Default | Description |\n| -- | -- | -- | -- |\n| `BRANCH` | String | `main` | Sets the `paas-cf` branch which will be used in the pipeline |\n| `DEPLOY_ENV` | String | `null` for short-lived dev envs, fixed in `Makefile` for other envs | Sets the name of the environment |\n| `SELF_UPDATE_PIPELINE` | Bool | `true` | Whether the pipeline should update its own definition from the current branch at runtime. Disable this if you're making a pipeline change which has not been pushed to branch yet |\n| `SLIM_DEV_DEPLOYMET` | Bool| `true` in dev, `false` elsewhere | If `true`, reduces the number and size of VMs created for each component to 2. In dev, set this to `false` when testing the impact of a change on platform availability |\n|`DISABLED_AZS` | String list, space separated | `\"\"` | \u003cp\u003eDisables the given availability zones in Bosh. This is used when an availability zone goes away, and we need to redistribute virtual machines away from that AZ. \u003c/p\u003e\u003cp\u003e Set to a value like `\"z1 z2\"`\u003c/p\u003e|\n|`ENABLE_AUTODELETE` | Bool | `true` in dev, `false` elsewhere | \u003cp\u003eIf `true`, deploys a pipeline which tears down Cloud Foundry at 8pm each day as a cost saving measure.\u003c/p\u003e\u003cp\u003eThis should absolutely never be set to `true` in a staging or production deployment\u003c/p\u003e |\n|`ENABLE_DESTROY` | Bool | `true` in dev, `false` elsewhere |\u003cp\u003eIf `true`, deploys a pipeline which, when run, will completely destroy Cloud Foundry and all of its data\u003c/p\u003e\u003cp\u003eThis should absolutely never be set to `true` in a staging or production deployment\u003c/p\u003e|\n\n## Accessing Concourse\nOnce deployed, Concourse can be accessed from the URLs below. By default, authentication with Github is enabled.\n\n| Environment type | Environment name | URL |\n| ---------------- | ---------------- | --- |\n| Dev | Unique name | https://deployer.$NAME.dev.cloudpipeline.digital/ |\n| Dev | Dev[0-9]+ | https://deployer.dev$NUMBER.dev.cloudpipeline.digital/ |\n| Staging | `stg-lon` | https://deployer.london.staging.cloudpipeline.digital/ |\n| CI | `build` | https://concourse.build.ci.cloudpipeline.digital/ |\n| Production | `prod` | https://deployer.cloud.service.gov.uk/ |\n| Production | `prod-lon` | https://deployer.london.cloud.service.gov.uk/ |\n\nNon-development URLs are also accessible via the `gds paas open` command.\n\n## Finding configuration\nThe following table outlines some important directories in the repository, their purpose, and when you might need to look in them.\n\n| Directory | Purpose | I will need this when .. |\n| -- | -- | -- |\n| `concourse/pipelines/` | YAML definitions of the Concourse pipelines | I want to make a change to how the platform is deployed, monitored, or torn down |\n| `manifests/cf-manifest/` | The Bosh manifest configuration for Cloud Foundry | See specific directories below |\n| `manifests/cf-manifest/operations.d/` | Customisations applied to `cf-deployment`, applicable to all environments | \u003cul\u003e\u003cli\u003eI want to make a configuration change that will affect every environment\u003c/li\u003e\u003cli\u003eI want to deploy a new piece of software with a Bosh release\u003c/li\u003e\u003c/ul\u003e\n| `manifests/cf-manifest/operations` | Customisations applied to `cf-deployment` [based on some condition](https://github.com/alphagov/paas-cf/blob/main/manifests/cf-manifest/scripts/generate-manifest.sh#L18) | I want to make a configuration change that will only be applied in certain circumstances |\n|`manifests/cf-manifest/spec`| Unit tests applied to the generated manifest file | I want to make sure a property of the manifest is not invalidated (for example, correct number of instances of some VM) |\n|`manifests/cf-manifest/env-specific`| Values of variables per environment | I want to change things like the number of Diego cells deployed in an environment |\n| `terraform/az-monitoring` | Terraform configuration for out availability zone monitoring solution | I want to make a change to how we monitoring how alive an availability zone is |\n| `terraform/cloudfoundry` | Terraform configuration for the AWS resources associated with running Cloud Foundry | \u003cul\u003e\u003cli\u003eI want to set/unset DNS records\u003c/li\u003e\u003cli\u003eI want to configure ingress for a new service broker\u003c/li\u003e\u003cli\u003eI want to alter Cloud Foundry's AWS network architecture\u003c/li\u003e\u003c/ul\u003e|\n| `terraform/spec` | Unit tests applied to Terraform configuration | I want to make an assertion about Terraform configuration as part of the unit tests\n| `terraform/vpc-peering` | Terraform configuration for VPC peering between the Cloud Foundry VPC and others | I want to change a property of our existing VPC peers, and future ones |\n| `tools/buildpacks` | Golang implementation of our regular buildpack update emails | I want to make a change to the email we send to tenants about buildpack updates |\n| `tools/metrics` | A Prometheus exporter which exposes a variety of platform-level metrics collected from different sources | \u003cul\u003e\u003cli\u003eI want to add a new metrics\u003c/li\u003e\u003cli\u003eI want to change the frequency of the measurement of an existing metric\u003c/li\u003e\u003c/ul\u003e|\n\n## Utility Scripts\n\n### Add a permissions boundary policy to paas-s3-broker users\n\nConfigure the POLICY_NAME variable within the Makefile with the name of the Permissions Boundary policy that you wish to add to the paas-s3-broker users.\n\nRun this command to add a permissions boundary to paas-s3-broker users:\n\n```\ngds aws paas-\u003cENV-ROLE\u003e -- make \u003cBUILD_ENV\u003e add_permissions_boundary_to_existing_s3_broker_users ARGS=\"\u003c--dry-run\u003e\"\n```\n\nReplace:\n\n* `\u003cENV-ROLE\u003e` with the environment and role that you want to use e.g. prod-admin.\n* `\u003cBUILD_ENV\u003e` with the environment that you want to update e.g. prod-lon.\n* Only use the --dry-run flag if you would like the script to run but not change anything.\n\nIf the command is successful, the output will look similar to this:\n\n```\nDry run? false\nPolicy attached to user: paas-s3-broker-dev05-0a094c73-7ae7-42cc-b028-6c78b93985d7\nPolicy attached to user: paas-s3-broker-dev05-dad332ff-3557-4f13-a768-5dc0e8421cd4\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falphagov%2Fpaas-cf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falphagov%2Fpaas-cf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falphagov%2Fpaas-cf/lists"}