{"id":26059727,"url":"https://github.com/orange-cloudfoundry/cf-ops-automation","last_synced_at":"2025-06-21T03:39:44.287Z","repository":{"id":24453902,"uuid":"99213306","full_name":"orange-cloudfoundry/cf-ops-automation","owner":"orange-cloudfoundry","description":"a collaboration framework for operating cloudfoundry and services at scale","archived":false,"fork":false,"pushed_at":"2025-06-17T18:28:20.000Z","size":7101,"stargazers_count":23,"open_issues_count":90,"forks_count":6,"subscribers_count":7,"default_branch":"develop","last_synced_at":"2025-06-17T19:39:06.333Z","etag":null,"topics":["bosh","cloudfoundry","concourse","gitops-framework","terraform"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orange-cloudfoundry.png","metadata":{"files":{"readme":"Readme.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-2.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-08-03T09:06:57.000Z","updated_at":"2025-06-10T12:12:18.000Z","dependencies_parsed_at":"2023-02-10T17:15:47.823Z","dependency_job_id":"45811bc8-0adf-4c20-8a24-b39914a9cc54","html_url":"https://github.com/orange-cloudfoundry/cf-ops-automation","commit_stats":null,"previous_names":[],"tags_count":74,"template":false,"template_full_name":null,"purl":"pkg:github/orange-cloudfoundry/cf-ops-automation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orange-cloudfoundry","download_url":"https://codeload.github.com/orange-cloudfoundry/cf-ops-automation/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261058991,"owners_count":23103947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bosh","cloudfoundry","concourse","gitops-framework","terraform"],"created_at":"2025-03-08T13:28:12.067Z","updated_at":"2025-06-21T03:39:39.273Z","avatar_url":"https://github.com/orange-cloudfoundry.png","language":"Ruby","readme":"# cf-ops-automation (COA)\n\n[![Maintainability](https://api.codeclimate.com/v1/badges/98afd0def1915fd16f71/maintainability)](https://codeclimate.com/github/orange-cloudfoundry/cf-ops-automation/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/98afd0def1915fd16f71/test_coverage)](https://codeclimate.com/github/orange-cloudfoundry/cf-ops-automation/test_coverage)\n[![Inline docs](http://inch-ci.org/github/orange-cloudfoundry/cf-ops-automation.svg?branch=develop\u0026style=shields)](http://inch-ci.org/github/orange-cloudfoundry/cf-ops-automation)\n[![Docker Pull](https://img.shields.io/docker/pulls/coabot/cf-ops-automation.svg)](https://hub.docker.com/r/coabot/cf-ops-automation/)\n[![GitHub release](https://img.shields.io/github/release/orange-cloudfoundry/cf-ops-automation.svg)](https://github.com/orange-cloudfoundry/cf-ops-automation/releases)\n\n[![GitHub Page](https://img.shields.io/badge/Github%20Page-Here-green.svg)](https://orange-cloudfoundry.github.io/cf-ops-automation/)\n\n## Table of Contents\n\n\u003c!-- TOC initiated with [gh-md-toc](https://github.com/ekalinin/github-markdown-toc) --\u003e\n\u003c!--ts--\u003e\n* [cf-ops-automation (COA)](#cf-ops-automation-coa)\n   * [Table of Contents](#table-of-contents)\n   * [Introduction](#introduction)\n      * [Core principles](#core-principles)\n      * [Skills prereqs](#skills-prereqs)\n      * [Model Overview](#model-overview)\n      * [Sample deployment topology](#sample-deployment-topology)\n   * [Generated pipelines](#generated-pipelines)\n      * [singletons](#singletons)\n      * [per-root-deployment](#per-root-deployment)\n   * [Template engine reference documentation](#template-engine-reference-documentation)\n      * [Multi deployer support](#multi-deployer-support)\n         * [Bosh deployer](#bosh-deployer)\n         * [Concourse pipeline deployer](#concourse-pipeline-deployer)\n      * [Resource lifecycle overview](#resource-lifecycle-overview)\n      * [Bosh deployment resources template format](#bosh-deployment-resources-template-format)\n      * [Iaas specifics support](#iaas-specifics-support)\n         * [Terraform specific](#terraform-specific)\n      * [Bosh cli v2 specific features support](#bosh-cli-v2-specific-features-support)\n         * [ops-files](#ops-files)\n         * [vars-files](#vars-files)\n         * [Cloud and runtime config](#cloud-and-runtime-config)\n      * [git submodules](#git-submodules)\n         * [enable deployment format (enable-deployment.yml)](#enable-deployment-format-enable-deploymentyml)\n         * [deployment dependencies format (deployment-dependencies.yml)](#deployment-dependencies-format-deployment-dependenciesyml)\n      * [Cloudfoundry application resources template format](#cloudfoundry-application-resources-template-format)\n      * [pipeline auto-update](#pipeline-auto-update)\n      * [Concourse team](#concourse-team)\n      * [terraform](#terraform)\n         * [file format](#file-format)\n      * [delete lifecycle support](#delete-lifecycle-support)\n         * [bosh deployment](#bosh-deployment)\n         * [cf-app deployment](#cf-app-deployment)\n      * [shared and private configuration](#shared-and-private-configuration)\n      * [\"paas-templates\" and \"secrets\" repo structure examples](#paas-templates-and-secrets-repo-structure-examples)\n      * [Stemcell management](#stemcell-management)\n* [COA development](#coa-development)\n   * [Status and roadmap](#status-and-roadmap)\n   * [Interns](#interns)\n   * [Running the Test Suite](#running-the-test-suite)\n   * [Generating pipelines locally and uploading a test version](#generating-pipelines-locally-and-uploading-a-test-version)\n   * [Local terraform development](#local-terraform-development)\n   * [Submitting Pull Requests](#submitting-pull-requests)\n   * [Releasing COA](#releasing-coa)\n      * [Standard release](#standard-release)\n      * [Hotfix release](#hotfix-release)\n   * [Bootstrapping a COA env](#bootstrapping-a-coa-env)\n      * [How to use it](#how-to-use-it)\n      * [Prerequisites](#prerequisites)\n      * [Connecting to Concourse](#connecting-to-concourse)\n      * [Known issues](#known-issues)\n         * [VMs access issues](#vms-access-issues)\n         * [Stemcell loading creates a timeout](#stemcell-loading-creates-a-timeout)\n         * [Some Concourse resource won't load](#some-concourse-resource-wont-load)\n* [FAQ](#faq)\n   * [How to initialize a new bosh deployment template ?](#how-to-initialize-a-new-bosh-deployment-template-)\n   * [How to enable a bosh deployment template ?](#how-to-enable-a-bosh-deployment-template-)\n   * [How to upload a bosh release not available on bosh.io?](#how-to-upload-a-bosh-release-not-available-on-boshio)\n   * [How to generate a tfvars in json from a yaml template?](#how-to-generate-a-tfvars-in-json-from-a-yaml-template)\n      * [Sample](#sample)\n   * [How to bootstrap pipelines to a new concourse?](#how-to-bootstrap-pipelines-to-a-new-concourse)\n      * [pre requisite](#pre-requisite)\n   * [How to create a new root deployment?](#how-to-create-a-new-root-deployment)\n      * [pre requisite](#pre-requisite-1)\n* [Credits](#credits)\n* [Changelog](#changelog)\n* [Upgrade](#upgrade)\n   * [pre requisite](#pre-requisite-2)\n\n\u003c!-- Created by https://github.com/ekalinin/github-markdown-toc --\u003e\n\n\u003c!--te--\u003e\n\n## Introduction\n\nThis repo provides a collaboration framework for operating cloudfoundry and services at scale.\n\nThe goal is to automate most (if not all) interactive operations of Bosh/CF API/Iaas APIs through concourse pipelines, while keeping volume of boilerplate code low, and limit skills\n prereqs required to contribute services automation.\n\n### Core principles\n\n* foster **open-source** collaboration across operators through\n  * automation of bosh/terraform best practices\n    * **simplified operations** of large scale complex deployments (e.g. cloudfoundry + n dataservices incl. monitoring/alerting/logs ...):\n    * portable distribution which bootstraps from raw Iaas with minimal external prereqs\n    * concourse \u0026 git as primary UIs for operators\n    * consistent operations of multiple environments (e.g. preprod/prod-region1/prod-region2/...)\n    * scalability through multiple bosh directors\n    * high avaibility through indepent regions support\n    * offline/airgap support\n  * **opiniated collaboration** framework\n    * strict separation between template and configuration\n      * template: shared across operation teams\n      * configuration: specific to each operated environment\n    * differentiated support for template authors and service operators:\n      * feature branches, template release versions\n  * **declarative idempotent** approach, leveraging:\n    * terraform configurations (for Iaas, CF, K8S resources)\n    * bosh deployment manifests\n    * cloudfoundry application manifests\n  * convention-over-configuration\n    * concourse pipelines are generated and updated from directory/file naming conventions\n    * consolidated bosh release version management\n* **diversity** of supported cloud and operators **customizations**\n  * multiple iaas support (eg: openstack / vsphere / cloudstack)\n  * rich yaml manifest templating support:\n    * spruce enable leveraging bosh manifests\n    * vars \u0026 operators, bosh cli v2 interpolate enable leveraging current bosh v2 syntax\n  * conditional activation of deployments\n  * extensibility through terraform providers\n\n### Skills prereqs\n\n\u003c!-- table initiated through https://www.tablesgenerator.com/markdown_tables# --\u003e\n\nCOA framework strives to limit skills prereqs for each persona, which the following table summarizes\n\n| Persona                | Skills                                                            |\n| :----------------------| :-----------------------------------------------------------------|\n| service operator       | git \u003cbr\u003econcourse concepts (UI usage, no authoring as pipelines are generated)|\n| template author        | git \u003cbr\u003ebosh \u003cbr\u003eterraform (\\*) \u003cbr\u003ecf app manifest (\\*) \u003cbr\u003econcourse-pipelines (\\*) \u003cbr\u003ek8s (\\*)|\n| COA framework developer | ruby \u003cbr\u003econcourse \u003cbr\u003eshell \u003cbr\u003ebosh \u003cbr\u003eterraform \u003cbr\u003egit|\n\n(\\*) optional depending on contributed templates\n\n### Model Overview\n\nCOA takes templates and configurations as inputs, and generates concourse pipelines as outputs. Generated templates automatically reload and execute. As a result, resources (Iaas, Bosh, CF) gets provisioned and\n operated through pipelines.\n\n\u003c!-- edit image source at https://www.draw.io/?lightbox=1\u0026highlight=0000ff\u0026edit=_blank\u0026layers=1\u0026nav=1\u0026title=coab#Uhttps%3A%2F%2Fdrive.google.com%2Fuc%3Fid%3D1qaLM6Ca-3RerlEsQ-BEJHHfIYVWZdeVF%26export%3Ddownload --\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/coa.png\" width=\"300\"\u003e\n \u003cem\u003eDiagram inspired from \u003ca title=\"Dreftymac [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or CC BY 2.5 (http://creativecommons.org/licenses/by/2.5)],\n  via Wikimedia Commons\" href=\"https://commons.wikimedia.org/wiki/File%3ATempEngGen015.svg\"\u003eDreftymac\u003c/a\u003e\n\u003c/em\u003e\n\u003c/p\u003e\n\n* Templates are specified in a git repo (referred to as \"paas-templates\"). It contains a hierarchical structure with root deployment and nested deployment templates.\n* Configurations are specified in a git repo (referred to as \"secrets\"). Their structure mimics the template structure, indicating which deployment template should instanciated. See\n* Generated pipeline triggers provisioning of resources whose credentials and secrets are currently pushed into a git repo (referred to as \"secrets\"). The plan is to move all credentials to credhub.\n\nA `root deployment` contains infrastructure to operate `nested deployment`s.\n\n* A root deployment typically contains Iaas prereqs, Bosh director and its cloud-config, DNS infrastrucure, private git server, Concourse, log collection, monitoring/alerting, credhub, etc...\n* Nested deployments are resources created by a root deployment. This typically include cloudfoundry, admin-ui, services, ...\n\n### Sample deployment topology\n\nThis section provides an overview of the deployment topology and bootstrapping process used by the Orange CloudFoundy skill center team.\n\n![Overview of pipeline generation for bosh deployments](http://www.plantuml.com/plantuml/proxy?src=https://raw.githubusercontent.com/orange-cloudfoundry/cf-ops-automation/master/docs/overview.puml?lastRefreshOn=17082017).\n\nNb: source is in the [plantuml](http://plantuml.com/) file: [bosh overview](docs/overview.puml), see [caching tips](https://stackoverflow.com/questions/32203610/how-to-integrate-uml-diagrams-into-gitlab-or-github)\n\nThe `inception`, `micro-depls`, `master-depls`, `ops-depls`, `expe-depls` are `root deployment`s in Orange CF skill center infrastructure (each associated with a dedicated bosh director)\n\nThe nested deployment model enables a split of responsibility as the operations team scales.\n\nThe plan is to open source the Orange's CF skill center team template git repo in the near future (once the remaining secrets get cleaned up), watch\n [paas-templates](https://github.com/orange-cloudfoundry/paas-templates) repo for incoming commits.\n\n## Generated pipelines\n\nThis sections describes the pipelines that COA generates and loads into concourse. Some are singletons while others are templated and instanciated for each root deployment.\n\n### singletons\n\n* [bootstraps-all-init-pipelines](concourse/pipelines/bootstraps-all-init-pipelines.yml): bootstraps all init pipelines\n* [micro-bosh-init-pipeline](concourse/pipelines/micro-bosh-init-pipeline.yml): !!! WIP !!! manages bosh-micro\n* [sync-feature-branches](concourse/pipelines/sync-feature-branches.yml): manages a WIP branch based on develop and any feature branches\n* [sync-hotfix-branches](concourse/pipelines/sync-hotfix-branches.yml): manages a WIP branch based on master and any hotfix branches\n\n### per-root-deployment\n\n* [cf-apps-pipeline](concourse/pipelines/templates/cf-apps-pipeline.yml.erb): manages cf-apps associated to a root-deployment\n* [bosh-pipeline](concourse/pipelines/templates/bosh-pipeline.yml.erb): manages bosh deployments associated to a root-deployment\n* [update-pipeline](concourse/pipelines/templates/update-pipeline.yml.erb): generates updated pipelines related to a root-deployment\n* [news-pipeline](concourse/pipelines/templates/news-pipeline.yml.erb): notifies of new bosh release versions for a root-deployment\n\nThe following diagram illustrates the sequence of pipeline generation and loading\n\n![Overview of pipeline generation for bosh deployments](http://www.plantuml.com/plantuml/proxy?src=https://raw.githubusercontent.com/orange-cloudfoundry/cf-ops-automation/master/docs/bootstrap.puml?lastRefreshOn=29092017).\n\n## Template engine reference documentation\n\nThis section details the format supported by the templating engine in both the template repo and the secrets repo.\n\n### Multi deployer support\n\nUsing this feature for a deployment to aggregate several operation to multiple \"deployer\". Currently we support:\n\n* bosh\n* concourse pipeline\n\nThere is no dependencies between deployer, it is processed independently.\n\n#### Bosh deployer\n\nFiles included in `template` dir, in a deployment, are used by `bosh-deployer`\n\n#### Concourse pipeline deployer\n\nFiles included in `concourse-pipeline-config` dir, in a deployment, are used by `concourse-deployer`\n\nThe base concourse file name should named like the directory and end with `.yml`\n\nBy default, this pipeline is not enabled, you need to activate it.\n\n### Resource lifecycle overview\n\nThe diagram below illustrates the concourse pipeline generation for two types of supported resources (Bosh deployments and CF apps). The diagram includes the main hooks that templating\n engine supports during the resources life cycle.\n\n![script lifecycle overview](http://www.plantuml.com/plantuml/proxy?src=https://raw.githubusercontent.com/orange-cloudfoundry/cf-ops-automation/master/docs/script-lifecycle-overview.puml?lastRefreshOn=17082017).\n\n### Bosh deployment resources template format\n\nFor each boshrelease, when an `enable-deployment.yml` file is found in the secrets repo, it is going to spruce all template files in the corresponding template repo dir\n (template files need to end with ```-tpl.yml``` extension).\n\nIf a template directory contains hook scripts with specific name, then these scripts get executed in the following order :\n\n  1: `post-generate.sh`: can execute shell operation or spruce task.\n     **Restrictions**: as the post-generation script is executed in the same docker image running spruce, no spiff is available.\n     Environment variables available:\n\n      * GENERATE_DIR: directory holding generated files. It's an absolute path.\n      * BASE_TEMPLATE_DIR: directory where `post-generate.sh` is located. It's an relative path.\n\n  2: `pre-deploy.sh`: can execute shell operation (bosh, credhub, cf and spruce).\n     **Legacy support**: script named `pre-bosh-deploy.sh` are still supported\n     Environment variables available:\n\n      * GENERATE_DIR: directory holding generated files. It's an absolute path.\n      * BASE_TEMPLATE_DIR: directory where `pre-deploy.sh` is located. It's an relative path.\n      * SECRETS_DIR: directory holding secrets related to current deployment. It's an relative path.\n\n  3: `post-deploy.sh`: can execute shell operation (including curl).\n     **Legacy support**: script named `post-bosh-deploy.sh` are still supported\n     Environment variables available:\n\n      * GENERATE_DIR: directory holding generated files. It's an absolute path.\n      * BASE_TEMPLATE_DIR: directory where `post-deploy.sh` is located. It's an relative path.\n      * SECRETS_DIR: directory holding secrets related to current deployment. It's an relative path.\n\n* to generate an additional errand step, in a `deployment-dependencies.yml` file, insert a key ```errands``` with a subkey named like the errand job to execute\n\n### Iaas specifics support\n\nManifest generation supports dedicated part to\nAdd directories (like openstack, cloudstack, etc...) for each specific iaas, in the template directory.\nSet a `iaas-type` credential in secrets repo to match the directory name.\n\n#### Terraform specific\n\nAs all spec subdirectories are processed by terraform, it is not possible to use the same convention. So to support\n`iaas-type` with terraform, a directory called `spec-\u003ciaas-type\u003e` is required.\nOther [terraform](#terraform) mechanisms applies.\n\n### Bosh cli v2 specific features support\n\nThe newest bosh feature are not implemented in bosh cli v1. So some feature are only available to deployments using bosh cli v2.\nThis can be combined with iaas specifics support\n\n#### ops-files\n\nBy convention, all files in template dir matching `*-operators.yml` are used by `bosh-deployment`\nas ```ops-files``` inputs. Theses files **are not processed by spruce**.\n\n#### vars-files\n\nBy convention, all files in template dir matching `*-vars-tpl.yml` are processed by spruce and generate `*-vars.yml` files.\nAs spruce is no longer required, it is also possible to include vars files directly, files matching `*-vars.yml` are used by `bosh-deployment`  but ignored by spruce.\nTheses files are used by ```bosh-deployment``` as ```vars-files``` inputs.\n\n**Warning**: if there is a naming conflict between `*-vars-tpl.yml` and `*-vars.yml`, the `tpl` wins !\n\n#### Cloud and runtime config\n\nRules for ops-files and vars-files here.\nTo support operators and vars files for cloud and runtime config, we have to define addition convention, as there are in the same directory.\n\n* ```*cloud-operators.yml```: operators for cloud-config\n* ```*cloud-vars.yml```: vars for cloud-config\n* ```*runtime-operators.yml```: operators for runtime-config\n* ```*runtime-vars.yml```: vars for runtime-config\n\n**Migration v1 to v2 tips**: empty vars-files and ops-files are generated to avoid an error message\n\n### git submodules\n\nBy default, git submodules are not checked out (this can be very time consuming). But some bosh releases require these\n  submodule. There is a mechanism to detect submodule for a release and include it only for this bosh release\n\n#### enable deployment format (enable-deployment.yml)\n\nthis is expected to be an empty yaml file !\n\n#### deployment dependencies format (deployment-dependencies.yml)\n\nIn `deployment-dependencies.yml`, it is possible to:\n\n* add secrets path to trigger the build:\n\n```yaml\n        resources:\n          secrets:\n            extented_scan_path: [\"ops-depls/cloudfoundry\", \"....\"]\n```\n\nFollowing is a `deployment-dependencies.yml` sample (should be placed in the boshrelease deployment dir):\n\n```yaml\n\n---\ndeployment:\n  bosh-deployment: # or micro-bosh:\n    releases:\n      route-registrar-boshrelease:\n        base_location: https://bosh.io/d/github.com/\n        repository: cloudfoundry-community/route-registrar-boshrelease\n      shield:\n        base_location: https://bosh.io/d/github.com/\n        repository: starkandwayne/shield-boshrelease\n      xxx_boshrelease:\n        base_location: https://bosh.io/d/github.com/\n        repository: xxx/yyyy\n    errands:\n      smoke_tests:\n```\n\nStaring with COA [2.2](https://github.com/orange-cloudfoundry/cf-ops-automation/releases/tag/v2.2.0), it is possible to use\na generic key (`bosh-deployment`) or `\u003cdeployment-name\u003e` (ie: the directory name) under `deployment`\n\n### Cloudfoundry application resources template format\n\nFor each cf-application, when a `enable-cf-app.yml` file is found, it is going to spruce all files in the template dir ending with ```-tpl.yml```\n\nIf a template directory contains a `pre-cf-push.sh` file, then this script is run:\n\n* you are already logged in CF,\n* you have to download your binaries before uploading to CF\n\nEnvironment variables available:\n\n* GENERATE_DIR: directory holding generated files. It's an absolute path.\n* BASE_TEMPLATE_DIR: directory where `pre-cf-push.sh` is located. It's an relative path.\n* SECRETS_DIR: directory holding secrets related to current deployment. It's an relative path.\n* CF_API_URL: current application Cloud Foundry API url\n* CF_USERNAME: current Cloud Foundry application user\n* CF_PASSWORD: current Cloud Foundry application user password\n* CF_ORG: current Cloud Foundry application organization\n* CF_SPACE: current Cloud Foundry application space\n\nIt is also possible to use a `post-deploy.sh`, it is like a 'post-cf-push' script with an inconsistent name (we reuse the same concourse task)...\n To interact easily with cloudfoundry, previously listed environment variables are available.\n\n#### `enable-cf-app.yml` file format\n\n```yaml\n\n---\ncf-app:\n  probe-apps-domains:\n    cf_api_url:\n    cf_username:\n    cf_password:\n    cf_organization:\n    cf_space:\n```\n\n### pipeline auto-update\n\nIf a ci_deployments descriptor (i.e. a file called `ci-deployment-overview.yml`) is detected in `secrets_dir/\u003cdepls\u003e`, then an\nauto-update job is generated.\n\n### Concourse team\n\nBy default all pipelines deploy into `main` team. But it is possible to add a `team` key to specify another team. See File format bellow.\n\n**WARNING**: `bootstrap` or `*-init` pipelines must belong to `main` team\n\nPre-requisite: team to deploy must exist\n\n### terraform\n\n`ci-deployment-overview.yml` may include a `terraform_config` key to generate a terraform  pipeline.The `terraform_config` key\n must include a `state_file_path` key to indicate tfstate file path. It assumes that a spec dir is also included alongside\n the tfstate file.\n\n#### file format\n\n``` yaml\n\n---\nci-deployment:\n  ops-depls:\n    target_name: concourse-ops\n    terraform_config:\n      state_file_path: ops-depls/tf-config-dir\n    pipelines:\n      ops-depls-generated:\n        team: bootstrap #optional - Default: main\n        config_file: xxxx/pipelines/ops-depls-generated.yml\n        vars_files:\n        - xxx/pipelines/credentials-ops-depls-pipeline.yml\n        - xxx/root-deployment.yml\n      ops-depls-cf-apps-generated:\n        config_file: xxx/pipelines/ops-depls-cf-apps-generated.yml\n        vars_files:\n        - xxx/pipelines/credentials-ops-depls-pipeline.yml\n        - xxx/root-deployment.yml\n```\n\n### delete lifecycle support\n\n#### bosh deployment\n\nIt `scans secrets/\u003croot_deployment\u003e` for directories. If a `enable-deployment.yml` is found, deployment status is set\nto `enabled`, otherwise to `disabled`.\n`disabled` deployment are going to be candidates for deletion.\n\n#### cf-app deployment\n\nNYI\n\n### shared and private configuration\n\nwe provide a new config mechanism shared across all root deployments. A [default](lib/config.rb#L11) will be provided by cf-ops-automation,\nbut it is possible to override these values with a `shared-config.yml` file located in paas-template root directory. It\nalso possible to override again with a `private-config.yml` file located in secrets root directory.\n\n### \"paas-templates\" and \"secrets\" repo structure examples\n\nIn [/docs/reference_dataset](/docs/reference_dataset), you find a set of\nMarkdown files describing structure examples for the repos, links to example\nfiles as well as the lists of credentials needed by the generated pipelines to\nbe deployed.\n\nThose files are generated automatically following the specs given in\n[features/](features).\n\n### Stemcell management\n\nIn order to be portable across multiple infrastructures, and allow for centralized stemcell upgrades, deployment authors\n rely on COA to manage stemcells including offline replication, uploading to bosh director and purge, and stemcell\n version selection at deployment time. Deployment authors therefore use the following syntax in their deployment manifests:\n\n```yml\nstemcells:\n- alias: trusty\n  os: ubuntu-trusty\n  version: latest\n```\n\nAt deployment time, the deployment manifest is transformed by COA to load the expected stemcell version (say `3586.25`).\n\nThe same stemcell generation (currrently `ubuntu-trusty`) is used for all deployments as defined by authors in\n [shared-config.yml](docs/reference_dataset/template_repository/shared-config.yml)\nor overloaded by operators in [private-config.yml](docs/reference_dataset/config_repository/private-config.yml).\n\nThe exact stemcell version (say `3586.25`) is used within a root-deployment as defined by authors in\n[root-deployment.yml](docs/reference_dataset/template_repository/hello-world-root-depls/root-deployment.yml)\n\n# COA development\n\n## Status and roadmap\n\nSee [github issues](https://github.com/orange-cloudfoundry/cf-ops-automation/issues).\n\n## Interns\n\n[Readme.md](docs/Readme.md)\n\n## Running the Test Suite\n\nPrereqs:\n\n* install ruby, gems and bundler (version \u003e= 1.15.1):\n  * `gem install bundler`\n* install dependent gems: `bundle install --path vendor/bundle`\n\nIf you are running the full test suite, some of the integration tests are dependent on the fly CLI.\n\nTo login to the Fly CLI and target the cf-ops-automation CI:\n\n```sh\nfly -t cf-ops-automation login\n```\n\nYou will be prompted to select either the Github, UAA or Basic Auth authentication methods.\n\nAfter these are set up, you will be able to run the test suite via:\n\n```sh\nbundler exec rspec\n```\n\n## Generating pipelines locally and uploading a test version\n\nWhile developing new pipelines, it might be easier to generate them locally and upload them manually to a concourse instance\n\n```bash\nfly -t preprod login -u login -p password -c concourse-url\n./scripts/generate-depls.rb --depls cloudflare-depls -t ../paas-template/ -p ../bosh-cloudwatt-preprod-secrets/ --no-dump -i ./concourse/pipelines/template/tf-pipeline.yml.erb --iaas openstack\nSECRETS=../bosh-cloudwatt-preprod-secrets/ TARGET_NAME=preprod ./scripts/concourse-manual-pipelines-update.rb -dcloudflare-depls\n```\n\nOnce pipelines are correct, commit, pipelines would perform automated deployment, see [scripts/concourse-generate-all-pipelines.sh](scripts/concourse-generate-all-pipelines.sh)\n\n## Local terraform development\n\nIn order to leverage IDE capabilities for terraform when editing TF config files (completion, syntax highlighting, etc.)\n, a consistent local environment is required, with spruce templates interpolated,\nand tf config files merged from template and secrets repo.\n\nThis also enables local execution for `terraform plan` and `tf apply` providing shorter feedback cycles.\n\nThe `scripts/setUpTfDevEnv.sh` script partially automates the set up of such local environment:\n\n```bash\nsource scripts/setUpTfDevEnv.sh\n```\n\n## Submitting Pull Requests\n\n* All pull requests should be made against the `develop` branch. Only Pull Requests based on this branch trigger automated builds.\n\n## Releasing COA\n\n### Standard release\n\nUse cf-ops-automation pipeline to perform a release. You may need to bump the version using one of the following jobs: `major`, `minor` or `patch`.\nOnce the version is ok, simply launch `ship-it` job\n\n### Hotfix release\n\nThis type of release requires manual work.\n\n  1. checkouts the hotfix branch, and sets it to the expected tag\n  1. fixes the issue\n  1. adds release description files :\n     1. `hotfix.version`: add the expected release version. Format is `\u003cmajor\u003e.\u003cminor\u003e.\u003cpatch\u003e`, e.g. 1.1.0 or 1.7.2\n     1. `hotfix_release_notes.md`: the release note to publish on github\n  1. commits and push\n  1. ensures `run-tests-for-hotfix-branch` is successful\n  1. triggers `ship-hotfix` to publish the release on github\n\n## Bootstrapping a COA env\n\n### How to use it\n\nIn order to quickly create an environment in which you can use the COA engine,\nyou can use the \"bootstrap_coa_env.rb\" script. By running\n`ruby scripts/bootstrap_coa_env.rb /path/to/prereqs1.yml /path/to/prereqs2.yml ... /path/to/prereqsn.yml`\nwhere the prereqs YAML are files containing configuration information for the\nbootstrapping, pipelines will be created from the reference dataset data.\n\n### Prerequisites\n\nThe prerequisites YAML files are expected to contain some information that will\nhelp the script to build the environment. You can write it all in a single file\nor in multiple files. An example file can be found at [/lib/coa_env_bootstrapper/prereqs.example.yml](/lib/coa_env_bootstrapper/prereqs.example.yml).\n\nIt can contain up to 8 main keys:\n\n* inactive_steps, _optional_: pass a list of steps that will be deactivated in case you wouldn't need them to run, for instance in case you have some resources already installed. You can deactivate:\n    * deploy_transient_infra: you can deactivate this step if you already have an infrastructure with BOSH and Concourse\n    * upload_stemcell: you can deactivate this step if you don't want a new stemcell to be uploaded to the BOSH Director\n    * upload_cloud_config: you can deactivate this step if you don't want to overwrite the cloud config of the BOSH Director\n    * deploy_git_server: you can deactivate this step if you have the git-server deployment already deployed on the BOSH Director\n* bucc, _optional_: you have to pass this key unless you deactivate the deploy_transient_infra step:\n    * bin_path: the path to your installation of the bucc project. The project can be found at [https://github.com/starkandwayne/bucc](https://github.com/starkandwayne/bucc)\n    * cpi: the cpi you want to use for the deployment. The list of existing CPIs can be found on the bucc GitHub project\n    * cpi_specific_options: options you want to pass to the bucc CLI for the bucc deployment\n* stemcell, _optional_: the stemcell that will be uploaded to the BOSH director and that will be sued to deploy the git server. You can find the stemcell for your usecase on [https://bosh.io/stemcells/](https://bosh.io/stemcells/)\n    * name\n    * version\n    * uri\n    * sha\n* git_server_manifest, _mandatory_:  a BOSH manifest to deploy the git-server deployment. The example one can be used to be deployed with VirtualBox, in any other case, you will want to adapt it for your IaaS.\n* cloud_config, _optional_: you have to pass this key unless you deactivate the \"upload_cloud_config\" step. This will be used by the BOSH CLI to upload a cloud-config to the BOSH Director\n* pipeline_credentials, _mandatory_: this passes a list of credentials that will by used by fly to upload the pipelines to Concourse.\n* concourse, _optional_: you have to pass this object if you deactivated the deploy_transient_infra step or if you want to overwrite the BUCC Concourse and want to use another one. The options are self-explanatory.\n    * target\n    * url\n    * username\n    * password\n* bosh, _optional_: you have to pass this key if you deactivate the deploy_transient_infra step or if you want to overwrite BUCC's BOSH and want to use another one. The options are self-explanatory.\n    * bosh_environment\n    * bosh_client\n    * bosh_client_secret\n    * bosh_ca_cert\n\n### Connecting to Concourse\n\nOnce the script is done running, it displays information about how to connect to\nthe Concourse it has installed. If you wish to display those information, you\ncan run `bucc info`.\n\n### Known issues\n\n#### VMs access issues\n\nIf you're using VirtualBox as a IaaS on OS X, you may have trouble connectiong\nto the VMs installed by BUCC's BOSH. For instance, when the script is trying to\npush the config repository to the Git server it had installed. In this case, run\nthe `bucc routes` command to create the proper routes and enable communication\nto the VMs.\n\n#### Stemcell loading creates a timeout\n\nSome stemcells are very large and here we're downloading it manually which can\ntake a lot of time if the script is downloading it from the internet. This can\nlead to some timeouts. To prevent this, you can manually upload the stemcell to\nthe BOSH Director and desctivate the *upload_stemcell* step.\n\n#### Some Concourse resource won't load\n\nIf you're observing a Concourse error saying\n`pq: insert or update on table \"worker_resource_config_check_sessions\" violates foreign key constraint \"worker_resource_config_check__resource_config_check_sessio_fkey\"`,\nit should resolve itself in a matter of seconds.\n\nThere is another error where GitHub resources as well as Docker images won't\nload. In this case, it was sufficient to restart the VirtualBox image.\n\n# FAQ\n\n## How to initialize a new bosh deployment template ?\n\nrun `./init-template.sh`, and it creates empty placeholder.\n\n## How to enable a bosh deployment template ?\n\n`deployment-dependencies.yml` sample:\n\n```yaml\n\n---\ndeployment:\n  bosh-deployment: #or micro-bosh:\n    releases:\n      route-registrar-boshrelease:\n        base_location: https://bosh.io/d/github.com/\n        repository: cloudfoundry-community/route-registrar-boshrelease\n      shield:\n        base_location: https://bosh.io/d/github.com/\n        repository: starkandwayne/shield-boshrelease\n      xxx_boshrelease:\n        base_location: https://bosh.io/d/github.com/\n        repository: xxx/yyyy\n    errands:\n      smoke_tests:\n```\n\n## How to upload a bosh release not available on bosh.io?\n\nuse `deploy.sh` script like [this](docs/reference_dataset/template_repository/hello-world-root-depls/template/deploy.sh) to manually upload release.\n`deploy.sh` use bosh cli v2 syntax.\n\n## How to generate a tfvars in json from a yaml template?\n\nYou can use spruce embedded with post-generate.sh to do it !\nSee [post-generate.sh](micro-depls/terraform-config/template/post-generate.sh) script t\n\n### Sample\n\nThere is no yet public template sample. Orange employees can have a look to the [post-generate.sh](micro-depls/terraform-config/template/post-generate.sh) in the private paas-template repo.\n\n## How to bootstrap pipelines to a new concourse?\n\nsimply run [concourse-bootstrap.sh](scripts/concourse-bootstrap.sh) with the appropriate environment variable set. It loads the\n`bootstrap-all-init-pipelines` pipeline and triggers it.\n\nThis script must also be run when git is updated (branch or url)\n\n```sh\nSECRETS=\u003cpath_to_your_secret_dir\u003e FLY_TARGET=\u003cyour_target\u003e ./concourse-bootstrap.sh\n```\n\n### pre requisite\n\nThe following tools are required to run [concourse-bootstrap.sh](scripts/concourse-bootstrap.sh)\n\n* git\n* fly, the concourse CLI\n  * Login to concourse in main team\n\n## How to create a new root deployment?\n\nTo setup a new paas-template repo, a new secrets repo or to add a new root deployment, you can run\n[create-root-depls](scripts/create-root-depls.rb) script to create empty files.\n\n### pre requisite\n\nThe following tools are required to run [create-root-depls](scripts/create-root-depls.rb)\n\n* ruby\n\n# Credits\n\nThis repo was inspired by great work shared in:\n\n* [alphagov/paas-cf](https://github.com/alphagov/paas-cf)\n* [cloudfoundry/buildpacks-ci](https://github.com/cloudfoundry/buildpacks-ci/)\n\n# Changelog\n\nSee [CHANGELOG.md](CHANGELOG.md)\n\n# Upgrade\n\nLook into [upgrade](upgrade) directory, and run required scripts from Cf-Ops-Automation root directory to benefit\nfrom default values\n\n## pre requisite\n\nThe following tools are required to run upgrade scripts:\n\n* ruby\n* git\n\nIt is also required to have a paas-templates repository clone and/or a config repository clone to be able to perform\nupgrade operations.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Fcf-ops-automation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forange-cloudfoundry%2Fcf-ops-automation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Fcf-ops-automation/lists"}