{"id":26059749,"url":"https://github.com/orange-cloudfoundry/cf-ops-automation-broker","last_synced_at":"2025-04-11T06:50:36.403Z","repository":{"id":26264103,"uuid":"103646540","full_name":"orange-cloudfoundry/cf-ops-automation-broker","owner":"orange-cloudfoundry","description":"On demand dedicated services through COA concourse pipeline engine","archived":false,"fork":false,"pushed_at":"2024-02-14T15:39:12.000Z","size":5912,"stargazers_count":4,"open_issues_count":62,"forks_count":0,"subscribers_count":6,"default_branch":"develop","last_synced_at":"2025-03-25T04:41:32.410Z","etag":null,"topics":["bosh","cloudfoundry","concourse","osbapi","service-broker"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orange-cloudfoundry.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-15T10:47:15.000Z","updated_at":"2022-01-03T17:28:14.000Z","dependencies_parsed_at":"2023-02-12T20:15:31.587Z","dependency_job_id":null,"html_url":"https://github.com/orange-cloudfoundry/cf-ops-automation-broker","commit_stats":null,"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation-broker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation-broker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation-broker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Fcf-ops-automation-broker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orange-cloudfoundry","download_url":"https://codeload.github.com/orange-cloudfoundry/cf-ops-automation-broker/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248358548,"owners_count":21090401,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bosh","cloudfoundry","concourse","osbapi","service-broker"],"created_at":"2025-03-08T13:28:20.365Z","updated_at":"2025-04-11T06:50:36.381Z","avatar_url":"https://github.com/orange-cloudfoundry.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cf-ops-automation-broker \n\n[![CI](https://circleci.com/gh/orange-cloudfoundry/cf-ops-automation-broker.svg?style=shield\u0026circle-token=:circle-token)](https://circleci.com/gh/orange-cloudfoundry/cf-ops-automation-broker) [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Forange-cloudfoundry%2Fcf-ops-automation-broker.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Forange-cloudfoundry%2Fcf-ops-automation-broker?ref=badge_shield) [![Coverity Scan](https://img.shields.io/coverity/scan/3997.svg)](https://scan.coverity.com/projects/orange-cloudfoundry-cf-ops-automation-broker)\n\nOn demand dedicated services through COA concourse pipeline engine\n\n## Overview\n\nCOAB is a generic service broker which enables operators to provide on-demand dedicated services from existing bosh releases with service brokers offering shared service plans (e.g [cassandra-cf-service-boshrelease](https://github.com/orange-cloudfoundry/cassandra-cf-service-boshrelease/))\n\nCOAB leverages concourse-based pipelines to deploy and operate the dedicated resources, managed by the [cf-ops-automation (COA)](https://github.com/orange-cloudfoundry/cf-ops-automation) collaboration framework.\n\nHere are some slides to provide more background to this project:\n\n[![image](https://user-images.githubusercontent.com/4748380/46364595-85ead380-c676-11e8-947d-2c6b32516a43.png)](https://docs.google.com/presentation/d/1ChrpHQRxwdLzpt4m4sLe2PVJICqP0bCo9B5ffCPv57Q/edit?usp=sharing)\n\nThe following diagram illustrates COAB interactions\n![Overview of COAB interactions](coab-overview.png)\n\n\u003cbr/\u003e 1- A user requests a dedicated service instance (or binding) through its prefered platform (CF or K8S service catalog) which in turn propagate the request to COAB as an [OSB API call](https://github.com/openservicebrokerapi/servicebroker)\n\u003cbr/\u003e 2- COAB requests an on-demand dedicated pipeline to the COA templating engine, by writing to Git repos\n\u003cbr/\u003e 3- COA templating engine reads the git repos \n\u003cbr/\u003e 4- COA generates on-demand pipelines that deploys and operate the requested dedicate service resources\n\u003cbr/\u003e \u003cbr/\u003e 5- The dedicated concourse pipeline provisions resources supporting the dedicated resource (in form of a bosh deployment, a terraform module, or a CF app)\n\u003cbr/\u003e 7- the dedicated concourse pipeline records the outcome of the dedicated service in git\n\u003cbr/\u003e 8- the dedicated concourse pipeline (and its underlying tools such as bosh director) records the credentials necessary to consumme the decicated service instance/binding\n\u003cbr/\u003e 9- COAB pulls the dedicated service provisionning completion status from git\n\u003cbr/\u003e 10a)- COAB delegates OSB API calls to nested service brokers, delegating credentials management to them\n\u003cbr/\u003e 10b)- alternatively COAB fetches dedicated service credentials generated by the pipeline from credhub\n\u003cbr/\u003e 11- COAB returns service instance/bindings to user\n\n\n\n\n\n## Status\n\nThis project is still in a beta state providing POC broker releases for CloudFlare and Cassandra use-cases above. Next step is to generalize the approach.\n\n## Operating COAB\n\nDeploy the broker as a CF app:\n* The CF manifest.yml file would be available in [orange-cloudfoundry/paas-templates](https://github.com/orange-cloudfoundry/paas-templates) repo\n   * The documentation reference for supported and required environment variables is in the [integration tests properties](cf-ops-automation-bosh-broker/src/test/resources/application.properties). \n   * The broker does not require stateful data service to be bound. \n\n### Configuring the service broker catalog\n\nUse `spring.cloud.openservicebroker.catalog` environment variable to set catalog config in a YAML format. See [spring-cloud-open-service-broker](https://docs.spring.io/spring-cloud-open-service-broker/docs/current/reference/html5/#_providing_a_catalog_using_properties) for a sample YML and raw properties configuration\n\n### Git clone pooling\n\nBy default coab perform pooling of git clone, including eager pooling at startup (i.e. prefetching a clone), and asynchronously maintaining the min idle pool size. This is designed to avoid getting the git clone elapsed time as part of the user-facing experience and running on the OSB sync timeout (which defaults to 60s)\n\nPooling may be disabled, through  [GitProperties.java](cf-ops-automation-broker-core/src/main/java/com/orange/oss/cloudfoundry/broker/opsautomation/ondemandbroker/git/GitProperties.java) and eager pool configured in [PoolingProperties.java](cf-ops-automation-broker-core/src/main/java/com/orange/oss/cloudfoundry/broker/opsautomation/ondemandbroker/git/PoolingProperties.java).\n\n### Read-only mode\n\nWhen the service instance read-only mode is configured (see [DeploymentProperties.java](cf-ops-automation-broker-core/src/main/java/com/orange/oss/cloudfoundry/broker/opsautomation/ondemandbroker/pipeline/DeploymentProperties.java)), then service instance operations (create/update/delete) are rejected, while service binding operations (create/delete) are still accepted. This enables to perform maintenance on the underlying COA/git branches infrastructure while not risking corrupted COA inventory and not imposing full control plan downtime to coab users. \n\n### skipDeProvision opt-in\n\nWhen `pipeline.skipDeProvision` is set to `true`, then the inner broker will not receive unprovision calls (i.e. upon `cf delete-service-instance`). Default is `false`\n\nThis is designed to support undeletes by operators until they explicitly approve the deletion the associated COA deployment (and the associated underlying bosh deployment).\n\nSome brokers with highly sensitive data (e.g. implementing data erasure) should prefer to not opt-in for this flag, as this could increase risk of data recovery after the service deprovision phase. \n\n### Disable input validation as a transient workaround\n\nCoab strives to perform strict fail-fast input validation to prevent security-related injections in paas-templates.\n\nWhen the input validation is too restrictive and blocks legitimate use-case, it is possible to transiently disable input validation through setting the `deployment.coabVarsInputValidationDisabled` property to `true` (default is `false`)\n\n### Troubleshooting COAB\n\n#### Spring boot debug mode\n\nEnable [springboot debug/trace mode](https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html#boot-features-logging-console-output) to diagnose beans configuration. When running COAB as a cf app, use the \ncommands below to turn on spring boot debug mode. \n\n\n```bash\ncf set-env my-app DEBUG true\n```\n\nand when finished\n\n```bash\ncf unset-env my-app DEBUG true\n```\n\n#### Increase logging levels\n\nIncrease verbosity for COAB classes:\n- default levels are in [cf-ops-automation-bosh-broker/src/main/resources/application.yml](cf-ops-automation-bosh-broker/src/main/resources/application.yml)\n- see some frequently used levels in [cf-ops-automation-bosh-broker/src/test/resources/application.properties](cf-ops-automation-bosh-broker/src/test/resources/application.properties) and [cf-ops-automation-broker-core/src/test/resources/application.yml](cf-ops-automation-broker-core/src/test/resources/application.yml) \n\n#### Expose more spring actuator endpoints\n\nBy default, COAB only exposed the health and info endpoints and make them accessible without authentication.\n\nFor debug purposes only, expose all actuator endpoints which are disabled by default, see related [spring manual](https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html#production-ready-endpoints-exposing-endpoints) They currently require broker basic auth.\n\n```yml\nmanagement.endpoints.web.exposure.include: \"*\"\n```\n\nSee also, associated [actuator endpoints API doc](https://docs.spring.io/spring-boot/docs/current/actuator-api/html/)\n\n\n## Authoring a new COAB-based service\n\nPrepare a standard COA bosh deployment, see [COA documentation](https://github.com/orange-cloudfoundry/cf-ops-automation#template-engine-reference-documentation)\n\nsuch as \n\n```\n$ tree coab-depls/mongodb\n\ncoab-depls/mongodb\n|-- deployment-dependencies.yml\n`-- template\n    |-- coab-operators.yml\n    |-- add-prometheus-operators.yml\n    |-- add-shield-operators.yml\n    |-- mongodb-vars.yml\n    `-- mongodb.yml\n    \n    \n$ cat coab-depls/mongodb/template/coab-operators.yml \n\n#replace deployment name\n- type: replace\n  path: /name\n  value: ((deployment_name))\n\n#replace external host\n- type: replace\n  path: /instance_groups/name=mongodb-broker/jobs/name=route-registrar/properties/route_registrar/external_host\n  # deployment name is of format model-deployment-short-alias + \"_\" + \u003cservice-instance-id\u003e\n  value: mongodb-broker-((deployment_name)).((!/secrets/cloudfoundry_system_domain))\n\n```\n\nCOAB would generate the following for service instance provisionning/updates:\n\n```\n$tree coab-depls/m_f49911f7-b69a-4aba-afdf-23fd11014278\n\ncoab-depls/m_f49911f7-b69a-4aba-afdf-23fd11014278\n|-- deployment-dependencies.yml -\u003e ../../template/mongodb/deployment-dependencies.yml\n`-- template\n    |-- coab-operators.yml -\u003e ../../mongodb/operators/coab-operators.yml\n    |-- mongodb-vars.yml -\u003e ../../mongodb/template/mongodb-vars.yml\n    |-- m_f49911f7-b69a-4aba-afdf-23fd11014278.yml -\u003e ../../mongodb/template/mongodb.yml\n    `-- coab-vars.yml\n    \n```\n\nThe `coab-vars.yml` file contains the open service broker api standardized input for both service [provisionning](https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#provisioning) and [update](https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#updating-a-service-instance). The following example below shows a dummy example with a user passing arbirary params in a `cf update-service-instance` along with a service plan update. \n\n```\n$cat coab-depls/m_f49911f7-b69a-4aba-afdf-23fd11014278/coab-vars.yml\n\n---\ndeployment_name: \"m_f49911f7-b69a-4aba-afdf-23fd11014278\"\ninstance_id: \"f49911f7-b69a-4aba-afdf-23fd11014278\"\nservice_id: \"mongodb-service_definition_id\"\nplan_id: \"mongodb-plan_guid\"\ncontext:\n  platform: \"cloudfoundry\"\n  user_guid: \"user_guid1\"\n  space_guid: \"space_guid1\"\n  organization_guid: \"org_guid1\"\nparameters:\n  cacheRatio: 0.8642\n  cacheSizeMb: 10\n  slowQuery: false\nprevious_values:\n  plan_id: \"previous_plan_guid\"     \n```\n\nNote: for security reasons, input validation is applied on the name and value of arbitrary params to prevent various injections (such as reading from credhub, spruce, file system injection, yml loading or reference expansions), and only so alphabetical characters, numbers are supported.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) document","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Fcf-ops-automation-broker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forange-cloudfoundry%2Fcf-ops-automation-broker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Fcf-ops-automation-broker/lists"}