{"id":28395129,"url":"https://github.com/databiosphere/terra-resource-buffer","last_synced_at":"2026-02-26T03:54:29.455Z","repository":{"id":40483138,"uuid":"297686383","full_name":"DataBiosphere/terra-resource-buffer","owner":"DataBiosphere","description":"Terra Resource Buffering Service","archived":false,"fork":false,"pushed_at":"2025-06-16T19:38:48.000Z","size":1167,"stargazers_count":4,"open_issues_count":2,"forks_count":2,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-06-16T19:47:47.641Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-09-22T15:08:33.000Z","updated_at":"2025-06-05T15:33:09.000Z","dependencies_parsed_at":"2023-11-10T18:23:46.178Z","dependency_job_id":"4d784d6c-1f16-47c1-9c55-a9fa9920b79a","html_url":"https://github.com/DataBiosphere/terra-resource-buffer","commit_stats":null,"previous_names":[],"tags_count":337,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/terra-resource-buffer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-buffer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-buffer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-buffer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-buffer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/terra-resource-buffer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-buffer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172389,"owners_count":23269996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:14.958Z","updated_at":"2026-02-26T03:54:29.448Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Terra Resource Buffering Server\nCloud Resource Buffering Server for Terra.\n\nRBS Cheat sheet:\n- RBS (server) creates cloud resources based on config provided by clients. Resource details are stored in a database. \n- Common clients of RBS: [Workspace Manager](https://github.com/DataBiosphere/terra-workspace-manager), [Terra Data Repo](https://github.com/DataBiosphere/jade-data-repo), [Rawls](https://github.com/broadinstitute/rawls)\n- Local testing: may use a local Postgres DB (native installation) or within Docker. Resource specifics are also notified to Janitor service by writing a pub-sub message if it is to be auto-deleted\n- [Buffer Service envs](https://docs.google.com/document/d/1FyYpASNoKW2iY1IIuRmWpRL6-ST06_n5iYg0m_3LP-c/)\n- [DSP playbook](https://docs.google.com/document/d/1fo5Q92sJe-8BZNRPuI2GtvrUwWqHKDY4UouD86KtVcM)\n\n## Static (Build Time) Pool Configuration\n### File Structure\nPool configuration manages the pool size, and configuration of resources in the pool. All static configuration files are under [src/main/resources/config](src/main/resources/config) folder.  These static configurations are built into the Resource Buffer service at compile time.\nThe folder structure is:\n```\n-{env}\n    - pool_schema.yml\n    - resource-config\n        - resource_config.yml\n- resource_schema.yaml\n```\n* `{env}` is the static configuration folder Buffer service will use. Set `BUFFER_POOL_CONFIG_PATH=config/{env}` as environment variable to change folder to use.\nIn Broad deployment, the value can be found at [Broad helmfile repo](https://github.com/broadinstitute/terra-helmfile/blob/326c7f220c4d3ef5466b12e8a6f8b5fa3f255ec0/values/app/buffer/live/dev.yaml#L27)\n* `resource_schema.yaml` is the resource config template\n* `pool_schema.yml` lists all pools under that environment. It includes the pool size and resource config to use for that pool.\n* `resource-config` folder contains all resource configs all pools are using or used before.\n\n### Upgrade Static Pool Configuration\nWhen using static pools, a configuration update is required to build a new docker image and redeploy the server.\n\nTo update pool size, just update the pool size in the configuration file.\n\nTo update resource configs, it is the same process as creating a new pool using a new resource config. The recommended process is:\n1. Add a new resource config and a new pool in configuration file.\n2. Wait for next Buffer Service release, and it will create resources using the new config.\n3. Client switch to use the new pool id when ready.\n4. Remove the old pool from `pool_schema.yml` and delete old resource config(optional).\n5. Next Buffer Service release will delete resoruces in the old pool\n\n## Runtime Pool Configuration\nSince version [`0.176.0`](https://github.com/DataBiosphere/terra-resource-buffer/releases/tag/0.176.0), Resource Buffer has supported runtime pool configuration.  Instead of requiring the use of a static pool configuration created at build time (and thus requiring a new version of the Resource Buffer service container to be built and published), runtime pool configuration allows for the specification of pool configurations at service runtime, read from a directory on the local filesystem.\n\nIn order to use a runtime pool configuration, the environment variable `BUFFER_POOL_SYSTEM_FILE_PATH` should point at a directory on the local file system in the RBS container which stores the pool configuration.  *This will override the use of any static pool configurations built into the Resource Buffer Jarfile.*\n\n### Runtime Pool Example\n* `BUFFER_POOL_SYSTEM_FILE_PATH` is set to `/etc/config/staging`\n* Local file `/etc/config/staging/pool_schema.yaml` contains two pools,\n  ```\n  # RBS Pools Schema for staging environment\n  ---\n  poolConfigs:\n    - poolId: \"resource_staging_v1\"\n      size: 100\n      resourceConfigName: \"resource_staging_v1\"\n  poolConfigs:\n    - poolId: \"resource_staging_v2\"\n      size: 100\n      resourceConfigName: \"resource_staging_v2\"\n  ```\n* Local file `/etc/config/staging/resource_config/resource_staging_v1.yaml` contains the configuration for pool `resource_staging_v1`\n* Local file `/etc/config/staging/resource_config/resource_staging_v2.yaml` contains the configuration for pool `resource_staging_v2`\n\n### Notes\n* Pool configurations are only read at Resource Buffer start time, so changes to files in the configuration directory will require the service to be restarted for these configurations to take effect.\n* The same rules around pool modification and deletion apply for dynamically configured pools as do for static pools:\n  * For existing pools, pool sizes may change, but resource configurations may not.\n  * If a previously created pool does not exist under `BUFFER_POOL_SYSTEM_FILE_PATH`, it will be deleted.\n\n## GitHub Interactions\n\nWe currently have these workflows:\n\nWorkflow      | Triggers                  | Work\n--------------|---------------------------|-------\n_master_push_ | on PR merge to master     | tags, version bumps, publishes client to artifactory, pushes image to GCR\n_test-runner-nightly-perf_ | nightly at 1am ET         | runs the TestRunner-based integration test suite\n_test_ | on PR and merge to master | runs the unit and integration tests\n_trivy_ | on PR                     | Broad app-security scan on base Docker image\n\n## Development, testing \u0026 deployment\n\n### Authenticating to Dev Buffer Service\n[Dev Buffer Service Swagger](https://buffer.dsde-dev.broadinstitute.org/swagger-ui.html)\n\nIn Broad deployment, use a valid Google Service Account(created by [Terraform](https://github.com/broadinstitute/terraform-ap-modules/blob/master/buffer/sa.tf#L83)) is required for service authorization. This can be retrieved in Vault.\nTo get client service account access token:\n\nStep 1:\n```\nexport DEV_SERVICE_ACCOUNT_PATH=./src/test/resources/rendered/dev-buffer-client-sa.json\ngcloud secrets versions access latest --secret=buffer-client-sa-b64 --project=broad-dsde-dev | jq -r '.key' | base64 -d \u003e ${DEV_SERVICE_ACCOUNT_PATH}\n```\nStep2:\n```\ngcloud auth activate-service-account --key-file=${DEV_SERVICE_ACCOUNT_PATH}\n```\nStep3:\n```\ngcloud auth print-access-token\n```\n\nTo access Buffer Service in other environment, lookup for `vault.pathPrefix` in [helmfile repo](https://github.com/broadinstitute/terra-helmfile/tree/master/values/app/buffer) to find the correct vault path.\n\n### Configs Rendering\nLocal Testing and Github Action tests require credentials to be able to call GCP, run\n``` local-dev/render-config.sh``` first for local testing. It generates:\n* A Google Service Account Secret to create/delete cloud resources in test.\n* A Google Service Account Secret to publish message to Janitor instance.\n\n### Run Locally\n\nUse JDK 17, [instructions](https://github.com/DataBiosphere/terra-workspace-manager/blob/main/DEVELOPMENT.md#jdk)\n\nSet executable permissions:\n```\nchmod +x gradlew\n```\n\nTo spin up the local postgres (within docker), run:\n```\nlocal-dev/run_postgres.sh start\n```\nStart local server. Environment variables are added into this file\n```\nlocal-dev/run_local.sh\n```\n\nThis starts a local instance of the Resource Buffer Service. Check the status of service on your browser\n```\nhttp://localhost:8080/status\n```\n\nNext, go to the Cloud Console to find the cloud project folder, identified by the parentFolderId in resource_schema.yml.\nPool schema to be used is set via environment variable `BUFFER_POOL_CONFIG_PATH`\n\nThe name of the new Google Project created by the local RBS instance above in found in console logs, search for the same on Google cloud console\n\n### Deploy to GKE cluster:\nThe provided setup script clones the terra-helm and terra-helmfile git repos,\nand templates in the desired Terra environment/k8s namespace to target.\nIf you need to pull changes to either terra-helm or terra-helmfile, rerun this script.\n\nTo use this, first ensure Skaffold is installed on your local machine\n(available at https://skaffold.dev/).\n\n\u003e Older versions of Skaffold (v1.4.0 and earlier) do not have support for Helm 3 and will fail to deploy your\nchanges. If you're seeing errors like `UPGRADE FAILED: \"(Release name)\" has no\ndeployed releases`, try updating Skaffold.\n\nYou may need to use gcloud to provide GCR credentials with `gcloud auth configure-docker`.\nAlternately use `gcloud auth login \u003cyou\u003e@broadinstitute.org` or `gcloud auth application-default login` for fine grain control of the login credentials to be used (account must have permissions to push to GKE)\n```\ncd local-dev\n```\nThe script itself permits pushing to any valid environment as the target. Developers however only have access to push to their [personal environment](https://github.com/DataBiosphere/terra/blob/main/docs/dev-guides/personal-environments.md)\n\n```\n./setup_gke_deploy.sh \u003cenvironment\u003e\n```\nYou can now push to the specified environment by running\n```\nskaffold run\n```\n\n### Connecting psql client using the Cloud SQL Proxy:\nFollow [Installing this instruction](https://cloud.google.com/sql/docs/mysql/sql-proxy#macos-64-bit)\nto install Cloud SQL Proxy. This is used to connect to the remote SQL database used by RBS\n\nStep 1: Go to cloud console to get the instance name you want to connect to, then start the proxy. If connecting to the dev instance, first run:\n```\ngcloud auth login \u003cbroad email\u003e\n```\nThen run the Cloud SQL Proxy with the instance connection name. For example, to connect to Dev Buffer Database, run:\n```\ncloud-sql-proxy $INSTANCE_CONNECTION_NAME -g\n```\nStep 2: Then set database password as `PGPASSWORD`:\n```\nexport PGPASSWORD={BUFFER_DB_PASSWORD}\n```\nStep 3.1: To connect to Buffer Database, run:\n```\npsql \"host=127.0.0.1 sslmode=disable dbname=buffer user=buffer\"\n```\nStep 3.2: To connect to Buffer Stairway Database, run:\n```\npsql \"host=127.0.0.1 sslmode=disable dbname=buffer-stairway user=buffer-stairway\"\n```\n#### Connect to Broad Deployment Buffer Database\nFor Broad engineer, the INSTANCE_CONNECTION_NAME and BUFFER_DB_PASSWORD can be found in gsm. For example, to connect to Dev Buffer Database, run:\n```\nexport DB_REGION=$(gcloud secrets versions access latest --secret=buffer-postgres-instance --project=broad-dsde-dev | jq -r \".region\")\nexport DB_INSTANCE=$(gcloud secrets versions access latest --secret=buffer-postgres-instance --project=broad-dsde-dev | jq -r \".name\")\nexport INSTANCE_CONNECTION_NAME=\"broad-dsde-dev:${DB_REGION}:${DB_INSTANCE}\"\nexport PGPASSWORD=$(gcloud secrets versions access latest --secret=buffer-postgres-creds --project=broad-dsde-dev | jq -r \".password\")\n```\n\nThen follow the steps above to connect to the database.\n\nNote that you must stop the local postgres first to free the 5432 port.\nSee [this document](https://cloud.google.com/sql/docs/postgres/connect-admin-proxy) for more details.\n\n### Jacoco\nWe use [Jacoco](https://www.eclemma.org/jacoco/) as code coverage library\n\n## SourceClear\n\n[SourceClear](https://srcclr.github.io) is a static analysis tool that scans a project's Java\ndependencies for known vulnerabilities. If you are working on addressing dependency vulnerabilities\nin response to a SourceClear finding, you may want to run a scan off of a feature branch and/or local code.\n\n### Github Action\n\nYou can trigger RBS's SCA scan on demand via its\n[Github Action](https://github.com/broadinstitute/dsp-appsec-sourceclear-github-actions/actions/workflows/z-manual-terra-resource-buffer.yml),\nand optionally specify a Github ref (branch, tag, or SHA) to check out from the repo to scan.  By default,\nthe scan is run off of RBS's `master` branch.\n\nHigh-level results are outputted in the Github Actions run.\n\n### Running Locally\n\nYou will need to get the API token from Vault before running the Gradle `srcclr` task.\n\n```sh\nexport SRCCLR_API_TOKEN=$(vault read -field=api_token secret/secops/ci/srcclr/gradle-agent)\n./gradlew srcclr\n```\n\nHigh-level results are outputted to the terminal.\n\n### Veracode\n\nFull results including dependency graphs are uploaded to\n[Veracode](https://sca.analysiscenter.veracode.com/workspaces/jppForw/projects/544768/issues)\n(if running off of a feature branch, navigate to Project Details \u003e Selected Branch \u003e Change to select your feature branch).\nYou can request a Veracode account to view full results from #dsp-infosec-champions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-resource-buffer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fterra-resource-buffer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-resource-buffer/lists"}