{"id":28395183,"url":"https://github.com/databiosphere/terra-resource-janitor","last_synced_at":"2026-02-20T15:31:37.271Z","repository":{"id":37897552,"uuid":"273027394","full_name":"DataBiosphere/terra-resource-janitor","owner":"DataBiosphere","description":"Janitor service to cleanup resources created by Cloud Resource Library (CRL) ","archived":false,"fork":false,"pushed_at":"2025-10-02T14:45:16.000Z","size":745,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-10-30T21:40:25.393Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-06-17T16:38:09.000Z","updated_at":"2025-10-02T14:45:19.000Z","dependencies_parsed_at":"2023-02-14T00:45:54.448Z","dependency_job_id":"620dceff-4a09-4d9d-b20b-9ce5743d6065","html_url":"https://github.com/DataBiosphere/terra-resource-janitor","commit_stats":null,"previous_names":[],"tags_count":163,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/terra-resource-janitor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-janitor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-janitor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-janitor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-janitor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/terra-resource-janitor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-resource-janitor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29655301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T09:27:29.698Z","status":"ssl_error","status_checked_at":"2026-02-20T09:26:12.373Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:25.985Z","updated_at":"2026-02-20T15:31:37.245Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# terra-resource-janitor\nJanitor service to cleanup resources created by [Cloud Resource Library (CRL)](https://github.com/DataBiosphere/terra-cloud-resource-lib) \n\n## Overview\n![](https://app.lucidchart.com/publicSegments/view/057ef338-e869-4fb3-acd9-6ad2d7d560ba/image.jpeg)\n\n1. When a CRL client configures it in cleanup mode, when a cloud resource is about to be created,\nCRL will publish a message to the Janitor's pubsub with the cloud resource's unique id.\n2. The Janitor Service subscribes to pub/sub, persisting tracking received resources for eventual\ncleanup.\n3. Once the resource has expired, the Janitor Service uses Stairway to delete the cloud resource. \n\n## Primary \u0026 Secondary Instances\nThe Janitor is expected to be deployed with one primary instance and 0-many secondary instances.\nThe primary instance controls the lifecycle of tracked resources. Having a single primary instance\nlive at a given time makes it easier to reason about concurrency. Only actions that should not be\ndone by multiple instances should be confined to the primary.\n\n# Development\n\n## Configs Rendering\n\nLocal Testing and Github Action tests require credentials to be able to call GCP, run\n``` local-dev/render-config.sh``` first for local testing. It generates:\n\n* A Google Service Account Secret to create/delete cloud resources in test.\n* A Google Service Account Secret to publish message to 'prod' Janitor instance.\n* A Google Service Account Secret to publish message to 'test' Janitor instance.\n\n## Run Locally\n\nUse JDK 17, [instructions](https://github.com/DataBiosphere/terra-workspace-manager/blob/main/DEVELOPMENT.md#jdk)\n\nSet executable permissions:\n\n```\nchmod +x gradlew\n```\n\nTo spin up the local postgres, run:\n\n```\nlocal-dev/run_postgres.sh start\n```\nStart local server\n```\nlocal-dev/run_local.sh\n```\nAnd then check http://127.0.0.1:8080/status for service status.\n\nSwagger in local server: http://127.0.0.1:8080/swagger-ui.html\n\nYou can connect to local janitor DB by running: \n```\npsql postgresql://127.0.0.1:5432/testdb -U dbuser\n```\nenter `dbpwd` as password\n\nTo connect to stairway db, run: \n```\npsql postgresql://127.0.0.1:5432/testdb_stairway -U dbuser_stairway\n```\nenter `dbpwd_stairway` as password\n\n## Deploy to GKE cluster:\nThe provided setup script clones the terra-helm and terra-helmfile git repos,\nand templates in the desired Terra environment/k8s namespace to target.\nIf you need to pull changes to either terra-helm or terra-helmfile, rerun this script.\n\nTo use this, first ensure the following tools are installed on your local machine:\n * Skaffold (https://skaffold.dev/)\n * Helm (https://helm.sh/docs/intro/install/)\n\n\u003e Older versions of Skaffold (v1.4.0 and earlier) do not have support for Helm 3 and will fail to deploy your \nchanges. If you're seeing errors like `UPGRADE FAILED: \"(Release name)\" has no \ndeployed releases`, try updating Skaffold.\n\nYou may need to use gcloud to provide GCR\n credentials with `gcloud auth configure-docker`. Finally, run local-run.sh with\n  your target environment as the first argument:\n\n```\nlocal-dev/setup_gke_deploy.sh \u003cenvironment\u003e\n```\n\nwhere `environment` is\nyour [personal environment](https://github.com/DataBiosphere/terra/blob/main/docs/dev-guides/personal-environments.md) (\ne.g. `gjordan`) or an existing Terra env (e.g. `toolsalpha`). You can now push to the specified environment by running\n\n```\nskaffold run\n```\n\n## Connecting psql client using the Cloud SQL Proxy:\nFollow [Installing this instruction](https://cloud.google.com/sql/docs/mysql/sql-proxy#macos-64-bit)\nto install Cloud SQL Proxy \n\nGo to cloud console to get the instance name you want to connect to, then start the proxy:\n```\n./cloud-sql-proxy \u003cINSTANCE_CONNECTION_STRING\u003e --port 5432 \n```\nStart the client session\n```\npsql \"host=127.0.0.1 sslmode=disable dbname=\u003cDB_NAME\u003e user=\u003cUSER_NAME\u003e\"\n```\nFor Broad engineer, DB_NAME and USER_NAME can be found in vault. \n```\ndocker run -e VAULT_TOKEN=$(cat ~/.vault-token) -it broadinstitute/dsde-toolbox:dev vault read secret/dsde/terra/kernel/integration/{$NAMESPACE}/crl_janitor/postgres/{db-creds|stairway-db-creds}\n```\nThe db instance name can be also found under `...crl_janitor/postgres/instance` in vault.\n\nNote that you must stop the local postgres first to free the 5432 port.\nSee [this document](https://cloud.google.com/sql/docs/postgres/connect-admin-proxy) for more details.\n\n### Dependencies\nWe use [Gradle's dependency locking](https://docs.gradle.org/current/userguide/dependency_locking.html)\nto ensure that builds use the same transitive dependencies, so they're reproducible. This means that\nadding or updating a dependency requires telling Gradle to save the change. If you're getting errors\nthat mention \"dependency lock state\" after changing a dep, you need to do this step.\n\n```\n./gradlew dependencies --write-locks\n```\n\n## Testing\n\n### Unit tests\n\nSpin up the local postgres:\n```\nlocal-dev/run_postgres.sh start\n```\n\nThen run unit tests:\n```\n./gradlew test\n```\nStop the local postgres:\n```\nlocal-dev/run_postgres.sh stop\n```\n## Authorization\nJanitor uses a list of user email address as administrator user to access its admin endpoint.\nThe value can be set in `iam.adminUserList` configuration.\n### For Broad Engineers:\n#### To connect to `tools` Janitor database\n```\n./cloud_sql_proxy -instances=terra-kernel-k8s:us-central1:crljanitor-db-tools-aa84446ffb6a5fd1=tcp:5432\n```\nPull secret from Vault\n```\nexport PGPASSWORD=$(docker run -e VAULT_TOKEN=$(cat ~/.vault-token) -it broadinstitute/dsde-toolbox:dev vault read -field='password' secret/dsde/terra/kernel/integration/tools/crl_janitor/postgres/db-creds)\n```\nConnect to postgres DB\n```\npsql \"host=127.0.0.1 sslmode=disable dbname=crljanitor user=crljanitor\"\n```\n\n#### To use `tools` Janitor\n`tools` Janitor is used by Broad deployed Terra APPs. `tools` Janitor's client service account(created by [Terraform](https://github.com/broadinstitute/terraform-ap-modules/blob/54bf1f9669ade3d4f5e8fb0197f1dd4239448dea/crl-janitor/sa.tf#L76)) has permission to access\nadmin endpoint. To use this:\n\nStep 1:\n```\ndocker run --rm --cap-add IPC_LOCK -e \"VAULT_TOKEN=$(cat ~/.vault-token)\" -e \"VAULT_ADDR=https://clotho.broadinstitute.org:8200\" vault:1.1.0 vault read -format json secret/dsde/terra/kernel/integration/tools/crl_janitor/client-sa | jq -r '.data.key' | base64 --decode \u003e janitor-client-sa.json```\n```\nStep2: \n```\ngcloud auth activate-service-account --key-file=janitor-client-sa.json\n```\nStep3:\n```\ngcloud auth print-access-token\n```\nGo to [tools Janitor Swagger](https://crljanitor.tools.integ.envs.broadinstitute.org/swagger-ui.html) Paste this token into Swagger's `bearerAuth`\n\n#### To use other environment Janitor\nThe admin user list file is stored in Vault. \nTo read user list, run:\n```\ndocker run -e VAULT_TOKEN=$(cat ~/.vault-token) -it broadinstitute/dsde-toolbox:dev vault read config/terra/crl-janitor/common/iam \n```\nTo request admin access, please contact [mc-terra-janitor-admins](https://github.com/orgs/broadinstitute/teams/mc-terra-janitor-admins) members.\n\nTo update user list, run:\n```\ndocker run --rm --cap-add IPC_LOCK -e \"VAULT_TOKEN=$(cat ~/.vault-token)\" -e \"VAULT_ADDR=https://clotho.broadinstitute.org:8200\" -v $(pwd):/current vault:1.1.0 vault write config/terra/crl-janitor/common/iam admin-users=@/current/{USER_FILE}}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-resource-janitor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fterra-resource-janitor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-resource-janitor/lists"}