{"id":28395134,"url":"https://github.com/databiosphere/terra-workspace-data-service","last_synced_at":"2025-06-12T12:39:06.432Z","repository":{"id":37078895,"uuid":"499230676","full_name":"DataBiosphere/terra-workspace-data-service","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-11T18:49:17.000Z","size":5674,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-06-11T20:18:09.652Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-06-02T17:34:58.000Z","updated_at":"2025-06-03T14:02:51.000Z","dependencies_parsed_at":"2023-09-27T03:01:06.887Z","dependency_job_id":"ea6097f8-a671-45ef-8fd5-a3c7e979f18b","html_url":"https://github.com/DataBiosphere/terra-workspace-data-service","commit_stats":null,"previous_names":[],"tags_count":462,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/terra-workspace-data-service","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-workspace-data-service","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-workspace-data-service/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-workspace-data-service/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-workspace-data-service/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/terra-workspace-data-service/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-workspace-data-service/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259466792,"owners_count":22862392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:16.210Z","updated_at":"2025-06-12T12:39:06.420Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Workspace Data Service\n\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=DataBiosphere_workspace-data-service\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=DataBiosphere_workspace-data-service)\n\nWorkspace Data Service provides an API to manage records in Terra.\n\nDesign docs and other references may be\nfound [here](https://broadworkbench.atlassian.net/wiki/spaces/AS/pages/2437742606/Workspace+Data+Service).\n\n## Setup\n\n### Prerequisites\n\nMake sure you have Java 17 installed. Our favorite way to do this is\nwith [SDKMAN](https://sdkman.io/jdks). Once it is installed , use `sdk list java` to see available\nversions, and `sdk install java 17.0.2-tem` to install, for example, the Temurin version of Java 17.\n\n[jenv](https://www.jenv.be/) is also helpful for managing active versions. On a Mac,jenv can be\ninstalled and used like so:\n\n```\nbrew install jenv\n# follow postinstall instructions to activate jenv...\n\n# to add previously installed versions of Java to jEnv, list them:\n# /usr/libexec/java_home -V\n# and then add them:\n# jenv add /Library/Java/JavaVirtualMachines/\u003cJAVA VERSION HERE\u003e/Contents/Home\n\nbrew install homebrew/cask-versions/temurin17\n\njenv add /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home\n```\n\nWe are using Postgres 14. You do not need to have postgres installed on your system.\nInstead use the `run_postgres.sh` script to set up a docker container running postgres (see below).\n\n#### Environment Variables\n\nFor WDS to work properly, several environment variables need to be set, and exactly one of the\nfollowing active profiles must be set:\n\n- control-plane\n- data-plane\n\nTo run WDS locally and therefore use the local application properties, set the profile as follows:\n\n```bash\nexport SPRING_PROFILES_ACTIVE=local,data-plane\n```\n\nOther profiles that are available are:\n\n- staging\n- dev\n- prod\n- bee\n- data-plane OR control-plane\n\nYou are unlikely to use prod and bee when running locally, those profiles are leveraged when WDS is\ndeployed in Terra.\n\nIf you would like to not use the local profile, you can set some of the environment variables\nmanually. The variables that need to be set are described below. You can also add them to your\n`~/.zshrc` or similar shell profile. However, in order for the app to run correctly, you still\nhave to at least specify data-plane or control-plane.\n\n##### Control Plane Setup\n\nIf you are running locally with the `control-plane` profile, you'll need some additional setup to\nuse PubSub.\n\n1. [Set up application default credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).\n   ```bash\n   gcloud auth application-default login\n   ```\n\n2. Configure a Google Cloud Project for CWDS to use. This can be done by setting a default project\n   with `gcloud` or by setting the `GOOGLE_CLOUD_PROJECT` environment variable.\n   ```bash\n   gcloud config set project broad-dsde-dev\n   export GOOGLE_CLOUD_PROJECT=broad-dsde-dev\n   ```\n\n3. Configure PubSub topics by setting environment variables.\n   ```bash\n   # Topic for outgoing notifications to Rawls\n   export RAWLS_NOTIFY_TOPIC=rawls-async-import-topic-dev\n   # Topic for incoming notifications from Rawls\n   export IMPORT_STATUS_UPDATES_TOPIC=cwds-import-job-status-updates-dev\n   # Create a subscription dedicated to your local CWDS\n   export IMPORT_STATUS_UPDATES_SUBSCRIPTION=cwds-import-job-status-updates-sub-local-$(whoami)\n   ```\n\n   This will send imports the dev Rawls; you may want to use a dummy project/topic instead.\n\n4. Configure the Google Cloud Storage bucket to write Rawls JSON to.\n   ```bash\n   export SERVICE_GOOGLE_BUCKET=cwds-batchupsert-dev\n   ```\n\n##### SAM_URL\n\nWDS contacts Sam for permission checks. You will need to configure Sam's URL by setting an\nenvironment variable, such as:\n\n```\nexport SAM_URL=https://sam.dsde-dev.broadinstitute.org/\n```\n\n##### WORKSPACE_ID\n\nWDS requires a valid workspace id to check permissions in Sam and to import snapshots from the Terra\nData Repo.\nThis is controlled by a `WORKSPACE_ID` environment variable. You should set this to the UUID\nof a workspace you own, e.g.\n\n```\nexport WORKSPACE_ID=123e4567-e89b-12d3-a456-426614174000\n```\n\n## Running\n\nTo just build the code, from the root directory run\n\n```bash\n./gradlew build --exclude-task test\n```\n\nTo run the application, first a postgres database must be running:\n\n```bash\n./local-dev/run_postgres.sh start\n```\n\nTo bypass Sam permission checks locally, you can run a fake Sam nginx backend that will allow\neverything. Run the following from WDS root folder to launch the docker container and export the\nenvironment variable WDS will use on the next startup to connect to Sam.\n\n```bash\n# start the SAM mock as a docker container in detached mode\ndocker run -v `pwd`/service/src/test/resources/nginx.conf:/etc/nginx/nginx.conf -p 9889:80 -d nginx:1.23.3\nexport SAM_URL=http://localhost:9889\n```\n\nTo run WDS locally, you can either use the command line:\n\n```bash\n./gradlew bootRun\n```\n\nOr, from Intellij, go\nto `src/main/java/org.databiosphere.workspacedataservice/WorkspaceDataServiceApplication` and click\nthe green play button.\nThis will launch the service on port 8080.\n\nAt the moment, WDS is only available through this port. It can be reached from the command line:\n\nTo query for a single record:\n\n```bash\ncurl http://localhost:8080/\u003cinstanceid\u003e/records/v0.2/\u003ctable/type\u003e/\u003crecord_id\u003e\n```\n\nTo add new attribute or update values for existing attributes (this won't create a new record\nhowever):\n\n``` bash\ncurl -H \"Content-type: application/json\" -X PATCH \"http://localhost:8080/\u003cinstanceid guid\u003e/records/v0.2/\u003ctable/type\u003e/\u003crecord_id\u003e\" -d '{\n\"id\": \"\u003crecord_name\u003e\",\n\"type\": \"\u003crecord_type\u003e\",\n\"attributes\": {\n\"new-int\": -77,\n\"new-date-time\": \"2011-01-11T11:00:50\",\n\"new-double\": -122.45,\n\"new-json\": \"{\\\"key\\\": \\\"Richie\\\"}\"\n}\n}'\n```\n\nNote that attribute and record type names are subject to the logic\nin [validateName](https://github.com/DataBiosphere/terra-workspace-data-service/blob/5375c8c846ad163e38a5540c4487ea05fb292f45/service/src/main/java/org/databiosphere/workspacedataservice/dao/RecordDao.java#L80)\n\nWhen done, stop postgres:\n\n```bash\n./local-dev/run_postgres.sh stop\n```\n\n## Tests\n\nTo run unit tests locally, first make sure your local postgres is up and running:\n\n```bash\n./local-dev/run_postgres.sh start\n```\n\nFrom the command line, run\n\n```bash\n./gradlew test\n```\n\nTo run one test suite, run\n\n```bash\n./gradlew test --tests '*RecordDaoTest'\n```\n\nTo run a single test, run\n\n```bash\n./gradlew test --tests '*RecordDaoTest.testGetSingleRecord'\n```\n\n## Troubleshooting\n\nSome problems during build and test may be solved by running the Gradle `clean` task:\n\n```bash\n./gradlew clean\n```\n\n## Postgres\n\nWhen running the local docker postgres, you can access the shell directly:\n\n```bash\n./local-dev/run_posgres.sh shell\n```\n\n## Swagger UI\n\nWhen running locally, a Swagger UI is available at http://localhost:8080/swagger/swagger-ui.html.\n\nYou can also view the swagger definitions via this third-party hosted\nUI: https://petstore.swagger.io/?url=https://raw.githubusercontent.com/DataBiosphere/terra-workspace-data-service/main/service/src/main/resources/static/swagger/openapi-docs.yaml.\nNote that in this third-party UI, the APIs are not active; this is only valuable for viewing the\ndefinitions.\n\n## Docker\n\nTo build the WDS docker image _(replace \"wdsdocker\" with your desired image name)_:\n\n```bash\n./gradlew --build-cache jibDockerBuild --image=wdsdocker -Djib.console=plain\n```\n\nTo run the built docker image (_replace \"wdsdocker\" the image name specified during build_):\n\n* Docker for Mac:\n\n```bash\ndocker run -e WDS_DB_HOST='host.docker.internal' -p 8080:8080 wdsdocker\n```\n\n* Docker on Linux:\n\n```bash\ndocker run --network=host -e WDS_DB_HOST='127.0.0.1' -p 8080:8080 wdsdocker\n```\n\n## Using Workspace Data Service\n\nA client of WDS is published in the Broad Artifactory. To include it in your Gradle project, add the\nfollowing to your `build.gradle` file:\n\n```\nrepositories {\n    maven {\n        url \"https://us-central1-maven.pkg.dev/dsp-artifact-registry/libs-snapshot/\"\n    }\n}\n\ndependencies {\n    implementation(group: 'org.databiosphere', name: 'workspacedataservice-client', version: 'x.x.x')\n```\n\nThe latest version can be seen\nunder [Tags](https://github.com/DataBiosphere/terra-workspace-data-service/tags).\n\n## For Developers\n\n### Code Style\n\nWe use follow the [Google Java style guide](https://google.github.io/styleguide/javaguide.html) and\nimplement it with a combination of the [spotless](https://github.com/diffplug/spotless) plugin and\nIDE tooling.\n\nTo run spotless from the CLI:\n\n```bash\n./gradlew spotlessApply\n```\n\nFor details on how to make the most of your IDE to apply the style guide automatically,\nsee [CONTRIBUTING.md](CONTRIBUTING.md).\n\n### API Definitions and Code Generation\n\nWDS's `v0.2` APIs are documented in `service/src/main/resources/static/swagger/openapi-docs.yaml`\nand\ntheir implementations are hand-coded.\n\nAPIs at `v1` or later use code generation. To create or update an API, you must:\n\n1. Define the API using OpenAPI syntax\n   at `service/src/main/resources/static/swagger/apis-v1.yaml`\n2. Include the API, via `$ref:`, in `service/src/main/resources/static/swagger/openapi-docs.yaml`\n3. Run the `generateApiInterfaces` Gradle task\n4. Note the new/updated Java interfaces and models under the `o.d.w.generated` package\n5. Create or update a controller class that implements the new Java interface\n\nThe `generateApiInterfaces` Gradle task is configured to run as a dependency of other\nimportant Gradle tasks, such as `compileJava`. You should not have to run it manually\nin the course of most development; however, if you are actively changing definitions for\nan API you may need to run it to see your changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-workspace-data-service","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fterra-workspace-data-service","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-workspace-data-service/lists"}