{"id":28395138,"url":"https://github.com/databiosphere/welder","last_synced_at":"2025-07-28T22:34:19.935Z","repository":{"id":37897377,"uuid":"175476487","full_name":"DataBiosphere/welder","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-01T02:52:28.000Z","size":1222,"stargazers_count":3,"open_issues_count":2,"forks_count":3,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-06-01T06:52:20.553Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-03-13T18:23:43.000Z","updated_at":"2025-04-17T13:14:13.000Z","dependencies_parsed_at":"2024-06-14T03:25:19.821Z","dependency_job_id":"ac6fed99-5b44-4f74-a192-1f08dadbea58","html_url":"https://github.com/DataBiosphere/welder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/welder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwelder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwelder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwelder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwelder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/welder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fwelder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172363,"owners_count":23269992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:16.736Z","updated_at":"2025-06-27T01:31:15.476Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Welder\n\nWelder is a lightweight webservice intended to be run on Dataproc Clusters and GCE VMs spun up by [Leonardo](https://github.com/DataBiosphere/leonardo). Compute spun up by Leonardo allows users to run interactive tools, such as Jupyter, in the cloud. Welder's primary purpose is to facilitate the persistence of files generated by these tools to a bucket in Google Cloud Storage. See the API section below for a list of endpoints and functions available in Welder.\n\n# Try it out\n\nSetup env vars and auth\n```bash\nexport OWNER_EMAIL=\"fake@gmail.com\"\nexport STAGING_BUCKET=\"jc-sample-bucket\"\nexport SHOULD_BACKGROUND_SYNC=false # false if you don't want the file syncing background process to run\nexport CLOUD_PROVIDER=gcp\nexport WSM_URL=\"dummy\" # if doesn't work use the real WSM url\nexport WORKSPACE_ID=\"dummy\"\nexport STORAGE_CONTAINER_RESOURCE_ID=\"dummy\"\nexport PORT=8080\nexport LOCKING_ENABLED=true\ngcloud auth application-default login\n```\n\n* Start welder-api-server `sbt server/run`\n** you may want to create `server/src/main/resources/application.conf` with the following content\n```bash\npath-to-storage-links-json = \"storage_links.json\"\npath-to-gcs-metadata-json = \"gcs_metadata.json\"\nobject-service.working-directory = \"/tmp\"\n```\n\n\n# Publish container image to Google container registry\n* Set up auth for publishing docker image to GCR\n`gcloud auth configure-docker`\n* Publish welder. \n   * **IMPORTANT** make sure you have a new commit so that you don't override the HEAD hash in gcr\n   * For local development: `sbt server/docker:publishLocal`\n   * If you'd like to share the image:  `sbt server/docker:publish` \n\n# Development\n\n## Using git secrets\nMake sure git secrets is installed:\n```bash\nbrew install git-secrets\n```\nEnsure git-secrets is run:\n\u003ci\u003eIf you use the rsync script to run locally you can skip this step\u003c/i\u003e\n```bash\ncp -r hooks/ .git/hooks/\nchmod 755 .git/hooks/apply-git-secrets.sh\n```\n\n## After merging your PR\nMake sure all builds are green after you merge PR. Look for `welder-update-hash` notification in `dsp-callisto-internal` channel, and open the PR for updating welder hash in leonardo. Make sure to include relevant Jira ticket number in the PR title if necessary.\n\n## Run unit tests\nSet up environment variables\n```\nexport OWNER_EMAIL=\"fake@gmail.com\"\nexport STAGING_BUCKET=\"jc-sample-bucket\"\nexport SHOULD_BACKGROUND_SYNC=false # false if you don't want the file syncing background process to run\nexport CLOUD_PROVIDER=gcp\nexport WSM_URL=\"dummy\" # if doesn't work use the real WSM url\nexport WORKSPACE_ID=\"dummy\"\nexport STORAGE_CONTAINER_RESOURCE_ID=\"dummy\"\n```\nRun tests\n```\nsbt test\n```\n\n# UI workflow\n![UI workflow](UI_Interaction.png)\n\n# How to modify the workflow image?\n* Enable [IntelliJ plugin](https://plugins.jetbrains.com/plugin/7017-plantuml-integration)\n* modify [UI_Interaction.puml](server/src/main/resources/UI_Interaction.puml)\n* Save rendered image as `UI_Interaction.png`\n\n# For Clients\n\n## Hashed Metadata\nFor the the sake of privacy, some of the fields stored on the GCS object metadata are hashed. Specifically, the `lastLockedBy` field is hashed in combination with the GCS bucket name. The formula for this hash is `sha256(my-bucket-name:my-email-address@gmail.com)`. Note that `my-bucket-name` does not contain the `gs://` prefix. You can use this formula to check whether or not known email addresses hold locks in known buckets.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fwelder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fwelder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fwelder/lists"}