{"id":28395170,"url":"https://github.com/databiosphere/topmed-workflows","last_synced_at":"2026-01-27T20:14:12.161Z","repository":{"id":28826709,"uuid":"117876946","full_name":"DataBiosphere/topmed-workflows","owner":"DataBiosphere","description":"a place for topmed workflows","archived":false,"fork":false,"pushed_at":"2022-02-03T00:28:26.000Z","size":5905,"stargazers_count":12,"open_issues_count":9,"forks_count":23,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-06-27T01:38:11.790Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"wdl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-17T18:38:50.000Z","updated_at":"2023-12-23T00:00:47.000Z","dependencies_parsed_at":"2022-07-27T17:03:48.434Z","dependency_job_id":null,"html_url":"https://github.com/DataBiosphere/topmed-workflows","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/topmed-workflows","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ftopmed-workflows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ftopmed-workflows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ftopmed-workflows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ftopmed-workflows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/topmed-workflows/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ftopmed-workflows/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28820434,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T18:44:20.126Z","status":"ssl_error","status_checked_at":"2026-01-27T18:44:09.161Z","response_time":168,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:22.844Z","updated_at":"2026-01-27T20:14:12.142Z","avatar_url":"https://github.com/DataBiosphere.png","language":"wdl","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Topmed Workflows\n\n## About\nThe original pipelines were assembled and written by Hyun Min Kang (hmkang@umich.edu) and Adrian Tan (atks@umich.edu) \nat the [Abecasis Lab at the University of Michigan](https://genome.sph.umich.edu/wiki/Abecasis_Lab)\n\nSee the [variant calling pipeline](https://github.com/statgen/topmed_freeze3_calling) and [alignment pipeline](https://github.com/statgen/docker-alignment) repositories\n\n## Installing dependencies on your local system\n\n### 1. Cloud SDK (`gcloud`, `gsutil`)\nIf you are on Debian / Ubuntu, follow the instructions on [Cloud SDK](https://cloud.google.com/sdk/downloads#apt-get). \nAfter you execute `gcloud init` the installer asks you to log in and you should respond with `Y`, head to the provided URL, copy the code and past it to the prompt. After that it will ask you for the cloud project you want to use, so you need to input the GCP _Project ID_. I picked `us-west1-b` as the region.\n\n#### Configuration and credentials file\n```bash\nexport CLOUD_SDK_REPO=\"cloud-sdk-$(lsb_release -c -s)\"\necho \"deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main\" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list\ncurl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\ngcloud auth login\n```\nAfter that run `gcloud auth application-default --help` and follow the instructions. Briefly, run\n```bash\ngcloud iam service-accounts create \u003cpick-a-username\u003e\ngcloud iam service-accounts keys create key.json --iam-account=\u003cthe-username-you-just-picked\u003e@\u003cyour-service-account-name\u003e.iam.gserviceaccount.com\n```\n\nThat should print something like \n```bash\ncreated key [\u003csome long integer\u003e] of type [json] as [key.json] for [\u003cusername-you-picked\u003e@\u003cyour-service-account-name\u003e.iam.gserviceaccount.com]\n```\nYou can check in the Google Cloud Platform console under IAM Service Accounts. That account you just created should be in the list.\n\nNext create an environment variable that points to the file `key.json`:\n```bash\nexport GOOGLE_APPLICATION_CREDENTIALS=key.json\n```\n\n#### Providing credentials to your application\nTo run workflows of data stored on `gcloud` you need to set an environment variable `GOOGLE_APPLICATION_CREDENTIALS`, which holds the path to the credentials file.\n\n### 2. Broad's execution engine `cromwell`\n`cromwell` is a Java executable and requires a Java Runtime Engine. Follow the instruction [here](http://cromwell.readthedocs.io/en/develop/tutorials/FiveMinuteIntro/) for a complete installation.\n\n### 3. Dockstore\nFor Dockstore to run you need to install the [Java Runtime Engine](https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04). Find installation instructions for Dockstore [here](https://dockstore.org/quick-start).\n\n## Running workflows\n\n### Provisioning reference files\nTo copy contents of a SDK bucket to your local system (or a VM) use\n```bash\ngsutil -u [PROJECT_ID] cp gs://[BUCKET_NAME]/[OBJECT_NAME] [OBJECT_DESTINATION]\n```\n\n### Checker workflows\n\n\nA WDL and a JSON file to test checker workflows are in the `test_data` directory. You need to adjust all paths in the JSON file to the paths on your system before running the checker. It has been tested with `cromwell-31.jar`. To run the checker workflow for the WDL aligner navigate to respective directory (usually it has _checker_ in its name) and run\n```bash\njava -Dconfig.file=\u003clocation_to_file\u003e -jar ~/bin/\u003ccromwell_version\u003e.jar run \u003cchecker-workflow\u003e.wdl -i  \u003cchecker-workflow\u003e.json\n```\n\n## Cost estimates for Terra\nPlease keep in mind that your costs may vary depending on how your data is formatted and what parameters you use. In addition, if you are using preemptibles, there is some element of randomness here -- a preemptible may or may not be stopped by Google at any given time, causing an in-progress task to need to restart.\n\n### Aligner (WDL)\nWhen running the aligner workflow on 10 full-size CRAMs from the PharmaHD study imported from Gen3, running on the aligner's default settings, the cost was $80.38 as reported by Terra. The most expensive of those ten files cost $10.82 and the least expensive cost $5.74.\n\n### Aligner Checker (WDL)\nAs the aligner checker runs the aligner and then simply preforms an md5sum, the cost for the aligner checker will be about the same as that of the aligner.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Ftopmed-workflows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Ftopmed-workflows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Ftopmed-workflows/lists"}