{"id":18016472,"url":"https://github.com/saresend/deepvariant-challenge","last_synced_at":"2025-08-17T12:41:39.593Z","repository":{"id":125383325,"uuid":"250871808","full_name":"saresend/deepvariant-challenge","owner":"saresend","description":"A small repo that eases use of google's deepvariant with GWA metastudies about hypertension","archived":false,"fork":false,"pushed_at":"2020-03-29T15:55:47.000Z","size":3816,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-10T01:18:26.790Z","etag":null,"topics":["deepvariant","genomics"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saresend.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-28T19:00:38.000Z","updated_at":"2020-03-29T15:55:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"3a0f9a09-916b-49ec-b5f6-9cea3ad4367a","html_url":"https://github.com/saresend/deepvariant-challenge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saresend%2Fdeepvariant-challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saresend%2Fdeepvariant-challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saresend%2Fdeepvariant-challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saresend%2Fdeepvariant-challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saresend","download_url":"https://codeload.github.com/saresend/deepvariant-challenge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247201245,"owners_count":20900572,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepvariant","genomics"],"created_at":"2024-10-30T04:18:09.281Z","updated_at":"2025-04-04T15:28:24.637Z","avatar_url":"https://github.com/saresend.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/cover.png\" /\u003e\n\u003c/p\u003e\n\n![alt](https://img.shields.io/github/license/saresend/deepvariant-challenge)\n\n# Introduction\n\n### Submission Info\n\nFor submission info, please see [submission.md](submission.md)\n\nThe following steps provide step by step instructions for how to run\ngoogle's deepvariant project with hypertension journal information. Project proposed by Office Ally,\nand is a submission for LA Hacks 2020.\n\n### Notes and Disclaimers:\n\nThis project has only been tested with Ubuntu 18.X. It should work as-is with most linux distros, and can certainly be ported to Windows and Mac. However, instructions are written with linux in mind.\n\n## Step 1: Clone this repo:\n\n```\ngit clone https://github.com/saresend/deepvariant-challenge\n```\n\n### Step 2.1: Install docker\n\nIf you are running on a fresh image, you may need to install docker. If so, please run the following:\n\n```\nsudo apt -y update\nsudo apt-get -y install docker.io\n```\n\n### Step 2.2: Download Sample data (Optional)\n\nThe following command will load a truncated dataset provided by google for demonstration purposes. If running on an individual dataset please substitute with that.\n\n```\ncd deepvariant-challenge\n./load_test_data.sh\n```\n\n**Note:** You may need to give this execute privilege by running `chmod +x load_test_data.sh`. If you get issues that say something about `permission denied`, this is probably the issue.\n\n## Step 3: Run Docker Build\n\nThis command will build the image and run the analysis on the data provided in the previous steps\n\n```\ncd ..\ndocker build deepvariant-challenge --tag lahacks:0.1\n```\n\n## Step 4: Instantiate docker image\n\nThe following command instantiates our image, allowing us to collect the data that is the result of the analysis\n\n```\ncd deepvariant-challenge\ndocker run -dit lahacks:0.1\n```\n\n## Step 5: Copy output to local machine\n\n### Step 5.1: Get Container Identifier\n\nRun the following command:\n\n```\ndocker ps\n```\n\nThis provides a list of all currently running containers. Look for one tagged lahacks:0.1, and find the entry under name. Common ones are things like `mythical_tree`, etc.\n\n### Step 5.2: Copy files from Container\n\n```\ndocker cp \u003ccontainer_name\u003e:/output .\n```\n\nThis command will take the output analysis and VCF files and copy it locally. If all is well, you should be able to open a `output.visual_report.html` file in the `output/` directory, and see something like the following:\n\n![alt](images/output1.png)\n\n## Step 6: Run Similarity analysis:\n\n### Step 6.1: Jupyter Notebook installation\n\nFor installation instructions for jupyter notebooks, please refer to: https://jupyter.org/install\n\n### Step 6.2: Run The notebook\n\nThe following command analyzes the resulting VCF file, and compares it with the data in the journal loaded from `hypertension_markers.csv`.\n\nThe following command requires you to have jupyter notebook installed:\n\n```\njupyter notebook vcf_compare.ipynb\n```\n\n**Note:** This notebook uses `python3`\n\nThe notebook provides some preliminary analysis and allows for flexibility to play around with the resulting data and conduct additional analyses.\n\n## Full genomic Sequencing\n\nIn addition to this small test dataset, we also have the full dataset available. The modifications to run with this dataset are first, instead of running `./load_test_data.sh`, please run `./load_full_genome.sh`, which will pull the entire dataset.\n\n**Warning**: it is about 110 GB in size.\n\nYou can then run the following command to build the actual docker container:\n\n```\nsudo docker build deepvariant-challenge/ --build-arg ref_file=testdata/hs37d5.fa.gz --build-arg bam_file=testdata/HG002_NIST_150bp_50x.bam --tag lahacks:0.1\n```\n\nOtherwise, the commands remain the same and you should be able to produce VCF files for the full genomic sequence, rather than just the test dataset.\n\n## Benchmarks for the project\n\nDuring development for this project, the following instances were used:\n\n| Instance                      | CPUs | Memory | Success |\n| ----------------------------- | ---- | ------ | ------- |\n| n1-highmem-32 (GCP)           | 32   | 208 GB | No      |\n| standard - 40 (Digital Ocean) | 4    | 160 GB | No      |\n| standard - 80 (Digital Ocean) | 6    | 320 GB | No      |\n\nBelow is a speed reference for a full genome sequenced by deepvariant, as a sense of how long one might expect this to take for a given instance. It seems that it scales fairly linearly, meaning that we might expect a `32 CPU` machine to run in about double the time of a `64 CPU` machine.\n\n| Step                               | Hardware | Wall time |\n| ---------------------------------- | -------- | --------- |\n| `make_examples`                    | 64 CPUs  | ~ 1h 46m  |\n| `call_variants`                    | 64 CPUs  | ~ 3h 09m  |\n| `postprocess_variants` (with gVCF) | 1 CPU    | ~ 53m     |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaresend%2Fdeepvariant-challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaresend%2Fdeepvariant-challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaresend%2Fdeepvariant-challenge/lists"}