https://github.com/saresend/deepvariant-challenge
A small repo that eases use of google's deepvariant with GWA metastudies about hypertension
https://github.com/saresend/deepvariant-challenge
deepvariant genomics
Last synced: 11 months ago
JSON representation
A small repo that eases use of google's deepvariant with GWA metastudies about hypertension
- Host: GitHub
- URL: https://github.com/saresend/deepvariant-challenge
- Owner: saresend
- License: mit
- Created: 2020-03-28T19:00:38.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-03-29T15:55:47.000Z (over 6 years ago)
- Last Synced: 2025-02-10T01:18:26.790Z (over 1 year ago)
- Topics: deepvariant, genomics
- Language: Jupyter Notebook
- Homepage:
- Size: 3.64 MB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
README
# Introduction
### Submission Info
For submission info, please see [submission.md](submission.md)
The following steps provide step by step instructions for how to run
google's deepvariant project with hypertension journal information. Project proposed by Office Ally,
and is a submission for LA Hacks 2020.
### Notes and Disclaimers:
This project has only been tested with Ubuntu 18.X. It should work as-is with most linux distros, and can certainly be ported to Windows and Mac. However, instructions are written with linux in mind.
## Step 1: Clone this repo:
```
git clone https://github.com/saresend/deepvariant-challenge
```
### Step 2.1: Install docker
If you are running on a fresh image, you may need to install docker. If so, please run the following:
```
sudo apt -y update
sudo apt-get -y install docker.io
```
### Step 2.2: Download Sample data (Optional)
The following command will load a truncated dataset provided by google for demonstration purposes. If running on an individual dataset please substitute with that.
```
cd deepvariant-challenge
./load_test_data.sh
```
**Note:** You may need to give this execute privilege by running `chmod +x load_test_data.sh`. If you get issues that say something about `permission denied`, this is probably the issue.
## Step 3: Run Docker Build
This command will build the image and run the analysis on the data provided in the previous steps
```
cd ..
docker build deepvariant-challenge --tag lahacks:0.1
```
## Step 4: Instantiate docker image
The following command instantiates our image, allowing us to collect the data that is the result of the analysis
```
cd deepvariant-challenge
docker run -dit lahacks:0.1
```
## Step 5: Copy output to local machine
### Step 5.1: Get Container Identifier
Run the following command:
```
docker ps
```
This provides a list of all currently running containers. Look for one tagged lahacks:0.1, and find the entry under name. Common ones are things like `mythical_tree`, etc.
### Step 5.2: Copy files from Container
```
docker cp :/output .
```
This command will take the output analysis and VCF files and copy it locally. If all is well, you should be able to open a `output.visual_report.html` file in the `output/` directory, and see something like the following:
## Step 6: Run Similarity analysis:
### Step 6.1: Jupyter Notebook installation
For installation instructions for jupyter notebooks, please refer to: https://jupyter.org/install
### Step 6.2: Run The notebook
The following command analyzes the resulting VCF file, and compares it with the data in the journal loaded from `hypertension_markers.csv`.
The following command requires you to have jupyter notebook installed:
```
jupyter notebook vcf_compare.ipynb
```
**Note:** This notebook uses `python3`
The notebook provides some preliminary analysis and allows for flexibility to play around with the resulting data and conduct additional analyses.
## Full genomic Sequencing
In addition to this small test dataset, we also have the full dataset available. The modifications to run with this dataset are first, instead of running `./load_test_data.sh`, please run `./load_full_genome.sh`, which will pull the entire dataset.
**Warning**: it is about 110 GB in size.
You can then run the following command to build the actual docker container:
```
sudo docker build deepvariant-challenge/ --build-arg ref_file=testdata/hs37d5.fa.gz --build-arg bam_file=testdata/HG002_NIST_150bp_50x.bam --tag lahacks:0.1
```
Otherwise, the commands remain the same and you should be able to produce VCF files for the full genomic sequence, rather than just the test dataset.
## Benchmarks for the project
During development for this project, the following instances were used:
| Instance | CPUs | Memory | Success |
| ----------------------------- | ---- | ------ | ------- |
| n1-highmem-32 (GCP) | 32 | 208 GB | No |
| standard - 40 (Digital Ocean) | 4 | 160 GB | No |
| standard - 80 (Digital Ocean) | 6 | 320 GB | No |
Below is a speed reference for a full genome sequenced by deepvariant, as a sense of how long one might expect this to take for a given instance. It seems that it scales fairly linearly, meaning that we might expect a `32 CPU` machine to run in about double the time of a `64 CPU` machine.
| Step | Hardware | Wall time |
| ---------------------------------- | -------- | --------- |
| `make_examples` | 64 CPUs | ~ 1h 46m |
| `call_variants` | 64 CPUs | ~ 3h 09m |
| `postprocess_variants` (with gVCF) | 1 CPU | ~ 53m |