{"id":15060707,"url":"https://github.com/googlegenomics/gcp-variant-transforms","last_synced_at":"2025-05-07T16:07:42.794Z","repository":{"id":26891609,"uuid":"110025283","full_name":"googlegenomics/gcp-variant-transforms","owner":"googlegenomics","description":"GCP Variant Transforms","archived":false,"fork":false,"pushed_at":"2022-03-31T21:01:41.000Z","size":21460,"stargazers_count":139,"open_issues_count":113,"forks_count":55,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-05-07T16:07:03.475Z","etag":null,"topics":["beam","bigquery","dataflow","vcf-files"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/googlegenomics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-08T20:12:08.000Z","updated_at":"2025-03-25T00:12:39.000Z","dependencies_parsed_at":"2022-08-07T12:01:14.957Z","dependency_job_id":null,"html_url":"https://github.com/googlegenomics/gcp-variant-transforms","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googlegenomics%2Fgcp-variant-transforms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googlegenomics%2Fgcp-variant-transforms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googlegenomics%2Fgcp-variant-transforms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/googlegenomics%2Fgcp-variant-transforms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/googlegenomics","download_url":"https://codeload.github.com/googlegenomics/gcp-variant-transforms/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252912993,"owners_count":21824065,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beam","bigquery","dataflow","vcf-files"],"created_at":"2024-09-24T23:03:24.659Z","updated_at":"2025-05-07T16:07:42.741Z","avatar_url":"https://github.com/googlegenomics.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# GCP Variant Transforms\n\n## Overview\n\nThis is a tool for transforming and processing\n[VCF](https://samtools.github.io/hts-specs/VCFv4.3.pdf) files in a scalable\nmanner based on [Apache Beam](https://beam.apache.org/) using \n[Dataflow](https://cloud.google.com/dataflow/) on Google Cloud Platform.\n\nIt can be used to directly load VCF files to\n[BigQuery](https://cloud.google.com/bigquery/) supporting hundreds of thousands\nof files, millions of samples, and billions of records. Additionally, it\nprovides a preprocess functionality to validate the VCF files such that the\ninconsistencies can be easily identified.\n\nPlease see the following links for more information:\n*  Presentation: [Overview of BigQuery and how to effectively use Variant Transforms and BigQuery](docs/Variant_Transforms_and_BigQuery.pdf)\n*  Blog post: [How Color uses the new Variant Transforms tool for breakthrough clinical data science with BigQuery](https://cloud.google.com/blog/big-data/2018/03/how-color-uses-the-new-variant-transforms-tool-for-breakthrough-clinical-data-science-with-bigquery).\n*  Blog post: [Accelerating Mayo Clinic’s data platform with BigQuery and Variant Transforms](https://cloud.google.com/blog/products/data-analytics/genome-data-analytics-with-google-cloud).\n*  Jupyter notebook: [Sample queries to explore variant data in BigQuery](docs/sample_queries)\n* The underlying pipeline uses\n[Cloud Dataflow](https://cloud.google.com/dataflow/). You can navigate to the\n[Dataflow Console](https://console.cloud.google.com/dataflow), to see more\ndetailed view of the pipeline (e.g. number of records being processed, number of workers, more detailed error logs).\n\n\n## Prerequisites\n\n1.  Follow the [getting started](https://cloud.google.com/genomics/docs/how-tos/getting-started)\n    instructions on the Google Cloud page.\n1.  Enable the [Genomics, Compute Engine, Cloud Storage, and Dataflow APIs](https://console.cloud.google.com/flows/enableapi?apiid=genomics,storage_component,storage_api,compute_component,dataflow)\n1.  Create a new BigQuery dataset by visiting the\n    [BigQuery web UI](https://bigquery.cloud.google.com/), clicking on the\n    down arrow icon next to your project name in the navigation, and clicking on\n    _Create new dataset_.\n\n## Loading VCF files to BigQuery\n\n### Using docker\n\nThe easiest way to run the VCF to BigQuery pipeline is to use the\n[docker](https://www.docker.com/) image, as it has the binaries and all\ndependencies pre-installed. Please ensure you have the latest `gcloud` tool by\nrunning `gcloud components update` (more details [here](https://cloud.google.com/sdk/gcloud/reference/components/update)).\n\nUse the following command to get the latest version of Variant Transforms.\n```bash\ndocker pull gcr.io/cloud-lifesciences/gcp-variant-transforms\n```\n\nRun the script below and replace the following parameters:\n\n* Dataflow's [required inputs](https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#configuring-pipelineoptions-for-execution-on-the-cloud-dataflow-service):\n  * `GOOGLE_CLOUD_PROJECT`: This is your project ID that contains the BigQuery\n  dataset.\n  * `GOOGLE_CLOUD_REGION`: You must choose a geographic region for Cloud Dataflow\n  to process your data, for example: `us-west1`. For more information please refer to\n  [Setting Regions](docs/setting_region.md).\n  * `GOOGLE_CLOUD_LOCATION`: You may choose a geographic location for Cloud Life\n  Sciences API to orchestrate job from. This is not where the data will be processed,\n  but where some operation metadata will be stored. This can be the same or different from\n  the region chosen for Cloud Dataflow. If this is not set, the metadata will be stored in\n  us-central1. See the list of [Currently Available Locations](https://cloud.google.com/life-sciences/docs/concepts/locations).\n  * `TEMP_LOCATION`: This can be any folder in Google Cloud Storage that your\n  project has write access to. It's used to store temporary files and logs\n  from the pipeline.\n* `INPUT_PATTERN`: A location in Google Cloud Storage where the\n  VCF file are stored. You may specify a single file or provide a pattern to\n  load multiple files at once. Please refer to the\n  [Variant Merging](docs/variant_merging.md) documentation if you want\n  to merge samples across files. The pipeline supports gzip, bzip, and\n  uncompressed VCF formats. However, it runs slower for compressed files as they\n  cannot be sharded.\n* `OUTPUT_TABLE`: The full path to a BigQuery table to store the output.\n\n```bash\n#!/bin/bash\n# Parameters to replace:\nGOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT\nGOOGLE_CLOUD_REGION=GOOGLE_CLOUD_REGION\nGOOGLE_CLOUD_LOCATION=GOOGLE_CLOUD_LOCATION\nTEMP_LOCATION=gs://BUCKET/temp\nINPUT_PATTERN=gs://BUCKET/*.vcf\nOUTPUT_TABLE=GOOGLE_CLOUD_PROJECT:BIGQUERY_DATASET.BIGQUERY_TABLE\n\nCOMMAND=\"vcf_to_bq \\\n  --input_pattern ${INPUT_PATTERN} \\\n  --output_table ${OUTPUT_TABLE} \\\n  --job_name vcf-to-bigquery \\\n  --runner DataflowRunner\"\n\ndocker run -v ~/.config:/root/.config \\\n  gcr.io/cloud-lifesciences/gcp-variant-transforms \\\n  --project \"${GOOGLE_CLOUD_PROJECT}\" \\\n  --location \"${GOOGLE_CLOUD_LOCATION}\" \\\n  --region \"${GOOGLE_CLOUD_REGION}\" \\\n  --temp_location \"${TEMP_LOCATION}\" \\\n  \"${COMMAND}\"\n```\n`--project`, `--region`, and `--temp_location` are required inputs. You must set all of them, unless your project and region default values\nare set in your local `gcloud` configuration. You may set the default project\nand region using the following commands:\n```bash\ngcloud config set project GOOGLE_CLOUD_PROJECT\ngcloud config set compute/region REGION\n```\n\nThere are options to control which service account, subnet and similar in the [Advanced Flags](docs/advanced_flags.md) documentation.\n\n\n### Running from github\n\nIn addition to using the docker image, you may run the pipeline directly from\nsource. First install git, python, pip, and virtualenv:\n\n```bash\nsudo apt-get install -y git python3-pip python3-venv python3.7-venv python-dev build-essential\n```\n\nNote that python 3.8 is not yet supported, so ensure you are using Python 3.7.\n\nRun virtualenv, clone the repo, and install pip packages:\n\n```bash\npython3 -m venv venv3\nsource venv3/bin/activate\ngit clone https://github.com/googlegenomics/gcp-variant-transforms.git\ncd gcp-variant-transforms\npython -m pip install --upgrade pip\npython -m pip install --upgrade wheel\npython -m pip install --upgrade .\n```\n\nYou may use the\n[DirectRunner](https://beam.apache.org/documentation/runners/direct/)\n(aka local runner) for small (e.g. 10,000 records) files or\n[DataflowRunner](https://beam.apache.org/documentation/runners/dataflow/)\nfor larger files. Files should be stored on Google Cloud Storage if using\nDataflow, but may be stored locally for DirectRunner.\n\nExample command for DirectRunner:\n\n```bash\npython -m gcp_variant_transforms.vcf_to_bq \\\n  --input_pattern gcp_variant_transforms/testing/data/vcf/valid-4.0.vcf \\\n  --output_table GOOGLE_CLOUD_PROJECT:BIGQUERY_DATASET.BIGQUERY_TABLE \\\n  --job_name vcf-to-bigquery-direct-runner \\\n  --temp_location \"${TEMP_LOCATION}\"\n```\n\nExample command for DataflowRunner:\n\n```bash\npython -m gcp_variant_transforms.vcf_to_bq \\\n  --input_pattern gs://BUCKET/*.vcf \\\n  --output_table GOOGLE_CLOUD_PROJECT:BIGQUERY_DATASET.BIGQUERY_TABLE \\\n  --job_name vcf-to-bigquery \\\n  --setup_file ./setup.py \\\n  --runner DataflowRunner \\\n  --project \"${GOOGLE_CLOUD_PROJECT}\" \\\n  --region \"${GOOGLE_CLOUD_REGION}\" \\\n  --temp_location \"${TEMP_LOCATION}\"\n```\n\n## Running VCF files preprocessor\n\nThe VCF files preprocessor is used for validating the datasets such that the\ninconsistencies can be easily identified. It can be used as a standalone\nvalidator to check the validity of the VCF files, or as a helper tool for\n[VCF to BigQuery pipeline](#loading-vcf-files-to-bigquery). Please refer to\n[VCF files preprocessor](docs/vcf_files_preprocessor.md) for more details.\n\n\n## Running BigQuery to VCF\n\nThe BigQuery to VCF pipeline is used to export variants in BigQuery to one VCF file.\nPlease refer to [BigQuery to VCF pipeline](docs/bigquery_to_vcf.md) for more\ndetails.\n\n\n## Additional topics\n\n* [Understanding the BigQuery Variants Table\n  Schema](https://cloud.google.com/life-sciences/docs/how-tos/bigquery-variants-schema)\n* [Loading multiple files](docs/multiple_files.md)\n* [Variant merging](docs/variant_merging.md)\n* [Handling large inputs](docs/large_inputs.md)\n* [Appending data to existing tables](docs/data_append.md)\n* [Variant Annotation](docs/variant_annotation.md)\n* [Sharding](docs/sharding.md)\n* [Flattening the BigQuery table](docs/flattening_table.md)\n* [Troubleshooting](docs/troubleshooting.md)\n\n## Development\n\n* [Development Guide](docs/development_guide.md)\n* [Release process](docs/release.md)\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglegenomics%2Fgcp-variant-transforms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglegenomics%2Fgcp-variant-transforms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglegenomics%2Fgcp-variant-transforms/lists"}