{"id":28395293,"url":"https://github.com/databiosphere/clinvar-ingest","last_synced_at":"2025-06-27T04:31:51.180Z","repository":{"id":36073620,"uuid":"221254658","full_name":"DataBiosphere/clinvar-ingest","owner":"DataBiosphere","description":"Batch ETL pipeline to mirror ClinVar releases into the Jade Data Repository.","archived":false,"fork":false,"pushed_at":"2024-01-22T18:29:13.000Z","size":670,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-06-01T06:52:51.408Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-11-12T15:49:54.000Z","updated_at":"2022-03-14T13:40:09.000Z","dependencies_parsed_at":"2023-01-16T12:50:13.104Z","dependency_job_id":"ac1130d1-7f01-4674-8ba4-b2cba4a70314","html_url":"https://github.com/DataBiosphere/clinvar-ingest","commit_stats":null,"previous_names":[],"tags_count":238,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/clinvar-ingest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fclinvar-ingest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fclinvar-ingest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fclinvar-ingest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fclinvar-ingest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/clinvar-ingest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fclinvar-ingest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262188396,"owners_count":23272341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:45.722Z","updated_at":"2025-06-27T04:31:51.170Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ClinVar Ingest\nBatch ETL pipeline to mirror ClinVar releases first into the Terra Data Repository (TDR),\nthen into ClinGen's Kafka processing pipeline.\n\n## Schema Design\nThe schema used for this dataset was (largely) designed by the ClinGen team. JSON\ndefinitions of the resulting tables can be found under [`schema/`](./schema). Some\npieces of the design are worth calling out here:\n1. Every table includes a `release_date` column\n2. Most tables contain a `content` column to store unmodeled data\n3. Some tables are used to track processing provenance, instead of primary data\n\n### Using Release Date\nEvery table in the ClinVar dataset includes a `release_date` column, used as (part of)\nthe primary key. This means that the base TDR dataset shows the latest data for _every_\nprocessed release, not just the latest processed release. We do this to enable:\n* Parallel processing \u0026 ingest of raw archives\n* Targeted reprocessing of archives\n\nThe downsides of this scheme are:\n* A significant amount of data is duplicated from release to release\n* Creating a snapshot of a single release is more complicated\n\n### Handling Unmodeled Data\nMost tables in the ClinVar dataset include a `content` column. We use these columns to\nstore unmodeled data as string-encoded JSON. This allows us to fully ETL the pieces of\nthe data we're most interested in, without throwing any information away. It also insulates\nus against future additions in the NCBI source schema.\n\n### Tracking Provenance\nThe `xml_archive` and `processing_history` tables both track provenance-related\ninformation in the dataset.\n* `xml_archive` tracks references to raw XML payloads per release\n* `processing_history` tracks the pipeline version and run date of processing per release\n\n\nThe ingest pipeline uses information in these tables to short-circuit when appropriate.\n* If a release date is already present in `xml_archive`, the corresponding raw XML is\n  not re-ingested\n* If a release date / pipeline version pair is already present in `processing_history`, the\n  corresponding raw XML is not re-processed using that pipeline version\n\n## Pipeline Architecture\nThe ingest pipeline is orchestration through Argo, with most data processing logic\ndelegated to Dataflow and BigQuery. The high level flow looks like:\n![Architecture diagram](./ingest-flow.png)\n\nAs shown, end-to-end ingest is broken into 3 distinct sub-workflows, to mitigate\nfailures and reduce the work needed to run a partial reprocessing. The 3 phases are:\n1. Raw data ingest\n2. Data processing and re-ingest\n3. Export to ClinGen\n\n### Raw Ingest\nStage 1 of ingest prioritizes stashing the raw XML payloads from NCBI in the TDR. We package\nthis as a focused end-to-end workflow because NCBI weekly releases are only available during\nthe same month as their release. If we tightly coupled ingesting these raw payloads with their\ndownstream processing, and that processing suddenly broke, it could take us \"too long\" to fix\nthe logic and the weekly release could disappear. Separating the XML ingest into its own step\nminimizes the chances of this scenario.\n\n### Data Processing\nStage 2 of ingest runs the bulk of our business logic. For a target release date, it:\n1. Downloads a raw XML release from the TDR, to a local volume in k8s\n2. Converts the XML to JSON-list, and uploads it back to GCS\n3. Runs a Dataflow pipeline to process the JSON into our target schema\n4. Per-table:\n   1. Uses BigQuery to diff staged data against the current TDR state\n   2. Uses the TDR APIs to ingest/soft-delete data so that the TDR state\n      matches staged data\n5. Cuts a TDR snapshot for the release\n\n### Export to ClinGen\nStage 3 of ingest exports processed data to ClinGen. For a pair of release dates, it:\n1. Per-table:\n   1. Compares data across the two releases to get a diff of created/updated/deleted rows\n   2. Exports the three \"slices\" of data from the diff into GCS\n2. Produces a Kafka message to ClinGen, pointing at the exported data\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fclinvar-ingest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fclinvar-ingest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fclinvar-ingest/lists"}