{"id":16618623,"url":"https://github.com/brentp/echtvar","last_synced_at":"2025-04-04T23:09:15.996Z","repository":{"id":40642446,"uuid":"430700380","full_name":"brentp/echtvar","owner":"brentp","description":"using all the bits for echt rapid variant annotation and filtering","archived":false,"fork":false,"pushed_at":"2025-03-21T15:53:41.000Z","size":219,"stargazers_count":150,"open_issues_count":8,"forks_count":10,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-28T22:14:16.581Z","etag":null,"topics":["genetic-variants","genomics","variant-analysis","variant-annotations"],"latest_commit_sha":null,"homepage":"https://academic.oup.com/nar/advance-article/doi/10.1093/nar/gkac931/6775383","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brentp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-22T12:37:52.000Z","updated_at":"2025-03-27T19:17:20.000Z","dependencies_parsed_at":"2024-02-06T19:28:21.148Z","dependency_job_id":"57b69868-045c-4b1e-8ec8-7009af18bdd0","html_url":"https://github.com/brentp/echtvar","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fechtvar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fechtvar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fechtvar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fechtvar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brentp","download_url":"https://codeload.github.com/brentp/echtvar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247261612,"owners_count":20910108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["genetic-variants","genomics","variant-analysis","variant-annotations"],"created_at":"2024-10-12T02:20:56.895Z","updated_at":"2025-04-04T23:09:15.965Z","avatar_url":"https://github.com/brentp.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Echtvar: Really, truly rapid variant annotation and filtering \n[![Rust](https://github.com/brentp/echtvar/actions/workflows/ci.yml/badge.svg)](https://github.com/brentp/echtvar/actions/workflows/ci.yml)\n\nEchtvar efficiently encodes variant allele frequency and other information from huge population datasets to enable rapid (1M variants/second) annotation of genetic variants.\nIt chunks the genome into 1\u003c\u003c20 (~1 million) bases,\n[encodes each variant into a 32 bit integer](https://github.com/brentp/echtvar/blob/02774b8d1cd3703b65bd2c8d7aab93af05b7940f/src/lib/var32.rs#L9-L21) (with a [supplemental table](https://github.com/brentp/echtvar/blob/02774b8d1cd3703b65bd2c8d7aab93af05b7940f/src/lib/var32.rs#L33-L38)\nfor those that can't fit due to large REF and/or ALT alleles). It uses the zip format, [delta\nencoding](https://en.wikipedia.org/wiki/Delta_encoding)\nand [integer compression\n](https://lemire.me/blog/2017/09/27/stream-vbyte-breaking-new-speed-records-for-integer-compression/)\nto create a compact and searchable format of any integer, float, or low-cardinality string columns\nselected from the population file.\n\nread more at the [why of echtvar](https://github.com/brentp/echtvar/wiki/why)\n\n### Getting started.\n\nGet a static binary and pre-encoded echtvar files for gnomad v3.1.2 (hg38) here: https://github.com/brentp/echtvar/releases/latest\nThat page contains exact instructions to get started with the static binary.\n\n\u003cdetails\u003e\n  \u003csummary\u003e:arrow_down:Download or Build instructions for linux\u003c/summary\u003e\n\nThe linux binary is available via:\n\n```\nwget -O ~/bin/echtvar https://github.com/brentp/echtvar/releases/latest/download/echtvar \\\n    \u0026\u0026 chmod +x ~/bin/echtvar \\\n    \u0026\u0026 ~/bin/echtvar # show help\n ```\n\nUsers can make their own *echtvar* archives with `echtvar encode`, and pre-made archives for\ngnomAD version 3.1.2 are [here](https://github.com/brentp/echtvar/release)\n\nRust users can build on linux with:\n\n```\ncargo build --release --target x86_64-unknown-linux-gnu\n```\n\n\u003c/details\u003e\n\nTo run echtvar with an existing archive (we have several available in [releases](https://github.com/brentp/echtvar/releases/latest)) is as simple as\n```\nechtvar anno -e gnomad.echtvar.zip -e other.echtvar.zip input.vcf output.annotated.bcf\n```\n\nan optional filter that utilizes fields available any of the zip files can be added like:\n```\n-i \"gnomad_popmax_af \u003c 0.01\"\n```\n\nechtvar can also accept input from stdin using \"-\" or \"/dev/stdin\" for the input argument.\n\n### usage\n\n##### encode \n\nmake (`encode`) a new echtvar file. This is usually done once  (or download from those provided in the [Release pages](https://github.com/brentp/echtvar/releases/latest)) \nand then the file can be re-used for the annotation (`echtvar anno`) step with each new query file.\nNote that input VCFs must be [decomposed](https://github.com/brentp/echtvar/wiki/decompose).\n\n```\nechtvar \\\n   encode \\\n   gnomad.v3.1.2.echtvar.zip \\\n   conf.json # this defines the columns to pull from $input_vcf, and how to\n   $input_population_vcf[s] \\ can be split by chromosome or all in a single file.\nname and encode them\n\n```\n\nSee below for a description of the json file that defines which columns are\npulled from the population VCF.\n\n##### annotate \n\nAnnotate a [**decomposed** (and normalized)](https://github.com/brentp/echtvar/wiki/decompose) VCF with an echtvar file and only output variants where `gnomad_af`\nfrom the echtvar file is \u003c 0.01. Note that multiple echtvar files can be specified\nand the `-i` expression is optional and can be elided to output all variants.\n\n```\nechtvar anno \\\n   -e gnomad.v3.1.2.echtvar.v2.zip \\\n   -e dbsnp.echtvar.zip \\\n   -i 'gnomad_popmax_af \u003c 0.01' \\\n   $cohort.input.bcf \\\n   $cohort.echtvar-annotated.filtered.bcf\n```\n\n#### Configuration File for Encode\n\nWhen running `echtvar encode`, a [json5](https://json5.org/) (json with\ncomments and other nice features) determines which columns are pulled from the\ninput VCF and how they are stored.\n\nA simple example is to pull a single integer field and give it a new name (`alias`):\n\n```\n[{\"field\": \"AC\", \"alias\": \"gnomad_AC\"}]\n```\n\nThis will extract the \"AC\" field from the INFO and labeled as \"gnomad_AC\" when\nlater used to annotate a VCF. Note that it's important to give a description/unique prefix lke \"`gnomad_`\" so\nas not to collide with fields already in the query VCF.\n\n\u003cdetails\u003e\n  \u003csummary\u003e:arrow_down:Expand this section for detail on additional fields, including float and string types\u003c/summary\u003e\n\n```\n[\n    {\"field\": \"AC\", \"alias\": \"gnomad_AC\"},\n    // this JSON file is json 5 and so can have comments\n    // the missing value will default to -1, but the value: -2147483648 will\n    // result in '.' as it is the missing value for VCF.\n    {\"field\": \"AN\", \"alias\":, gnomad_AN\", missing_value: -2147483648},\n    {\n           field: \"AF\",\n           alias: \"gnomad_AF\",\n           missing_value: -1,\n           // since all values (including floats) are stored as integers, echtvar internally converts\n           // any float to an integer by multiplying by `multiplier`.\n           // higher values give better precision and worse compression.\n           // upon annotation, the score is divided by multiplier to give a number close to the original float.\n           multiplier: 2000000,\n           // set zigzag to true if your data has negative values\n           zigzag: true,\n   }\n    // echtvar will save strings as integers along with a lookup. this can work for fields with a low cardinality.\n    {\"field\": \"string_field\", \"alias\":, gnomad_string_field\", missing_string: \"UNKNOWN\"},\n    // \"FILTER\" is a special case that indicates that echtvar should extract the FILTER column from the annotation vcf.\n    {\"field\": \"FILTER\", \"alias\": \"gnomad_filter\"},\n]\n```\n\nThe above file will extract 5 fields, but the user can chooose as many as they like when encoding.\nAll fields in an `echtvar` file will be added (with the given alias) to any VCF it is used to annotate.\n\n\u003c/details\u003e\n\nOther examples are available [here](https://github.com/brentp/echtvar/tree/main/examples)\n\nAnd full examples are in the [wiki](https://github.com/brentp/echtvar/wiki)\n\n#### Expressions\n\nAn optional expression will determine which variants are written. It can utilize any (and only) integer or float fields present in the\nechtvar file (not those present in the query VCF). An example could be:\n\n```\n-i 'gnomad_af \u003c 0.01 \u0026\u0026 gnomad_nhomalts \u003c 10'\n```\n\nThe expressions are enabled by [fasteval](https://github.com/likebike/fasteval) with supported syntax detailed [here](https://docs.rs/fasteval/latest/fasteval/). \n\nIn brief, the normal operators: (`\u0026\u0026, ||, +, -, *, /, \u003c, \u003c=, \u003e, \u003e=` and groupings `(, )`, etc) are supported and can be used to\ncraft an expression that returns true or false as above.\n\n# References and Acknowledgements\n\nWithout these (and other) critical libraries, `echtvar` would not exist.\n\n+ [htslib](https://github.com/samtools/htslib) is used for reading and writing BCF and VCF via [rust-htslib](https://github.com/rust-bio/rust-htslib)\n+ [stream-vbyte](https://lemire.me/blog/2017/09/27/stream-vbyte-breaking-new-speed-records-for-integer-compression/) is used for integer compression via the [excellent rust bindings](https://bitbucket.org/marshallpierce/stream-vbyte-rust/src/master/)\n+ [fasteval](https://github.com/likebike/fasteval) is used for the expressions. It is fast and simple and awesome.\n+ [bincode](https://docs.rs/bincode/latest/bincode/) is used for rapid serialization of large variants.\n\n\n`echtvar` is developed in the [Jeroen De Ridder lab](https://www.deridderlab.nl/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrentp%2Fechtvar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrentp%2Fechtvar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrentp%2Fechtvar/lists"}