{"id":17835999,"url":"https://github.com/shiltemann/virtual-normal-correction","last_synced_at":"2025-03-19T17:32:56.152Z","repository":{"id":71194335,"uuid":"44116602","full_name":"shiltemann/Virtual-Normal-Correction","owner":"shiltemann","description":"Set of commandline tools for analysis of tumor variants without a matching normal. Method described in Genome Research: http://genome.cshlp.org/content/25/9/1382.full","archived":false,"fork":false,"pushed_at":"2022-05-10T11:48:08.000Z","size":19642,"stargazers_count":6,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-17T09:45:32.473Z","etag":null,"topics":["cancer-genomics","genomics","variant-analysis"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shiltemann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-12T15:32:33.000Z","updated_at":"2021-12-09T14:52:46.000Z","dependencies_parsed_at":"2023-11-03T22:45:23.681Z","dependency_job_id":null,"html_url":"https://github.com/shiltemann/Virtual-Normal-Correction","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiltemann%2FVirtual-Normal-Correction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiltemann%2FVirtual-Normal-Correction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiltemann%2FVirtual-Normal-Correction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shiltemann%2FVirtual-Normal-Correction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shiltemann","download_url":"https://codeload.github.com/shiltemann/Virtual-Normal-Correction/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244474123,"owners_count":20458591,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cancer-genomics","genomics","variant-analysis"],"created_at":"2024-10-27T20:29:54.722Z","updated_at":"2025-03-19T17:32:56.141Z","avatar_url":"https://github.com/shiltemann.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Virtual Normal Correction\n\n## Deprecation Notice\n\nThis tool is currently deprecated and no longer useful. Please have a look at alternative tools, such as ones described and compared in [Little et. al. 2021, UNMASC: tumor-only variant calling with unmatched normal controls](https://pubmed.ncbi.nlm.nih.gov/34632388/ )\n\n## Documentation\nThis code performs a virtual normal correction on a set of input variants. For more information see our paper in Genome Research [here](http://genome.cshlp.org/content/25/9/1382.abstract)\n\nThis analysis can be run on both small variants (i.e. single nucleotide variants and indels of up to ~50 bps) and the larger structural variations (SVs)\n\nFor any questions, please contact Saskia Hiltemann (s.hiltemann@erasmusmc.nl)\n\nTools have moved to the main Galaxy ToolShed: https://toolshed.g2.bx.psu.edu/\n\n## Example\n\nThere is an example run using just 3 normals (chr21 only) and input from HCC1187 (chr21 only) available in the folder \"example\". See the README document in the example folder for more information about running the example data.\n\n\n## How to Run\n\n### Preliminaries\n\n**Reference Genome**\nDownload the reference genome (.crr file) of the desired build (hg18=build36, hg19=build37) from Complete Genomics: [ftp://ftp.completegenomics.com/ReferenceFiles/](ftp://ftp.completegenomics.com/ReferenceFiles/). \n\n**CGATools binary**\nThe cgatools binary version 1.8 is included in this repository under bin/cgatools18, alternatively it may be obtained from [cgatools.sourceforge.net](cgatools.sourceforge.net).\n\nMake sure the cgatools binary is available on your PATH. For example do something like:\n\nThe program expects a binary called *cgatools18* to be present on the PATH. To change the expected name of this binary, edit the *cgatools* variable at the top of the scripts: \n\n```\n#!/bin/bash\n\n### location of cgatools binary, change as needed, binaries may be downloaded from Complete Genomics website here: http://cgatools.sourceforge.net/\n# latest version as of the writing of this code is included in this repository (v1.8)\ncgatools=\"cgatools18\"\n```\n\n**Normal Samples**\nConfigure your Virtual Normal set (see section *Virtual Normal Sets* below)\n\n\n### Small Variants\n\nRunning the VN correction on small variant data:\n\n```bash\n$ bash virtual-normal-correction-smallvariants.sh \\\n    --variants \u003cinput list of variants\u003e \\\n    --reference \u003creference .crr file\u003e \\\n    --VN_varfiles_list \u003clist with paths to CG-sequenced normals, one file per line\u003e \\\n    --threshold \u003cfilter out variants appearing in at least this many normals\u003e \\\n    --threshold_highconf \u003conly output variants fully called in at least this many normals\u003e \\\n    --outputfile_all \u003cwhere to store final output\u003e \\\n    --outputfile_filtered \u003cwhere to store final output filtered for the two threshold values\u003e \\\n```\n\n### Structural Variations\n\nRunning the VN correction on small variant data:\n\n```bash\n$ bash virtual-normal-correction-SVs.sh \\\n          --variants \u003cCG junctions file\u003e \\\n          --reference \u003creference .crr file\u003e \\\n          --VN_junctionfiles_list \u003clist with paths to CG-sequenced normals, one file per line\u003e \\\n          --output_prefix \u003cwhere to store final output, for example output/mysample\u003e         \n```\n\nOptionally the following additional parameters may also be given, if any of these parameters are not specified, the default values are used:\n\n```bash\n--scoreThresholdA \u003cdefault 10. The minimum number of discordant mate pair alignments supporting the junction from input genome\u003e \n--scoreThresholdB \u003cdefault 10. The minimum number of discordant mate pair alignments supporting the junction from normal genoes\u003e \n--distance \u003cdefault 200. Maximum distance between coordinates of potentially compatible junctions.\u003e\n--minlength \u003cdefault 500. Minimum deletion junctions length to be included into the difference file\u003e\n```\n\n## Input format\n\n### Small Variants\n\nThe input format may be obtained from a CG varfile by running the CGAtools ListVariants tool, or may be obtained from a VCF file by running the conversion script included in this repository (VCF-2-Variantlist.sh).\n\nExample of VCF conversion:\n\n```\n$ bash VCF-2- Variantlist.sh \u003cVCF file\u003e \u003coutput variantlist\u003e\n```\n\nExample CG varfile conversion:\n\n```\ncgatools18 listvariants --beta --variants \u003cvarfile\u003e --reference \u003creference crr file\u003e \n```\n\nThe input format must be a tab-delimited file with the following header line:\n\n```\nvariantId - chromosome - begin - end - varType - reference - alleleSeq - xRef\n```\n\n**variantID** - may be any unique identifier  \n**chromosome** - must have a *chr* prefix  \n**begin** - starting cooridinate in 0-based half-open format  \n**end** - end coordinate of the variant in 0-based half-open format  \n**varType** - must be one of the following values: snp,ins,del,sub  \n**reference** - nucleotide sequence of reference genome at given locus  \n**alleleSeq** - observed nucleotide sequence in sample at given locus  \n**xRef** - is not used by this program but may contain any annotations you wish or may be left blank or omitted altogether  \n\nExample:\n\n```\nvariantId       chromosome      begin   end     varType reference       alleleSeq       xRef\n1               chr1            38231   38232   snp     A               G               dbsnp.86:rs806727;dbsnp.131:rs77823476\n2               chr1            46669   46670   snp     A               G               dbsnp.100:rs2548905\n3               chr1            47107   47108   snp     G               C               dbsnp.100:rs2531241\n4               chr1            47291   47292   snp     T               G               dbsnp.100:rs2691275\n5               chr1            49271   49272   snp     G               A               dbsnp.100:rs2531245\n6               chr1            49290   49291   snp     C               T               dbsnp.130:rs71240757\n7               chr1            49313   49315   sub     GT              AC                \n... \n```\n\n### Structural Variations\n\nFor the SV analysis, input must be a Complete Genomics junctions file.\n\n\n## Output format\n\n### Small Variants\n\nThere are 3 output files for the small variant analysis:\n\n1. The main output file  \n   containing all variants found in the sample, annotated with the frequency in the normal sets.\n2. The filtered output file  \n   same format as 1 but all variants which occur in more than the specified number of normals were removed, as well as all variant which were no-called or half-called in more than the specified number of normals\n3. The expanded file  \n   This is the input file with a column added per normal genome specifying the genotype (00 for not present, 01 for heterozygous, 11 for homozygous, NN for no-called, etc)\n\n\n*Example of main output file:*\n\n```\nvariantId chromosome begin   end     varType reference alleleSeq xRef                  VN_occurrences VN_frequency VN_fullycalled_count VN_fullycalled_frequency VN_00 VN_01 VN_11 VN_0N VN_1N VN_NN VN_0 VN_1 VN_N\n3155150   chr21      9412628 9412629 snp     C         T         dbsnp.130:rs71220886  1              0.333333     1                    0.333333                 1     0     0     0     1     1     0    0    0\n[..]\n```\n\nDescription of the columns added:\n\n**VN_occurrences** - Number of normals this variants was present in  \n**VN_frequency** - Fraction of normals this variant was present in   \n**VN_fullycalled_count** - Number of normals that were fully-called at this locus  \n**VN_fullycalled_frequency** - Franction of normals that were fully-called at this locus  \n**VN_00** - Number of normals without the variant and fully called at the locus.  \n**VN_01** - Number of normals with heterozygous occurrence of the variant.  \n**VN_11** - Number of normals with homozygous occurrence of the variant.  \n**VN_0N** - Number of normals without variant but half-called at the locus.  \n**VN_1N** - Number of normals with the variant, but half-called at the locus.  \n**VN_NN** - Number of normals without the variant, but nocalled at the locus.  \n**VN_0** - Number of normals without the variant, but with only one allele present (e.g. X and Y chromosomes).  \n**VN_1** - Number of normals with the variant, but with only one allele present (e.g. X and Y chromosomes).  \n**VN_N** - Number of normals without the variant, but no-called at the locus with only one allele present (e.g. X and Y chromosomes).  \n\n\n*Example of expanded output file:*\n\n```\nvariantId chromosome begin   end     varType reference alleleSeq xRef                 NA06985-200-37-ASM NA06994-200-37-ASM     NA07357-200-37-ASM\n3155150   chr21      9412628 9412629 snp     C         T         dbsnp.130:rs71220886 00                 NN                     1N\n[..]\n```\n\n\n### Structural Variations\n\nThere are two output files for the SV analysis:\n\n1. Filtered Junctions  \n2. Report  \n\nThe output junctions are the same format as the input; CG junctions file, but with SVs that were present in one or more of the normals removed.\n\nThe report tells you how many variants were removed at each step. \n\n*Example Report*\n\n```\nreport of run 1:\n----------------------\n#GENERATED_BY\tcgatools\n#GENERATED_AT\t2015-Oct-12 16:06:03.542623\n#CGATOOLS_VERSION\t1.8.0\n#FORMAT_VERSION\t2.4\n#TYPE\tJUNCTIONDIFF_REPORT\n\n\u003efileId\tinputJunctions\tfilteredJunctions\tincompatible\tfilteredIncompatible\tcompatible\tscoreThreshold\tmaxDistance\tminDelLength\n65\t1581\t1581\t746\t621\t835\t10\t200\t500\n66\t5376\t2426\t1590\t0\t836\t10\t200\t500\n\nreport of run 2:\n----------------------\n#GENERATED_BY\tcgatools\n#GENERATED_AT\t2015-Oct-12 16:06:03.597889\n#CGATOOLS_VERSION\t1.8.0\n#FORMAT_VERSION\t2.4\n#TYPE\tJUNCTIONDIFF_REPORT\n\n\u003efileId\tinputJunctions\tfilteredJunctions\tincompatible\tfilteredIncompatible\tcompatible\tscoreThreshold\tmaxDistance\tminDelLength\n65\t621\t621\t504\t504\t117\t10\t200\t500\n66\t6489\t2163\t2047\t0\t116\t10\t200\t500\n\nreport of run 3:\n----------------------\n#GENERATED_BY\tcgatools\n#GENERATED_AT\t2015-Oct-12 16:06:03.655524\n#CGATOOLS_VERSION\t1.8.0\n#FORMAT_VERSION\t2.4\n#TYPE\tJUNCTIONDIFF_REPORT\n\n\u003efileId\tinputJunctions\tfilteredJunctions\tincompatible\tfilteredIncompatible\tcompatible\tscoreThreshold\tmaxDistance\tminDelLength\n65\t504\t504\t447\t447\t57\t10\t200\t500\n66\t4883\t2088\t2031\t0\t57\t10\t200\t500\n```\n\nfor each normal genome, the first line indicates the current set of junctions, and the second line indicates the normal junctionsfile. So in this example, in the first run we started with 1581 input junctions, we compared this to a normal containing 5376 junctions, and were left with 621 junctions after correction.\n\n\n## Virtual Normal Sets\n\n### How to specify\n\nDownload all CG varfiles you wish to use as a virtual normal. Create a text file with their locations, one per line. For instance, for the example data this text file looks like\n\n```\nnormals/smallvariants/var-NA06985-200-37-ASM_chr21.tsv\nnormals/smallvariants/var-NA06994-200-37-ASM_chr21.tsv\nnormals/smallvariants/var-NA07357-200-37-ASM_chr21.tsv\n```\n\nBe careful not to have any empty lines in your file.\n\n\n### Where to download\n\n#### 1000 Genomes Project Data\nThe set of CG-sequenced normals used in the paper may be downloaded from the 1000 Genomes Project site. Only for build hg19.\n\nHere is a list of all the locations of the CG normals on the 1000Genomes site:\n\nftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/historical_data/former_toplevel/complete_genomics_indices/20130725.cg_data.untar.index\n\n(just download all files starting with \"var-\" (and for SVs download everything starting with \"allJunctions\") )\n\nNOTE: it seems they have recently moved the files, and not yet updated the list, and you should now put \"phase3\" before all locations it seems, so if the file says:\ndata/NA12812/cg_data/ASM_lcl/var-GS000016405-ASM.tsv.bz2\nthe address now is:\nftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA12812/cg_data/ASM_lcl/var-GS000016405-ASM.tsv.bz2\netc.\n\n#### Complete Genomics Diversity Panel\nA smaller set of normals, has some overlap with the set of normals in 1000 Genomes Project, but also available on build hg18. This data may be downloaded from the Complete Genomics FTP server: [ftp://ftp2.completegenomics.com/Diversity/](ftp://ftp2.completegenomics.com/Diversity/)\n\n### Complete Genomics Trios\nThe parents from the trios available from Complete Genomics may also be used in the virtual normal and can be obtained from the Complete Genomics FTP server: [ftp://ftp2.completegenomics.com/YRI_trio](ftp://ftp2.completegenomics.com/YRI_trio/) and\n[ftp://ftp2.completegenomics.com/PUR_trio/](ftp://ftp2.completegenomics.com/PUR_trio/)\n\nBe careful not to include the child, as this will skew the frequencies of occurrence of variants within the population; you should only ever use unrelated individuals.\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshiltemann%2Fvirtual-normal-correction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshiltemann%2Fvirtual-normal-correction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshiltemann%2Fvirtual-normal-correction/lists"}