{"id":22504972,"url":"https://github.com/bcbi/genbank_holobiome","last_synced_at":"2025-03-27T23:21:43.374Z","repository":{"id":44352259,"uuid":"402057828","full_name":"bcbi/genbank_holobiome","owner":"bcbi","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-23T21:22:25.000Z","size":5142,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-02T01:27:33.228Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bcbi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-01T12:46:31.000Z","updated_at":"2022-07-19T04:33:05.000Z","dependencies_parsed_at":"2024-12-07T00:12:38.348Z","dependency_job_id":"0b6eb4e7-8d50-4c27-8367-e7bcab086962","html_url":"https://github.com/bcbi/genbank_holobiome","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_holobiome","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_holobiome/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_holobiome/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_holobiome/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bcbi","download_url":"https://codeload.github.com/bcbi/genbank_holobiome/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245938756,"owners_count":20697086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T00:12:30.349Z","updated_at":"2025-03-27T23:21:43.346Z","avatar_url":"https://github.com/bcbi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Genbank Database Pipeline \n#### Authors: Vivek Ramanan and Shanti Mechery \n\nThe GenBank DataBase Pipeline has two purposes: to map source site data\nfrom the National Center for Biotechnology Information's GenBank Database\nto their preferred name according to the the National Institute of Health's\nUnited Medical Language System and to create a matrix of bacteria species\nand host species data, both also from the National Center for Biotechnology\nInformation's GenBank Database, pertaining to a specific source site. The first\npart is the Upstream Process and the second part is the Downstream Process.\nThis matrix created in the Downstream Process will be a part of a NEXUS file,\nwhich can be used as input for PAUP*, a phylogenetic analysis software that\ncreates phlyogentic trees. Each section of this guide explains the processes\nin further detail. If the most up-to-date input data from the GenBank database\nand the UMLS for the pipeline are needed, start with the Upstream Process.\nNote that this process only needs to be run once. If all the data is up-to-date,\nonly the Downstream process needs to be run. The last GenBank download was \nperformed in November 2020. \n\nThe data from the GenBank Database can be downloaded using a GenBank Loader to\na MySQL database (details in the github repo BCBI/genbank_loader). In the GenBank\nDatabase, an entry has the following information: Locus, Organism, Source Site \n(isolation_source or tissue_type), and Host. The GenBank Loader imports this data into\na MySQL database named 'genbank' with the following tables: annotations, authors, \ndbxrefs, journals, keywords, and basic. Only the annotations table is needed, with the \nfollowing columns: partitionKey, locus, name, indexedValue, and value. Each row of the\ntable contains a single piece of type information (name column) with a value (value column)\npertaining to a locus number (locus column). The types of information included in the\nname column are organism, source site (isolation_source or tissue_type) and host. \nFor the pipeline, use a cleaned version of the host data, which is found in the table \ncleaned_host_annotations. The last cleaned_host_annotations was created in March 2021\nand can be updated accordingly using the hostClean scripts included. \n\nPlease refer first to the \"GenBank_Write_Up.pdf\" for the first version \nof this pipeline. \n\nDependencies: \n1. Julia\n2. Python3\n    - BioPython (Entrez) and NCBI Entrez Account information\n    - Pandas and Numpy\n    - SKLearn, markov_clustering, and NetworkX\n3. R\n    - ggTree and treeIO\n4. TimeTree.org\n\n** If using Oscar, run ALL of these only using Batch jobs (or interact jobs for testing) **\n\n## Data Download and Processing Process: \n\nThis section processes the source site data (isolation_source or tissue_type) into\nCSV files using the UMLS MetaMap program. This process must be run individually\nfor isolation_source and tissue_type. MetaMap installation can be found at \nhttps://lhncbc.nlm.nih.gov/ii/tools/MetaMap/documentation/Installation.html and \ncan be downloaded with the binary onto Oscar or any other remote server. This section\nis built in Julia solely and requires some MySQL commands as well. \n\n1. Downloading the GenBank data (if needs to be updated): \n    - Refer to https://github.com/bcbi/genbank_loader\n    - Original source code: https://bitbucket.org/UVM-BIRD/genbank-loader/src\n\n2. MySQL Queries for source site and bacteria species: \n\nThe following files have been given with their source type from annotations: \n    - isolationLocus.csv: isolation_source\n    - tissueLocus.csv: tissue_type\n    - allBacteriaSpecies.csv: organism\n\nGeneral Query: \n```\nselect locus,value from annotations where name=\"isolation_source\"\nselect locus,value from annotations where name=\"tissue_type\"\nselect locus,value from annotations where name=\"organism\"\n```\n\nHOWEVER, if you are not able to access the MySQL queries directly or more likely, \nbecause the tables are too large to run directly on the command line, please use\nthe following command line batch jobs instead. \n\nex: IsolationLocus.CSV command\nFirst create the table in mySQL if it does not exist: \n$ create table isolationLocus(locus varchar(20), value longtext)\nThen we run the following command outside of mySQL from command line: \n```\nmysql --enable-cleantext-plugin -h \u003cDBName\u003e -u \u003cusername\u003e -p \u003cpassword\u003e -D genbank\n-e \"insert into isolationLocus select locus,value from annotations where name='isolation_source';\"\n```\n\nThis will create the mySQL table in the database properly. Sometimes you will get a timed\nout output, but this is likely because the table finished loading and the mySQL server\ntimed out after that so it should've run correctly. \n\n**IF YOU ARE UPDATING cleaned_host_annotations: also run hostLocus.csv (\"where name='host'\")\n\n3. Downloading the mySQL tables:\n\nIf you are working off of command line and on a remote server, and cannot download the mySQL\ntables directly, you will need to use MySQLDump. \n```\nmysqldump --enable-cleartext-plugin -h \u003cDBname\u003e -u \u003cusername\u003e -p \u003cpassword\u003e genbank tissueLocus \u003e tissueLocus.sql\n```\n** occasionally I have noticed that the space between \"-p\" and the password can cause errors. \nTry using without a space between the -p flag and the password if this occurs. **\n\n4. Converting .SQL files into .CSV\n\nTo convert .SQL files into .CSV, I used a preexisting Python program called mysqldump-to-csv \nthat I updated for this project specifically. \n```\npython3 mysqldump-to-csvV2.py \u003cfile.SQL\u003e\n```\nNow the data should be in a CSV format and ready to process moving forward. These are the files\nyou should have: \n    - isolationLocus.csv\n    - tissueLocus.csv\n    - allBacteriaSpecies.csv\n    - (if updating cleaned_host_annotations: hostLocus.csv)\nThe Locus files are the ones that will go through the upstream process (not allBacteriaSpecies.csv)\n\n## Upstream Process:\n\n1. upstream_1_valueCSV_parser.jl\n\nThis file processes CSVs into TXT files for MetaMap to take in as input. \nInput: source site CSV (ex: tissueLocus.csv, isolationLocus.csv, or hostLocus.csv)\nOutput: appropriate MetaMap input format for text file (sourceSite_MetaMap_input.txt)\n```\njulia upstream_1_valueCSV_parser.jl sourceSiteLocus.csv sourceSite_MetaMap_input.txt\n```\n2. Running MetaMap\n```\n$ ./public_mm/bin/wsdserverctl start\n$ ./public_mm/bin/skrmedpostctl start\n$ ./public_mm/bin/metamap -I sourceSite_MetaMap_input.txt sourceSite_MetaMap_output.txt\n$ ./public_mm/bin/wsdserverctl stop\n$ ./public_mm/bin/skrmedpostctl stop\n```\n** In particular, the isolationLocus.CSV was much larger than the other files for me. I\nhad to split this file into 5 files to get MetaMap to run on each one individually,\nthen joined them all together after for the following steps. \n\n3. upstream_2_metamapParser.jl\n\nEach UMLS concept has a Semantic Type and this script filters for the Semantic Types that we want,\nwhich is specific to body parts and organs. \n** If updating cleaned_host_annotations, please use semanticHost.txt here instead **\n\nInput: sourceSite_MetaMap_output.txt\nOutput: 1) sourceSite_MetaMap_parsed.csv\n            - columns: original, preferred, semanticType\n            - original: original source site from Genbank\n            - preferred: source site preferred MEtaMap name, all concept names concatenated\n            - semanticType: semantic type associated with each MetaMap concept\n        2) sourceSite_CUI.csv\n            - CUI: concept unique identifiers\n```\n$ julia upstream_2_metamapParser.jl sourceSite_MetaMap_output.txt semanticTypes.txt sourceSite_MetaMap_parse.csv sourceSite_CUI.csv\n```\n4. upstream_3_createBacteriaSpecieFile.jl\n\nNow the appropriate bacteria species data need to be filtered out from allBacteriaSpecies.csv\nfile for each source site. This will be a large file. \n```\n$ julia upstream_3_createBacteriaSpecieFile.jl sourceSiteLocus.csv allBacteriaSpecies.csv sourceSite_bacteriaSpecies.csv\n```\n5. upstream_4_speciesPreferredMapping.jl\n\nFinally, we can merge the bacteria species data for each source site from upstream_3 and the \nmetamap preferred names from upstream_2 metamap parsing. \n```\n$ julia upstream_4_speciesPreferredMapping.jl sourceSite_MetaMap_parsed.csv sourceSite_bacteriaSpecies.csv sourceSite_bacteriaSpecies_preferred.csv\n```\nYou finished the Upstream Processes! \n\n## Updating Cleaned_Host_Annotations\n\nIf you are updating this file, you should now have the host_bacteriaSpecies_preferred.csv file - this\nis after running hostLocus.csv through all the upstream steps and using semanticHost.txt for the processing \nof MetaMap concepts. \n\n1. hostClean_1_cleaningHosts.py\n\nThis script is a first manual run-through of cleaning the host names. It also changes the column names\nof the host_bacteriaSpecies_preferred.csv file to the columns of the original cleaned_host_annotations.csv\nfile and adds a cleaned \"updatedScientificName\" column that is the most up-to-date host name. \n\n```\npython3 hostClean_1_cleaningHosts.pyy hostData/host_bacteriaSpecies_preferred.csv hostTaxDict.txt\n```\n\nInput: 1) host_bacteriaSpecies_preferred.csv\n        2) hostTaxDict.txt\n            This is a preexisting taxonomic dictionary that I used for the host names. No need to replicate. \n        3) hostSpeciesNames.txt\n            A manual creation for common misspellings of the big host categories \nOutput: updated_cleanedhost.csv\n\n2. hostClean_2_searchHosts.py\n\nThis script is an extra check of the host names in Entrez to find scientific names for all the hosts. This is\nnot always successful however due to variation and grammatical issues in the way the host name was entered.\nI used it because it was valuable in coordinating common names vs. scientific names, however when the host\nnames are kept as genus rather than species, we did not merge them because of different taxonomical levels.  \n\n```\npython3 hostClean_2_searchHosts.py cleaned_hostData.csv\n```\n\nInput: updated_cleanedhost.csv\nOutput: cleaned_hostData.csv\n\n## Downstream Processes: \n\n1. downstream_1_createHostSpecies.jl\n\nThis script maps the data from sourceSite_bacteriaSpecies_preferred.csv to the data in \ncleaned_host_annotations.csv\n```\n$ julia downstream_1_createHostSpecies.jl cleaned_host_annotations.csv sourceSite_bacteriaSpecies_preferred.csv sourceSite_Host_Bacteria.csv\n```\n2. downstream_2_mergeAndGroup.py\n\nThis script takes the two source site CSVs (isolation_source and tissue_type) and merges them together. \nIt also adds a grouping category for the source site, in which body organs of interest (GI in particular), \nare grouped and given group names in a separate column \"group\". \n\nInput: 1) isolation_Host_Bacteria.csv\n        2) tissue_Host_Bacteria.csv\n        3) sourceGroupings.txt\nOutput: merged_Host_Bacteria.csv\n```\n$ python3 downstream_2_mergeAndGroup.py isolation_Host_Bacteria.csv tissue_Host_Bacteria.csv sourceGroupings.txt merged_Host_Bacteria.csv \n```\n### Cleaning Species Names and Processing Taxonomically\n\nThe bacterial species names are quite varied and also do not allow us to look at groupings of bacteria \n(such as family or phyla) rather than individual species. Because of that, we have an additional \nmicroorganism species cleaning step to add taxonomical data on top of our current dataset.  \n\n1. speciesClean_1_splitSpecies.py\n\n```\n$ python3 speciesClean_1_splitSpecies.py merged_Host_Bacteria.csv microSpeciesNames.txt\n```\nThis script simply takes the species names in the merged_Host_Bacteria.csv and sorts them into their \nvalue counts (most abundant at top and least at bottom). Then, it writes them to the file microSpeciesNames.txt. \nFinally, using the split command, it splits this file into files of 15,000 species in total so that the\nnext step can be run sequentially. \n\n2. speciesClean_2_searchSpecies.py\n\nThis program runs the Entrez search to find each microorganism's taxonomic information, \naka it takes a super long time and can often time out from NCBI's side if you run it at \nthe wrong time or for too long. I suggest splitting up the microSpeciesNames.txt file\nup into around 15,000 entries each (done in prev step) and run on weekdays after 5 pm or weekends. \n\nInput: a text file with all of the unique species names available\nOutput: taxonomyDict.txt in the same directory\n```\n$ python3 speciesClean_2_searchSpecies.py speciesNameFile.txt\n```\n3. speciesClean_3_updateCSV.py\n\nNow, we update the merged_Host_Bacteria.csv file with the taxonomy information generated by\nthe speciesClean_1 step and the taxonomyDict.txt file. I recommend running a \"wc -l taxonomyDict.txt\"\nbefore hand to make sure that the length of that file matches the length of the original species\nnames file (before splitting) to make sure the file is correct. \n\nOutput: merged_Host_Bacteria_Species.csv \n```\n$ python3 speciesClean_3_updateCSV.py merged_Host_Bacteria.csv taxonomyDict.txt \n```\n#### Going back to the Downstream Process\n\n3. downstream_3_createMatrix.jl\n\nIn downstream 3, we create the matrix file that PAUP will input to create a phylogenetic tree. \nThis script has been updated from the previous version to include more command line arguments\nto customize the grouping and host requirements before making the PAUP file. \n```\n$ julia downstream_3_createMatrix.jl \u003csearch term\u003e merged_Host_Bacteria_Spcies.csv \u003coutput.nex\u003e \u003cgroup or specific\u003e \u003ctaxonomic resolution\u003e \u003crow number limit\u003e\n```\n- Search Term: this will be the term that is searched in either the group or species column\n    - Group example: GI\n    - Specific example: Intestine, Stomach\n- Output.nex: name of output file in NEXUS format\n- Group or Specific: decides whether you will be searching by Group or by specific source site\n    - This is a binary switch - if group is NOT input, it will use the source site column\n- Taxonomic resolution: species, genus, family, order, class, phylum \n- Row number limit: this ensures that every host included in the file has AT LEAST the row number limit \n    entries of microorganisms (I used 4 for GI)\n\nOutputs: \n    1) output.nex\n    2) hostFiles/\u003cgroup\u003e_coverage.txt\n\n4. downstream_4_coverage.py\n\nDownstream 4 creates a coverage file from the raw coverage file created by downstream 3 (group_coverage.txt) \nin hostFiles. This coverage file is used by downstream 5 to ensure equal coverage amongst hosts that have\nhigh amounts of data and hosts that have low amounts of data. \n```\n$ python3 downstream_4_coverage.py hostFiles/\u003cgroup\u003e_coverage.txt \u003coutput.txt\u003e\n``` \n5. downstream_5_dataType.py\n\nFinally, downstream 5 updates the matrix nexus file with a few pieces of information for final visualization.\n```\n$ python3 downstream_5_dataType.py \u003coutputDS3.nex\u003e hostUpdates.csv \u003coutput names\u003e \u003cupper threshold for coverage\u003e \u003coutputDS4.txt\u003e\n```\n- outputDS3.nex: output from Downstream 3\n- hostUpdates.csv: This is a CSV I manually created to ensure which hosts ended up in the final file\n    - Columns: Newick_label, updatedScientificName, family, order, class, correctedName\n    - Newick_label: the updatedScientificName but with all spaces replaced with underscore so it is one word \n    - correctedName: in case the updatedScientificName is incorrect, I used this to update the name\n        or merge two labels together (such as one scientific name and one common name)\n- output names: there are multiple output files, so I inputted a phrase that will start all the output files\n    - Output files: \u003cgroup\u003e_updated.nex and hostFiles/\u003cgroup\u003e_updated.txt\n    - First is the nexus file for use, second is a list of all hosts \n    - The second file is necessary for finding out the outgroup in tree creation\n- upper threshold for coverage: all hosts ABOVE this number will be coverage updated to this number\n    - EX: 20 for GI\n- outputDS4.txt: output coverage file from downstream 4\n\n# Visualizaton\n\n1. vis1_speciesComposition.py\n\nThis script finds the species composition at the **SPECIES** level. To get the species file, make sure to run\ndownstream step 3 (need not run downstream steps 4 and 5 because coverage normalization is not required here).\nThis is an example command:\n```\n$ julia downstream_3_createMatrix.jl GI merged_Host_Bacteria_Spcies.csv gi_species.nex group species 4\n```\n\nUsing this output file (gi_species.nex), we can analyze the number of unique species present in each phylum. \nThis script will create a percentage based stacked bar chart to visualize the output of host groups of interest.\n\n```\n$ python3 vis1_speciesComposition.py gi_species.nex hostUpdates.csv taxonomyDict.txt \u003coutputFile\u003e\n```\nhostUpdates.csv and taxonomyDict.txt are both located in the Data/ folder.\n\n2. vis2_diversityAnalysis.py\n\nRuns a hill numbers approach of biodiversity estimation across the sample dataset. \n```\npython3 vis2_diversityAnalysis.py gi_phyla.nex hostUpdates.csv \u003coutputFile\u003e\n```\nThis can be run on both the phyla and species level files to contrast the difference between \nbiodiversity estimation in both. \n\n3. vis3_networkClustering.py\n\nRuns the Markov Clustering Algorithm on phylum level data to find clustering groups of hosts. \nAlso analyzes cluster microbiome composition similarities as well as creates three visualization\nfiles in a directory of choice: MCL cluster network, presence-absence general network, and \nmammalian similarity network. Cosine similarity is the metric for pairwise similarity. \n```\npython3 vis3_networkClustering.py gi_phyla.nex hostUpdates.csv \u003coutputDirectory\u003e\n```\nAll network files can be visualized in CytoScape. The basic process I used in Cytoscape is the following:\n- Load in the network\n- Layout -\u003e Grouped Layout -\u003e Cluster / Taxonomy (cluster for MCL, taxonomy for other two networks)\n- Tools -\u003e Analyze Network\n- Style -\u003e coloring based on taxonomy and node size based on degree\n\n4. vis4_phyloTree.r - Steps for Phylogenetic Trees\n\nThe creation of the microbiome tree requires a few different steps: \n- Using gi hosts file (output from Downstream 5) in timetree.org to find outgroup (I used Octopus_vulgaris)\n- Export this tree in newick format to save as true evolutionary tree for hosts\n    - there may be hosts that do not have the same naming conventions as hosts in the dataset \n    - I dealt with this by manually including data for these new host names\n- Running the phylum presence absence data (gi_phyla.nex) through PAUP neighbor joining with outgroup\n- Visualizing both microbiome and evolutionary tree with vis4_phyloTree.r\n\n#### Timetree.org\nInput in host file (each line is different host)\n** make sure host names (genus and species) are separated by a space and NOT UNDERSCORE **\nNote down oldest evolutionary branch and organism (Octupus_vulgaris)\nExport tree in newick format in bottom left\n\n#### PAUP Neighbor Joining Commands: \nRun PAUP neighbor joining on the gi_phyla.nex data\n```\npaup\nexecute \u003cfilename\u003e \nset criterion=distance maxtrees=200 increase=no\nhsearch\nset root=outgroup\noutgroup \u003cname of single host or multiple\u003e\nnj brlens=yes treefile=\u003coutputFile\u003e\n```\n#### Tree Visualization: vis4_phyloTree.r\n\nVisualization of trees using TreeIO and GGTree, using a slightly different hostUpdates CSV file.\nI modified this to include hosts with different names from TimeTree.org. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcbi%2Fgenbank_holobiome","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbcbi%2Fgenbank_holobiome","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcbi%2Fgenbank_holobiome/lists"}