{"id":17174971,"url":"https://github.com/blinard-bioinfo/ncbitaxonomy","last_synced_at":"2026-03-06T04:31:35.203Z","repository":{"id":206254474,"uuid":"247133519","full_name":"blinard-BIOINFO/NCBITaxonomy","owner":"blinard-BIOINFO","description":"Tools to build a copy of the NCBI Taxonomy database in a local SQL schema and query it via a Java package.","archived":false,"fork":false,"pushed_at":"2020-09-02T16:35:50.000Z","size":80,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-04T01:54:24.931Z","etag":null,"topics":["blast","blast-searches","ncbi","species-lists","taxid","taxonomy"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blinard-BIOINFO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-03-13T18:06:36.000Z","updated_at":"2023-12-23T12:27:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"7c8f506c-73ac-43b5-8c01-021b2e21397f","html_url":"https://github.com/blinard-BIOINFO/NCBITaxonomy","commit_stats":null,"previous_names":["blinard-bioinfo/ncbitaxonomy"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/blinard-BIOINFO/NCBITaxonomy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blinard-BIOINFO%2FNCBITaxonomy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blinard-BIOINFO%2FNCBITaxonomy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blinard-BIOINFO%2FNCBITaxonomy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blinard-BIOINFO%2FNCBITaxonomy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blinard-BIOINFO","download_url":"https://codeload.github.com/blinard-BIOINFO/NCBITaxonomy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blinard-BIOINFO%2FNCBITaxonomy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30161737,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T04:22:03.816Z","status":"ssl_error","status_checked_at":"2026-03-06T04:22:00.183Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blast","blast-searches","ncbi","species-lists","taxid","taxonomy"],"created_at":"2024-10-14T23:55:23.574Z","updated_at":"2026-03-06T04:31:35.184Z","avatar_url":"https://github.com/blinard-BIOINFO.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"**Contributions are welcome !**\n\n# NCBI Taxonomy SQL/Java Tools\n\nThis set of tools aims to create a local SQL copy of the NCBI Taxonomy database and uses Java scripts to query the database.\n \nIt aims to answer many common operations when working on systematics or species identification, such as :\n  * extract a lineage for a species or taxonomic id\n  * extract all species below a certain taxonomy node\n  * extract every lineages of every species below a certain taxonomy node\n  * build a Blast \"Delimitation File\", intended to be used with commands `blastdb_aliastool -gilist ` to build Blast database indexes focuses on particular NCBI Taxonomy clades.\n  * ... etc ...\n  \nThis package was initially developed for the following academic projects :\n\n* *The contribution of mitochondrial metagenomics to large-scale data mining and phylogenetic analysis of Coleoptera. Linard B et al. Mol Phylogenet Evol. 2018 Nov;128:1-11.*\n* *Lessons from genome skimming of arthropod-preserving ethanol. Linard B. et al.  Mol Ecol Resour. 2016 Nov;16(6):1365-1377.*\n* *Metagenome skimming of insect specimen pools: potential for comparative genomics. Linard et al. Genome Biol Evol. 2015 May 14;7(6):1474-89.*\n\nIf these sources are of any use in your own project, the authors would greatly appreciate that you cite one of these.\n\n## Requirements\n\n* Postgresql server (Java code should be compatible with other SGBDs after adapting COPY statements in `update_taxonomy.sh` and the SQL schema in `taxonomy_schema.sql`)\n* ADMIN or COPY rights associated to your SGBD user/role to copy NCBI dumps to your local database.\n* Java JDK 1.8\n\n## HOW TO USE\n\n**Its main purpose is the treatment of very large lists of species names, sequence identifiers and the export of large chunks of taxonomic data.**\n**Basically, the idea is 1) load the full NCBI taxonomy tree in memory and 2) rapidly query this tree using a list of input queries**\n\n**Good approach:**\nBuild a list of thousands of species names or NCBI sequence identifiers as a text file.\nThen all one of the functions of this package ONCE.\n\n**Bad approach:**\nIn a bash script call the functions of this package at each iteration.\nThis will be super slow... Why? because this would load the full NCBI tree in memory at each iteration !\n\n\n## NCBI Taxonomy operations\n\n### Available operations\n\n* **ScientificNamesToLineages** : From a list of Scientific Names, (written in a file, 1 identifier per line) extract the corresponding NCBI lineages.\n\n* **TaxidToLineage** : Extract the lineage from a simple taxonomic id.\n\n* **TaxidToSubTreeLeavesLineages** : Using a taxonomic id of an internal node of NCBI Taxonomy (for instance, 7041 which is Coleopteran order), extract the lineages of every species belonging to the subtree of this node (with, 7041, extract lineages of every Coleoptaran species.\n\n* **IdentifiersToLineages** : From a list of NCBI GIs or ACCESSIONs identifiers (written in a file, 1 identifier per line) extract the corresponding NCBI lineages. WARNING: queries will be fast only IF `index_accession2taxid` was set to 1 (default is 0) during installation. If not, you can index later the column of this table (corresponding SQL lines are in `database_schema.sql`).\n\n* **GenerateTaxonomyDelimitationForBlast** : One can require a copy of the NCBI Blast database focused on a particular clade. For instance, you may download the Nematodes Blast database but you are actually only interested by C. elegans sequences. A command `blastdb_aliastool -db nematode_mrna -gilist c_elegans_mrna.gi` can help you to build the corresponding Blast index (see NCBI documentation) BUT the annoying part is to build the `gilist` which targets every single sequence of C. elegans. The present operation does exactly that, from a taxonomic id, it will extract every gi numbers associated to the subtree so that you can build later a Blst database focused on a particular clade and accelerate you Blast searches. WARNING: queries will be fast only IF `index_index_gi_taxid_nucl` or `index_index_gi_taxid_prot` were set to 1 during installation (default is 1). If not, you can index later the column of this table (corresponding SQL lines are in `database_schema.sql`).\n\n* More to come ...\n\n\n### Calling an operation\n\n```\njava -cp NCBITaxonomy.jar op.[operation_name] \n```\nFor instance:\n```\njava -cp NCBITaxonomy.jar op.TaxidToLineage --help\n```\nWill show the usage of this operation:\n```\nUsage: TaxidToLineage [-hrV] [-f=[1|2]] [-o=\u003cout\u003e] -t=int\nExtract NCBI lineage from a NCBI taxid.\n  -f, --format=[1|2]   Format used to output ranks:\n                       1 = 'Homo[Genus];sapiens[Species]'\n                       2 = 'Homo;sapiens' (line 1)\n                            'genus;species' (line 2)\n  -h, --help           Show this help message and exit.\n  -o, --out=\u003cout\u003e      Output results in file instead of stdout.\n  -r, --ranks          Add rank names to scientific names.\n  -t, --taxid=int      The taxonomic id.\n  -V, --version        Print version information and exit.\n```\n\nAvailable operations can be listed by writing the following line in a terminal, followed by 2 pushes of the TAB key when your cursor is just on the right of the last dot :\n```\njava -cp NCBITaxonomy.jar op.\n```\nIf your system is correctly configured for Java autocompletion, you should see a list of all available operations (op).\n```\nop.DBConnectionTest\nop.GenerateTaxonomyDelimitationForBlast\nop.IdentifiersToLineages\nop.ScientificNamesToLineages\nop.TaxidToLineage\nop.TaxidToSubTreeLeavesLineages\n```\n\n## Installation\n\nThe installation process is done in 4 steps:\n\n1. Configure the header of 'update_taxonomy.sh'. Execute to download the NCBI taxonomy dumps and copy them to a new database in your SQL server. (requires ADMIN or COPY rights associated to your SGBD user/role).\n2. Optionnal: Create a user granted with SELECT permissions on the created database.\n3. Write the database credentials of this user in a database.properties file (see below).\n4. Use the Java package to query your new NCBI Taxonomy database via different operations (see below).\n\n\n**Step 1: update_taxonomy.sh**\n\nJust edit the `### SCRIPT CONFIG` section. Some important points:\n\n* The database user set in this file MUST have COPY rights to dump data to the database. In most recent version of Postgres, this can be done by granting `pg_write_server_files` privileges to this user.\n* `index_*` options are intended to avoid the create of giant indexes that are not necessarily useful to your applications. In particular, the table `index_accession2taxid=0` will avoid to index the corresponding table wich contains todays ~ 2 milliards of lines (march 2020), leading to a \u003e100 Gb index while the database itself is less than this size !\n* `step_*` options are there only if one of the step fails and you need to relaunch the script. For instance, to avoid downloading again the NACBI taxonomy dumps when the script failed in a later step.\n* By default the create database name will follow the pattern `ncbi_taxoniomy_YYYY_MM_DD` where YYYY_MM_DD are year-month-date in numerical caracters. This can be changed by changing `dbname=\"\"` with a non-empty string.\n\n**Step 2: (Optionnal) NCBI taxonomy user**\n\nIt can be useful to create a user dedicated to this new datbase, in particular when someone not intended to modify your SQL datbases just want to interogate NCBI taxonomy. In the database prompt, and a role holding 'CREATE USER' rights, do the following :\n\n```\nCREATE USER taxonomypublic WITH PASSWORD 'taxonomypublic' ;\nGRANT CONNECT ON DATABASE ncbi_taxonomy_YYYY_MM_DD TO taxonomypublic ;\nGRANT SELECT ON ALL TABLES IN SCHEMA public TO taxonomypublic ;\n```\n\n**Step 3: database.properties file**\n\nEdit the file `database.properties` to set valid database credentials. The Java code will require this file to connect to the database.  You can either use your own database user or a dedicated-user as sdhown in step 2. \nBy default, the Java code will look for database credentials in a database.properties file in the current directory.\n```\njdbc.drivers=org.postgresql.Driver                           #driver\njdbc.url=jdbc:postgresql://ip:port/ncbi_taxonomy_xxxx_xx_x   #taxonomy DB address\njdbc.username=taxonomypublic                                 #login\njdbc.password=taxonomypublic                                 #password\n```\nMoreover, if you plan to use something else than Postgresl (MySQL, Oracle ...) do not forget to change the driver accordingly. \n\nBy default, this file will be looked in the same directory where is present the jar file.\nFor all operations, this behaviour can be changed by targeting a particular property file with option -d .\n\n**Step 4: Java compilation**\n\nInstall Java JDK and compiler is not already done.\n```\n#install java DK and gradle for compilation\nsudo apt-get update\nsudo apt-get install openjdk-8-jdk\nsudo apt-get install gradle\n```\nCompile sources. To use another JDBC driver (MySQL, Oracle ...) edit `gradle.build` to add the corresponding driver in the dependancies. \n\n``` \ngit clone https://github.com/blinard-BIOINFO/NCBITaxonomy.git\ncd ./NCBITaxonomy\ngradle build \u0026\u0026 gradle clean\n```\nRapid test. The command help should appear.\n```\njava -cp NCBITaxonomy-0.1.0.jar op.TaxidToLineage --help\n```\n\nRapid connection test. If your setup is correct, you should see : \n```\njava -cp NCBITaxonomy-0.1.0.jar op.DBConnectionTest\n\nTesting postgres database connection...\nconnected on: jdbc:postgresql://127.0.0.1:5432/ncbi_taxonomy_2020_03_12\nwith user: taxonomypublic\n```\n\n# License\n\nThis code is distributed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblinard-bioinfo%2Fncbitaxonomy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblinard-bioinfo%2Fncbitaxonomy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblinard-bioinfo%2Fncbitaxonomy/lists"}