{"id":22504984,"url":"https://github.com/bcbi/genbank_loader","last_synced_at":"2025-09-06T09:47:18.172Z","repository":{"id":40998862,"uuid":"402057456","full_name":"bcbi/genbank_loader","owner":"bcbi","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-23T21:21:18.000Z","size":96,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-27T23:23:34.067Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bcbi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-01T12:45:18.000Z","updated_at":"2021-10-05T21:20:17.000Z","dependencies_parsed_at":"2024-12-07T00:12:57.023Z","dependency_job_id":"c5164ccb-487c-4e76-801a-54f99fc3af3f","html_url":"https://github.com/bcbi/genbank_loader","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bcbi/genbank_loader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bcbi","download_url":"https://codeload.github.com/bcbi/genbank_loader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bcbi%2Fgenbank_loader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273804511,"owners_count":25171556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T00:12:46.987Z","updated_at":"2025-09-06T09:47:18.122Z","avatar_url":"https://github.com/bcbi.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GenBank Loader #\n\nGenBank Loader is a Java-based command-line utility that may be used to download the latest [GenBank](http://www.ncbi.nlm.nih.gov/genbank/) data from the [National Center for Biotechnology Information](http://www.ncbi.nlm.nih.gov/), and to load those publicly available DNA sequences into a local relational database to facilitate advanced analysis. \n\n[ORIGINAL CODE HERE, written by Matthew Storer](https://bitbucket.org/UVM-BIRD/genbank-loader/src/master/)\n\nThis is an updated version, edited by Vivek Ramanan in 2021, for the Brown Center of Biomedical Informatics (BCBI). \nI have added comments with ** before them so they hopefully make running the process easier. \n\n## Building GenBank Loader ##\n\nBuilding the GenBank Loader is accomplished by running the following from the project's base directory:\n\n    $ mvn package\n\n## Preparation ##\n\nTwo preparatory steps must be taken before GenBank Loader may be used.  These are creating the local database into which GenBank data will be ultimately stored, and ensuring your system has sufficient disk space available to store everything.\n\n### 1. Create Local Database ###\n\nBefore the GenBank Loader can be used, the database into which GenBank DNA sequences will be stored must be created.  GenBank Loader is pre-configured to use MySQL as its database back-end.  If you don't already have it installed, you should first download and install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/).\n\nOnce MySQL is installed, run the following command from the project's base directory:\n\n    $ mysql -uroot \u003c createdb.sql\n\nThis will create the _genbank_ database, with appropriate default values to integrate with the GenBank Loader's default settings.\n\n** You should need to use this command because you will be using the BCBI DB pursamydbcit. \n\n### 2. Ensure You Have Sufficient Free Disk Space ###\n\nThe GenBank Loader's intermediate files and target database tables can take _hundreds of gigabytes_ of disk space.  Please be sure that you have at least **500 gigabytes** of disk space available before starting this process!\n\n## Usage ##\n\nTo use the GenBank Loader and to see a list of command-line switches, run the program without any arguments:\n\n    $ java -jar genbank-loader-1.0.jar \n    GenBank Loader\n    --------------\n    Copyright 2015 The University of Vermont and State Agricultural College.  All rights reserved.\n    \n    usage: genbank-loader [-d \u003cstring\u003e] [-h \u003cstring\u003e] --load | --prepare [-p \u003cstring\u003e]  [-u \u003cstring\u003e]\n     -d,--db \u003cstring\u003e     the database name (default 'genbank')\n     -h,--host \u003cstring\u003e   the database host (default 'localhost')\n        --load            only load prepared files into the target database\n     -p,--pass \u003cstring\u003e   the database user password (default 'genbank')\n        --prepare         only prepare database files for import\n     -u,--user \u003cstring\u003e   the database user name (default 'genbank')\n\n## 3. Prepare GenBank Files For Import ##\n\nThe GenBank Loader has two primary modes of operation: `prepare` and `load`.  The preparation stage involves downloading GenBank files and transforming them such that they may be loaded into the target database.  The loading stage involves taking those prepared files and importing them into the target database.\n \nTo download and prepare files for import, run the following command:\n\n    $ java -Xmx1000m -jar genbank-loader-1.0.jar --prepare\n    GenBank Loader\n    --------------\n    Copyright 2015 The University of Vermont and State Agricultural College.  All rights reserved.\n    \n    INFO  Load - process started at Thu Jun 18 10:09:31 EDT 2015\n    INFO  FTPClient - connected to 'ftp.ncbi.nlm.nih.gov'\n    INFO  FTPClient - disconnected from 'ftp.ncbi.nlm.nih.gov'\n    INFO  FTPClient - connected to 'ftp.ncbi.nlm.nih.gov'\n    INFO  FTPClient - disconnected from 'ftp.ncbi.nlm.nih.gov'\n    INFO  AbstractFTPLoader - [2]  start\n    INFO  AbstractFTPLoader - [1]  start\n    INFO  FTPClient - connected to 'ftp.ncbi.nlm.nih.gov'\n    INFO  FTPClient - connected to 'ftp.ncbi.nlm.nih.gov'\n    INFO  AbstractFTPLoader - [2]  (1/3749, 0%)  processing 'gbbct1.seq.gz'\n    INFO  AbstractFTPLoader - [1]  (2/3749, 0%)  processing 'gbbct10.seq.gz'\n    INFO  AbstractFTPLoader - [1]  (3/3749, 0%)  processing 'gbbct100.seq.gz'\n    INFO  AbstractFTPLoader - [2]  (4/3749, 0%)  processing 'gbbct101.seq.gz'\n    ...\n    INFO  AbstractFTPLoader - [2]  (3748/3749, 99%)  processing 'complete.999.genomic.gbff.gz'\n    INFO  AbstractFTPLoader - [1]  (3749/3749, 100%)  processing 'complete.wgs_mstr.gbff.gz'\n    INFO  AbstractFTPLoader - [1]  done.\n    INFO  FTPClient - disconnected from 'ftp.ncbi.nlm.nih.gov'\n    INFO  AbstractFTPLoader - [2]  done.\n    INFO  FTPClient - disconnected from 'ftp.ncbi.nlm.nih.gov'\n    INFO  AbstractLoader - processing 3749 files across 2 threads took 13 hours, 24 minutes, 19 seconds\n    INFO  Load - preparing GenBank files finished at Thu Jun 18 23:33:54 EDT 2015 (took 13 hours, 24 minutes, 23 seconds).\n\nDownloaded files will be stored in a temporary folder while they are being processed, and are deleted after processing is completed.  Resultant files that will be imported into the local database are stored in the _out_ folder in the current working directory:\n\n    $ ls out\n    annotations.txt\t\tauthors.txt\t\tbasic.txt\t\tdbxrefs.txt\t\tjournals.txt\t\tkeywords.txt\n\nEach of these files is associated with a corresponding table in the target database.\n\n## Running Prepare on Oscar ##\n\n#### Batch Job Command:\n\nI ran the Prepare and Load steps through batch jobs on Oscar, which allowed them to run in the background and not take up much time. I used a time maximum of 100 hours and then the given command for prepare or load. \n\n```\nJava -jar target/genbank-loader-1.0.jar —prepare\n```\n#### Changes to Pom.XML: \n- This file holds information on Maven, MySQL, and the Oracle MySQL connection\n- Change version to 8.0.21\n- Added the following repository section to the end of the Pom.XML for Oscar specifically (pom_Oscar.xml) \n\n#### Changes to Load.java: src/main/java/edu/uvm/ccts/genbank/Load.java\n- This holds information on the links to download and access the MySQL database\n- private static final String JDBC_DRIVER = \"com.mysql.cj.jdbc.Driver\"\n- Added \"?serverTimezone=UTC\u0026allowLoadLocalInfile=true\" to the String url JDBC\n\n#### Changes to Record.java: src/main/java/edu/uvm/ccts/genbank/model/Record.java\n- This file goes through the individual Genbank records and processes them into files\n- Changed populateGInumber to [0] because there is no index 1 anymore\n- This change is due to GenBank changing their data organization \n\nWith this, hopefully, prepare will work! If it doesn’t, try changing some of these variables, particularly the set of added strings to the String url JDBC in Load.java. Some of these variables are server dependent. \n\n#### Prepare Stage is Resumable ####\n\nDuring the preparation stage, thousands of files are downloaded over potentially many hours.  It is possible that during this time, the process might be interrupted (perhaps by an `OutOfMemoryError`, lost network connection, user error, etc.).  Should any such interruption occur, it is important to know that the download and processing of data files may be resumed where it left off with an extremely remote chance of data corruption.\n\nThis is because metadata is generated about each successfully processed file (allowing the system to know where to resume operations), and because data generated from each downloaded file is processed independently, the results from which are appended to the master target files for database import only after all individual file processing has completed.\n\nCorruption of the master files can occur **only** if the GenBank Loader is interrupted during the few milliseconds required to append individual file's data to its respective master file.  This is extremely unlikely to occur, as while the processing of an individual file may take many seconds, appending those data to the master file takes just a few milliseconds.\n\n#### Important Note Regarding Memory and Performance ####\n\nGenBank Loader is a multi-threaded process that can leverage the cores of your CPU to download and prepare GenBank files in parallel, which can dramatically improve performance.\n\nWhen GenBank Loader executes its `prepare` function, it determines how much memory it has been allocated, and creates as many threads as it can safely use without negatively impacting other user and system processes.  GenBank Loader will use at least one core, but may use up to (_N_-2) cores, where _N_ is the total number of cores in your system's CPU.\n\nGenBank Loader's preparation stage requires **400 megabytes of Java heap memory per thread** to properly execute, and may otherwise suffer severely degraded performance, or even fail with an `OutOfMemoryError`.  It is therefore suggested to allocate _at least 2.5 times_ the required memory per thread to the JVM, but more is better and will translate to improved performance, especially on systems with many CPU cores.\n\nTo allocate the minimum suggested memory to the GenBank Loader process, use the JVM option `-Xmx1000m` as in the example above.\n \nSee [Oracle's Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html) for details about `-Xmx` and other JVM options.\n\n## 4. Load Prepared Files Into Local Database ##\n\nTo load the prepared GenBank files into the local database, execute the following:\n\n    $ java -jar genbank-loader-1.0.jar --load\n    GenBank Loader\n    --------------\n    Copyright 2015 The University of Vermont and State Agricultural College.  All rights reserved.\n    \n    INFO  Load - process started at Wed Jun 17 16:55:32 EDT 2015\n    INFO  AbstractLoader - populating database from './out' -\n    INFO  AbstractLoader -  loading './out/basic.txt' into table 'basic'\n    INFO  DataSource - getConnection : establishing connection to 'genbank'\n    INFO  AbstractLoader -  building indexes for table 'basic'\n    INFO  AbstractLoader -  loading './out/keywords.txt' into table 'keywords'\n    INFO  AbstractLoader -  building indexes for table 'keywords'\n    INFO  AbstractLoader -  loading './out/dbxrefs.txt' into table 'dbxrefs'\n    INFO  AbstractLoader -  building indexes for table 'dbxrefs'\n    INFO  AbstractLoader -  loading './out/journals.txt' into table 'journals'\n    INFO  AbstractLoader -  building indexes for table 'journals'\n    INFO  AbstractLoader -  loading './out/authors.txt' into table 'authors'\n    INFO  AbstractLoader -  building indexes for table 'authors'\n    INFO  AbstractLoader -  loading './out/annotations.txt' into table 'annotations'\n    INFO  AbstractLoader -  building indexes for table 'annotations'\n    INFO  AbstractLoader - finished populating database.  took 12 hours, 6 minutes, 59 seconds\n    INFO  Load - populating database with release '207' finished at Thu Jun 18 05:02:35 EDT 2015 (took 12 hours, 7 minutes, 3 seconds).\n\nNote that as soon as the `load` process completes, you may safely delete intermediate database import files in the _out_ folder.\n\n## Load in Oscar ##\n\nLoad required a few more changes for me particularly to get this process working. The main issue is that the current set of code in GenBank loader relies on a set of background scripts by UVM that I was able to get access to in order to change some of them. \nLink: https://bitbucket.org/UVM-BIRD/ccts-common/src/master/\n\nIn particular, the DB folder within src/ is the scripts that I was able to edit to fix the load process. The DB folder can be copied and placed into your local genbank-loader (within the genbank folder), where I edited the following: \n\n#### Changes to: MetaGenbankLoader.java: \n- import edu.uvm.ccts.genbank.db.loader.AbstractFTPLoader\n\n#### Changes to DB Loader: \n- Initialize the following files in loader with “package edu.uvm.ccts.genbank.db.loader;”\n  - Loader/AbstractFileLoader.java\n  - Loader/AbstractFTPLoader.java\n  - Loader/AbstractLoader.java: This code has the MySQL commands that loads the data\n- In AbstractLoader.java, update dbLoad (line 94) by removing the alter statements starting with DBUtil.executeUpdate\n```\nDBUtil.executeUpdate(\"set session sql_log_bin = OFF\", dataSource); \nDBUtil.executeUpdate(\"load data local infile '\" + filename + \"' into table \" + table, dataSource);\nDBUtil.executeUpdate(\"set session sql_log_bin = ON\", dataSource);\n```\n- FeatureTableParser.java holds the variable names for each of the files to be processed - if you want to remove one or change a filename, do so here\n\n#### Changes to overall process: \n- Use truncate from MySQL command line to clean out the data tables you are going to update\n- However, this requires additional permission for the “drop” command from the DB team\n- Once you have gotten permissions, use truncate on the command line and not in the script as I had some issues with that and allows the manual operations used later on to add onto pre existing tables with data\n\n#### Batch job Command: \n```\njava -jar target/genbank-loader-1.0.jar --load -h pursamydbcit.services.brown.edu -u vramanan -p \u003cpassword\u003e\n```\n*I had to use my password on the command line to get this work with the batch job\n\nChanges to the file loading: \n- I was able to load the following tables easily: Basic, keywords, journals and dbxrefs\n- Authors and annotations were too large to load so I split them into 5 and 10 files respectively and manually loaded them after that. \n- I do not advise trying to mass load them at once. I tried it twice and each time it crashed due to communication failures just a day before they were completed (approximately 11-12 days)\n\n#### Splitting authors and annotations: \nThese were the results of the “wc -l” command on the authors and annotations files to find their line count: \n- Authors: 3,322,431,281\n- Annotations: 6,712,480,648\nThis is the split command for annotations – again use a batch job rather than command line\n```\nSplit -a 1 -l 700,000,000 annotations.txt annotations\n```\nApply same logic to authors\n\nOnce split, make sure to *not* truncate authors and annotations as you are adding on each individual file to the previously loaded table\n\n#### Flowchart for Updating Authors and Annotations\nI made a flowchart so I could quickly go through the motions for each manual update: \n1. Rm \u003cpreviousfile\u003e\n2. Mv \u003cnextfile\u003e \u003cnextfile.txt\u003e \n3. Adds “.txt” extension\n4. Update FeatureTableParser.java with new name of file\n5. Remove \"truncate\" or \"delete\" from AbstractLoader.java in dbLoad\n6. Comment out the defining variables of the files you don’t need\n7. Change the name of the annotations or authors file you are using with, for example “authorsa.txt” (a is first file of the 5, followed by bcde)\n8. Maven package\n9. Update the batch file script\n10. Run script\n\n#### Checking MySQL for Updates ####\n    \nUse the following command to check the row count of the tables to see the updates as you’re going allowing: \n```\nSelect table_rows \"Rows Count\" from information_schema.tables where table_name=\"annotations\" and table_schema=\"genbank\";\n```\nTry not to use the “select count(*) from \u003ctable\u003e” because that’s a long linear time\n\n\n## License and Copyright ##\n\nGenBank Loader is Copyright 2015 [The University of Vermont and State Agricultural College](https://www.uvm.edu/).  All rights reserved.\n\nGenBank Loader is licensed under the terms of the [GNU General Public License (GPL) version 3](https://www.gnu.org/licenses/gpl-3.0-standalone.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcbi%2Fgenbank_loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbcbi%2Fgenbank_loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcbi%2Fgenbank_loader/lists"}