{"id":13692835,"url":"https://github.com/naver/biobert-pretrained","last_synced_at":"2026-01-28T15:01:30.384Z","repository":{"id":66176413,"uuid":"167883658","full_name":"naver/biobert-pretrained","owner":"naver","description":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","archived":false,"fork":false,"pushed_at":"2020-06-02T03:28:07.000Z","size":23,"stargazers_count":689,"open_issues_count":7,"forks_count":91,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-05-31T13:08:24.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/naver.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-01-28T02:05:23.000Z","updated_at":"2025-05-11T01:32:09.000Z","dependencies_parsed_at":"2023-02-21T16:45:23.460Z","dependency_job_id":null,"html_url":"https://github.com/naver/biobert-pretrained","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/naver/biobert-pretrained","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fbiobert-pretrained","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fbiobert-pretrained/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fbiobert-pretrained/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fbiobert-pretrained/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/naver","download_url":"https://codeload.github.com/naver/biobert-pretrained/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/naver%2Fbiobert-pretrained/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28846082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-28T13:02:32.985Z","status":"ssl_error","status_checked_at":"2026-01-28T13:02:04.945Z","response_time":57,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T17:01:02.509Z","updated_at":"2026-01-28T15:01:30.367Z","avatar_url":"https://github.com/naver.png","language":null,"funding_links":[],"categories":["Uncategorized","Techniques and Models"],"sub_categories":["Uncategorized","BERT models"],"readme":"# BioBERT Pre-trained Weights\n\nThis repository provides pre-trained weights of BioBERT, a language representation model for biomedical domain, especially designed for biomedical text mining tasks such as biomedical named entity recognition, relation extraction, question answering, etc.\nPlease refer to our paper [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://arxiv.org/abs/1901.08746) for more details.\n\n## Downloading pre-trained weights\nGo to [releases](https://github.com/naver/biobert-pretrained/releases) section of this repository or click links below to download pre-trained weights of BioBERT.\nWe provide three combinations of pre-trained weights: BioBERT (+ PubMed), BioBERT (+ PMC), and BioBERT (+ PubMed + PMC).\nPre-training was based on the [original BERT code](https://github.com/google-research/bert) provided by Google, and training details are described in our paper. Currently available versions of pre-trained weights are as follows:\n\n* **[BioBERT-Base v1.1 (+ PubMed 1M)](https://drive.google.com/file/d/1R84voFKHfWV9xjzeLzWBbmY1uOMYpnyD/view?usp=sharing)** - based on BERT-base-Cased (same vocabulary)\n* **[BioBERT-Large v1.1 (+ PubMed 1M)](https://drive.google.com/file/d/1GJpGjQj6aZPV-EfbiQELpBkvlGtoKiyA/view?usp=sharing)** - based on BERT-large-Cased (custom 30k vocabulary), [NER/QA Results](https://github.com/dmis-lab/biobert/wiki/BioBERT-Large-Results)\n* **[BioBERT-Base v1.0 (+ PubMed 200K)](https://drive.google.com/file/d/17j6pSKZt5TtJ8oQCDNIwlSZ0q5w7NNBg/view?usp=sharing)** - based on BERT-base-Cased (same vocabulary)\n* **[BioBERT-Base v1.0 (+ PMC 270K)](https://drive.google.com/file/d/1LiAJklso-DCAJmBekRTVEvqUOfm0a9fX/view?usp=sharing)** - based on BERT-base-Cased (same vocabulary)\n* **[BioBERT-Base v1.0 (+ PubMed 200K + PMC 270K)](https://drive.google.com/file/d/1jGUu2dWB1RaeXmezeJmdiPKQp3ZCmNb7/view?usp=sharing)** - based on BERT-base-Cased (same vocabulary)\n\nMake sure to specify the versions of pre-trained weights used in your works.\nIf you have difficulty choosing which one to use, we recommend using **BioBERT-Base v1.1 (+ PubMed 1M)** or **BioBERT-Large v1.1 (+ PubMed 1M)** depending on your GPU resources.\nNote that for BioBERT-Base, we are using WordPiece vocabulary (`vocab.txt`) provided by Google as any new words in biomedical corpus can be represented with subwords (for instance, Leukemia =\u003e Leu + ##ke + ##mia).\nMore details are in the closed [issue #1](https://github.com/naver/biobert-pretrained/issues/1).\n\n## Pre-training corpus\nWe do not provide pre-processed version of each corpus. However, each pre-training corpus could be found in the following links:\n*   **`PubMed Abstracts1`**: ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/\n*   **`PubMed Abstracts2`**: ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/\n*   **`PubMed Central Full Texts`**: ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/\n\nEstimated size of each corpus is 4.5 billion words for **`PubMed Abstracts1`** + **`PubMed Abstracts2`**, and 13.5 billion words for **`PubMed Central Full Texts`**.\n\n## Fine-tuning BioBERT\nTo fine-tunine BioBERT on biomedical text mining tasks using provided pre-trained weights, refer to the [DMIS GitHub repository for BioBERT](https://github.com/dmis-lab/biobert).\n\n## Citation\n```\n@article{10.1093/bioinformatics/btz682,\n    author = {Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo},\n    title = \"{BioBERT: a pre-trained biomedical language representation model for biomedical text mining}\",\n    journal = {Bioinformatics},\n    year = {2019},\n    month = {09},\n    issn = {1367-4803},\n    doi = {10.1093/bioinformatics/btz682},\n    url = {https://doi.org/10.1093/bioinformatics/btz682},\n}\n```\n\n## Contact information\nFor help or issues using pre-trained weights of BioBERT, please submit a GitHub issue. Please contact Jinhyuk Lee\n(`lee.jnhk@gmail.com`), or Sungdong Kim (`sungdong.kim@navercorp.com`) for communication related to pre-trained weights of BioBERT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaver%2Fbiobert-pretrained","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnaver%2Fbiobert-pretrained","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnaver%2Fbiobert-pretrained/lists"}