{"id":50244866,"url":"https://github.com/neuralmind-ai/portuguese-bert","last_synced_at":"2026-06-29T20:00:57.784Z","repository":{"id":37818676,"uuid":"233952052","full_name":"neuralmind-ai/portuguese-bert","owner":"neuralmind-ai","description":"Portuguese pre-trained BERT models","archived":false,"fork":false,"pushed_at":"2022-06-16T11:30:25.000Z","size":949,"stargazers_count":765,"open_issues_count":16,"forks_count":117,"subscribers_count":53,"default_branch":"master","last_synced_at":"2024-03-22T03:10:32.553Z","etag":null,"topics":["bert","bert-model","deep-learning","natural-language-processing","nlp-resources","portuguese"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neuralmind-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-14T22:56:00.000Z","updated_at":"2024-03-20T19:50:13.000Z","dependencies_parsed_at":"2022-06-22T18:51:56.191Z","dependency_job_id":null,"html_url":"https://github.com/neuralmind-ai/portuguese-bert","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/neuralmind-ai/portuguese-bert","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralmind-ai%2Fportuguese-bert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralmind-ai%2Fportuguese-bert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralmind-ai%2Fportuguese-bert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralmind-ai%2Fportuguese-bert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neuralmind-ai","download_url":"https://codeload.github.com/neuralmind-ai/portuguese-bert/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neuralmind-ai%2Fportuguese-bert/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34941027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-29T02:00:05.398Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","bert-model","deep-learning","natural-language-processing","nlp-resources","portuguese"],"created_at":"2026-05-26T23:00:19.776Z","updated_at":"2026-06-29T20:00:57.778Z","avatar_url":"https://github.com/neuralmind-ai.png","language":"Python","funding_links":[],"categories":["NLP per Language"],"sub_categories":["Models"],"readme":"\n# BERTimbau - Portuguese BERT\n\nThis repository contains pre-trained [BERT](https://github.com/google-research/bert) models trained on the Portuguese language. BERT-Base and BERT-Large Cased variants were trained on the [BrWaC (Brazilian Web as Corpus)](https://www.researchgate.net/publication/326303825_The_brWaC_Corpus_A_New_Open_Resource_for_Brazilian_Portuguese), a large Portuguese corpus, for 1,000,000 steps, using whole-word mask. Model artifacts for TensorFlow and PyTorch can be found below.\n\nThe models are a result of an ongoing Master's Program. The [text submission for Qualifying Exam](qualifying_exam-portuguese_named_entity_recognition_using_bert_crf.pdf) is also included in the repository in PDF format, which contains more details about the pre-training procedure, vocabulary generation and downstream usage in the task of Named Entity Recognition.\n\n## Download\n\nThe base and large models are available at [Hugging Face](https://huggingface.co/neuralmind)\n\n\n## Evaluation benchmarks\n\nThe models were benchmarked on three tasks (Sentence Textual Similarity, Recognizing Textual Entailment and Named Entity Recognition) and compared to previous published results and [Multilingual BERT](https://github.com/google-research/bert/blob/master/multilingual.md). Metrics are: Pearson's correlation for STS and F1-score for RTE and NER.\n\n| Task | Test Dataset           | BERTimbau-Large | BERTimbau-Base | mBERT  |      Previous SOTA    |\n|:----:|:----------------------:|:---------------:|:-------------: | :-----:| :--------------------:| \n| STS  | ASSIN2                 |    **0.852**    |     0.836      |  0.809 | 0.83 [[1]](#References) |\n| RTE  | ASSIN2                 |    **90.0**     |     89.2       |  86.8  | 88.3 [[1]](#References) |\n| NER  | MiniHAREM (5 classes)  |    **83.7**     |     83.1       |  79.2  | 82.3 [[2]](#References) |\n| NER  | MiniHAREM (10 classes) |    **78.5**     |     77.6       |  73.1  | 74.6 [[2]](#References) |\n\n### NER experiments code\n\nCode and instructions to reproduce the Named Entity Recognition experiments are in [`ner_evaluation/`](ner_evaluation/) directory.\n\n\n## PyTorch usage example\n\nOur PyTorch artifacts are compatible with the [🤗Huggingface Transformers](https://github.com/huggingface/transformers) library and are also available on the [Community models](https://huggingface.co/models):\n\n- [BERTimbau Base model card](https://huggingface.co/neuralmind/bert-base-portuguese-cased)\n- [BERTimbau Large model card](https://huggingface.co/neuralmind/bert-large-portuguese-cased)\n\n```python\nfrom transformers import AutoModel, AutoTokenizer\n\n# Using the community model\n# BERT Base\ntokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-base-portuguese-cased')\nmodel = AutoModel.from_pretrained('neuralmind/bert-base-portuguese-cased')\n\n# BERT Large\ntokenizer = AutoTokenizer.from_pretrained('neuralmind/bert-large-portuguese-cased')\nmodel = AutoModel.from_pretrained('neuralmind/bert-large-portuguese-cased')\n\n# or, using BertModel and BertTokenizer directly\nfrom transformers import BertModel, BertTokenizer\n\ntokenizer = BertTokenizer.from_pretrained('path/to/vocab.txt', do_lower_case=False)\nmodel = BertModel.from_pretrained('path/to/bert_dir')  # Or other BERT model class\n```\n\n## Acknowledgement\n\nWe would like to thank Google for Cloud credits under a research grant that allowed us to train these models.\n\n## References\n\n[1] [Multilingual Transformer Ensembles for Portuguese Natural Language Task](https://www.researchgate.net/publication/340236502_Multilingual_Transformer_Ensembles_for_Portuguese_Natural_Language_Tasks)\n\n[2] [Assessing the Impact of Contextual Embeddings for Portuguese Named Entity Recognition](https://github.com/jneto04/ner-pt)\n\n\n## How to cite this work\n\n    @InProceedings{souza2020bertimbau,\n        author=\"Souza, F{\\'a}bio and Nogueira, Rodrigo and Lotufo, Roberto\",\n        editor=\"Cerri, Ricardo and Prati, Ronaldo C.\",\n        title=\"BERTimbau: Pretrained BERT Models for Brazilian Portuguese\",\n        booktitle=\"Intelligent Systems\",\n        year=\"2020\",\n        publisher=\"Springer International Publishing\",\n        address=\"Cham\",\n        pages=\"403--417\",\n        isbn=\"978-3-030-61377-8\"\n    }\n\n\n\n    @article{souza2019portuguese,\n        title={Portuguese Named Entity Recognition using BERT-CRF},\n        author={Souza, F{\\'a}bio and Nogueira, Rodrigo and Lotufo, Roberto},\n        journal={arXiv preprint arXiv:1909.10649},\n        url={http://arxiv.org/abs/1909.10649},\n        year={2019}\n    }\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneuralmind-ai%2Fportuguese-bert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneuralmind-ai%2Fportuguese-bert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneuralmind-ai%2Fportuguese-bert/lists"}