{"id":13394762,"url":"https://github.com/google-research/bert","last_synced_at":"2025-05-12T03:50:31.724Z","repository":{"id":37390895,"uuid":"154747577","full_name":"google-research/bert","owner":"google-research","description":"TensorFlow code and pre-trained models for BERT","archived":false,"fork":false,"pushed_at":"2024-07-23T23:39:41.000Z","size":317,"stargazers_count":39081,"open_issues_count":887,"forks_count":9674,"subscribers_count":998,"default_branch":"master","last_synced_at":"2025-05-01T13:54:46.513Z","etag":null,"topics":["google","natural-language-processing","natural-language-understanding","nlp","tensorflow"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1810.04805","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-25T22:57:34.000Z","updated_at":"2025-05-01T12:31:13.000Z","dependencies_parsed_at":"2022-07-13T19:10:30.771Z","dependency_job_id":"45a25c87-3136-477a-8edc-9b91abbaa17c","html_url":"https://github.com/google-research/bert","commit_stats":{"total_commits":76,"total_committers":31,"mean_commits":"2.4516129032258065","dds":0.5526315789473684,"last_synced_commit":"eedf5716ce1268e56f0a50264a88cafad334ac61"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fbert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fbert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fbert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fbert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/bert/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253672536,"owners_count":21945476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["google","natural-language-processing","natural-language-understanding","nlp","tensorflow"],"created_at":"2024-07-30T17:01:30.744Z","updated_at":"2025-05-12T03:50:31.695Z","avatar_url":"https://github.com/google-research.png","language":"Python","readme":"# BERT\n\n**\\*\\*\\*\\*\\* New March 11th, 2020: Smaller BERT Models \\*\\*\\*\\*\\***\n\nThis is a release of 24 smaller BERT models (English only, uncased, trained with WordPiece masking) referenced in [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://arxiv.org/abs/1908.08962).\n\nWe have shown that the standard BERT recipe (including model architecture and training objective) is effective on a wide range of model sizes, beyond BERT-Base and BERT-Large. The smaller BERT models are intended for environments with restricted computational resources. They can be fine-tuned in the same manner as the original BERT models. However, they are most effective in the context of knowledge distillation, where the fine-tuning labels are produced by a larger and more accurate teacher.\n\nOur goal is to enable research in institutions with fewer computational resources and encourage the community to seek directions of innovation alternative to increasing model capacity.\n\nYou can download all 24 from [here][all], or individually from the table below:\n\n|   |H=128|H=256|H=512|H=768|\n|---|:---:|:---:|:---:|:---:|\n| **L=2**  |[**2/128 (BERT-Tiny)**][2_128]|[2/256][2_256]|[2/512][2_512]|[2/768][2_768]|\n| **L=4**  |[4/128][4_128]|[**4/256 (BERT-Mini)**][4_256]|[**4/512 (BERT-Small)**][4_512]|[4/768][4_768]|\n| **L=6**  |[6/128][6_128]|[6/256][6_256]|[6/512][6_512]|[6/768][6_768]|\n| **L=8**  |[8/128][8_128]|[8/256][8_256]|[**8/512 (BERT-Medium)**][8_512]|[8/768][8_768]|\n| **L=10** |[10/128][10_128]|[10/256][10_256]|[10/512][10_512]|[10/768][10_768]|\n| **L=12** |[12/128][12_128]|[12/256][12_256]|[12/512][12_512]|[**12/768 (BERT-Base)**][12_768]|\n\nNote that the BERT-Base model in this release is included for completeness only; it was re-trained under the same regime as the original model.\n\nHere are the corresponding GLUE scores on the test set:\n\n|Model|Score|CoLA|SST-2|MRPC|STS-B|QQP|MNLI-m|MNLI-mm|QNLI(v2)|RTE|WNLI|AX|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|BERT-Tiny|64.2|0.0|83.2|81.1/71.1|74.3/73.6|62.2/83.4|70.2|70.3|81.5|57.2|62.3|21.0|\n|BERT-Mini|65.8|0.0|85.9|81.1/71.8|75.4/73.3|66.4/86.2|74.8|74.3|84.1|57.9|62.3|26.1|\n|BERT-Small|71.2|27.8|89.7|83.4/76.2|78.8/77.0|68.1/87.0|77.6|77.0|86.4|61.8|62.3|28.6|\n|BERT-Medium|73.5|38.0|89.6|86.6/81.6|80.4/78.4|69.6/87.9|80.0|79.1|87.7|62.2|62.3|30.5|\n\nFor each task, we selected the best fine-tuning hyperparameters from the lists below, and trained for 4 epochs:\n- batch sizes: 8, 16, 32, 64, 128\n- learning rates: 3e-4, 1e-4, 5e-5, 3e-5\n\nIf you use these models, please cite the following paper:\n\n```\n@article{turc2019,\n  title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},\n  author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},\n  journal={arXiv preprint arXiv:1908.08962v2 },\n  year={2019}\n}\n```\n\n[2_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-128_A-2.zip\n[2_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-256_A-4.zip\n[2_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-512_A-8.zip\n[2_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-768_A-12.zip\n[4_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-128_A-2.zip\n[4_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-256_A-4.zip\n[4_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-512_A-8.zip\n[4_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-768_A-12.zip\n[6_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-128_A-2.zip\n[6_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-256_A-4.zip\n[6_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-512_A-8.zip\n[6_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-768_A-12.zip\n[8_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-128_A-2.zip\n[8_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-256_A-4.zip\n[8_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-512_A-8.zip\n[8_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-768_A-12.zip\n[10_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-128_A-2.zip\n[10_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-256_A-4.zip\n[10_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-512_A-8.zip\n[10_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-768_A-12.zip\n[12_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-128_A-2.zip\n[12_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-256_A-4.zip\n[12_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-512_A-8.zip\n[12_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-768_A-12.zip\n[all]: https://storage.googleapis.com/bert_models/2020_02_20/all_bert_models.zip\n\n**\\*\\*\\*\\*\\* New May 31st, 2019: Whole Word Masking Models \\*\\*\\*\\*\\***\n\nThis is a release of several new models which were the result of an improvement\nthe pre-processing code.\n\nIn the original pre-processing code, we randomly select WordPiece tokens to\nmask. For example:\n\n`Input Text: the man jumped up , put his basket on phil ##am ##mon ' s head`\n`Original Masked Input: [MASK] man [MASK] up , put his [MASK] on phil\n[MASK] ##mon ' s head`\n\nThe new technique is called Whole Word Masking. In this case, we always mask\n*all* of the the tokens corresponding to a word at once. The overall masking\nrate remains the same.\n\n`Whole Word Masked Input: the man [MASK] up , put his basket on [MASK] [MASK]\n[MASK] ' s head`\n\nThe training is identical -- we still predict each masked WordPiece token\nindependently. The improvement comes from the fact that the original prediction\ntask was too 'easy' for words that had been split into multiple WordPieces.\n\nThis can be enabled during data generation by passing the flag\n`--do_whole_word_mask=True` to `create_pretraining_data.py`.\n\nPre-trained models with Whole Word Masking are linked below. The data and\ntraining were otherwise identical, and the models have identical structure and\nvocab to the original models. We only include BERT-Large models. When using\nthese models, please make it clear in the paper that you are using the Whole\nWord Masking variant of BERT-Large.\n\n*   **[`BERT-Large, Uncased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip)**:\n    24-layer, 1024-hidden, 16-heads, 340M parameters\n\n*   **[`BERT-Large, Cased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_cased_L-24_H-1024_A-16.zip)**:\n    24-layer, 1024-hidden, 16-heads, 340M parameters\n\nModel                                    | SQUAD 1.1 F1/EM | Multi NLI Accuracy\n---------------------------------------- | :-------------: | :----------------:\nBERT-Large, Uncased (Original)           | 91.0/84.3       | 86.05\nBERT-Large, Uncased (Whole Word Masking) | 92.8/86.7       | 87.07\nBERT-Large, Cased (Original)             | 91.5/84.8       | 86.09\nBERT-Large, Cased (Whole Word Masking)   | 92.9/86.7       | 86.46\n\n**\\*\\*\\*\\*\\* New February 7th, 2019: TfHub Module \\*\\*\\*\\*\\***\n\nBERT has been uploaded to [TensorFlow Hub](https://tfhub.dev). See\n`run_classifier_with_tfhub.py` for an example of how to use the TF Hub module,\nor run an example in the browser on\n[Colab](https://colab.sandbox.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb).\n\n**\\*\\*\\*\\*\\* New November 23rd, 2018: Un-normalized multilingual model + Thai +\nMongolian \\*\\*\\*\\*\\***\n\nWe uploaded a new multilingual model which does *not* perform any normalization\non the input (no lower casing, accent stripping, or Unicode normalization), and\nadditionally inclues Thai and Mongolian.\n\n**It is recommended to use this version for developing multilingual models,\nespecially on languages with non-Latin alphabets.**\n\nThis does not require any code changes, and can be downloaded here:\n\n*   **[`BERT-Base, Multilingual Cased`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**:\n    104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters\n\n**\\*\\*\\*\\*\\* New November 15th, 2018: SOTA SQuAD 2.0 System \\*\\*\\*\\*\\***\n\nWe released code changes to reproduce our 83% F1 SQuAD 2.0 system, which is\ncurrently 1st place on the leaderboard by 3%. See the SQuAD 2.0 section of the\nREADME for details.\n\n**\\*\\*\\*\\*\\* New November 5th, 2018: Third-party PyTorch and Chainer versions of\nBERT available \\*\\*\\*\\*\\***\n\nNLP researchers from HuggingFace made a\n[PyTorch version of BERT available](https://github.com/huggingface/pytorch-pretrained-BERT)\nwhich is compatible with our pre-trained checkpoints and is able to reproduce\nour results. Sosuke Kobayashi also made a\n[Chainer version of BERT available](https://github.com/soskek/bert-chainer)\n(Thanks!) We were not involved in the creation or maintenance of the PyTorch\nimplementation so please direct any questions towards the authors of that\nrepository.\n\n**\\*\\*\\*\\*\\* New November 3rd, 2018: Multilingual and Chinese models available\n\\*\\*\\*\\*\\***\n\nWe have made two new BERT models available:\n\n*   **[`BERT-Base, Multilingual`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)\n    (Not recommended, use `Multilingual Cased` instead)**: 102 languages,\n    12-layer, 768-hidden, 12-heads, 110M parameters\n*   **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:\n    Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M\n    parameters\n\nWe use character-based tokenization for Chinese, and WordPiece tokenization for\nall other languages. Both models should work out-of-the-box without any code\nchanges. We did update the implementation of `BasicTokenizer` in\n`tokenization.py` to support Chinese character tokenization, so please update if\nyou forked it. However, we did not change the tokenization API.\n\nFor more, see the\n[Multilingual README](https://github.com/google-research/bert/blob/master/multilingual.md).\n\n**\\*\\*\\*\\*\\* End new information \\*\\*\\*\\*\\***\n\n## Introduction\n\n**BERT**, or **B**idirectional **E**ncoder **R**epresentations from\n**T**ransformers, is a new method of pre-training language representations which\nobtains state-of-the-art results on a wide array of Natural Language Processing\n(NLP) tasks.\n\nOur academic paper which describes BERT in detail and provides full results on a\nnumber of tasks can be found here:\n[https://arxiv.org/abs/1810.04805](https://arxiv.org/abs/1810.04805).\n\nTo give a few numbers, here are the results on the\n[SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/) question answering\ntask:\n\nSQuAD v1.1 Leaderboard (Oct 8th 2018) | Test EM  | Test F1\n------------------------------------- | :------: | :------:\n1st Place Ensemble - BERT             | **87.4** | **93.2**\n2nd Place Ensemble - nlnet            | 86.0     | 91.7\n1st Place Single Model - BERT         | **85.1** | **91.8**\n2nd Place Single Model - nlnet        | 83.5     | 90.1\n\nAnd several natural language inference tasks:\n\nSystem                  | MultiNLI | Question NLI | SWAG\n----------------------- | :------: | :----------: | :------:\nBERT                    | **86.7** | **91.1**     | **86.3**\nOpenAI GPT (Prev. SOTA) | 82.2     | 88.1         | 75.0\n\nPlus many other tasks.\n\nMoreover, these results were all obtained with almost no task-specific neural\nnetwork architecture design.\n\nIf you already know what BERT is and you just want to get started, you can\n[download the pre-trained models](#pre-trained-models) and\n[run a state-of-the-art fine-tuning](#fine-tuning-with-bert) in only a few\nminutes.\n\n## What is BERT?\n\nBERT is a method of pre-training language representations, meaning that we train\na general-purpose \"language understanding\" model on a large text corpus (like\nWikipedia), and then use that model for downstream NLP tasks that we care about\n(like question answering). BERT outperforms previous methods because it is the\nfirst *unsupervised*, *deeply bidirectional* system for pre-training NLP.\n\n*Unsupervised* means that BERT was trained using only a plain text corpus, which\nis important because an enormous amount of plain text data is publicly available\non the web in many languages.\n\nPre-trained representations can also either be *context-free* or *contextual*,\nand contextual representations can further be *unidirectional* or\n*bidirectional*. Context-free models such as\n[word2vec](https://www.tensorflow.org/tutorials/representation/word2vec) or\n[GloVe](https://nlp.stanford.edu/projects/glove/) generate a single \"word\nembedding\" representation for each word in the vocabulary, so `bank` would have\nthe same representation in `bank deposit` and `river bank`. Contextual models\ninstead generate a representation of each word that is based on the other words\nin the sentence.\n\nBERT was built upon recent work in pre-training contextual representations —\nincluding [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432),\n[Generative Pre-Training](https://blog.openai.com/language-unsupervised/),\n[ELMo](https://allennlp.org/elmo), and\n[ULMFit](http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html)\n— but crucially these models are all *unidirectional* or *shallowly\nbidirectional*. This means that each word is only contextualized using the words\nto its left (or right). For example, in the sentence `I made a bank deposit` the\nunidirectional representation of `bank` is only based on `I made a` but not\n`deposit`. Some previous work does combine the representations from separate\nleft-context and right-context models, but only in a \"shallow\" manner. BERT\nrepresents \"bank\" using both its left and right context — `I made a ... deposit`\n— starting from the very bottom of a deep neural network, so it is *deeply\nbidirectional*.\n\nBERT uses a simple approach for this: We mask out 15% of the words in the input,\nrun the entire sequence through a deep bidirectional\n[Transformer](https://arxiv.org/abs/1706.03762) encoder, and then predict only\nthe masked words. For example:\n\n```\nInput: the man went to the [MASK1] . he bought a [MASK2] of milk.\nLabels: [MASK1] = store; [MASK2] = gallon\n```\n\nIn order to learn relationships between sentences, we also train on a simple\ntask which can be generated from any monolingual corpus: Given two sentences `A`\nand `B`, is `B` the actual next sentence that comes after `A`, or just a random\nsentence from the corpus?\n\n```\nSentence A: the man went to the store .\nSentence B: he bought a gallon of milk .\nLabel: IsNextSentence\n```\n\n```\nSentence A: the man went to the store .\nSentence B: penguins are flightless .\nLabel: NotNextSentence\n```\n\nWe then train a large model (12-layer to 24-layer Transformer) on a large corpus\n(Wikipedia + [BookCorpus](http://yknzhu.wixsite.com/mbweb)) for a long time (1M\nupdate steps), and that's BERT.\n\nUsing BERT has two stages: *Pre-training* and *fine-tuning*.\n\n**Pre-training** is fairly expensive (four days on 4 to 16 Cloud TPUs), but is a\none-time procedure for each language (current models are English-only, but\nmultilingual models will be released in the near future). We are releasing a\nnumber of pre-trained models from the paper which were pre-trained at Google.\nMost NLP researchers will never need to pre-train their own model from scratch.\n\n**Fine-tuning** is inexpensive. All of the results in the paper can be\nreplicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU,\nstarting from the exact same pre-trained model. SQuAD, for example, can be\ntrained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of\n91.0%, which is the single system state-of-the-art.\n\nThe other important aspect of BERT is that it can be adapted to many types of\nNLP tasks very easily. In the paper, we demonstrate state-of-the-art results on\nsentence-level (e.g., SST-2), sentence-pair-level (e.g., MultiNLI), word-level\n(e.g., NER), and span-level (e.g., SQuAD) tasks with almost no task-specific\nmodifications.\n\n## What has been released in this repository?\n\nWe are releasing the following:\n\n*   TensorFlow code for the BERT model architecture (which is mostly a standard\n    [Transformer](https://arxiv.org/abs/1706.03762) architecture).\n*   Pre-trained checkpoints for both the lowercase and cased version of\n    `BERT-Base` and `BERT-Large` from the paper.\n*   TensorFlow code for push-button replication of the most important\n    fine-tuning experiments from the paper, including SQuAD, MultiNLI, and MRPC.\n\nAll of the code in this repository works out-of-the-box with CPU, GPU, and Cloud\nTPU.\n\n## Pre-trained models\n\nWe are releasing the `BERT-Base` and `BERT-Large` models from the paper.\n`Uncased` means that the text has been lowercased before WordPiece tokenization,\ne.g., `John Smith` becomes `john smith`. The `Uncased` model also strips out any\naccent markers. `Cased` means that the true case and accent markers are\npreserved. Typically, the `Uncased` model is better unless you know that case\ninformation is important for your task (e.g., Named Entity Recognition or\nPart-of-Speech tagging).\n\nThese models are all released under the same license as the source code (Apache\n2.0).\n\nFor information about the Multilingual and Chinese model, see the\n[Multilingual README](https://github.com/google-research/bert/blob/master/multilingual.md).\n\n**When using a cased model, make sure to pass `--do_lower=False` to the training\nscripts. (Or pass `do_lower_case=False` directly to `FullTokenizer` if you're\nusing your own script.)**\n\nThe links to the models are here (right-click, 'Save link as...' on the name):\n\n*   **[`BERT-Large, Uncased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip)**:\n    24-layer, 1024-hidden, 16-heads, 340M parameters\n*   **[`BERT-Large, Cased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_cased_L-24_H-1024_A-16.zip)**:\n    24-layer, 1024-hidden, 16-heads, 340M parameters\n*   **[`BERT-Base, Uncased`](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip)**:\n    12-layer, 768-hidden, 12-heads, 110M parameters\n*   **[`BERT-Large, Uncased`](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip)**:\n    24-layer, 1024-hidden, 16-heads, 340M parameters\n*   **[`BERT-Base, Cased`](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip)**:\n    12-layer, 768-hidden, 12-heads , 110M parameters\n*   **[`BERT-Large, Cased`](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-24_H-1024_A-16.zip)**:\n    24-layer, 1024-hidden, 16-heads, 340M parameters\n*   **[`BERT-Base, Multilingual Cased (New, recommended)`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**:\n    104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters\n*   **[`BERT-Base, Multilingual Uncased (Orig, not recommended)`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)\n    (Not recommended, use `Multilingual Cased` instead)**: 102 languages,\n    12-layer, 768-hidden, 12-heads, 110M parameters\n*   **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:\n    Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M\n    parameters\n\nEach .zip file contains three items:\n\n*   A TensorFlow checkpoint (`bert_model.ckpt`) containing the pre-trained\n    weights (which is actually 3 files).\n*   A vocab file (`vocab.txt`) to map WordPiece to word id.\n*   A config file (`bert_config.json`) which specifies the hyperparameters of\n    the model.\n\n## Fine-tuning with BERT\n\n**Important**: All results on the paper were fine-tuned on a single Cloud TPU,\nwhich has 64GB of RAM. It is currently not possible to re-produce most of the\n`BERT-Large` results on the paper using a GPU with 12GB - 16GB of RAM, because\nthe maximum batch size that can fit in memory is too small. We are working on\nadding code to this repository which allows for much larger effective batch size\non the GPU. See the section on [out-of-memory issues](#out-of-memory-issues) for\nmore details.\n\nThis code was tested with TensorFlow 1.11.0. It was tested with Python2 and\nPython3 (but more thoroughly with Python2, since this is what's used internally\nin Google).\n\nThe fine-tuning examples which use `BERT-Base` should be able to run on a GPU\nthat has at least 12GB of RAM using the hyperparameters given.\n\n### Fine-tuning with Cloud TPUs\n\nMost of the examples below assumes that you will be running training/evaluation\non your local machine, using a GPU like a Titan X or GTX 1080.\n\nHowever, if you have access to a Cloud TPU that you want to train on, just add\nthe following flags to `run_classifier.py` or `run_squad.py`:\n\n```\n  --use_tpu=True \\\n  --tpu_name=$TPU_NAME\n```\n\nPlease see the\n[Google Cloud TPU tutorial](https://cloud.google.com/tpu/docs/tutorials/mnist)\nfor how to use Cloud TPUs. Alternatively, you can use the Google Colab notebook\n\"[BERT FineTuning with Cloud TPUs](https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\".\n\nOn Cloud TPUs, the pretrained model and the output directory will need to be on\nGoogle Cloud Storage. For example, if you have a bucket named `some_bucket`, you\nmight use the following flags instead:\n\n```\n  --output_dir=gs://some_bucket/my_output_dir/\n```\n\nThe unzipped pre-trained model files can also be found in the Google Cloud\nStorage folder `gs://bert_models/2018_10_18`. For example:\n\n```\nexport BERT_BASE_DIR=gs://bert_models/2018_10_18/uncased_L-12_H-768_A-12\n```\n\n### Sentence (and sentence-pair) classification tasks\n\nBefore running this example you must download the\n[GLUE data](https://gluebenchmark.com/tasks) by running\n[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)\nand unpack it to some directory `$GLUE_DIR`. Next, download the `BERT-Base`\ncheckpoint and unzip it to some directory `$BERT_BASE_DIR`.\n\nThis example code fine-tunes `BERT-Base` on the Microsoft Research Paraphrase\nCorpus (MRPC) corpus, which only contains 3,600 examples and can fine-tune in a\nfew minutes on most GPUs.\n\n```shell\nexport BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12\nexport GLUE_DIR=/path/to/glue\n\npython run_classifier.py \\\n  --task_name=MRPC \\\n  --do_train=true \\\n  --do_eval=true \\\n  --data_dir=$GLUE_DIR/MRPC \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --max_seq_length=128 \\\n  --train_batch_size=32 \\\n  --learning_rate=2e-5 \\\n  --num_train_epochs=3.0 \\\n  --output_dir=/tmp/mrpc_output/\n```\n\nYou should see output like this:\n\n```\n***** Eval results *****\n  eval_accuracy = 0.845588\n  eval_loss = 0.505248\n  global_step = 343\n  loss = 0.505248\n```\n\nThis means that the Dev set accuracy was 84.55%. Small sets like MRPC have a\nhigh variance in the Dev set accuracy, even when starting from the same\npre-training checkpoint. If you re-run multiple times (making sure to point to\ndifferent `output_dir`), you should see results between 84% and 88%.\n\nA few other pre-trained models are implemented off-the-shelf in\n`run_classifier.py`, so it should be straightforward to follow those examples to\nuse BERT for any single-sentence or sentence-pair classification task.\n\nNote: You might see a message `Running train on CPU`. This really just means\nthat it's running on something other than a Cloud TPU, which includes a GPU.\n\n#### Prediction from classifier\n\nOnce you have trained your classifier you can use it in inference mode by using\nthe --do_predict=true command. You need to have a file named test.tsv in the\ninput folder. Output will be created in file called test_results.tsv in the\noutput folder. Each line will contain output for each sample, columns are the\nclass probabilities.\n\n```shell\nexport BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12\nexport GLUE_DIR=/path/to/glue\nexport TRAINED_CLASSIFIER=/path/to/fine/tuned/classifier\n\npython run_classifier.py \\\n  --task_name=MRPC \\\n  --do_predict=true \\\n  --data_dir=$GLUE_DIR/MRPC \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$TRAINED_CLASSIFIER \\\n  --max_seq_length=128 \\\n  --output_dir=/tmp/mrpc_output/\n```\n\n### SQuAD 1.1\n\nThe Stanford Question Answering Dataset (SQuAD) is a popular question answering\nbenchmark dataset. BERT (at the time of the release) obtains state-of-the-art\nresults on SQuAD with almost no task-specific network architecture modifications\nor data augmentation. However, it does require semi-complex data pre-processing\nand post-processing to deal with (a) the variable-length nature of SQuAD context\nparagraphs, and (b) the character-level answer annotations which are used for\nSQuAD training. This processing is implemented and documented in `run_squad.py`.\n\nTo run on SQuAD, you will first need to download the dataset. The\n[SQuAD website](https://rajpurkar.github.io/SQuAD-explorer/) does not seem to\nlink to the v1.1 datasets any longer, but the necessary files can be found here:\n\n*   [train-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)\n*   [dev-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json)\n*   [evaluate-v1.1.py](https://github.com/allenai/bi-att-flow/blob/master/squad/evaluate-v1.1.py)\n\nDownload these to some directory `$SQUAD_DIR`.\n\nThe state-of-the-art SQuAD results from the paper currently cannot be reproduced\non a 12GB-16GB GPU due to memory constraints (in fact, even batch size 1 does\nnot seem to fit on a 12GB GPU using `BERT-Large`). However, a reasonably strong\n`BERT-Base` model can be trained on the GPU with these hyperparameters:\n\n```shell\npython run_squad.py \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --do_train=True \\\n  --train_file=$SQUAD_DIR/train-v1.1.json \\\n  --do_predict=True \\\n  --predict_file=$SQUAD_DIR/dev-v1.1.json \\\n  --train_batch_size=12 \\\n  --learning_rate=3e-5 \\\n  --num_train_epochs=2.0 \\\n  --max_seq_length=384 \\\n  --doc_stride=128 \\\n  --output_dir=/tmp/squad_base/\n```\n\nThe dev set predictions will be saved into a file called `predictions.json` in\nthe `output_dir`:\n\n```shell\npython $SQUAD_DIR/evaluate-v1.1.py $SQUAD_DIR/dev-v1.1.json ./squad/predictions.json\n```\n\nWhich should produce an output like this:\n\n```shell\n{\"f1\": 88.41249612335034, \"exact_match\": 81.2488174077578}\n```\n\nYou should see a result similar to the 88.5% reported in the paper for\n`BERT-Base`.\n\nIf you have access to a Cloud TPU, you can train with `BERT-Large`. Here is a\nset of hyperparameters (slightly different than the paper) which consistently\nobtain around 90.5%-91.0% F1 single-system trained only on SQuAD:\n\n```shell\npython run_squad.py \\\n  --vocab_file=$BERT_LARGE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_LARGE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \\\n  --do_train=True \\\n  --train_file=$SQUAD_DIR/train-v1.1.json \\\n  --do_predict=True \\\n  --predict_file=$SQUAD_DIR/dev-v1.1.json \\\n  --train_batch_size=24 \\\n  --learning_rate=3e-5 \\\n  --num_train_epochs=2.0 \\\n  --max_seq_length=384 \\\n  --doc_stride=128 \\\n  --output_dir=gs://some_bucket/squad_large/ \\\n  --use_tpu=True \\\n  --tpu_name=$TPU_NAME\n```\n\nFor example, one random run with these parameters produces the following Dev\nscores:\n\n```shell\n{\"f1\": 90.87081895814865, \"exact_match\": 84.38978240302744}\n```\n\nIf you fine-tune for one epoch on\n[TriviaQA](http://nlp.cs.washington.edu/triviaqa/) before this the results will\nbe even better, but you will need to convert TriviaQA into the SQuAD json\nformat.\n\n### SQuAD 2.0\n\nThis model is also implemented and documented in `run_squad.py`.\n\nTo run on SQuAD 2.0, you will first need to download the dataset. The necessary\nfiles can be found here:\n\n*   [train-v2.0.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json)\n*   [dev-v2.0.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json)\n*   [evaluate-v2.0.py](https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/)\n\nDownload these to some directory `$SQUAD_DIR`.\n\nOn Cloud TPU you can run with BERT-Large as follows:\n\n```shell\npython run_squad.py \\\n  --vocab_file=$BERT_LARGE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_LARGE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \\\n  --do_train=True \\\n  --train_file=$SQUAD_DIR/train-v2.0.json \\\n  --do_predict=True \\\n  --predict_file=$SQUAD_DIR/dev-v2.0.json \\\n  --train_batch_size=24 \\\n  --learning_rate=3e-5 \\\n  --num_train_epochs=2.0 \\\n  --max_seq_length=384 \\\n  --doc_stride=128 \\\n  --output_dir=gs://some_bucket/squad_large/ \\\n  --use_tpu=True \\\n  --tpu_name=$TPU_NAME \\\n  --version_2_with_negative=True\n```\n\nWe assume you have copied everything from the output directory to a local\ndirectory called ./squad/. The initial dev set predictions will be at\n./squad/predictions.json and the differences between the score of no answer (\"\")\nand the best non-null answer for each question will be in the file\n./squad/null_odds.json\n\nRun this script to tune a threshold for predicting null versus non-null answers:\n\npython $SQUAD_DIR/evaluate-v2.0.py $SQUAD_DIR/dev-v2.0.json\n./squad/predictions.json --na-prob-file ./squad/null_odds.json\n\nAssume the script outputs \"best_f1_thresh\" THRESH. (Typical values are between\n-1.0 and -5.0). You can now re-run the model to generate predictions with the\nderived threshold or alternatively you can extract the appropriate answers from\n./squad/nbest_predictions.json.\n\n```shell\npython run_squad.py \\\n  --vocab_file=$BERT_LARGE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_LARGE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \\\n  --do_train=False \\\n  --train_file=$SQUAD_DIR/train-v2.0.json \\\n  --do_predict=True \\\n  --predict_file=$SQUAD_DIR/dev-v2.0.json \\\n  --train_batch_size=24 \\\n  --learning_rate=3e-5 \\\n  --num_train_epochs=2.0 \\\n  --max_seq_length=384 \\\n  --doc_stride=128 \\\n  --output_dir=gs://some_bucket/squad_large/ \\\n  --use_tpu=True \\\n  --tpu_name=$TPU_NAME \\\n  --version_2_with_negative=True \\\n  --null_score_diff_threshold=$THRESH\n```\n\n### Out-of-memory issues\n\nAll experiments in the paper were fine-tuned on a Cloud TPU, which has 64GB of\ndevice RAM. Therefore, when using a GPU with 12GB - 16GB of RAM, you are likely\nto encounter out-of-memory issues if you use the same hyperparameters described\nin the paper.\n\nThe factors that affect memory usage are:\n\n*   **`max_seq_length`**: The released models were trained with sequence lengths\n    up to 512, but you can fine-tune with a shorter max sequence length to save\n    substantial memory. This is controlled by the `max_seq_length` flag in our\n    example code.\n\n*   **`train_batch_size`**: The memory usage is also directly proportional to\n    the batch size.\n\n*   **Model type, `BERT-Base` vs. `BERT-Large`**: The `BERT-Large` model\n    requires significantly more memory than `BERT-Base`.\n\n*   **Optimizer**: The default optimizer for BERT is Adam, which requires a lot\n    of extra memory to store the `m` and `v` vectors. Switching to a more memory\n    efficient optimizer can reduce memory usage, but can also affect the\n    results. We have not experimented with other optimizers for fine-tuning.\n\nUsing the default training scripts (`run_classifier.py` and `run_squad.py`), we\nbenchmarked the maximum batch size on single Titan X GPU (12GB RAM) with\nTensorFlow 1.11.0:\n\nSystem       | Seq Length | Max Batch Size\n------------ | ---------- | --------------\n`BERT-Base`  | 64         | 64\n...          | 128        | 32\n...          | 256        | 16\n...          | 320        | 14\n...          | 384        | 12\n...          | 512        | 6\n`BERT-Large` | 64         | 12\n...          | 128        | 6\n...          | 256        | 2\n...          | 320        | 1\n...          | 384        | 0\n...          | 512        | 0\n\nUnfortunately, these max batch sizes for `BERT-Large` are so small that they\nwill actually harm the model accuracy, regardless of the learning rate used. We\nare working on adding code to this repository which will allow much larger\neffective batch sizes to be used on the GPU. The code will be based on one (or\nboth) of the following techniques:\n\n*   **Gradient accumulation**: The samples in a minibatch are typically\n    independent with respect to gradient computation (excluding batch\n    normalization, which is not used here). This means that the gradients of\n    multiple smaller minibatches can be accumulated before performing the weight\n    update, and this will be exactly equivalent to a single larger update.\n\n*   [**Gradient checkpointing**](https://github.com/openai/gradient-checkpointing):\n    The major use of GPU/TPU memory during DNN training is caching the\n    intermediate activations in the forward pass that are necessary for\n    efficient computation in the backward pass. \"Gradient checkpointing\" trades\n    memory for compute time by re-computing the activations in an intelligent\n    way.\n\n**However, this is not implemented in the current release.**\n\n## Using BERT to extract fixed feature vectors (like ELMo)\n\nIn certain cases, rather than fine-tuning the entire pre-trained model\nend-to-end, it can be beneficial to obtained *pre-trained contextual\nembeddings*, which are fixed contextual representations of each input token\ngenerated from the hidden layers of the pre-trained model. This should also\nmitigate most of the out-of-memory issues.\n\nAs an example, we include the script `extract_features.py` which can be used\nlike this:\n\n```shell\n# Sentence A and Sentence B are separated by the ||| delimiter for sentence\n# pair tasks like question answering and entailment.\n# For single sentence inputs, put one sentence per line and DON'T use the\n# delimiter.\necho 'Who was Jim Henson ? ||| Jim Henson was a puppeteer' \u003e /tmp/input.txt\n\npython extract_features.py \\\n  --input_file=/tmp/input.txt \\\n  --output_file=/tmp/output.jsonl \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --layers=-1,-2,-3,-4 \\\n  --max_seq_length=128 \\\n  --batch_size=8\n```\n\nThis will create a JSON file (one line per line of input) containing the BERT\nactivations from each Transformer layer specified by `layers` (-1 is the final\nhidden layer of the Transformer, etc.)\n\nNote that this script will produce very large output files (by default, around\n15kb for every input token).\n\nIf you need to maintain alignment between the original and tokenized words (for\nprojecting training labels), see the [Tokenization](#tokenization) section\nbelow.\n\n**Note:** You may see a message like `Could not find trained model in model_dir:\n/tmp/tmpuB5g5c, running initialization to predict.` This message is expected, it\njust means that we are using the `init_from_checkpoint()` API rather than the\nsaved model API. If you don't specify a checkpoint or specify an invalid\ncheckpoint, this script will complain.\n\n## Tokenization\n\nFor sentence-level tasks (or sentence-pair) tasks, tokenization is very simple.\nJust follow the example code in `run_classifier.py` and `extract_features.py`.\nThe basic procedure for sentence-level tasks is:\n\n1.  Instantiate an instance of `tokenizer = tokenization.FullTokenizer`\n\n2.  Tokenize the raw text with `tokens = tokenizer.tokenize(raw_text)`.\n\n3.  Truncate to the maximum sequence length. (You can use up to 512, but you\n    probably want to use shorter if possible for memory and speed reasons.)\n\n4.  Add the `[CLS]` and `[SEP]` tokens in the right place.\n\nWord-level and span-level tasks (e.g., SQuAD and NER) are more complex, since\nyou need to maintain alignment between your input text and output text so that\nyou can project your training labels. SQuAD is a particularly complex example\nbecause the input labels are *character*-based, and SQuAD paragraphs are often\nlonger than our maximum sequence length. See the code in `run_squad.py` to show\nhow we handle this.\n\nBefore we describe the general recipe for handling word-level tasks, it's\nimportant to understand what exactly our tokenizer is doing. It has three main\nsteps:\n\n1.  **Text normalization**: Convert all whitespace characters to spaces, and\n    (for the `Uncased` model) lowercase the input and strip out accent markers.\n    E.g., `John Johanson's, → john johanson's,`.\n\n2.  **Punctuation splitting**: Split *all* punctuation characters on both sides\n    (i.e., add whitespace around all punctuation characters). Punctuation\n    characters are defined as (a) Anything with a `P*` Unicode class, (b) any\n    non-letter/number/space ASCII character (e.g., characters like `$` which are\n    technically not punctuation). E.g., `john johanson's, → john johanson ' s ,`\n\n3.  **WordPiece tokenization**: Apply whitespace tokenization to the output of\n    the above procedure, and apply\n    [WordPiece](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder.py)\n    tokenization to each token separately. (Our implementation is directly based\n    on the one from `tensor2tensor`, which is linked). E.g., `john johanson ' s\n    , → john johan ##son ' s ,`\n\nThe advantage of this scheme is that it is \"compatible\" with most existing\nEnglish tokenizers. For example, imagine that you have a part-of-speech tagging\ntask which looks like this:\n\n```\nInput:  John Johanson 's   house\nLabels: NNP  NNP      POS NN\n```\n\nThe tokenized output will look like this:\n\n```\nTokens: john johan ##son ' s house\n```\n\nCrucially, this would be the same output as if the raw text were `John\nJohanson's house` (with no space before the `'s`).\n\nIf you have a pre-tokenized representation with word-level annotations, you can\nsimply tokenize each input word independently, and deterministically maintain an\noriginal-to-tokenized alignment:\n\n```python\n### Input\norig_tokens = [\"John\", \"Johanson\", \"'s\",  \"house\"]\nlabels      = [\"NNP\",  \"NNP\",      \"POS\", \"NN\"]\n\n### Output\nbert_tokens = []\n\n# Token map will be an int -\u003e int mapping between the `orig_tokens` index and\n# the `bert_tokens` index.\norig_to_tok_map = []\n\ntokenizer = tokenization.FullTokenizer(\n    vocab_file=vocab_file, do_lower_case=True)\n\nbert_tokens.append(\"[CLS]\")\nfor orig_token in orig_tokens:\n  orig_to_tok_map.append(len(bert_tokens))\n  bert_tokens.extend(tokenizer.tokenize(orig_token))\nbert_tokens.append(\"[SEP]\")\n\n# bert_tokens == [\"[CLS]\", \"john\", \"johan\", \"##son\", \"'\", \"s\", \"house\", \"[SEP]\"]\n# orig_to_tok_map == [1, 2, 4, 6]\n```\n\nNow `orig_to_tok_map` can be used to project `labels` to the tokenized\nrepresentation.\n\nThere are common English tokenization schemes which will cause a slight mismatch\nbetween how BERT was pre-trained. For example, if your input tokenization splits\noff contractions like `do n't`, this will cause a mismatch. If it is possible to\ndo so, you should pre-process your data to convert these back to raw-looking\ntext, but if it's not possible, this mismatch is likely not a big deal.\n\n## Pre-training with BERT\n\nWe are releasing code to do \"masked LM\" and \"next sentence prediction\" on an\narbitrary text corpus. Note that this is *not* the exact code that was used for\nthe paper (the original code was written in C++, and had some additional\ncomplexity), but this code does generate pre-training data as described in the\npaper.\n\nHere's how to run the data generation. The input is a plain text file, with one\nsentence per line. (It is important that these be actual sentences for the \"next\nsentence prediction\" task). Documents are delimited by empty lines. The output\nis a set of `tf.train.Example`s serialized into `TFRecord` file format.\n\nYou can perform sentence segmentation with an off-the-shelf NLP toolkit such as\n[spaCy](https://spacy.io/). The `create_pretraining_data.py` script will\nconcatenate segments until they reach the maximum sequence length to minimize\ncomputational waste from padding (see the script for more details). However, you\nmay want to intentionally add a slight amount of noise to your input data (e.g.,\nrandomly truncate 2% of input segments) to make it more robust to non-sentential\ninput during fine-tuning.\n\nThis script stores all of the examples for the entire input file in memory, so\nfor large data files you should shard the input file and call the script\nmultiple times. (You can pass in a file glob to `run_pretraining.py`, e.g.,\n`tf_examples.tf_record*`.)\n\nThe `max_predictions_per_seq` is the maximum number of masked LM predictions per\nsequence. You should set this to around `max_seq_length` * `masked_lm_prob` (the\nscript doesn't do that automatically because the exact value needs to be passed\nto both scripts).\n\n```shell\npython create_pretraining_data.py \\\n  --input_file=./sample_text.txt \\\n  --output_file=/tmp/tf_examples.tfrecord \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --do_lower_case=True \\\n  --max_seq_length=128 \\\n  --max_predictions_per_seq=20 \\\n  --masked_lm_prob=0.15 \\\n  --random_seed=12345 \\\n  --dupe_factor=5\n```\n\nHere's how to run the pre-training. Do not include `init_checkpoint` if you are\npre-training from scratch. The model configuration (including vocab size) is\nspecified in `bert_config_file`. This demo code only pre-trains for a small\nnumber of steps (20), but in practice you will probably want to set\n`num_train_steps` to 10000 steps or more. The `max_seq_length` and\n`max_predictions_per_seq` parameters passed to `run_pretraining.py` must be the\nsame as `create_pretraining_data.py`.\n\n```shell\npython run_pretraining.py \\\n  --input_file=/tmp/tf_examples.tfrecord \\\n  --output_dir=/tmp/pretraining_output \\\n  --do_train=True \\\n  --do_eval=True \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --train_batch_size=32 \\\n  --max_seq_length=128 \\\n  --max_predictions_per_seq=20 \\\n  --num_train_steps=20 \\\n  --num_warmup_steps=10 \\\n  --learning_rate=2e-5\n```\n\nThis will produce an output like this:\n\n```\n***** Eval results *****\n  global_step = 20\n  loss = 0.0979674\n  masked_lm_accuracy = 0.985479\n  masked_lm_loss = 0.0979328\n  next_sentence_accuracy = 1.0\n  next_sentence_loss = 3.45724e-05\n```\n\nNote that since our `sample_text.txt` file is very small, this example training\nwill overfit that data in only a few steps and produce unrealistically high\naccuracy numbers.\n\n### Pre-training tips and caveats\n\n*   **If using your own vocabulary, make sure to change `vocab_size` in\n    `bert_config.json`. If you use a larger vocabulary without changing this,\n    you will likely get NaNs when training on GPU or TPU due to unchecked\n    out-of-bounds access.**\n*   If your task has a large domain-specific corpus available (e.g., \"movie\n    reviews\" or \"scientific papers\"), it will likely be beneficial to run\n    additional steps of pre-training on your corpus, starting from the BERT\n    checkpoint.\n*   The learning rate we used in the paper was 1e-4. However, if you are doing\n    additional steps of pre-training starting from an existing BERT checkpoint,\n    you should use a smaller learning rate (e.g., 2e-5).\n*   Current BERT models are English-only, but we do plan to release a\n    multilingual model which has been pre-trained on a lot of languages in the\n    near future (hopefully by the end of November 2018).\n*   Longer sequences are disproportionately expensive because attention is\n    quadratic to the sequence length. In other words, a batch of 64 sequences of\n    length 512 is much more expensive than a batch of 256 sequences of\n    length 128. The fully-connected/convolutional cost is the same, but the\n    attention cost is far greater for the 512-length sequences. Therefore, one\n    good recipe is to pre-train for, say, 90,000 steps with a sequence length of\n    128 and then for 10,000 additional steps with a sequence length of 512. The\n    very long sequences are mostly needed to learn positional embeddings, which\n    can be learned fairly quickly. Note that this does require generating the\n    data twice with different values of `max_seq_length`.\n*   If you are pre-training from scratch, be prepared that pre-training is\n    computationally expensive, especially on GPUs. If you are pre-training from\n    scratch, our recommended recipe is to pre-train a `BERT-Base` on a single\n    [preemptible Cloud TPU v2](https://cloud.google.com/tpu/docs/pricing), which\n    takes about 2 weeks at a cost of about $500 USD (based on the pricing in\n    October 2018). You will have to scale down the batch size when only training\n    on a single Cloud TPU, compared to what was used in the paper. It is\n    recommended to use the largest batch size that fits into TPU memory.\n\n### Pre-training data\n\nWe will **not** be able to release the pre-processed datasets used in the paper.\nFor Wikipedia, the recommended pre-processing is to download\n[the latest dump](https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2),\nextract the text with\n[`WikiExtractor.py`](https://github.com/attardi/wikiextractor), and then apply\nany necessary cleanup to convert it into plain text.\n\nUnfortunately the researchers who collected the\n[BookCorpus](http://yknzhu.wixsite.com/mbweb) no longer have it available for\npublic download. The\n[Project Guttenberg Dataset](https://web.eecs.umich.edu/~lahiri/gutenberg_dataset.html)\nis a somewhat smaller (200M word) collection of older books that are public\ndomain.\n\n[Common Crawl](http://commoncrawl.org/) is another very large collection of\ntext, but you will likely have to do substantial pre-processing and cleanup to\nextract a usable corpus for pre-training BERT.\n\n### Learning a new WordPiece vocabulary\n\nThis repository does not include code for *learning* a new WordPiece vocabulary.\nThe reason is that the code used in the paper was implemented in C++ with\ndependencies on Google's internal libraries. For English, it is almost always\nbetter to just start with our vocabulary and pre-trained models. For learning\nvocabularies of other languages, there are a number of open source options\navailable. However, keep in mind that these are not compatible with our\n`tokenization.py` library:\n\n*   [Google's SentencePiece library](https://github.com/google/sentencepiece)\n\n*   [tensor2tensor's WordPiece generation script](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder_build_subword.py)\n\n*   [Rico Sennrich's Byte Pair Encoding library](https://github.com/rsennrich/subword-nmt)\n\n## Using BERT in Colab\n\nIf you want to use BERT with [Colab](https://colab.research.google.com), you can\nget started with the notebook\n\"[BERT FineTuning with Cloud TPUs](https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\".\n**At the time of this writing (October 31st, 2018), Colab users can access a\nCloud TPU completely for free.** Note: One per user, availability limited,\nrequires a Google Cloud Platform account with storage (although storage may be\npurchased with free credit for signing up with GCP), and this capability may not\nlonger be available in the future. Click on the BERT Colab that was just linked\nfor more information.\n\n## FAQ\n\n#### Is this code compatible with Cloud TPUs? What about GPUs?\n\nYes, all of the code in this repository works out-of-the-box with CPU, GPU, and\nCloud TPU. However, GPU training is single-GPU only.\n\n#### I am getting out-of-memory errors, what is wrong?\n\nSee the section on [out-of-memory issues](#out-of-memory-issues) for more\ninformation.\n\n#### Is there a PyTorch version available?\n\nThere is no official PyTorch implementation. However, NLP researchers from\nHuggingFace made a\n[PyTorch version of BERT available](https://github.com/huggingface/pytorch-pretrained-BERT)\nwhich is compatible with our pre-trained checkpoints and is able to reproduce\nour results. We were not involved in the creation or maintenance of the PyTorch\nimplementation so please direct any questions towards the authors of that\nrepository.\n\n#### Is there a Chainer version available?\n\nThere is no official Chainer implementation. However, Sosuke Kobayashi made a\n[Chainer version of BERT available](https://github.com/soskek/bert-chainer)\nwhich is compatible with our pre-trained checkpoints and is able to reproduce\nour results. We were not involved in the creation or maintenance of the Chainer\nimplementation so please direct any questions towards the authors of that\nrepository.\n\n#### Will models in other languages be released?\n\nYes, we plan to release a multi-lingual BERT model in the near future. We cannot\nmake promises about exactly which languages will be included, but it will likely\nbe a single model which includes *most* of the languages which have a\nsignificantly-sized Wikipedia.\n\n#### Will models larger than `BERT-Large` be released?\n\nSo far we have not attempted to train anything larger than `BERT-Large`. It is\npossible that we will release larger models if we are able to obtain significant\nimprovements.\n\n#### What license is this library released under?\n\nAll code *and* models are released under the Apache 2.0 license. See the\n`LICENSE` file for more information.\n\n#### How do I cite BERT?\n\nFor now, cite [the Arxiv paper](https://arxiv.org/abs/1810.04805):\n\n```\n@article{devlin2018bert,\n  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},\n  author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},\n  journal={arXiv preprint arXiv:1810.04805},\n  year={2018}\n}\n```\n\nIf we submit the paper to a conference or journal, we will update the BibTeX.\n\n## Disclaimer\n\nThis is not an official Google product.\n\n## Contact information\n\nFor help or issues using BERT, please submit a GitHub issue.\n\nFor personal communication related to BERT, please contact Jacob Devlin\n(`jacobdevlin@google.com`), Ming-Wei Chang (`mingweichang@google.com`), or\nKenton Lee (`kentonl@google.com`).\n","funding_links":[],"categories":["4 Reasoning Techniques","NLP Libraries and models with Swedish support","Python","\u003ca name=\"for-beginner\"\u003eFor Beginners\u003c/a\u003e","official implement:","2 Foundation Models","Official BERT Implementations","Python (144)","BERT优化","Implementations","Libraries and Frameworks","Technologies","Natural Language Processing (NLP)","Papers","📖 Natural Language Processing (NLP)","NLP","List of Most Starred Github Projects related to Deep Learning","Resources","\u003ca name='Models'\u003e\u003c/a\u003eModels","Codes","Uncategorized"],"sub_categories":["4.1 Pre-Training","2.1 Language Foundation Models","General","大语言对话模型及数据","Notable Projects","2022","Tools","2019","2024","Pretrained Language Models","\u003ca name='PreTrainedLanguageModels'\u003e\u003c/a\u003ePre-trained Language Models","Most QA systems have roughly 3 parts","Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fbert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fbert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fbert/lists"}