{"id":13793595,"url":"https://github.com/UBC-NLP/LMBERT","last_synced_at":"2025-05-12T20:31:06.941Z","repository":{"id":65821643,"uuid":"316316300","full_name":"UBC-NLP/LMBERT","owner":"UBC-NLP","description":null,"archived":false,"fork":false,"pushed_at":"2020-12-05T02:28:04.000Z","size":100,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-18T08:55:08.642Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UBC-NLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-26T19:05:42.000Z","updated_at":"2023-03-17T15:21:48.000Z","dependencies_parsed_at":"2023-02-12T09:25:11.204Z","dependency_job_id":null,"html_url":"https://github.com/UBC-NLP/LMBERT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UBC-NLP%2FLMBERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UBC-NLP%2FLMBERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UBC-NLP%2FLMBERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UBC-NLP%2FLMBERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UBC-NLP","download_url":"https://codeload.github.com/UBC-NLP/LMBERT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253816704,"owners_count":21968869,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:26.155Z","updated_at":"2025-05-12T20:31:06.643Z","avatar_url":"https://github.com/UBC-NLP.png","language":"Python","funding_links":[],"categories":["1.1 ARBERT \u0026 MARBERT"],"sub_categories":[],"readme":"# What is LMBERT?\nLMBERT is BERT pre-training masked language model *without next sentence prediction*. This code is a adjustment of Google's [original BERT code](https://github.com/google-research/bert) where we simply comment the next sentence prediction parts from the data preparation script ([create_pretraining_data.py](https://github.com/UBC-NLP/LMBERT/blob/main/create_pretraining_data.py)) and also edit the objective function to remove next sentence prediction in the running script ([run_pretraining.py](https://github.com/UBC-NLP/LMBERT/blob/main/run_pretraining.py)).\n\n\n# Pre-training with LMBERT\n\nThis code is to do \"masked LM\" on an arbitrary text corpus.\nFor convenience, ```we copy the below from Google's``` [GitHub](https://github.com/google-research/bert).\n\nHere's how to run the data generation. The input is a plain text file, with one\nsentence per line. (It is important that these be actual sentences for the \"next\nsentence prediction\" task). Documents are delimited by empty lines. The output\nis a set of `tf.train.Example`s serialized into `TFRecord` file format.\n\nYou can perform sentence segmentation with an off-the-shelf NLP toolkit such as\n[spaCy](https://spacy.io/). The `create_pretraining_data.py` script will\nconcatenate segments until they reach the maximum sequence length to minimize\ncomputational waste from padding (see the script for more details). However, you\nmay want to intentionally add a slight amount of noise to your input data (e.g.,\nrandomly truncate 2% of input segments) to make it more robust to non-sentential\ninput during fine-tuning.\n\nThis script stores all of the examples for the entire input file in memory, so\nfor large data files you should shard the input file and call the script\nmultiple times. (You can pass in a file glob to `run_pretraining.py`, e.g.,\n`tf_examples.tf_record*`.)\n\nThe `max_predictions_per_seq` is the maximum number of masked LM predictions per\nsequence. You should set this to around `max_seq_length` * `masked_lm_prob` (the\nscript doesn't do that automatically because the exact value needs to be passed\nto both scripts).\n\n```shell\npython create_pretraining_data.py \\\n  --input_file=./sample_text.txt \\\n  --output_file=/tmp/tf_examples.tfrecord \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --do_lower_case=True \\\n  --max_seq_length=128 \\\n  --max_predictions_per_seq=20 \\\n  --masked_lm_prob=0.15 \\\n  --random_seed=12345 \\\n  --dupe_factor=5\n```\n\nHere's how to run the pre-training. Do not include `init_checkpoint` if you are\npre-training from scratch. The model configuration (including vocab size) is\nspecified in `bert_config_file`. This demo code only pre-trains for a small\nnumber of steps (20), but in practice you will probably want to set\n`num_train_steps` to 10000 steps or more. The `max_seq_length` and\n`max_predictions_per_seq` parameters passed to `run_pretraining.py` must be the\nsame as `create_pretraining_data.py`.\n\n```shell\npython run_pretraining.py \\\n  --input_file=/tmp/tf_examples.tfrecord \\\n  --output_dir=/tmp/pretraining_output \\\n  --do_train=True \\\n  --do_eval=True \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --train_batch_size=32 \\\n  --max_seq_length=128 \\\n  --max_predictions_per_seq=20 \\\n  --num_train_steps=20 \\\n  --num_warmup_steps=10 \\\n  --learning_rate=2e-5\n```\n\nThis will produce an output like this:\n\n```\n***** Eval results *****\n  global_step = 20\n  loss = 0.0979674\n  masked_lm_accuracy = 0.985479\n  masked_lm_loss = 0.0979328\n  next_sentence_accuracy = 1.0\n  next_sentence_loss = 3.45724e-05\n```\n\nNote that since our `sample_text.txt` file is very small, this example training\nwill overfit that data in only a few steps and produce unrealistically high\naccuracy numbers.\n\n\n### Pre-training data\n\nWe will **not** be able to release the pre-processed datasets used in the paper.\nFor Wikipedia, the recommended pre-processing is to download\n[the latest dump](https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2),\nextract the text with\n[`WikiExtractor.py`](https://github.com/attardi/wikiextractor), and then apply\nany necessary cleanup to convert it into plain text.\n\nUnfortunately the researchers who collected the\n[BookCorpus](http://yknzhu.wixsite.com/mbweb) no longer have it available for\npublic download. The\n[Project Guttenberg Dataset](https://web.eecs.umich.edu/~lahiri/gutenberg_dataset.html)\nis a somewhat smaller (200M word) collection of older books that are public\ndomain.\n\n[Common Crawl](http://commoncrawl.org/) is another very large collection of\ntext, but you will likely have to do substantial pre-processing and cleanup to\nextract a usable corpus for pre-training BERT.\n\n### Learning a new WordPiece vocabulary\n\nThis repository does not include code for *learning* a new WordPiece vocabulary.\nThe reason is that the code used in the paper was implemented in C++ with\ndependencies on Google's internal libraries. For English, it is almost always\nbetter to just start with our vocabulary and pre-trained models. For learning\nvocabularies of other languages, there are a number of open source options\navailable. However, keep in mind that these are not compatible with our\n`tokenization.py` library:\n\n*   [Google's SentencePiece library](https://github.com/google/sentencepiece)\n\n*   [tensor2tensor's WordPiece generation script](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder_build_subword.py)\n\n*   [Rico Sennrich's Byte Pair Encoding library](https://github.com/rsennrich/subword-nmt)\n\n## Using BERT in Colab\n\nIf you want to use BERT with [Colab](https://colab.research.google.com), you can\nget started with the notebook\n\"[BERT FineTuning with Cloud TPUs](https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\".\n**At the time of this writing (October 31st, 2018), Colab users can access a\nCloud TPU completely for free.** Note: One per user, availability limited,\nrequires a Google Cloud Platform account with storage (although storage may be\npurchased with free credit for signing up with GCP), and this capability may not\nlonger be available in the future. Click on the BERT Colab that was just linked\nfor more information.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUBC-NLP%2FLMBERT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FUBC-NLP%2FLMBERT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUBC-NLP%2FLMBERT/lists"}