{"id":13535285,"url":"https://github.com/xu-song/bert-as-language-model","last_synced_at":"2025-04-02T00:33:15.195Z","repository":{"id":60404089,"uuid":"159764910","full_name":"xu-song/bert-as-language-model","owner":"xu-song","description":"BERT as language model, fork from https://github.com/google-research/bert","archived":false,"fork":false,"pushed_at":"2024-03-06T06:20:47.000Z","size":179,"stargazers_count":247,"open_issues_count":1,"forks_count":68,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-08-23T02:10:07.862Z","etag":null,"topics":["bert","language-model","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xu-song.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-11-30T03:43:38.000Z","updated_at":"2024-06-13T07:01:35.000Z","dependencies_parsed_at":"2024-01-14T02:37:24.094Z","dependency_job_id":"d7be78ae-e6f1-4b38-8535-beb3dfcae15a","html_url":"https://github.com/xu-song/bert-as-language-model","commit_stats":null,"previous_names":["xu-song/bert_as_language_model"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xu-song%2Fbert-as-language-model","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xu-song%2Fbert-as-language-model/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xu-song%2Fbert-as-language-model/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xu-song%2Fbert-as-language-model/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xu-song","download_url":"https://codeload.github.com/xu-song/bert-as-language-model/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222788514,"owners_count":17037777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","language-model","tensorflow"],"created_at":"2024-08-01T08:00:52.722Z","updated_at":"2024-11-02T23:31:16.036Z","avatar_url":"https://github.com/xu-song.png","language":"Python","funding_links":[],"categories":["BERT language model and embedding:"],"sub_categories":[],"readme":"**[🤗Demo](#demo)** |\n**[📖cases-en](#test-case)** |\n**[📖cases-zh](cases/test.zh.md)** |\n\n\n\n\n## BERT as Language Model\n\nFor a sentence \u003cimg src=\"https://www.zhihu.com/equation?tex=S%20=%20w_1,%20w_2,...,%20w_k\" alt=\"S = w_1, w_2,..., w_k\" eeimg=\"1\"\u003e , we have\n\n\u003cimg src=\"https://www.zhihu.com/equation?tex=p(S)%20=%20\\prod_{i=1}^{k}%20p(w_i%20|%20context)\" alt=\"p(S) = \\prod_{i=1}^{k} p(w_i | context)\" eeimg=\"1\"\u003e \n\n\nIn traditional language model, such as RNN,  \u003cimg src=\"https://www.zhihu.com/equation?tex=context%20=%20w_1,%20...,%20w_{i-1}\" alt=\"context = w_1, ..., w_{i-1}\" eeimg=\"1\"\u003e , \n\n\u003cimg src=\"https://www.zhihu.com/equation?tex=p(S)%20=%20\\prod_{i=1}^{k}%20p(w_i%20|%20w_1,%20...,%20w_{i-1})\" alt=\"p(S) = \\prod_{i=1}^{k} p(w_i | w_1, ..., w_{i-1})\" eeimg=\"1\"\u003e\n\n\nIn bidirectional language model, it has larger context, \u003cimg src=\"https://www.zhihu.com/equation?tex=context+%3d+w_1%2c+...%2c+w_%7bi-1%7d%2cw_%7bi%2b1%7d%2c...%2cw_k\" alt=\"context = w_1, ..., w_{i-1},w_{i+1},...,w_k\" eeimg=\"1\"\u003e.\n\nIn this implementation, we simply adopt the following approximation,\n\n\u003cimg src=\"https://www.zhihu.com/equation?tex=p(S)+%5capprox+%5cprod_%7bi%3d1%7d%5e%7bk%7d+p(w_i+%7c+w_1%2c+...%2c+w_%7bi-1%7d%2cw_%7bi%2b1%7d%2c+...%2cw_k)\" alt=\"p(S) \\approx \\prod_{i=1}^{k} p(w_i | w_1, ..., w_{i-1},w_{i+1}, ...,w_k)\" eeimg=\"1\"\u003e.\n\n\n\u003c!--\n1. 近似相等\n2. 句子越长，单个word预测的概率越大，ppl越大？传统的RNN也有这个问题\n--\u003e\n\n\u003c!-- n-gram\nn-gram models construct tables of conditional probabilities for the next word,\n\nUnder Markov assumption, the context is the all the \n--\u003e\n\n### Demo\n\nTry out the [Web Demo](https://huggingface.co/spaces/eson/bert-perplexity) at [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/eson/bert-perplexity)\n\n### test-case\n\n\u003e [more cases: 中文](cases/test.zh.md)\n\n\n```bash\nexport BERT_BASE_DIR=model/uncased_L-12_H-768_A-12\nexport INPUT_FILE=data/lm/test.en.tsv\npython run_lm_predict.py \\\n  --input_file=$INPUT_FILE \\\n  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n  --max_seq_length=128 \\\n  --output_dir=/tmp/lm_output/\n```\n\nfor the following test case\n\n```bash\n$ cat data/lm/test.en.tsv \nthere is a book on the desk\nthere is a plane on the desk\nthere is a book in the desk\n\n$ cat /tmp/lm/output/test_result.json\n```\noutput:\n\n```yml\n# prob: probability\n# ppl:  perplexity\n[\n  {\n    \"tokens\": [\n      {\n        \"token\": \"there\",\n        \"prob\": 0.9988962411880493\n      },\n      {\n        \"token\": \"is\",\n        \"prob\": 0.013578361831605434\n      },\n      {\n        \"token\": \"a\",\n        \"prob\": 0.9420605897903442\n      },\n      {\n        \"token\": \"book\",\n        \"prob\": 0.07452250272035599\n      },\n      {\n        \"token\": \"on\",\n        \"prob\": 0.9607976675033569\n      },\n      {\n        \"token\": \"the\",\n        \"prob\": 0.4983428418636322\n      },\n      {\n        \"token\": \"desk\",\n        \"prob\": 4.040586190967588e-06\n      }\n    ],\n    \"ppl\": 17.69329728285426\n  },\n  {\n    \"tokens\": [\n      {\n        \"token\": \"there\",\n        \"prob\": 0.996775209903717\n      },\n      {\n        \"token\": \"is\",\n        \"prob\": 0.03194097802042961\n      },\n      {\n        \"token\": \"a\",\n        \"prob\": 0.8877727389335632\n      },\n      {\n        \"token\": \"plane\",\n        \"prob\": 3.4907534427475184e-05   # low probability\n      },\n      {\n        \"token\": \"on\",\n        \"prob\": 0.1902322769165039\n      },\n      {\n        \"token\": \"the\",\n        \"prob\": 0.5981084704399109\n      },\n      {\n        \"token\": \"desk\",\n        \"prob\": 3.3164762953674654e-06\n      }\n    ],\n    \"ppl\": 59.646456254851806\n  },\n  {\n    \"tokens\": [\n      {\n        \"token\": \"there\",\n        \"prob\": 0.9969795942306519\n      },\n      {\n        \"token\": \"is\",\n        \"prob\": 0.03379646688699722\n      },\n      {\n        \"token\": \"a\",\n        \"prob\": 0.9095568060874939\n      },\n      {\n        \"token\": \"book\",\n        \"prob\": 0.013939591124653816\n      },\n      {\n        \"token\": \"in\",\n        \"prob\": 0.000823647016659379  # low probability\n      },\n      {\n        \"token\": \"the\",\n        \"prob\": 0.5844194293022156\n      },\n      {\n        \"token\": \"desk\",\n        \"prob\": 3.3361218356731115e-06\n      }\n    ],\n    \"ppl\": 54.65941516205144\n  }\n]\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxu-song%2Fbert-as-language-model","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxu-song%2Fbert-as-language-model","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxu-song%2Fbert-as-language-model/lists"}