{"id":13535299,"url":"https://github.com/hankcs/BERT-token-level-embedding","last_synced_at":"2025-04-02T00:33:15.406Z","repository":{"id":77342464,"uuid":"169492532","full_name":"hankcs/BERT-token-level-embedding","owner":"hankcs","description":"Generate BERT token level embedding without pain","archived":false,"fork":false,"pushed_at":"2019-02-07T02:03:46.000Z","size":8,"stargazers_count":28,"open_issues_count":1,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-10-31T02:11:40.455Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hankcs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-02-06T23:25:35.000Z","updated_at":"2024-01-10T09:08:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"7f3d036d-5832-4bba-a008-fbba7bd93c74","html_url":"https://github.com/hankcs/BERT-token-level-embedding","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2FBERT-token-level-embedding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2FBERT-token-level-embedding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2FBERT-token-level-embedding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hankcs%2FBERT-token-level-embedding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hankcs","download_url":"https://codeload.github.com/hankcs/BERT-token-level-embedding/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222788514,"owners_count":17037777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T08:00:53.006Z","updated_at":"2024-11-02T23:31:16.146Z","avatar_url":"https://github.com/hankcs.png","language":"Python","readme":"# BERT token level embedding\n\nThis tiny script demonstrates how to generate BERT token level embeddings. As BERT use word piece as tokenizer, OOV will be segmented into several word pieces. This script has implemented `2` strategies to handle that case.\n\nIn the following chapters, we'll show how to generate BERT embeddings for each token without pain.\n\n## 1. Install BERT server and download pre-trained BERT model\n\nRun following command:\n\n```bash\npip3 install bert-serving-server  # server\npip3 install bert-serving-client  # client, independent of `bert-serving-server`\nwget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\nunzip uncased_L-12_H-768_A-12.zip\ncd uncased_L-12_H-768_A-12/\nbert-serving-start -model_dir . -pooling_strategy NONE -show_tokens_to_client -max_seq_len 256\n```\n\n## 2. Generate embeddings\n\nPut the text you want to embed into `input.txt` :\n\n```\nHello World !\nI'm hankcs\n```\n\nRun `bert_token_embed.py`, you will get a pickle file called `output.pkl`. It stores a list of `numpy.ndarray`\n\n```\n\u003cclass 'list'\u003e: [array([[-0.10005677,  0.10111555,  0.3707362 , ..., -0.79261583,\n        -0.29630244, -0.43822828],\n       [-0.1717627 , -0.08768683,  0.57421064, ..., -0.43223655,\n        -0.02188881, -0.2638072 ],\n       [-1.2841692 , -1.4125137 , -0.92776453, ...,  0.26873824,\n        -0.03885475,  0.14489302]], dtype=float32), array([[-0.28254002, -0.01273985, -0.2916504 , ..., -0.99867177,\n         0.7456796 ,  0.3703635 ],\n       [-0.8595818 , -0.09063847, -0.14206652, ..., -0.10226044,\n         0.4216262 , -0.20428266]], dtype=float32)]\n```\n\nThe i-th `ndarray` is the BERT embedding of i-th sentence, of size n x c, where n is the sentence length, c is the dimension of BERT embedding(usually `768` or `1024`). For example, `Hello World !` is embedded as `3x768` tensor, since it contains `3` tokens.\n\nAs you may notice, `I'm hankcs` is embedded as `2x768` tensor. My ID `hankcs` is a typical OOV, which is segmented to `2` word pieces.\n\n```\n'hank', '##cs'\n```\n\nEach word piece has its own vector. There are `2` strategies implemented as `2` functions to merge them into one:\n\n1. `embed_last_token`: Use the last word piece as the representation of whole token.\n2. `embed_sum`: Use the average vector instead.\n\n## Licence\n\n#### Apache License 2.0\n\n","funding_links":[],"categories":["BERT language model and embedding:"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhankcs%2FBERT-token-level-embedding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhankcs%2FBERT-token-level-embedding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhankcs%2FBERT-token-level-embedding/lists"}