{"id":19131354,"url":"https://github.com/liyuanlucasliu/ld-net","last_synced_at":"2025-04-15T21:24:31.754Z","repository":{"id":88926706,"uuid":"125690304","full_name":"LiyuanLucasLiu/LD-Net","owner":"LiyuanLucasLiu","description":"Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling","archived":false,"fork":false,"pushed_at":"2020-02-29T22:34:45.000Z","size":613,"stargazers_count":146,"open_issues_count":4,"forks_count":13,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-29T01:24:24.131Z","etag":null,"topics":["contextualized-representation","language-model","model-compression","named-entity-recognition","ner","pytorch","sequence-labeling"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1804.07827","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LiyuanLucasLiu.png","metadata":{"files":{"readme":"ReadMe.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-18T03:46:19.000Z","updated_at":"2025-01-17T09:53:01.000Z","dependencies_parsed_at":"2023-06-13T04:30:54.860Z","dependency_job_id":null,"html_url":"https://github.com/LiyuanLucasLiu/LD-Net","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiyuanLucasLiu%2FLD-Net","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiyuanLucasLiu%2FLD-Net/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiyuanLucasLiu%2FLD-Net/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiyuanLucasLiu%2FLD-Net/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LiyuanLucasLiu","download_url":"https://codeload.github.com/LiyuanLucasLiu/LD-Net/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249155459,"owners_count":21221607,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contextualized-representation","language-model","model-compression","named-entity-recognition","ner","pytorch","sequence-labeling"],"created_at":"2024-11-09T06:14:54.573Z","updated_at":"2025-04-15T21:24:31.734Z","avatar_url":"https://github.com/LiyuanLucasLiu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LD-Net\n\n[![Documentation Status](https://readthedocs.org/projects/ld-net/badge/?version=latest)](http://ld-net.readthedocs.io/en/latest/?badge=latest)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n**Check Our New NER Toolkit🚀🚀🚀**\n- **Inference**:\n  - **[LightNER](https://github.com/LiyuanLucasLiu/LightNER)**: inference w. models pre-trained / trained w. *any* following tools, *efficiently*. \n- **Training**:\n  - **[LD-Net](https://github.com/LiyuanLucasLiu/LD-Net)**: train NER models w. efficient contextualized representations.\n  - **[VanillaNER](https://github.com/LiyuanLucasLiu/Vanilla_NER)**: train vanilla NER models w. pre-trained embedding.\n- **Distant Training**:\n  - **[AutoNER](https://shangjingbo1226.github.io/AutoNER/)**: train NER models w.o. line-by-line annotations and get competitive performance.\n\n--------------------------------\n\nLD-Net provides sequence labeling models featuring:\n- **Efficiency**: constructing *efficient contextualized representations* without retraining language models. \n- **Portability**: *well-organized*, *easy-to-modify* and *[well-documented](http://lm-lstm-crf.readthedocs.io/en/latest/)*.\n\nRemarkablely, our pre-trained NER model achieved:\n- **92.08** test F1 on the CoNLL03 NER task.\n- **160K words/sec** decoding speed (**6X** speedup compared to its original model).\n\nDetails about LD-Net can be accessed at: https://arxiv.org/abs/1804.07827.\n\n- [Model notes](#model-notes)\n- [Benchmarks](#benchmarks)\n- [Pretrained model](#pretrained-model)\n\t- [Language models](#language-models)\n\t- [Named Entity Recognition](#named-entity-recognition)\n\t- [Chunking](#chunking)\n- [Training](#model-training)\n\t- [Dependency](#dependency)\n\t- [Data](#data)\n\t- [Model](#model)\n\t- [Command](#command)\n- [Inference](#inference)\n- [Citation](#citation)\n\n## Model Notes\n\n![LD-Net Framework](docs/model_note.png)\n\n## Benchmarks\n\n| Model for CoNLL03 | #FLOPs| Mean(F1) | Std(F1) |\n| ------------- |-------------| -----| -----|\n| Vanilla NER w.o. LM | 3 M | 90.78 | 0.24 |\n| LD-Net (w.o. pruning) | 51 M | 91.86 | 0.15 |\n| LD-Net (origin, picked based on dev f1) | 51 M | 91.95 |  |\n| LD-Net (pruned) | **5 M** | 91.84 | 0.14 |\n\n| Model for CoNLL00 | #FLOPs| Mean(F1) | Std(F1) |\n| ------------- |-------------| -----| -----|\n| Vanilla NP w.o. LM | 3 M | 94.42 | 0.08 |\n| LD-Net (w.o. pruning) | 51 M | 96.01 | 0.07 |\n| LD-Net (origin, picked based on dev f1) | 51 M | 96.13 |  |\n| LD-Net (pruned) | **10 M** | 95.66 | 0.04 |\n\n## Pretrained Models\n\nHere we provide both pre-trained language models and pre-trained sequence labeling models.\n\n### Language Models\n\nOur pretrained language model contains word embedding, 10-layer densely-connected LSTM and adative softmax, and achieve an average PPL of 50.06 on the one billion benchmark dataset.\n\n| Forward Language Model | Backward Language Model |\n| ------------- |------------- |\n| [Download Link](http://dmserv4.cs.illinois.edu/ld0.th) | [Download Link](http://dmserv4.cs.illinois.edu/ld_0.th)|\n\n### Named Entity Recognition\n\nThe original pre-trained named entity tagger achieves 91.95 F1, the pruned tagged achieved 92.08 F1.\n\n| Original Tagger | Pruned Tagger |\n| ------------- |------------- |\n| [Download Link](http://dmserv4.cs.illinois.edu/ner.th) | [Download Link](http://dmserv4.cs.illinois.edu/pner0.th) |\n\n### Chunking\n\nThe original pre-trained named entity tagger achieves 96.13 F1, the pruned tagged achieved 95.79 F1.\n\n| Original Tagger | Pruned Tagger |\n| ------------- |------------- |\n| [Download Link](http://dmserv4.cs.illinois.edu/np.th) | [Download Link](http://dmserv4.cs.illinois.edu/pnp0.th) |\n\n## Training\n\n### Demo Scripts\n\nTo pruning the original LD-Net for the CoNLL03 NER, please run:\n```\nbash ldnet_ner_prune.sh\n```\n\nTo pruning the original LD-Net for the CoNLL00 Chunking, please run:\n```\nbash ldnet_np_prune.sh\n```\n\n### Dependency\n\nOur package is based on Python 3.6 and the following packages:\n```\nnumpy\ntqdm\ntorch-scope\ntorch==0.4.1\n```\n\n### Data\n\nPre-process scripts are available in ```pre_seq``` and ```pre_word_ada```, while pre-processed data has been stored in:\n\n| NER | Chunking |\n| ------------- |------------- |\n| [Download Link](http://dmserv4.cs.illinois.edu/ner_dataset.pk) | [Download Link](http://dmserv4.cs.illinois.edu/np_dataset.pk) |\n\n### Model\n\nOur implementations are available in ```model_seq``` and ```model_word_ada```, and the documentations are hosted in [ReadTheDoc](http://lm-lstm-crf.readthedocs.io/en/latest/)\n\n| NER | Chunking |\n| ------------- |------------- |\n| [Download Link](http://dmserv4.cs.illinois.edu/ner_dataset.pk) | [Download Link](http://dmserv4.cs.illinois.edu/np_dataset.pk) |\n\n## Inference\n\nFor model inference, please check our [LightNER package](https://github.com/LiyuanLucasLiu/LightNER) \n\n## Citation\n\nIf you find the implementation useful, please cite the following paper: [Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling](https://arxiv.org/abs/1804.07827)\n\n```\n@inproceedings{liu2018efficient,\n  title = \"{Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling}\", \n  author = {Liu, Liyuan and Ren, Xiang and Shang, Jingbo and Peng, Jian and Han, Jiawei}, \n  booktitle = {EMNLP}, \n  year = 2018, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliyuanlucasliu%2Fld-net","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliyuanlucasliu%2Fld-net","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliyuanlucasliu%2Fld-net/lists"}