{"id":13595203,"url":"https://github.com/monologg/JointBERT","last_synced_at":"2025-04-09T10:33:06.914Z","repository":{"id":42559046,"uuid":"221733013","full_name":"monologg/JointBERT","owner":"monologg","description":"Pytorch implementation of JointBERT: \"BERT for Joint Intent Classification and Slot Filling\"","archived":false,"fork":false,"pushed_at":"2024-01-11T07:51:17.000Z","size":475,"stargazers_count":665,"open_issues_count":16,"forks_count":186,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-06T17:46:25.485Z","etag":null,"topics":["bert","intent-classification","joint-bert","pytorch","slot-filling","slu","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monologg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-14T15:48:12.000Z","updated_at":"2024-11-01T04:10:54.000Z","dependencies_parsed_at":"2024-01-16T22:19:09.228Z","dependency_job_id":"5d9c939e-9af9-49ba-8269-fc2cc581a84d","html_url":"https://github.com/monologg/JointBERT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monologg%2FJointBERT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monologg%2FJointBERT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monologg%2FJointBERT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monologg%2FJointBERT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monologg","download_url":"https://codeload.github.com/monologg/JointBERT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248020593,"owners_count":21034459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","intent-classification","joint-bert","pytorch","slot-filling","slu","transformers"],"created_at":"2024-08-01T16:01:45.643Z","updated_at":"2025-04-09T10:33:01.904Z","avatar_url":"https://github.com/monologg.png","language":"Python","funding_links":[],"categories":["Python","实体识别NER、意图识别、槽位填充"],"sub_categories":["其他_文本生成、文本对话"],"readme":"# JointBERT\n\n(Unofficial) Pytorch implementation of `JointBERT`: [BERT for Joint Intent Classification and Slot Filling](https://arxiv.org/abs/1902.10909)\n\n## Model Architecture\n\n\u003cp float=\"left\" align=\"center\"\u003e\n    \u003cimg width=\"600\" src=\"https://user-images.githubusercontent.com/28896432/68875755-b2f92900-0746-11ea-8819-401d60e4185f.png\" /\u003e  \n\u003c/p\u003e\n\n- Predict `intent` and `slot` at the same time from **one BERT model** (=Joint model)\n- total_loss = intent_loss + coef \\* slot_loss (Change coef with `--slot_loss_coef` option)\n- **If you want to use CRF layer, give `--use_crf` option**\n\n## Dependencies\n\n- python\u003e=3.6\n- torch==1.6.0\n- transformers==3.0.2\n- seqeval==0.0.12\n- pytorch-crf==0.7.2\n\n## Dataset\n\n|       | Train  | Dev | Test | Intent Labels | Slot Labels |\n| ----- | ------ | --- | ---- | ------------- | ----------- |\n| ATIS  | 4,478  | 500 | 893  | 21            | 120         |\n| Snips | 13,084 | 700 | 700  | 7             | 72          |\n\n- The number of labels are based on the _train_ dataset.\n- Add `UNK` for labels (For intent and slot labels which are only shown in _dev_ and _test_ dataset)\n- Add `PAD` for slot label\n\n## Training \u0026 Evaluation\n\n```bash\n$ python3 main.py --task {task_name} \\\n                  --model_type {model_type} \\\n                  --model_dir {model_dir_name} \\\n                  --do_train --do_eval \\\n                  --use_crf\n\n# For ATIS\n$ python3 main.py --task atis \\\n                  --model_type bert \\\n                  --model_dir atis_model \\\n                  --do_train --do_eval\n# For Snips\n$ python3 main.py --task snips \\\n                  --model_type bert \\\n                  --model_dir snips_model \\\n                  --do_train --do_eval\n```\n\n## Prediction\n\n```bash\n$ python3 predict.py --input_file {INPUT_FILE_PATH} --output_file {OUTPUT_FILE_PATH} --model_dir {SAVED_CKPT_PATH}\n```\n\n## Results\n\n- Run 5 ~ 10 epochs (Record the best result)\n- Only test with `uncased` model\n- ALBERT xxlarge sometimes can't converge well for slot prediction.\n\n|           |                  | Intent acc (%) | Slot F1 (%) | Sentence acc (%) |\n| --------- | ---------------- | -------------- | ----------- | ---------------- |\n| **Snips** | BERT             | **99.14**      | 96.90       | 93.00            |\n|           | BERT + CRF       | 98.57          | **97.24**   | **93.57**        |\n|           | DistilBERT       | 98.00          | 96.10       | 91.00            |\n|           | DistilBERT + CRF | 98.57          | 96.46       | 91.85            |\n|           | ALBERT           | 98.43          | 97.16       | 93.29            |\n|           | ALBERT + CRF     | 99.00          | 96.55       | 92.57            |\n| **ATIS**  | BERT             | 97.87          | 95.59       | 88.24            |\n|           | BERT + CRF       | **97.98**      | 95.93       | 88.58            |\n|           | DistilBERT       | 97.76          | 95.50       | 87.68            |\n|           | DistilBERT + CRF | 97.65          | 95.89       | 88.24            |\n|           | ALBERT           | 97.64          | 95.78       | 88.13            |\n|           | ALBERT + CRF     | 97.42          | **96.32**   | **88.69**        |\n\n## Updates\n\n- 2019/12/03: Add DistilBert and RoBERTa result\n- 2019/12/14: Add Albert (large v1) result\n- 2019/12/22: Available to predict sentences\n- 2019/12/26: Add Albert (xxlarge v1) result\n- 2019/12/29: Add CRF option\n- 2019/12/30: Available to check `sentence-level semantic frame accuracy`\n- 2020/01/23: Only show the result related with uncased model\n- 2020/04/03: Update with new prediction code\n\n## References\n\n- [Huggingface Transformers](https://github.com/huggingface/transformers)\n- [pytorch-crf](https://github.com/kmkurn/pytorch-crf)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonologg%2FJointBERT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonologg%2FJointBERT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonologg%2FJointBERT/lists"}