{"id":13535276,"url":"https://github.com/yuanxiaosc/BERT-for-Sequence-Labeling-and-Text-Classification","last_synced_at":"2025-04-02T00:33:12.385Z","repository":{"id":34353600,"uuid":"171868025","full_name":"yuanxiaosc/BERT-for-Sequence-Labeling-and-Text-Classification","owner":"yuanxiaosc","description":"This is the template code to use BERT for sequence lableing and text classification, in order to facilitate BERT for more tasks. Currently, the template code has included conll-2003 named entity identification, Snips Slot Filling and Intent Prediction.","archived":false,"fork":false,"pushed_at":"2022-12-08T06:59:03.000Z","size":2882,"stargazers_count":466,"open_issues_count":37,"forks_count":97,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-02T08:10:06.390Z","etag":null,"topics":["atis-dataset","bert","conll-2003","sequence-labeling","snips-dataset","template-project","text-classification"],"latest_commit_sha":null,"homepage":"https://yuanxiaosc.github.io/2018/12/26/%E5%91%BD%E5%90%8D%E5%AE%9E%E4%BD%93%E8%AF%86%E5%88%ABCoNLL2003/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuanxiaosc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-21T12:29:30.000Z","updated_at":"2024-07-31T22:15:11.000Z","dependencies_parsed_at":"2023-01-15T06:31:12.862Z","dependency_job_id":null,"html_url":"https://github.com/yuanxiaosc/BERT-for-Sequence-Labeling-and-Text-Classification","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuanxiaosc","download_url":"https://codeload.github.com/yuanxiaosc/BERT-for-Sequence-Labeling-and-Text-Classification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222788514,"owners_count":17037777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atis-dataset","bert","conll-2003","sequence-labeling","snips-dataset","template-project","text-classification"],"created_at":"2024-08-01T08:00:52.578Z","updated_at":"2024-11-02T23:31:14.782Z","avatar_url":"https://github.com/yuanxiaosc.png","language":"Python","funding_links":[],"categories":["BERT chatbot :"],"sub_categories":[],"readme":"# Template Code: BERT-for-Sequence-Labeling-and-Text-Classification\nBERT is used for sequence annotation and text categorization template code to facilitate BERT for more tasks. The code has been tested on snips (intention recognition and slot filling task), ATIS (intention recognition and slot filling task) and conll-2003 (named entity recognition task) datasets. Welcome to use this BERT template to solve more NLP tasks, and then share your results and code here.\n\n这是使用BERT进行序列标注和文本分类的模板代码，方便大家将BERT用于更多任务。该代码已经在SNIPS（意图识别和槽填充任务）、ATIS（意图识别和槽填充任务）和conll-2003（命名实体识别任务）数据集上进行了实验。欢迎使用这个BERT模板解决更多NLP任务，然后在这里分享你的结果和代码。\n\n![](https://yuanxiaosc.github.io/2019/03/18/%E6%A7%BD%E5%A1%AB%E5%85%85%E5%92%8C%E6%84%8F%E5%9B%BE%E8%AF%86%E5%88%AB%E4%BB%BB%E5%8A%A1%E7%9A%84%E5%9F%BA%E6%9C%AC%E6%A6%82%E5%BF%B5/1.png)\n\n## Task and Dataset\nI have downloaded the data for you. Welcome to add new data set.\n\n|task name|dataset name|data source|\n|-|-|-|\n|CoNLL-2003 named entity recognition|conll2003ner|https://www.clips.uantwerpen.be/conll2003/ner/ |\n|Atis Joint Slot Filling and Intent Prediction|atis|https://github.com/MiuLab/SlotGated-SLU/tree/master/data/atis |\n|Snips Joint Slot Filling and Intent Prediction|snips|https://github.com/MiuLab/SlotGated-SLU/tree/master/data/snips |\n\n\n## Environment Requirements\nUse `pip install -r requirements.txt` to install dependencies quickly.\n+ python 3.6+\n+ Tensorflow 1.12.0+\n+ sklearn\n\n## Template Code Usage Method\n\n### Using pre training and fine-tuning model directly\n\u003e For example: Atis Joint Slot Filling and Intent Prediction\n\n1. Download model weight [atis_join_task_LSTM_epoch30_simple.zip](https://pan.baidu.com/s/1SZkQXP8NrOtZKVEMfDE4bw) and unzip then to  file `store_fine_tuned_model`, https://pan.baidu.com/s/1SZkQXP8NrOtZKVEMfDE4bw;\n2. Run Code!  You can change task_name and output_dir.\n```bash\npython run_slot_intent_join_task_LSTM.py \\\n  --task_name=Atis \\\n  --do_predict=true \\\n  --data_dir=data/atis_Intent_Detection_and_Slot_Filling \\\n  --vocab_file=pretrained_model/uncased_L-12_H-768_A-12/vocab.txt \\\n  --bert_config_file=pretrained_model/uncased_L-12_H-768_A-12/bert_config.json \\\n  --init_checkpoint=store_fine_tuned_model/atis_join_task_LSTM_epoch30_simple/model.ckpt-4198 \\\n  --max_seq_length=128 \\\n  --output_dir=./output_model_predict/atis_join_task_LSTM_epoch30_simple_ckpt4198\n```\n\nYou can find the file of model prediction and the score of model prediction in `output_dir` (You can find the content of model socres later).\n\n\n### Quick start(model train and predict)\n\u003e See [predefined_task_usage.md](predefined_task_usage.md) for more predefined task usage codes.\n\n1. Move google's [BERT code](https://github.com/google-research/bert) to  file `bert` (I've prepared a copy for you.);\n2. Download google's [BERT pretrained model](https://github.com/google-research/bert) and unzip then to  file `pretrained_model`, https://github.com/google-research/bert;\n3. Run Code!  You can change task_name and output_dir.\n\n**model training**\n```\npython run_sequence_labeling_and_text_classification.py \\\n  --task_name=snips \\\n  --do_train=true \\\n  --do_eval=true \\\n  --data_dir=data/snips_Intent_Detection_and_Slot_Filling \\\n  --vocab_file=pretrained_model/uncased_L-12_H-768_A-12/vocab.txt \\\n  --bert_config_file=pretrained_model/uncased_L-12_H-768_A-12/bert_config.json \\\n  --init_checkpoint=pretrained_model/uncased_L-12_H-768_A-12/bert_model.ckpt \\\n  --num_train_epochs=3.0 \\\n  --output_dir=./store_fine_tuned_model/snips_join_task_epoch3/\n```\n\nThen you can find the fine tuned model in the `output_dir=./store_fine_tuned_model/snips_join_task_epoch3/` folder.\n\n\n**model prediction**\n```\npython run_sequence_labeling_and_text_classification.py \\\n  --task_name=Snips \\\n  --do_predict=true \\\n  --data_dir=data/snips_Intent_Detection_and_Slot_Filling \\\n  --vocab_file=pretrained_model/uncased_L-12_H-768_A-12/vocab.txt \\\n  --bert_config_file=pretrained_model/uncased_L-12_H-768_A-12/bert_config.json \\\n  --init_checkpoint=output_model/snips_join_task_epoch3/model.ckpt-1000 \\\n  --max_seq_length=128 \\\n  --output_dir=./output_model_prediction/snips_join_task_epoch3_ckpt1000\n```\n\nThen you can find the predicted output of the model and the output test results (accuracy, recall, F1 value, etc.) in the `output_dir=./output_model_prediction/snips_join_task_epoch3_ckpt1000` folder.\n\n\n## File Structure\n\n|name|function|\n|-|-|\n| bert |store google's [BERT code](https://github.com/google-research/bert)|||\n| data |store task raw data set|\n|output_model_prediction|store model predict|\n|store_fine_tuned_model| store finet tuned model|\n|calculating_model_score||\n|pretrained_model |store [BERT pretrained model](https://github.com/google-research/bert)|\n|run_sequence_labeling.py |for Sequence Labeling Task|\n|run_text_classification.py| for Text Classification Task|\n|run_sequence_labeling_and_text_classification.py| for join task |\n|calculate_model_score.py |for evaluation model |\n\n\n## Model Socres\n\n**The following model scores are model scores without careful adjustment of model parameters, that is to say, the scores can continue to improve!**\n\n### CoNLL-2003 named entity recognition\neval_f = 0.926\neval_precision = 0.925\neval_recall = 0.928\n\n### Atis Joint Slot Filling and Intent Prediction\nIntent Prediction\nCorrect rate: 0.976\nAccuracy: 0.976\nRecall rate: 0.976\nF1-score: 0.976\n\nSlot Filling19\nCorrect rate:\t0.955\nAccuracy:\t0.955\nRecall rate:\t0.955\nF1-score: 0.955\n\n## How to add a new task\n\nJust write a small piece of code according to the existing template!\n\n### Data\nFor example, If you have a new classification task [QQP](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs).\n\nBefore running this example you must download the [GLUE data](https://gluebenchmark.com/tasks) by running [this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e).\n\n### Code\nNow, write code!\n\n```\nclass QqpProcessor(DataProcessor):\n    \"\"\"Processor for the QQP data set.\"\"\"\n\n    def get_train_examples(self, data_dir):\n        \"\"\"See base class.\"\"\"\n        return self._create_examples(\n            self._read_tsv(os.path.join(data_dir, \"train.tsv\")), \"train\")\n\n    def get_dev_examples(self, data_dir):\n        \"\"\"See base class.\"\"\"\n        return self._create_examples(\n            self._read_tsv(os.path.join(data_dir, \"dev.tsv\")), \"dev\")\n\n    def get_test_examples(self, data_dir):\n        \"\"\"See base class.\"\"\"\n        return self._create_examples(\n            self._read_tsv(os.path.join(data_dir, \"test.tsv\")), \"test\")\n\n    def get_labels(self):\n        \"\"\"See base class.\"\"\"\n        return [\"0\", \"1\"]\n\n    def _create_examples(self, lines, set_type):\n        \"\"\"Creates examples for the training and dev sets.\"\"\"\n        examples = []\n        for (i, line) in enumerate(lines):\n            if i == 0 or len(line)!=6:\n                continue\n            guid = \"%s-%s\" % (set_type, i)\n            text_a = tokenization.convert_to_unicode(line[3])\n            text_b = tokenization.convert_to_unicode(line[4])\n            if set_type == \"test\":\n                label = \"1\"\n            else:\n                label = tokenization.convert_to_unicode(line[5])\n            examples.append(\n                InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))\n        return examples\n ```\n\n Registration task\n\n ```\n def main(_):\n    tf.logging.set_verbosity(tf.logging.INFO)\n    processors = {\n        \"qqp\": QqpProcessor,\n    }\n```\n\n### Run\n```\npython run_text_classification.py \\\n--task_name=qqp \\\n--do_train=true \\\n--do_eval=true \\\n--data_dir=data/snips_Intent_Detection_and_Slot_Filling \\\n--vocab_file=pretrained_model/uncased_L-12_H-768_A-12/vocab.txt \\\n--bert_config_file=pretrained_model/uncased_L-12_H-768_A-12/bert_config.json \\\n--init_checkpoint=pretrained_model/uncased_L-12_H-768_A-12/bert_model.ckpt \\\n--max_seq_length=128 \\\n--train_batch_size=32 \\\n--learning_rate=2e-5 \\\n--num_train_epochs=3.0 \\\n--output_dir=./output/qqp_Intent_Detection/\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuanxiaosc%2FBERT-for-Sequence-Labeling-and-Text-Classification/lists"}