{"id":13567126,"url":"https://github.com/brightmart/text_classification","last_synced_at":"2025-05-14T02:06:06.954Z","repository":{"id":38862600,"uuid":"92841276","full_name":"brightmart/text_classification","owner":"brightmart","description":"all kinds of text classification models and more with deep learning","archived":false,"fork":false,"pushed_at":"2023-09-28T05:55:11.000Z","size":14810,"stargazers_count":7905,"open_issues_count":46,"forks_count":2569,"subscribers_count":298,"default_branch":"master","last_synced_at":"2025-04-10T16:48:35.047Z","etag":null,"topics":["attention-mechanism","classification","convolutional-neural-networks","fasttext","memory-networks","multi-class","multi-label","nlp","sentence-classification","tensorflow","text-classification","textcnn","textrnn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brightmart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-05-30T14:25:42.000Z","updated_at":"2025-04-10T09:28:26.000Z","dependencies_parsed_at":"2024-01-07T00:07:09.891Z","dependency_job_id":"8d2130d1-52aa-4b0a-9375-d2597dabe6d8","html_url":"https://github.com/brightmart/text_classification","commit_stats":{"total_commits":308,"total_committers":15,"mean_commits":"20.533333333333335","dds":0.09090909090909094,"last_synced_commit":"091ff9910839ba5053302383af99762c0c91a992"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightmart%2Ftext_classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightmart%2Ftext_classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightmart%2Ftext_classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brightmart%2Ftext_classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brightmart","download_url":"https://codeload.github.com/brightmart/text_classification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254052692,"owners_count":22006716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","classification","convolutional-neural-networks","fasttext","memory-networks","multi-class","multi-label","nlp","sentence-classification","tensorflow","text-classification","textcnn","textrnn"],"created_at":"2024-08-01T13:02:24.292Z","updated_at":"2025-05-14T02:06:01.925Z","avatar_url":"https://github.com/brightmart.png","language":"Python","readme":"Text Classification\n-------------------------------------------------------------------------\nThe purpose of this repository is to explore text classification methods in NLP with deep learning.\n\n#### Update: \n\nCustomize an NLP API in three minutes, for free: \u003ca href='https://www.cluebenchmarks.com/clueai.html'\u003eNLP API Demo\u003c/a\u003e\n\nLanguage Understanding Evaluation benchmark for Chinese(\u003ca href='https://www.CLUEbenchmarks.com'\u003eCLUE benchmark\u003ca/\u003e): run 10 tasks \u0026 9 baselines with one line of code, performance comparision with details.\n\nReleasing Pre-trained Model of \u003ca href=\"https://github.com/brightmart/albert_zh\"\u003eALBERT_Chinese\u003c/a\u003e Training with 30G+ Raw Chinese Corpus, xxlarge, xlarge and more, Target to match State of the Art performance in Chinese, 2019-Oct-7, During the National Day of China!\n \n\u003ca href='https://github.com/brightmart/nlp_chinese_corpus'\u003eLarge Amount of Chinese Corpus for NLP Available!\u003c/a\u003e\n\nGoogle's BERT achieved new state of art result on more than 10 tasks in NLP using pre-train in language model then \n\nfine-tuning. \u003ca href='https://github.com/brightmart/bert_language_understanding'\u003ePre-train TexCNN: idea from BERT for language understanding with running code and data set\u003c/a\u003e\n\n\n#### Introduction\nit has all kinds of baseline models for text classification.\n\nit also support for multi-label classification where multi labels associate with an sentence or document.\n\nalthough many of these models are simple, and may not get you to top level of the task. but some of these models are very \n\nclassic, so they may be good to serve as baseline models. each model has a test function under model class. you can run \n\nit to performance toy task first. the model is independent from data set.\n\n\u003ca href='https://github.com/brightmart/text_classification/blob/master/multi-label-classification.pdf'\u003echeck here for formal report of large scale multi-label text classification with deep learning\u003c/a\u003e\n\nseveral models here can also be used for modelling question answering (with or without context), or to do sequences generating. \n\nwe explore two seq2seq model(seq2seq with attention,transformer-attention is all you need) to do text classification. \n\nand these two models can also be used for sequences generating and other tasks. if your task is a multi-label classification, \n\nyou can cast the problem to sequences generating.\n\nwe implement two memory network. one is dynamic memory network. previously it reached state of art in question \n\nanswering, sentiment analysis and sequence generating tasks. it is so called one model to do several different tasks, \n\nand reach high performance. it has four modules. the key component is episodic memory module. it use gate mechanism to \n\nperformance attention, and use gated-gru to update episode memory, then it has another gru( in a vertical direction) to \n\nperformance hidden state update. it has ability to do transitive inference.\n\nthe second memory network we implemented is recurrent entity network: tracking state of the world. it has blocks of \n\nkey-value pairs as memory, run in parallel, which achieve new state of art. it can be used for modelling question \n\nanswering with contexts(or history). for example, you can let the model to read some sentences(as context), and ask a \n\nquestion(as query), then ask the model to predict an answer; if you feed story same as query, then it can do \n\nclassification task. \n\nTo discuss ML/DL/NLP problems and get tech support from each other, you can join QQ group: 836811304\n\nModels:\n-------------------------------------------------------------------------\n\n1) fastText\n2) TextCNN \n3) Bert:Pre-training of Deep Bidirectional Transformers for Language Understanding  \n4) TextRNN    \n5) RCNN     \n6) Hierarchical Attention Network    \n7) seq2seq with attention   \n8) Transformer(\"Attend Is All You Need\")\n9) Dynamic Memory Network\n10) EntityNetwork:tracking state of the world\n11) Ensemble models\n12) Boosting: \n\n    for a single model, stack identical models together. each layer is a model. the result will be based on logits added together. the only connection between layers are label's weights. the front layer's prediction error rate of each label will become weight for the next layers. those labels with high error rate will have big weight. so later layer's will pay more attention to those mis-predicted labels, and try to fix previous mistake of former layer. as a result, we will get a much strong model.\n    check a00_boosting/boosting.py\n\nand other models:\n\n1) BiLstmTextRelation;\n\n2) twoCNNTextRelation;\n\n3) BiLstmTextRelationTwoRNN\n\nPerformance\n-------------------------------------------------------------------------\n\n(mulit-label label prediction task,ask to prediction top5, 3 million training data,full score:0.5)\n\nModel   | fastText|TextCNN|TextRNN| RCNN | HierAtteNet|Seq2seqAttn|EntityNet|DynamicMemory|Transformer\n---     | ---     | ---   | ---   |---   |---         |---        |---      |---          |----\nScore   | 0.362   |  0.405| 0.358 | 0.395| 0.398      |0.322      |0.400    |0.392        |0.322\nTraining| 10m     |  2h   |10h    | 2h   | 2h         |3h         |3h       |5h           |7h\n--------------------------------------------------------------------------------------------------\n \n Bert model achieves 0.368 after first 9 epoch from validation set.\n \n Ensemble of TextCNN,EntityNet,DynamicMemory: 0.411\n \n Ensemble EntityNet,DynamicMemory: 0.403\n \n\n \n --------------------------------------------------------------------------------------------------\n \n Notice: \n \n `m` stand for **minutes**; `h` stand for **hours**;\n \n`HierAtteNet` means Hierarchical Attention Networkk;\n\n`Seq2seqAttn` means Seq2seq with attention;\n\n`DynamicMemory` means DynamicMemoryNetwork;\n\n`Transformer` stand for model from 'Attention Is All You Need'.\n\nUsage:\n-------------------------------------------------------------------------------------------------------\n1) model is in `xxx_model.py`\n2) run python `xxx_train.py` to train the model\n3) run python `xxx_predict.py` to do inference(test).\n\nEach model has a test method under the model class. you can run the test method first to check whether the model can work properly.\n\n-------------------------------------------------------------------------\n\nEnvironment:\n-------------------------------------------------------------------------------------------------------\npython 2.7+ tensorflow 1.8 \n\n(tensorflow 1.1 to 1.13 should also works; most of models should also work fine in other tensorflow version, since we \n\nuse very few features bond to certain version.\n\nif you use python3, it will be fine as long as you change print/try catch function in case you meet any error.\n\nTextCNN model is already transfomed to python 3.6\n\n\nSample data: \u003ca href='https://pan.baidu.com/s/1yWZf2eAPxq15-r2hHk2M-Q'\u003ecached file of baidu\u003c/a\u003e or \u003ca href=\"https://drive.google.com/drive/folders/0AKEuT4gza2AlUk9PVA\"\u003eGoogle Drive:\u003c/a\u003esend me an email\n-------------------------------------------------------------------------------------------------------\nto help you run this repository, currently we re-generate training/validation/test data and vocabulary/labels, and saved \n\nthem as cache file using h5py. we suggest you to download it from above link.\n\nit contain everything you need to run this repository: data is pre-processed, you can start to train the model in a minute.\n  \nit's a zip file about 1.8G, contains 3 million training data. although after unzip it's quite big, but with the help of \n\nhdf5, it only need a normal size of memory of computer(e.g.8 G or less) during training.\n\nwe use jupyter notebook: \u003ca href='https://github.com/brightmart/text_classification/blob/master/pre-processing.ipynb'\u003epre-processing.ipynb\u003c/a\u003e to pre-process data. you can have a better understanding of this task and \n\ndata by taking a look of it. you can also generate data by yourself in the way your want, just change few lines of code \n\nusing this jupyter notebook.\n\nIf you want to try a model now, you can dowload cached file from above, then go to folder 'a02_TextCNN', run \n        \n     python  p7_TextCNN_train.py \n   \nit will use data from cached files to train the model, and print loss and F1 score periodically.\n\nold sample data source:\nif you need some sample data and word embedding per-trained on word2vec, you can find it in closed issues, such as: \u003ca href=\"https://github.com/brightmart/text_classification/issues/3\"\u003eissue 3\u003c/a\u003e. \n\nyou can also find some sample data at folder \"data\". it contains two files:'sample_single_label.txt', contains 50k data \n\nwith single label; 'sample_multiple_label.txt', contains 20k data with multiple labels. input and label of is separate by \"   __label__\".\n\nif you want to know more detail about data set of text classification or task these models can be used, one of choose is below:\n\nhttps://biendata.com/competition/zhihu/\n\nRoad Map\n-------------------------------------------------------------------------------------------------------\nOne way you can use this repository:\n \nstep 1: you can read through this article. you will get a general idea of various classic models used to do text classification.\n\nstep 2: pre-process data and/or download cached file.\n\n      a. take a look a look of jupyter notebook('pre-processing.ipynb'), where you can familiar with this text \n\n           classification task and data set. you will also know how we pre-process data and generate training/validation/test \n           \n           set. there are a list of things you can try at the end of this jupyter.\n\n       b. download zip file that contains cached files, so you will have all necessary data, and can start to train models.\n\nstep 3: run some of models list here, and change some codes and configurations as you want, to get a good performance.\n\n      record performances, and things you done that works, and things that are not.\n\n      for example, you can take this sequence to explore: \n      \n      1) fasttext---\u003e 2)TextCNN---\u003e 3)Transformer---\u003e 4)BERT\n\nadditionally, write your article about this topic, you can follow paper's style to write. you may need to read some papers\n       \n       on the way, many of these papers list in the # Reference at the end of this article; or join  a machine learning \n       \n       competition, and apply it with what you've learned. \n       \nUse Your Own Data:\n-------------------------------------------------------------------------------------------------------\nreplace data in 'data/sample_multiple_label.txt', and make sure format as below:\n\n'word1 word2 word3 __label__l1 __label__l2 __label__l3'\n \nwhere part1: 'word1 word2 word3' is input(X), part2: '__label__l1 __label__l2 __label__l3' \n\nrepresenting there are three labels: [l1,l2,l3]. between part1 and part2 there should be a empty string: ' '.\n\nfor example: each line (multiple labels) like: \n\n'w5466 w138990 w1638 w4301 w6 w470 w202 c1834 c1400 c134 c57 c73 c699 c317 c184 __label__5626661657638885119 __label__4921793805334628695 __label__8904735555009151318'\n\nwhere '5626661657638885119','4921793805334628695'，‘8904735555009151318’ are three labels associate with this input string 'w5466 w138990...c699 c317 c184'\n\nNotice:\n\n\nSome util function is in data_util.py;  check load_data_multilabel() of data_util for how process input and labels from raw data.\n\nthere is a function to load and assign pretrained word embedding to the model,where word embedding is pretrained in word2vec or fastText. \n\nPretrain Work Embedding:\n-------------------------------------------------------------------------------------------------------\nif word2vec.load not works, you may load pretrained word embedding, especially for chinese word embedding use following lines:\n\nimport gensim\n\nfrom gensim.models import KeyedVectors\n\nword2vec_model = KeyedVectors.load_word2vec_format(word2vec_model_path, binary=True, unicode_errors='ignore')  #\n\nor you can turn off use pretrain word embedding flag to false to disable loading word embedding.\n\n\nModels Detail:\n-------------------------------------------------------------------------\n\n1.fastText:  \n-------------\nimplmentation of \u003ca href=\"https://arxiv.org/abs/1607.01759\"\u003eBag of Tricks for Efficient Text Classification\u003c/a\u003e\n\nafter embed each word in the sentence, this word representations are then averaged into a text representation, which is in turn fed to a linear classifier.it use softmax function to compute the probability distribution over the predefined classes. then cross entropy is used to compute loss. bag of word representation does not consider word order. in order to take account of word order, n-gram features is used to capture some partial information about the local word order; when the number of classes is large, computing the linear classifier is computational expensive. so it usehierarchical softmax to speed training process.\n1) use bi-gram and/or tri-gram\n2) use NCE loss to speed us softmax computation(not use hierarchy softmax as original paper)\n\nresult: performance is as good as paper, speed also very fast.\n\ncheck: p5_fastTextB_model.py\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/fastText.JPG)\n-------------------------------------------------------------------------\n\n2.TextCNN:\n-------------\nImplementation of \u003ca href=\"http://www.aclweb.org/anthology/D14-1181\"\u003e Convolutional Neural Networks for Sentence Classification \u003c/a\u003e\n\nStructure:embedding---\u003econv---\u003emax pooling---\u003efully connected layer--------\u003esoftmax\n\nCheck: p7_TextCNN_model.py\n\nIn order to get very good result with TextCNN, you also need to read carefully about this paper \u003ca href=\"https://arxiv.org/abs/1510.03820\"\u003eA Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification\u003c/a\u003e: it give you some insights of things that can affect performance. although you need to  change some settings according to your specific task.\n\nConvolutional Neural Network is main building box for solve problems of computer vision. Now we will show how CNN can be used for NLP, in in particular, text classification. Sentence length will be different from one to another. So we will use pad to get fixed length, n. For each token in the sentence, we will use word embedding to get a fixed dimension vector, d. So our input is a 2-dimension matrix:(n,d). This is similar with image for CNN. \n\nFirstly, we will do convolutional operation to our input. It is a element-wise multiply between filter and part of input. We use k number of filters, each filter size is a 2-dimension matrix (f,d). Now the output will be k number of lists. Each list has a length of n-f+1. each element is a scalar. Notice that the second dimension will be always the dimension of word embedding. We are using different size of filters to get rich features from text inputs. And this is something similar with n-gram features. \n\nSecondly, we will do max pooling for the output of convolutional operation. For k number of lists, we will get k number of scalars. \n\nThirdly, we will concatenate scalars to form final features. It is a fixed-size vector. And it is independent from the size of filters we use.\n\nFinally, we will use linear layer to project these features to per-defined labels.\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/TextCNN.JPG)\n\n-------------------------------------------------------------------------\n\n\n3.BERT: \n-------------------------------------------------------------------------\n#### Pre-training of Deep Bidirectional Transformers for Language Understanding \n\nBERT currently achieve state of art results on more than 10 NLP tasks. the key ideas behind this model is that we can \n\npre-train the model by using one kind of language model with huge amount of raw data, where you can find it easily.\n\nas most of parameters of the model is pre-trained, only last layer for classifier need to be need for different tasks.\n\nas a result, this model is generic and very powerful. you can just fine-tuning based on the pre-trained model within\n \na short period of time.\n \nhowever, this model is quite big. with sequence length 128, you may only able to train with a batch size of 32; for long\n\ndocument such as sequence length 512, it can only train a batch size 4 for a normal GPU(with 11G); and very few people\n\ncan pre-train this model from scratch, as it takes many days or weeks to train, and a normal GPU's memory is too small \n\nfor this model.\n\nSpecially, the backbone model is Transformer, where you can find it in Attention Is All You Need. it use two kind of \n\ntasks to pre-train the model.\n\n#### Masked Languge Model\ngenerally speaking, given a sentence, some percentage of words are masked, you will need to predict the masked words\n\nbased on this masked sentence. masked words are chosed randomly.\n\nwe feed the input through a deep Transformer encoder and then use the final hidden states corresponding to the masked \n\npositions to predict what word was masked, exactly like we would train a language model.\n\n    source_file each line is a sequence of token, can be a sentence.\n    \n    Input Sequence  : The man went to [MASK] store with [MASK] dog\n    Target Sequence :                  the                his\n         \n\n#### Next Sentence Prediction\nmany language understanding task, like question answering, inference, need understand relationship\n  \nbetween sentence. however, language model is only able to understand without a sentence. next sentence\n\nprediction is a sample task to help model understand better in these kinds of task.\n\n50% of chance the second sentence is tbe next sentence of the first one, 50% of not the next one.\n\ngiven two sentence, the model is asked to predict whether the second sentence is real next sentence of \n\nthe first one.\n  \n    Input : [CLS] the man went to the store [SEP] he bought a gallon of milk [SEP]\n    Label : IsNext\n\n    Input = [CLS] the man heading to the store [SEP] penguin [MASK] are flight ##less birds [SEP]\n    Label = NotNext\n    \n\u003cimg src=\"https://github.com/brightmart/text_classification/blob/master/images/bert_1.jpeg\"  width=\"65%\" height=\"65%\" /\u003e\n\n\u003cimg src=\"https://github.com/brightmart/text_classification/blob/master/images/bert_2.jpeg\"  width=\"65%\" height=\"65%\" /\u003e\n\n\n#### How to use BERT?\n\nbasically, you can download pre-trained model, can just fine-tuning on your task with your own data.\n\nfor classification task, you can add processor to define the format you want to let input and labels from source data.\n\n#### Use BERT for multi-label classification?\n\nrun the following command under folder a00_Bert:\n \n      python  train_bert_multi-label.py\n   \nIt achieve 0.368 after 9 epoch.\nor you can run multi-label classification with downloadable data using BERT from \n\n\u003ca href='https://github.com/brightmart/sentiment_analysis_fine_grain'\u003esentiment_analysis_fine_grain with BERT\u003c/a\u003e\n \n#### Use BERT for online prediction \n\nyou can use session and feed style to restore model and feed data, then get logits to make a online prediction.\n\n\u003ca href='https://github.com/brightmart/sentiment_analysis_fine_grain'\u003eonline prediction with BERT\u003c/a\u003e\n\noriginally, it train or evaluate model based on file, not for online.\n\n#### How to get better model for BERT?\n\nfirstly, you can use pre-trained model download from google. run a few epoch on you dataset, and find a suitable \n\nsequence length.\n\nsecondly, you can pre-train the base model in your own data as long as  you can find a dataset that is related to \n\nyour task, then fine-tuning on your specific task.\n\nthirdly, you can change loss function and last layer to better suit for your task.\n\nadditionally, you can add define some pre-trained tasks that will help the model understand your task much better.\n\nas experienced we got from experiments, pre-trained task is independent from model and pre-train is not limit to \n\nthe tasks above.\n\n-------------------------------------------------------------------------\n\n\n4.TextRNN\n-------------\nStructure v1:embedding---\u003ebi-directional lstm---\u003econcat output---\u003eaverage-----\u003esoftmax layer\n\ncheck: p8_TextRNN_model.py\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/bi-directionalRNN.JPG)\n\nStructure v2:embedding--\u003ebi-directional lstm----\u003edropout--\u003econcat ouput---\u003elstm---\u003edroput--\u003eFC layer--\u003esoftmax layer\n\ncheck: p8_TextRNN_model_multilayer.py\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/emojifier-v2.png)\n\n\n-------------------------------------------------------------------------\n\n\n5.BiLstmTextRelation\n-------------\nStructure same as TextRNN. but input is special designed. e.g.input:\"how much is the computer? EOS price of laptop\". where 'EOS' is a special\ntoken spilted question1 and question2.\n\ncheck:p9_BiLstmTextRelation_model.py\n\n\n-------------------------------------------------------------------------\n\n\n6.twoCNNTextRelation\n-------------\nStructure: first use two different convolutional to extract feature of two sentences. then concat two features. use linear\ntransform layer to out projection to target label, then softmax.\n\ncheck: p9_twoCNNTextRelation_model.py\n\n\n-------------------------------------------------------------------------\n\n\n7.BiLstmTextRelationTwoRNN\n-------------\nStructure: one bi-directional lstm for one sentence(get output1), another bi-directional lstm for another sentence(get output2). then:\nsoftmax(output1*M*output2)\n\ncheck:p9_BiLstmTextRelationTwoRNN_model.py\n\nfor more detail you can go to: \u003ca herf=\"http://www.wildml.com/2016/07/deep-learning-for-chatbots-2-retrieval-based-model-tensorflow\"\u003eDeep Learning for Chatbots, Part 2 – Implementing a Retrieval-Based Model in Tensorflow\u003ca\u003e\n\n\n-------------------------------------------------------------------------\n\n\n8.RCNN:\n-------------\nRecurrent convolutional neural network for text classification\n\nimplementation of \u003ca href=\"https://scholar.google.com.hk/scholar?q=Recurrent+Convolutional+Neural+Networks+for+Text+Classification\u0026hl=zh-CN\u0026as_sdt=0\u0026as_vis=1\u0026oi=scholart\u0026sa=X\u0026ved=0ahUKEwjpx82cvqTUAhWHspQKHUbDBDYQgQMIITAA\"\u003e Recurrent Convolutional Neural Network for Text Classification \u003c/a\u003e\n \nstructure:1)recurrent structure (convolutional layer) 2)max pooling 3) fully connected layer+softmax\n\nit learn represenation of each word in the sentence or document with left side context and right side context:\n\nrepresentation current word=[left_side_context_vector,current_word_embedding,right_side_context_vecotor].\n\nfor left side context, it use a recurrent structure, a no-linearity transfrom of previous word and left side previous context; similarly to right side context.\n\ncheck: p71_TextRCNN_model.py\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/RCNN.JPG)\n\n-------------------------------------------------------------------------\n\n9.Hierarchical Attention Network:\n-------------\nImplementation of \u003ca href=\"https://www.cs.cmu.edu/~diyiy/docs/naacl16.pdf\"\u003eHierarchical Attention Networks for Document Classification\u003c/a\u003e\n\nStructure:\n\n1) embedding \n\n2) Word Encoder: word level bi-directional GRU to get rich representation of words\n\n3) Word Attention:word level attention to get important information in a sentence\n\n4) Sentence Encoder: sentence level bi-directional GRU to get rich representation of sentences\n\n5) Sentence Attetion: sentence level attention to get important sentence among sentences\n\n5) FC+Softmax\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/HAN.JPG)\n\nIn NLP, text classification can be done for single sentence, but it can also be used for multiple sentences. we may call it document classification. Words are form to sentence. And sentence are form to document. In this circumstance, there may exists a intrinsic structure. So how can we model this kinds of task? Does all parts of document are equally relevant? And how we determine which part are more important than another?\n\nIt has two unique features: \n\n1)it has a hierarchical structure that reflect the hierarchical structure of documents; \n\n2)it has two levels of attention mechanisms used at the word and sentence-level. it enable the model to capture important information in different levels.\n\nWord Encoder:\nFor each words in a sentence, it is embedded into word vector in distribution vector space. It use a bidirectional GRU to encode the sentence. By concatenate vector from two direction, it now can form a representation of the sentence, which also capture contextual information.\n\nWord Attention:\nSame words are more important than another for the sentence. So attention mechanism is used. It first use one layer MLP to get uit hidden representation of the sentence, then measure the importance of the word as the similarity of uit with a word level context vector uw and get a normalized importance through a softmax function. \n\nSentence Encoder: \nfor sentence vectors, bidirectional GRU is used to encode it. Similarly to word encoder.\n\nSentence Attention: \nsentence level vector is used to measure importance among sentences. Similarly to word attention.\n\nInput of data: \n\nGenerally speaking, input of this model should have serveral sentences instead of sinle sentence. shape is:[None,sentence_lenght]. where None means the batch_size.\n\nIn my training data, for each example, i have four parts. each part has same length. i concat four parts to form one single sentence. the model will split the sentence into four parts, to form a tensor with shape:[None,num_sentence,sentence_length]. where num_sentence is number of sentences(equal to 4, in my setting).\n\ncheck:p1_HierarchicalAttention_model.py\n\nfor attentive attention you can check \u003ca href='https://github.com/brightmart/text_classification/issues/55'\u003eattentive attention\u003c/a\u003e\n\n-------------------------------------------------------------------------\n\n10.Seq2seq with attention\n-------------\nImplementation seq2seq with attention derived from \u003ca href=\"https://arxiv.org/pdf/1409.0473.pdf\"\u003eNEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE\u003c/a\u003e\n\nI.Structure:\n\n1)embedding 2)bi-GRU too get rich representation from source sentences(forward \u0026 backward). 3)decoder with attention.\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/seq2seqAttention.JPG)\n\nII.Input of data:\n\nthere are two kinds of three kinds of inputs:1)encoder inputs, which is a sentence; 2)decoder inputs, it is labels list with fixed length;3)target labels, it is also a list of labels.\n\nfor example, labels is:\"L1 L2 L3 L4\", then decoder inputs will be:[_GO,L1,L2,L2,L3,_PAD]; target label will be:[L1,L2,L3,L3,_END,_PAD]. length is fixed to 6, any exceed labels will be trancated, will pad if label is not enough to fill.\n\nIII.Attention Mechanism:\n\n1) transfer encoder input list and hidden state of decoder\n\n2) calculate similarity of hidden state with each encoder input, to get possibility distribution for each encoder input.\n\n3) weighted sum of encoder input based on possibility distribution.\n\n   go though RNN Cell using this weight sum together with decoder input to get new hidden state\n\nIV.How Vanilla Encoder Decoder Works:\n\nthe source sentence will be encoded using RNN as fixed size vector (\"thought vector\"). then during decoder:\n\n1) when it is training, another RNN will be used to try to get a word by using this \"thought vector\"  as init state, and take input from decoder input at each timestamp. decoder start from special token \"_GO\". \nafter one step is performanced, new hidden state will be get and together with new input, we can continue this process until we reach to a special token \"_END\". \nwe can calculate loss by compute cross entropy loss of logits and target label. logits is get through a projection layer for the hidden state(for output of decoder step(in GRU we can just use hidden states from decoder as output).\n\n2) when it is testing, there is no label. so we should feed the output we get from previous timestamp, and continue the process util we reached \"_END\" TOKEN.\n\nV.Notices:\n\n1) here i use two kinds of vocabularies. one is from words,used by encoder; another is for labels,used by decoder\n\n2) for vocabulary of lables, i insert three special token:\"_GO\",\"_END\",\"_PAD\"; \"_UNK\" is not used, since all labels is pre-defined.\n\n-------------------------------------------------------------------------\n\n11.Transformer(\"Attention Is All You Need\")\n-------------\nStatus: it was able to do task classification. and able to generate reverse order of its sequences in toy task. you can check it by running test function in the model. check: a2_train_classification.py(train) or a2_transformer_classification.py(model)\n\nwe do it in parallell style.layer normalization,residual connection, and mask are also used in the model. \n\nFor every building blocks, we include a test function in the each file below, and we've test each small piece successfully.\n\nSequence to sequence with attention is a typical model to solve sequence generation problem, such as translate, dialogue system. most of time, it use RNN as buidling block to do these tasks. util recently, people also apply convolutional Neural Network for sequence to sequence problem. Transformer, however, it perform these tasks solely on attention mechansim. it is fast and achieve new state-of-art result.\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/attention_is_all_you_need.JPG)\n\nIt also has two main parts: encoder and decoder. below is desc from paper:\n\nEncoder:\n\n6 layers.each layers has two sub-layers.\nthe first is multi-head self-attention mechanism;\nthe second is position-wise fully connected feed-forward network.\nfor each sublayer. use LayerNorm(x+Sublayer(x)). all dimension=512.\n\nDecoder:\n\n1. The decoder is composed of a stack of N= 6 identical layers.\n2. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\nattention over the output of the encoder stack.\n3. Similar to the encoder, we employ residual connections\naround each of the sub-layers, followed by layer normalization. We also modify the self-attention\nsub-layer in the decoder stack to prevent positions from attending to subsequent positions.  This\nmasking, combined with fact that the output embeddings are offset by one position, ensures that the\npredictions for position i can depend only on the known outputs at positions less than i.\n\nMain Take away from this model:\n\n1) multi-head self attention: use self attention, linear transform multi-times to get projection of key-values, then do ordinary attention; 2) some tricks to improve performance(residual connection,position encoding, poistion feed forward, label smooth, mask to ignore things we want to ignore).\n\nUse this model to do task classification:\n\nHere we only use encode part for task classification, removed resdiual connection, used only 1 layer.no need to use mask. we use multi-head attention and postionwise feed forward to extract features of input sentence, then use linear layer to project it to get logits.\n\nfor detail of the model, please check: a2_transformer_classification.py\n\n-------------------------------------------------------------------------\n\n12.Recurrent Entity Network\n-------------------------------------------------------------------------\nInput:1. story: it is multi-sentences, as context. 2.query: a sentence, which is a question, 3. ansewr: a single label.\n\nModel Structure:\n\n1) Input encoding: use bag of word to encode story(context) and query(question); take account of position by using position mask\n\n   by using bi-directional rnn to encode story and query, performance boost from 0.392 to 0.398, increase 1.5%.\n\n2) Dynamic memory: \n\na. compute gate by using 'similarity' of keys,values with input of story. \n\nb. get candidate hidden state by transform each key,value and input.\n\nc. combine gate and candidate hidden state to update current hidden state.\n\n3) Output moudle( use attention mechanism):\na. to get possibility distribution by computing 'similarity' of query and hidden state\n\nb. get weighted sum of hidden state using possibility distribution.\n\nc. non-linearity transform of query and hidden state to get predict label.\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/EntityNet.JPG)\n\nMain take away from this model:\n\n1) use blocks of keys and values, which is independent from each other. so it can be run in parallel.\n\n2) modelling context and question together. use memory to track state of world; and use non-linearity transform of hidden state and question(query) to make a prediction.\n\n3) simple model can also achieve very good performance. simple encode as use bag of word.\n\nfor detail of the model, please check: a3_entity_network.py\n\nunder this model, it has a test function, which ask this model to count numbers both for story(context) and query(question). but weights of story is smaller than query.\n\n-------------------------------------------------------------------------\n\n13.Dynamic Memory Network\n-------------------------------------------------------------------------\nOutlook of Model:\n\n1.Input Module: encode raw texts into vector representation\n\n2.Question Module: encode question into vector representation\n\n3.Episodic Memory Module: with inputs,it chooses which parts of inputs to focus on through the attention mechanism, taking into account of question and previous memory====\u003eit poduce a 'memory' vecotr.\n\n4.Answer Module:generate an answer from the final memory vector.\n\n![alt text](https://github.com/brightmart/text_classification/blob/master/images/DMN.JPG)\n\nDetail:\n\n1.Input Module:\n\n  a.single sentence: use gru to get hidden state\n  b.list of sentences: use gru to get the hidden states for each sentence. e.g. [hidden states 1,hidden states 2, hidden states...,hidden state n]\n  \n2.Question Module:\n  use gru to get hidden state\n  \n3.Episodic Memory Module:\n\n  use an attention mechanism and recurrent network to updates its memory. \n     \n  a. gate as attention mechanism:\n  \n     two-layer feed forward nueral network.input is candidate fact c,previous memory m and question q. features get by take: element-wise,matmul and absolute distance of q with c, and q with m.\n     \n  b.memory update mechanism: take candidate sentence, gate and previous hidden state, it use gated-gru to update hidden state. like: h=f(c,h_previous,g). the final hidden state is the input for answer module.\n  \n  c.need for multiple episodes===\u003etransitive inference. \n  \n  e.g. ask where is the football? it will attend to sentence of \"john put down the football\"), then in second pass, it need to attend location of john.\n\n4.Answer Module:\ntake the final epsoidic memory, question, it update hidden state of answer module.\n\n\nTODO \n-------------------------------------------------------------------------------------------------------\n1.Character-level Convolutional Networks for Text Classification\n\n2.Convolutional Neural Networks for Text Categorization:Shallow Word-level vs. Deep Character-level\n\n3.Very Deep Convolutional Networks for Text Classification\n\n4.Adversarial Training Methods For Semi-supervised Text Classification\n\n5.Ensemble Models\n\n\nConclusion:\n-------------------------------------------------------------------------\nDuring the process of doing large scale of multi-label classification, serveral lessons has been learned, and some list as below:\n\n1) What is most important thing to reach a high accuracy? \nIt depend the task you are doing. From the task we conducted here, we believe that ensemble models based on models trained from multiple features including word, character for title and description can help to reach very high accuarcy; However, in some cases,as just alphaGo Zero demonstrated, algorithm is more important then data or computational power, in fact alphaGo Zero did not use any humam data. \n\n2) Is there a ceiling for any specific model or algorithm?\nThe answer is yes. lots of different models were used here, we found many models have similar performances, even though there are quite different in structure. In some extent, the difference of performance is not so big.\n\n3) Is case study of error useful?\nI think it is quite useful especially when you have done many different things, but reached a limit. For example, by doing case study, you can find labels that models can make correct prediction, and where they make mistakes. And to imporove performance by  increasing weights of these wrong predicted labels or finding potential errors from data.\n\n4) How can we become expert in a specific of Machine Learning?\nIn my opinion,join a machine learning competation or begin a task with lots of data, then read papers and implement some, is a good starting point. So we will have some really experience and ideas of handling specific task, and know the challenges of it.\nBut what's more important is that we should not only follow ideas from papers, but to explore some new ideas we think may help to slove the problem. For example, by changing structures of classic models or even invent some new structures, we may able to tackle the problem in a much better way as it may more suitable for task we are doing.\n\nReference:\n-------------------------------------------------------------------------------------------------------\n1.Bag of Tricks for Efficient Text Classification\n\n2.Convolutional Neural Networks for Sentence Classification\n\n3.A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification\n\n4.Deep Learning for Chatbots, Part 2 – Implementing a Retrieval-Based Model in Tensorflow, from www.wildml.com\n\n5.Recurrent Convolutional Neural Network for Text Classification\n\n6.Hierarchical Attention Networks for Document Classification\n\n7.Neural Machine Translation by Jointly Learning to Align and Translate\n\n8.Attention Is All You Need\n\n9.Ask Me Anything:Dynamic Memory Networks for Natural Language Processing\n\n10.Tracking the state of world with recurrent entity networks\n\n11.Ensemble Selection from Libraries of Models\n\n12.\u003ca href='https://arxiv.org/abs/1810.04805'\u003eBERT:Pre-training of Deep Bidirectional Transformers for Language Understanding\u003c/a\u003e\n\n13.\u003ca href='https://github.com/google-research/bert'\u003egoogle-research/bert\u003c/a\u003e\n\n-------------------------------------------------------------------------\n\nto be continued. for any problem, concat brightmart@hotmail.com\n","funding_links":[],"categories":["Python","Table of Contents","Chinese NLP Toolkits 中文NLP工具","Repository \u0026 Tool"],"sub_categories":["Information Extraction 信息提取","Weakly-supervised \u0026 Semi-supervised Learning"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrightmart%2Ftext_classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrightmart%2Ftext_classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrightmart%2Ftext_classification/lists"}