{"id":15034803,"url":"https://github.com/kk7nc/text_classification","last_synced_at":"2025-05-14T06:14:05.467Z","repository":{"id":40897148,"uuid":"139912879","full_name":"kk7nc/Text_Classification","owner":"kk7nc","description":"Text Classification Algorithms: A Survey","archived":false,"fork":false,"pushed_at":"2025-04-01T00:35:13.000Z","size":14464,"stargazers_count":1810,"open_issues_count":2,"forks_count":543,"subscribers_count":72,"default_branch":"master","last_synced_at":"2025-04-11T01:41:48.053Z","etag":null,"topics":["boosting-algorithms","conditional-random-fields","convolutional-neural-networks","decision-trees","deep-belief-network","deep-learning","deep-neural-network","dimensionality-reduction","document-classification","hierarchical-attention-networks","k-nearest-neighbours","logistic-regression","naive-bayes-classifier","nlp-machine-learning","random-forest","recurrent-neural-networks","rocchio-algorithm","support-vector-machines","text-classification","text-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kk7nc.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-07-06T00:10:18.000Z","updated_at":"2025-04-03T14:52:40.000Z","dependencies_parsed_at":"2024-12-19T01:02:25.368Z","dependency_job_id":"e9021723-ddad-4e5e-a4ec-843ed88e10cd","html_url":"https://github.com/kk7nc/Text_Classification","commit_stats":{"total_commits":223,"total_committers":9,"mean_commits":24.77777777777778,"dds":"0.053811659192825156","last_synced_commit":"7092ca64619b305e0f184ceb6fd0341f5c16f57d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kk7nc%2FText_Classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kk7nc%2FText_Classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kk7nc%2FText_Classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kk7nc%2FText_Classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kk7nc","download_url":"https://codeload.github.com/kk7nc/Text_Classification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254083809,"owners_count":22011902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boosting-algorithms","conditional-random-fields","convolutional-neural-networks","decision-trees","deep-belief-network","deep-learning","deep-neural-network","dimensionality-reduction","document-classification","hierarchical-attention-networks","k-nearest-neighbours","logistic-regression","naive-bayes-classifier","nlp-machine-learning","random-forest","recurrent-neural-networks","rocchio-algorithm","support-vector-machines","text-classification","text-processing"],"created_at":"2024-09-24T20:26:23.618Z","updated_at":"2025-05-14T06:14:05.445Z","avatar_url":"https://github.com/kk7nc.png","language":"Python","readme":"\n################################################\nText Classification Algorithms: A Survey\n################################################\n\n|UniversityCube| |DOI| |Best| |medium| |mendeley| |contributions-welcome| |arXiv| |ansicolortags| |contributors| |twitter|\n \n \n.. figure:: docs/pic/WordArt.png \n \n \n Referenced paper : `Text Classification Algorithms: A Survey \u003chttps://arxiv.org/abs/1904.08067\u003e`__ \n \n|BPW| \n\n\n\n##################\nTable of Contents\n##################\n.. contents::\n :local:\n :depth: 4\n\n============\nIntroduction\n============\n\n.. figure:: docs/pic/OverviewTextClassification.png \n \n \n \n====================================\nText and Document Feature Extraction\n====================================\n\n----\n\n\nText feature extraction and pre-processing for classification algorithms are very significant. In this section, we start to talk about text cleaning since most of documents contain a lot of noise. In this part, we discuss two primary methods of text feature extractions- word embedding and weighted word.\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nText Cleaning and Pre-processing\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn Natural Language Processing (NLP), most of the text and documents contain many words that are redundant for text classification, such as stopwords, miss-spellings, slangs, and etc. In this section, we briefly explain some techniques and methods for text cleaning and pre-processing text documents. In many algorithms like statistical and probabilistic learning methods, noise and unnecessary features can negatively affect the overall perfomance. So, elimination of these features are extremely important.\n\n\n-------------\nTokenization\n-------------\n\nTokenization is the process of breaking down a stream of text into words, phrases, symbols, or any other meaningful elements called tokens. The main goal of this step is to extract individual words in a sentence. Along with text classifcation, in text mining, it is necessay to incorporate a parser in the pipeline which performs the tokenization of the documents; for example:\n\nsentence:\n\n.. code::\n\n After sleeping for four hours, he decided to sleep for another four\n\n\nIn this case, the tokens are as follows:\n\n.. code::\n\n {'After', 'sleeping', 'for', 'four', 'hours', 'he', 'decided', 'to', 'sleep', 'for', 'another', 'four'}\n\n\nHere is python code for Tokenization:\n\n.. code:: python\n\n from nltk.tokenize import word_tokenize\n text = \"After sleeping for four hours, he decided to sleep for another four\"\n tokens = word_tokenize(text)\n print(tokens)\n\n-----------\nStop words\n-----------\n\n\nText and document classification over social media, such as Twitter, Facebook, and so on is usually affected by the noisy nature (abbreviations, irregular forms) of the text corpuses.\n\nHere is an exmple from `geeksforgeeks \u003chttps://www.geeksforgeeks.org/removing-stop-words-nltk-python/\u003e`__\n\n.. code:: python\n\n from nltk.corpus import stopwords\n from nltk.tokenize import word_tokenize\n\n example_sent = \"This is a sample sentence, showing off the stop words filtration.\"\n\n stop_words = set(stopwords.words('english'))\n\n word_tokens = word_tokenize(example_sent)\n\n filtered_sentence = [w for w in word_tokens if not w in stop_words]\n\n filtered_sentence = []\n\n for w in word_tokens:\n if w not in stop_words:\n filtered_sentence.append(w)\n\n print(word_tokens)\n print(filtered_sentence)\n\n\n\nOutput:\n\n.. code:: python \n\n ['This', 'is', 'a', 'sample', 'sentence', ',', 'showing', \n 'off', 'the', 'stop', 'words', 'filtration', '.']\n ['This', 'sample', 'sentence', ',', 'showing', 'stop',\n 'words', 'filtration', '.']\n\n\n---------------\nCapitalization\n---------------\n\nSentences can contain a mixture of uppercase and lower case letters. Multiple sentences make up a text document. To reduce the problem space, the most common approach is to reduce everything to lower case. This brings all words in a document in same space, but it often changes the meaning of some words, such as \"US\" to \"us\" where first one represents the United States of America and second one is a pronoun. To solve this, slang and abbreviation converters can be applied.\n\n.. code:: python\n\n text = \"The United States of America (USA) or America, is a federal republic composed of 50 states\"\n print(text)\n print(text.lower())\n\nOutput:\n\n.. code:: python\n\n \"The United States of America (USA) or America, is a federal republic composed of 50 states\"\n \"the united states of america (usa) or america, is a federal republic composed of 50 states\"\n\n-----------------------\nSlangs and Abbreviations\n-----------------------\n\nSlangs and abbreviations can cause problems while executing the pre-processing steps. An abbreviation is a shortened form of a word, such as SVM stand for Support Vector Machine. Slang is a version of language that depicts informal conversation or text that has different meaning, such as \"lost the plot\", it essentially means that 'they've gone mad'. Common method to deal with these words is converting them to formal language.\n\n---------------\nNoise Removal\n---------------\n\n\nAnother issue of text cleaning as a pre-processing step is noise removal. Text documents generally contains characters like punctuations or special characters and they are not necessary for text mining or classification purposes. Although punctuation is critical to understand the meaning of the sentence, but it can affect the classification algorithms negatively.\n\n\nHere is simple code to remove standard noise from text:\n\n\n.. code:: python\n\n def text_cleaner(text):\n rules = [\n {r'\u003e\\s+': u'\u003e'}, # remove spaces after a tag opens or closes\n {r'\\s+': u' '}, # replace consecutive spaces\n {r'\\s*\u003cbr\\s*/?\u003e\\s*': u'\\n'}, # newline after a \u003cbr\u003e\n {r'\u003c/(div)\\s*\u003e\\s*': u'\\n'}, # newline after \u003c/p\u003e and \u003c/div\u003e and \u003ch1/\u003e...\n {r'\u003c/(p|h\\d)\\s*\u003e\\s*': u'\\n\\n'}, # newline after \u003c/p\u003e and \u003c/div\u003e and \u003ch1/\u003e...\n {r'\u003chead\u003e.*\u003c\\s*(/head|body)[^\u003e]*\u003e': u''}, # remove \u003chead\u003e to \u003c/head\u003e\n {r'\u003ca\\s+href=\"([^\"]+)\"[^\u003e]*\u003e.*\u003c/a\u003e': r'\\1'}, # show links instead of texts\n {r'[ \\t]*\u003c[^\u003c]*?/?\u003e': u''}, # remove remaining tags\n {r'^\\s+': u''} # remove spaces at the beginning\n ]\n for rule in rules:\n for (k, v) in rule.items():\n regex = re.compile(k)\n text = regex.sub(v, text)\n text = text.rstrip()\n return text.lower()\n \n\n\n-------------------\nSpelling Correction\n-------------------\n\n\nAn optional part of the pre-processing step is correcting the misspelled words. Different techniques, such as hashing-based and context-sensitive spelling correction techniques, or spelling correction using trie and damerau-levenshtein distance bigram have been introduced to tackle this issue.\n\n\n.. code:: python\n\n from autocorrect import spell\n\n print spell('caaaar')\n print spell(u'mussage')\n print spell(u'survice')\n print spell(u'hte')\n\nResult:\n\n.. code::\n\n caesar\n message\n service\n the\n\n\n------------\nStemming\n------------\n\n\nText Stemming is modifying a word to obtain its variants using different linguistic processeses like affixation (addition of affixes). For example, the stem of the word \"studying\" is \"study\", to which -ing.\n\n\nHere is an example of Stemming from `NLTK \u003chttps://pythonprogramming.net/stemming-nltk-tutorial/\u003e`__\n\n.. code:: python\n\n from nltk.stem import PorterStemmer\n from nltk.tokenize import sent_tokenize, word_tokenize\n\n ps = PorterStemmer()\n\n example_words = [\"python\",\"pythoner\",\"pythoning\",\"pythoned\",\"pythonly\"]\n \n for w in example_words:\n print(ps.stem(w))\n\n\nResult:\n\n.. code::\n\n python\n python\n python\n python\n pythonli\n\n-------------\nLemmatization\n-------------\n\n\nText lemmatization is the process of eliminating redundant prefix or suffix of a word and extract the base word (lemma).\n\n\n.. code:: python\n\n from nltk.stem import WordNetLemmatizer\n\n lemmatizer = WordNetLemmatizer()\n\n print(lemmatizer.lemmatize(\"cats\"))\n\n~~~~~~~~~~~~~~\nWord Embedding\n~~~~~~~~~~~~~~\n\nDifferent word embedding procedures have been proposed to translate these unigrams into consummable input for machine learning algorithms. A very simple way to perform such embedding is term-frequency~(TF) where each word will be mapped to a number corresponding to the number of occurrence of that word in the whole corpora. The other term frequency functions have been also used that represent word-frequency as Boolean or logarithmically scaled number. Here, each document will be converted to a vector of same length containing the frequency of the words in that document. Although such approach may seem very intuitive but it suffers from the fact that particular words that are used very commonly in language literature might dominate this sort of word representations.\n\n.. image:: docs/pic/CBOW.png\n\n\n--------\nWord2Vec\n--------\n\nOriginal from https://code.google.com/p/word2vec/\n\nI’ve copied it to a github project so that I can apply and track community\npatches (starting with capability for Mac OS X\ncompilation).\n\n- **makefile and some source has been modified for Mac OS X\n compilation** See\n https://code.google.com/p/word2vec/issues/detail?id=1#c5\n- **memory patch for word2vec has been applied** See\n https://code.google.com/p/word2vec/issues/detail?id=2\n- Project file layout altered\n\nThere seems to be a segfault in the compute-accuracy utility.\n\nTo get started:\n\n::\n\n cd scripts \u0026\u0026 ./demo-word.sh\n\nOriginal README text follows:\n\nThis tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research purposes. \n\n\nthis code provides an implementation of the Continuous Bag-of-Words (CBOW) and\nthe Skip-gram model (SG), as well as several demo scripts.\n\nGiven a text corpus, the word2vec tool learns a vector for every word in\nthe vocabulary using the Continuous Bag-of-Words or the Skip-Gram neural\nnetwork architectures. The user should specify the following: -\ndesired vector dimensionality (size of the context window for\neither the Skip-Gram or the Continuous Bag-of-Words model), training\nalgorithm (hierarchical softmax and / or negative sampling), threshold\nfor downsampling the frequent words, number of threads to use,\nformat of the output word vector file (text or binary).\n\nUsually, other hyper-parameters, such as the learning rate do not\nneed to be tuned for different training sets.\n\nThe script demo-word.sh downloads a small (100MB) text corpus from the\nweb, and trains a small word vector model. After the training is\nfinished, users can interactively explore the similarity of the\nwords.\n\nMore information about the scripts is provided at\nhttps://code.google.com/p/word2vec/\n\n\n----------------------------------------------\nGlobal Vectors for Word Representation (GloVe)\n----------------------------------------------\n\n.. image:: /docs/pic/Glove.PNG\n\nAn implementation of the GloVe model for learning word representations is provided, and describe how to download web-dataset vectors or train your own. See the `project page \u003chttp://nlp.stanford.edu/projects/glove/\u003e`__ or the `paper \u003chttp://nlp.stanford.edu/pubs/glove.pdf\u003e`__ for more information on glove vectors.\n\n\n------------------------------------\nContextualized Word Representations\n------------------------------------\n\nELMo is a deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). These word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. They can be easily added to existing models and significantly improve the state of the art across a broad range of challenging NLP problems, including question answering, textual entailment and sentiment analysis.\n\n\n**ELMo representations are:**\n\n- **Contextual:** The representation for each word depends on the entire context in which it is used.\n- **Deep:** The word representations combine all layers of a deep pre-trained neural network.\n- **Character based:** ELMo representations are purely character based, allowing the network to use morphological clues to form robust representations for out-of-vocabulary tokens unseen in training.\n\n\n**Tensorflow implementation**\n\nTensorflow implementation of the pretrained biLM used to compute ELMo representations from `\"Deep contextualized word representations\" \u003chttp://arxiv.org/abs/1802.05365\u003e`__.\n\nThis repository supports both training biLMs and using pre-trained models for prediction.\n\nWe also have a pytorch implementation available in `AllenNLP \u003chttp://allennlp.org/\u003e`__.\n\nYou may also find it easier to use the version provided in `Tensorflow Hub \u003chttps://www.tensorflow.org/hub/modules/google/elmo/2\u003e`__ if you just like to make predictions.\n\n**pre-trained models:**\n\nWe have got several pre-trained English language biLMs available for use. Each model is specified with two separate files, a JSON formatted \"options\" file with hyperparameters and a hdf5 formatted file with the model weights. Links to the pre-trained models are available `here \u003chttps://allennlp.org/elmo\u003e`__.\n\nThere are three ways to integrate ELMo representations into a downstream task, depending on your use case.\n\n1. Compute representations on the fly from raw text using character input. This is the most general method and will handle any input text. It is also the most computationally expensive.\n2. Precompute and cache the context independent token representations, then compute context dependent representations using the biLSTMs for input data. This method is less computationally expensive then #1, but is only applicable with a fixed, prescribed vocabulary.\n3. Precompute the representations for your entire dataset and save to a file.\n\nWe have used all of these methods in the past for various use cases. #1 is necessary for evaluating at test time on unseen data (e.g. public SQuAD leaderboard). #2 is a good compromise for large datasets where the size of the file in is unfeasible (SNLI, SQuAD). #3 is a good choice for smaller datasets or in cases where you'd like to use ELMo in other frameworks.\n\nIn all cases, the process roughly follows the same steps. First, create a ``Batcher`` (or ``TokenBatcher`` for #2) to translate tokenized strings to numpy arrays of character (or token) ids. Then, load the pretrained ELMo model (class ``BidirectionalLanguageModel``). Finally, for steps #1 and #2 use ``weight_layers`` to compute the final ELMo representations. For #3, use ``BidirectionalLanguageModel`` to write all the intermediate layers to a file.\n\n\n\n.. figure:: docs/pic/ngram_cnn_highway_1.png \nArchitecture of the language model applied to an example sentence [Reference: `arXiv paper \u003chttps://arxiv.org/pdf/1508.06615.pdf\u003e`__]. \n\n\n.. figure:: docs/pic/Glove_VS_DCWE.png \n\n--------\nFastText\n--------\n\n.. figure:: docs/pic/fasttext-logo-color-web.png\n\nfastText is a library for efficient learning of word representations and sentence classification.\n\n**Github:** `facebookresearch/fastText \u003chttps://github.com/facebookresearch/fastText\u003e`__\n\n**Models**\n\n- Recent state-of-the-art `English word vectors \u003chttps://fasttext.cc/docs/en/english-vectors.html\u003e`__.\n- Word vectors for `157 languages trained on Wikipedia and Crawl \u003chttps://github.com/facebookresearch/fastText/blob/master/docs/crawl-vectors.md\u003e`__.\n- Models for `language identification \u003chttps://fasttext.cc/docs/en/language-identification.html#content\u003e`__ and `various supervised tasks \u003chttps://fasttext.cc/docs/en/supervised-models.html#content\u003e`__.\n\n**Supplementary data :**\n\n\n- The preprocessed `YFCC100M data \u003chttps://fasttext.cc/docs/en/dataset.html#content\u003e`__ .\n\n**FAQ**\n\nYou can find `answers to frequently asked questions \u003chttps://fasttext.cc/docs/en/faqs.html#content\u003e`__ on Their project `website \u003chttps://fasttext.cc/\u003e`__.\n\n**Cheatsheet**\n\nAlso a `cheatsheet \u003chttps://fasttext.cc/docs/en/cheatsheet.html#content\u003e`__ is provided full of useful one-liners.\n\n\n\n~~~~~~~~~~~~~~\nWeighted Words\n~~~~~~~~~~~~~~\n\n\n--------------\nTerm frequency\n--------------\n\nTerm frequency is Bag of words that is one of the simplest techniques of text feature extraction. This method is based on counting number of the words in each document and assign it to feature space.\n\n\n-----------------------------------------\nTerm Frequency-Inverse Document Frequency\n-----------------------------------------\nThe mathematical representation of weight of a term in a document by Tf-idf is given:\n\n.. image:: docs/eq/tf-idf.gif\n :width: 10px\n \nWhere N is number of documents and df(t) is the number of documents containing the term t in the corpus. The first part would improve recall and the later would improve the precision of the word embedding. Although tf-idf tries to overcome the problem of common terms in document, it still suffers from some other descriptive limitations. Namely, tf-idf cannot account for the similarity between words in the document since each word is presented as an index. In the recent years, with development of more complex models, such as neural nets, new methods has been presented that can incorporate concepts, such as similarity of words and part of speech tagging. This work uses, word2vec and Glove, two of the most common methods that have been successfully used for deep learning techniques.\n\n\n.. code:: python\n\n from sklearn.feature_extraction.text import TfidfVectorizer\n def loadData(X_train, X_test,MAX_NB_WORDS=75000):\n vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)\n X_train = vectorizer_x.fit_transform(X_train).toarray()\n X_test = vectorizer_x.transform(X_test).toarray()\n print(\"tf-idf with\",str(np.array(X_train).shape[1]),\"features\")\n return (X_train,X_test)\n \n \n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nComparison of Feature Extraction Techniques\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **Model** | **Advantages** | **Limitation** |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **Weighted Words** | * Easy to compute | * It does not capture the position in the text (syntactic) |\n| | | |\n| | * Easy to compute the similarity between 2 documents using it | * It does not capture meaning in the text (semantics) |\n| | | |\n| | * Basic metric to extract the most descriptive terms in a document | |\n| | | * Common words effect on the results (e.g., “am”, “is”, etc.) |\n| | * Works with an unknown word (e.g., New words in languages) | |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **TF-IDF** | * Easy to compute | * It does not capture the position in the text (syntactic) |\n| | | |\n| | | |\n| | * Easy to compute the similarity between 2 documents using it | * It does not capture meaning in the text (semantics) |\n| | | |\n| | | |\n| | * Basic metric to extract the most descriptive terms in a document | |\n| | | |\n| | | |\n| | * Common words do not affect the results due to IDF (e.g., “am”, “is”, etc.) | |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **Word2Vec** | * It captures the position of the words in the text (syntactic) | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |\n| | | |\n| | * It captures meaning in the words (semantics) | * It cannot capture out-of-vocabulary words from corpus |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **GloVe (Pre-Trained)** | * It captures the position of the words in the text (syntactic) | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |\n| | | |\n| | * It captures meaning in the words (semantics) | |\n| | | * Memory consumption for storage |\n| | * Trained on huge corpus | |\n| | | |\n| | | * It cannot capture out-of-vocabulary words from corpus |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **GloVe (Trained)** | * It is very straightforward, e.g., to enforce the word vectors to capture sub-linear relationships in the vector space (performs better than Word2vec) | * Memory consumption for storage |\n| | | |\n| | * Lower weight for highly frequent word pairs, such as stop words like “am”, “is”, etc. Will not dominate training progress | * Needs huge corpus to learn |\n| | | |\n| | | * It cannot capture out-of-vocabulary words from the corpus |\n| | | |\n| | | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n| **FastText** | * Works for rare words (rare in their character n-grams which are still shared with other words | * It cannot capture the meaning of the word from the text (fails to capture polysemy) |\n| | | |\n| | | * Memory consumption for storage |\n| | * Solves out of vocabulary words with n-gram in character level | |\n| | | * Computationally is more expensive in comparing with GloVe and Word2Vec |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n|**Contextualized Word Representations**| * It captures the meaning of the word from the text (incorporates context, handling polysemy) | * Memory consumption for storage |\n| | | |\n| | | * Improves performance notably on downstream tasks. Computationally is more expensive in comparison to others |\n| | | |\n| | | * Needs another word embedding for all LSTM and feedforward layers |\n| | | |\n| | | * It cannot capture out-of-vocabulary words from a corpus |\n| | | |\n| | | |\n| | | * Works only sentence and document level (it cannot work for individual word level) |\n+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+\n\n\n========================\nDimensionality Reduction\n========================\n\n----\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nPrincipal Component Analysis (PCA)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nPrinciple component analysis~(PCA) is the most popular technique in multivariate analysis and dimensionality reduction. PCA is a method to identify a subspace in which the data approximately lies. This means finding new variables that are uncorrelated and maximizing the variance to preserve as much variability as possible.\n\n\nExample of PCA on text dataset (20newsgroups) from tf-idf with 75000 features to 2000 components:\n\n.. code:: python\n\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n\n def TFIDF(X_train, X_test, MAX_NB_WORDS=75000):\n vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)\n X_train = vectorizer_x.fit_transform(X_train).toarray()\n X_test = vectorizer_x.transform(X_test).toarray()\n print(\"tf-idf with\", str(np.array(X_train).shape[1]), \"features\")\n return (X_train, X_test)\n\n\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train,X_test = TFIDF(X_train,X_test)\n\n from sklearn.decomposition import PCA\n pca = PCA(n_components=2000)\n X_train_new = pca.fit_transform(X_train)\n X_test_new = pca.transform(X_test)\n\n print(\"train with old features: \",np.array(X_train).shape)\n print(\"train with new features:\" ,np.array(X_train_new).shape)\n \n print(\"test with old features: \",np.array(X_test).shape)\n print(\"test with new features:\" ,np.array(X_test_new).shape)\n\noutput:\n\n.. code:: python\n\n tf-idf with 75000 features\n train with old features: (11314, 75000)\n train with new features: (11314, 2000)\n test with old features: (7532, 75000)\n test with new features: (7532, 2000)\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nLinear Discriminant Analysis (LDA)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\nLinear Discriminant Analysis (LDA) is another commonly used technique for data classification and dimensionality reduction. LDA is particularly helpful where the within-class frequencies are unequal and their performances have been evaluated on randomly generated test data. Class-dependent and class-independent transformation are two approaches in LDA where the ratio of between-class-variance to within-class-variance and the ratio of the overall-variance to within-class-variance are used respectively. \n\n\n\n.. code:: python\n\n\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n\n\n def TFIDF(X_train, X_test, MAX_NB_WORDS=75000):\n vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)\n X_train = vectorizer_x.fit_transform(X_train).toarray()\n X_test = vectorizer_x.transform(X_test).toarray()\n print(\"tf-idf with\", str(np.array(X_train).shape[1]), \"features\")\n return (X_train, X_test)\n\n\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train,X_test = TFIDF(X_train,X_test)\n\n\n\n LDA = LinearDiscriminantAnalysis(n_components=15)\n X_train_new = LDA.fit(X_train,y_train)\n X_train_new = LDA.transform(X_train)\n X_test_new = LDA.transform(X_test)\n\n print(\"train with old features: \",np.array(X_train).shape)\n print(\"train with new features:\" ,np.array(X_train_new).shape)\n\n print(\"test with old features: \",np.array(X_test).shape)\n print(\"test with new features:\" ,np.array(X_test_new).shape)\n\n\noutput:\n\n.. code:: \n\n tf-idf with 75000 features\n train with old features: (11314, 75000)\n train with new features: (11314, 15)\n test with old features: (7532, 75000)\n test with new features: (7532, 15)\n \n \n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nNon-negative Matrix Factorization (NMF)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\n.. code:: python\n\n\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n from sklearn.decomposition import NMF\n\n\n def TFIDF(X_train, X_test, MAX_NB_WORDS=75000):\n vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)\n X_train = vectorizer_x.fit_transform(X_train).toarray()\n X_test = vectorizer_x.transform(X_test).toarray()\n print(\"tf-idf with\", str(np.array(X_train).shape[1]), \"features\")\n return (X_train, X_test)\n\n\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train,X_test = TFIDF(X_train,X_test)\n\n\n\n NMF_ = NMF(n_components=2000)\n X_train_new = NMF_.fit(X_train)\n X_train_new = NMF_.transform(X_train)\n X_test_new = NMF_.transform(X_test)\n\n print(\"train with old features: \",np.array(X_train).shape)\n print(\"train with new features:\" ,np.array(X_train_new).shape)\n\n print(\"test with old features: \",np.array(X_test).shape)\n print(\"test with new features:\" ,np.array(X_test_new))\n\noutput:\n\n.. code:: \n\n tf-idf with 75000 features\n train with old features: (11314, 75000)\n train with new features: (11314, 2000)\n test with old features: (7532, 75000)\n test with new features: (7532, 2000)\n \n \n\n~~~~~~~~~~~~~~~~~\nRandom Projection\n~~~~~~~~~~~~~~~~~\nRandom projection or random feature is a dimensionality reduction technique mostly used for very large volume dataset or very high dimensional feature space. Text and document, especially with weighted feature extraction, can contain a huge number of underlying features.\nMany researchers addressed Random Projection for text data for text mining, text classification and/or dimensionality reduction.\nWe start to review some random projection techniques. \n\n\n.. image:: docs/pic/Random%20Projection.png\n\n.. code:: python\n\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n\n def TFIDF(X_train, X_test, MAX_NB_WORDS=75000):\n vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)\n X_train = vectorizer_x.fit_transform(X_train).toarray()\n X_test = vectorizer_x.transform(X_test).toarray()\n print(\"tf-idf with\", str(np.array(X_train).shape[1]), \"features\")\n return (X_train, X_test)\n\n\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train,X_test = TFIDF(X_train,X_test)\n\n from sklearn import random_projection\n\n RandomProjection = random_projection.GaussianRandomProjection(n_components=2000)\n X_train_new = RandomProjection.fit_transform(X_train)\n X_test_new = RandomProjection.transform(X_test)\n\n print(\"train with old features: \",np.array(X_train).shape)\n print(\"train with new features:\" ,np.array(X_train_new).shape)\n\n print(\"test with old features: \",np.array(X_test).shape)\n print(\"test with new features:\" ,np.array(X_test_new).shape)\n\noutput:\n\n.. code:: python\n\n tf-idf with 75000 features\n train with old features: (11314, 75000)\n train with new features: (11314, 2000)\n test with old features: (7532, 75000)\n test with new features: (7532, 2000)\n \n~~~~~~~~~~~\nAutoencoder\n~~~~~~~~~~~\n\n\nAutoencoder is a neural network technique that is trained to attempt to map its input to its output. The autoencoder as dimensional reduction methods have achieved great success via the powerful reprehensibility of neural networks. The main idea is, one hidden layer between the input and output layers with fewer neurons can be used to reduce the dimension of feature space. Specially for texts, documents, and sequences that contains many features, autoencoder could help to process data faster and more efficiently.\n\n\n.. image:: docs/pic/Autoencoder.png\n\n\n\n.. code:: python\n\n from keras.layers import Input, Dense\n from keras.models import Model\n\n # this is the size of our encoded representations\n encoding_dim = 1500 \n\n # this is our input placeholder\n input = Input(shape=(n,))\n # \"encoded\" is the encoded representation of the input\n encoded = Dense(encoding_dim, activation='relu')(input)\n # \"decoded\" is the lossy reconstruction of the input\n decoded = Dense(n, activation='sigmoid')(encoded)\n\n # this model maps an input to its reconstruction\n autoencoder = Model(input, decoded)\n\n # this model maps an input to its encoded representation\n encoder = Model(input, encoded)\n \n\n encoded_input = Input(shape=(encoding_dim,))\n # retrieve the last layer of the autoencoder model\n decoder_layer = autoencoder.layers[-1]\n # create the decoder model\n decoder = Model(encoded_input, decoder_layer(encoded_input))\n \n autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')\n \n \n\nLoad data:\n\n\n.. code:: python\n\n autoencoder.fit(x_train, x_train,\n epochs=50,\n batch_size=256,\n shuffle=True,\n validation_data=(x_test, x_test))\n \n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nT-distributed Stochastic Neighbor Embedding (T-SNE)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\n\nT-distributed Stochastic Neighbor Embedding (T-SNE) is a nonlinear dimensionality reduction technique for embedding high-dimensional data which is mostly used for visualization in a low-dimensional space. This approach is based on `G. Hinton and ST. Roweis \u003chttps://www.cs.toronto.edu/~fritz/absps/sne.pdf\u003e`__ . SNE works by converting the high dimensional Euclidean distances into conditional probabilities which represent similarities.\n\n `Example \u003chttp://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html\u003e`__:\n\n\n.. code:: python\n\n import numpy as np\n from sklearn.manifold import TSNE\n X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])\n X_embedded = TSNE(n_components=2).fit_transform(X)\n X_embedded.shape\n\n\nExample of Glove and T-SNE for text:\n\n.. image:: docs/pic/TSNE.png\n\n===============================\nText Classification Techniques\n===============================\n\n----\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nRocchio classification\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe first version of Rocchio algorithm is introduced by rocchio in 1971 to use relevance feedback in querying full-text databases. Since then many researchers have addressed and developed this technique for text and document classification. This method uses TF-IDF weights for each informative word instead of a set of Boolean features. Using a training set of documents, Rocchio's algorithm builds a prototype vector for each class which is an average vector over all training document vectors that belongs to a certain class. Then, it will assign each test document to a class with maximum similarity that between test document and each of the prototype vectors.\n\n\nWhen in nearest centroid classifier, we used for text as input data for classification with tf-idf vectors, this classifier is known as the Rocchio classifier.\n\n.. code:: python\n\n from sklearn.neighbors.nearest_centroid import NearestCentroid\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', NearestCentroid()),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\n\n\n\nOutput:\n\n.. code:: python\n\n precision recall f1-score support\n\n 0 0.75 0.49 0.60 319\n 1 0.44 0.76 0.56 389\n 2 0.75 0.68 0.71 394\n 3 0.71 0.59 0.65 392\n 4 0.81 0.71 0.76 385\n 5 0.83 0.66 0.74 395\n 6 0.49 0.88 0.63 390\n 7 0.86 0.76 0.80 396\n 8 0.91 0.86 0.89 398\n 9 0.85 0.79 0.82 397\n 10 0.95 0.80 0.87 399\n 11 0.94 0.66 0.78 396\n 12 0.40 0.70 0.51 393\n 13 0.84 0.49 0.62 396\n 14 0.89 0.72 0.80 394\n 15 0.55 0.73 0.63 398\n 16 0.68 0.76 0.71 364\n 17 0.97 0.70 0.81 376\n 18 0.54 0.53 0.53 310\n 19 0.58 0.39 0.47 251\n\n avg / total 0.74 0.69 0.70 7532\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nBoosting and Bagging\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n---------\nBoosting\n---------\n\n.. image:: docs/pic/Boosting.PNG\n\n\n**Boosting** is a Ensemble learning meta-algorithm for primarily reducing variance in supervised learning. It is basically a family of machine learning algorithms that convert weak learners to strong ones. Boosting is based on the question posed by `Michael Kearns \u003chttps://en.wikipedia.org/wiki/Michael_Kearns_(computer_scientist)\u003e`__ and Leslie Valiant (1988, 1989) Can a set of weak learners create a single strong learner? A weak learner is defined to be a Classification that is only slightly correlated with the true classification (it can label examples better than random guessing). In contrast, a strong learner is a classifier that is arbitrarily well-correlated with the true classification.\n\n\n\n\n.. code:: python\n\n from sklearn.ensemble import GradientBoostingClassifier\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', GradientBoostingClassifier(n_estimators=100)),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\n\nOutput:\n \n.. code:: python\n\n precision recall f1-score support\n 0 0.81 0.66 0.73 319\n 1 0.69 0.70 0.69 389\n 2 0.70 0.68 0.69 394\n 3 0.64 0.72 0.68 392\n 4 0.79 0.79 0.79 385\n 5 0.83 0.64 0.72 395\n 6 0.81 0.84 0.82 390\n 7 0.84 0.75 0.79 396\n 8 0.90 0.86 0.88 398\n 9 0.90 0.85 0.88 397\n 10 0.93 0.86 0.90 399\n 11 0.90 0.81 0.85 396\n 12 0.33 0.69 0.45 393\n 13 0.87 0.72 0.79 396\n 14 0.87 0.84 0.85 394\n 15 0.85 0.87 0.86 398\n 16 0.65 0.78 0.71 364\n 17 0.96 0.74 0.84 376\n 18 0.70 0.55 0.62 310\n 19 0.62 0.56 0.59 251\n\n avg / total 0.78 0.75 0.76 7532\n\n \n-------\nBagging\n-------\n\n.. image:: docs/pic/Bagging.PNG\n\n\n.. code:: python\n\n from sklearn.ensemble import BaggingClassifier\n from sklearn.neighbors import KNeighborsClassifier\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', BaggingClassifier(KNeighborsClassifier())),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\n\nOutput:\n \n.. code:: python\n\n precision recall f1-score support\n 0 0.57 0.74 0.65 319\n 1 0.60 0.56 0.58 389\n 2 0.62 0.54 0.58 394\n 3 0.54 0.57 0.55 392\n 4 0.63 0.54 0.58 385\n 5 0.68 0.62 0.65 395\n 6 0.55 0.46 0.50 390\n 7 0.77 0.67 0.72 396\n 8 0.79 0.82 0.80 398\n 9 0.74 0.77 0.76 397\n 10 0.81 0.86 0.83 399\n 11 0.74 0.85 0.79 396\n 12 0.67 0.49 0.57 393\n 13 0.78 0.51 0.62 396\n 14 0.76 0.78 0.77 394\n 15 0.71 0.81 0.76 398\n 16 0.73 0.73 0.73 364\n 17 0.64 0.79 0.71 376\n 18 0.45 0.69 0.54 310\n 19 0.61 0.54 0.57 251\n\n avg / total 0.67 0.67 0.67 7532\n \n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nNaive Bayes Classifier\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNaïve Bayes text classification has been used in industry\nand academia for a long time (introduced by Thomas Bayes\nbetween 1701-1761). However, this technique\nis being studied since the 1950s for text and document categorization. Naive Bayes Classifier (NBC) is generative\nmodel which is widely used in Information Retrieval. Many researchers addressed and developed this technique\nfor their applications. We start with the most basic version\nof NBC which developed by using term-frequency (Bag of\nWord) fetaure extraction technique by counting number of\nwords in documents\n\n\n.. code:: python\n\n from sklearn.naive_bayes import MultinomialNB\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', MultinomialNB()),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n \n \nOutput:\n \n.. code:: python\n\n precision recall f1-score support\n\n 0 0.80 0.52 0.63 319\n 1 0.81 0.65 0.72 389\n 2 0.82 0.65 0.73 394\n 3 0.67 0.78 0.72 392\n 4 0.86 0.77 0.81 385\n 5 0.89 0.75 0.82 395\n 6 0.93 0.69 0.80 390\n 7 0.85 0.92 0.88 396\n 8 0.94 0.93 0.93 398\n 9 0.92 0.90 0.91 397\n 10 0.89 0.97 0.93 399\n 11 0.59 0.97 0.74 396\n 12 0.84 0.60 0.70 393\n 13 0.92 0.74 0.82 396\n 14 0.84 0.89 0.87 394\n 15 0.44 0.98 0.61 398\n 16 0.64 0.94 0.76 364\n 17 0.93 0.91 0.92 376\n 18 0.96 0.42 0.58 310\n 19 0.97 0.14 0.24 251\n\n avg / total 0.82 0.77 0.77 7532\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nK-nearest Neighbor\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nR\nIn machine learning, the k-nearest neighbors algorithm (kNN)\nis a non-parametric technique used for classification.\nThis method is used in Natural-language processing (NLP)\nas a text classification technique in many researches in the past\ndecades.\n\n.. image:: docs/pic/KNN.png\n\n.. code:: python\n\n from sklearn.neighbors import KNeighborsClassifier\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', KNeighborsClassifier()),\n ])\n\n text_clf.fit(X_train, y_train)\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\nOutput:\n\n.. code:: python\n\n precision recall f1-score support\n\n 0 0.43 0.76 0.55 319\n 1 0.50 0.61 0.55 389\n 2 0.56 0.57 0.57 394\n 3 0.53 0.58 0.56 392\n 4 0.59 0.56 0.57 385\n 5 0.69 0.60 0.64 395\n 6 0.58 0.45 0.51 390\n 7 0.75 0.69 0.72 396\n 8 0.84 0.81 0.82 398\n 9 0.77 0.72 0.74 397\n 10 0.85 0.84 0.84 399\n 11 0.76 0.84 0.80 396\n 12 0.70 0.50 0.58 393\n 13 0.82 0.49 0.62 396\n 14 0.79 0.76 0.78 394\n 15 0.75 0.76 0.76 398\n 16 0.70 0.73 0.72 364\n 17 0.62 0.76 0.69 376\n 18 0.55 0.61 0.58 310\n 19 0.56 0.49 0.52 251\n\n avg / total 0.67 0.66 0.66 7532\n\n\n\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nSupport Vector Machine (SVM)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\nThe original version of SVM was introduced by Vapnik and Chervonenkis in 1963. The early 1990s, nonlinear version was addressed by BE. Boser et al.. Original version of SVM was designed for binary classification problem, but Many researchers have worked on multi-class problem using this authoritative technique.\n\n\nThe advantages of support vector machines are based on scikit-learn page:\n\n* Effective in high dimensional spaces.\n* Still effective in cases where number of dimensions is greater than the number of samples.\n* Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.\n* Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.\n\n\nThe disadvantages of support vector machines include:\n\n* If the number of features is much greater than the number of samples, avoiding over-fitting via choosing kernel functions and regularization term is crucial.\n* SVMs do not directly provide probability estimates, these are calculated using an expensive five-fold cross-validation (see Scores and probabilities, below).\n\n\n\n.. image:: docs/pic/SVM.png\n\n\n.. code:: python\n\n\n from sklearn.svm import LinearSVC\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', LinearSVC()),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\n\noutput:\n\n\n.. code:: python\n\n precision recall f1-score support\n\n 0 0.82 0.80 0.81 319\n 1 0.76 0.80 0.78 389\n 2 0.77 0.73 0.75 394\n 3 0.71 0.76 0.74 392\n 4 0.84 0.86 0.85 385\n 5 0.87 0.76 0.81 395\n 6 0.83 0.91 0.87 390\n 7 0.92 0.91 0.91 396\n 8 0.95 0.95 0.95 398\n 9 0.92 0.95 0.93 397\n 10 0.96 0.98 0.97 399\n 11 0.93 0.94 0.93 396\n 12 0.81 0.79 0.80 393\n 13 0.90 0.87 0.88 396\n 14 0.90 0.93 0.92 394\n 15 0.84 0.93 0.88 398\n 16 0.75 0.92 0.82 364\n 17 0.97 0.89 0.93 376\n 18 0.82 0.62 0.71 310\n 19 0.75 0.61 0.68 251\n\n avg / total 0.85 0.85 0.85 7532\n\n\n\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nDecision Tree\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOne of earlier classification algorithm for text and data mining is decision tree. Decision tree classifiers (DTC's) are used successfully in many diverse areas of classification. The structure of this technique includes a hierarchical decomposition of the data space (only train dataset). Decision tree as classification task was introduced by `D. Morgan \u003chttp://www.aclweb.org/anthology/P95-1037\u003e`__ and developed by `JR. Quinlan \u003chttps://courses.cs.ut.ee/2009/bayesian-networks/extras/quinlan1986.pdf\u003e`__. The main idea is creating trees based on the attributes of the data points, but the challenge is determining which attribute should be in parent level and which one should be in child level. To solve this problem, `De Mantaras \u003chttps://link.springer.com/article/10.1023/A:1022694001379\u003e`__ introduced statistical modeling for feature selection in tree.\n\n\n.. code:: python\n\n from sklearn import tree\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', tree.DecisionTreeClassifier()),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\n\noutput:\n\n\n.. code:: python\n\n precision recall f1-score support\n\n 0 0.51 0.48 0.49 319\n 1 0.42 0.42 0.42 389\n 2 0.51 0.56 0.53 394\n 3 0.46 0.42 0.44 392\n 4 0.50 0.56 0.53 385\n 5 0.50 0.47 0.48 395\n 6 0.66 0.73 0.69 390\n 7 0.60 0.59 0.59 396\n 8 0.66 0.72 0.69 398\n 9 0.53 0.55 0.54 397\n 10 0.68 0.66 0.67 399\n 11 0.73 0.69 0.71 396\n 12 0.34 0.33 0.33 393\n 13 0.52 0.42 0.46 396\n 14 0.65 0.62 0.63 394\n 15 0.68 0.72 0.70 398\n 16 0.49 0.62 0.55 364\n 17 0.78 0.60 0.68 376\n 18 0.38 0.38 0.38 310\n 19 0.32 0.32 0.32 251\n\n avg / total 0.55 0.55 0.55 7532\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nRandom Forest\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\nRandom forests or random decision forests technique is an ensemble learning method for text classification. This method was introduced by `T. Kam Ho \u003chttps://doi.org/10.1109/ICDAR.1995.598994\u003e`__ in 1995 for first time which used t trees in parallel. This technique was later developed by `L. Breiman \u003chttps://link.springer.com/article/10.1023/A:1010933404324\u003e`__ in 1999 that they found converged for RF as a margin measure.\n\n\n.. image:: docs/pic/RF.png\n\n.. code:: python\n\n from sklearn.ensemble import RandomForestClassifier\n from sklearn.pipeline import Pipeline\n from sklearn import metrics\n from sklearn.feature_extraction.text import CountVectorizer\n from sklearn.feature_extraction.text import TfidfTransformer\n from sklearn.datasets import fetch_20newsgroups\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n text_clf = Pipeline([('vect', CountVectorizer()),\n ('tfidf', TfidfTransformer()),\n ('clf', RandomForestClassifier(n_estimators=100)),\n ])\n\n text_clf.fit(X_train, y_train)\n\n\n predicted = text_clf.predict(X_test)\n\n print(metrics.classification_report(y_test, predicted))\n\n\noutput:\n\n\n.. code:: python\n\n\n precision recall f1-score support\n\n 0 0.69 0.63 0.66 319\n 1 0.56 0.69 0.62 389\n 2 0.67 0.78 0.72 394\n 3 0.67 0.67 0.67 392\n 4 0.71 0.78 0.74 385\n 5 0.78 0.68 0.73 395\n 6 0.74 0.92 0.82 390\n 7 0.81 0.79 0.80 396\n 8 0.90 0.89 0.90 398\n 9 0.80 0.89 0.84 397\n 10 0.90 0.93 0.91 399\n 11 0.89 0.91 0.90 396\n 12 0.68 0.49 0.57 393\n 13 0.83 0.65 0.73 396\n 14 0.81 0.88 0.84 394\n 15 0.68 0.91 0.78 398\n 16 0.67 0.86 0.75 364\n 17 0.93 0.78 0.85 376\n 18 0.86 0.48 0.61 310\n 19 0.79 0.31 0.45 251\n\n avg / total 0.77 0.76 0.75 7532\n\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nConditional Random Field (CRF)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nConditional Random Field (CRF) is an undirected graphical model as shown in figure. CRFs state the conditional probability of a label sequence *Y* give a sequence of observation *X* *i.e.* P(Y|X). CRFs can incorporate complex features of observation sequence without violating the independence assumption by modeling the conditional probability of the label sequences rather than the joint probability P(X,Y). The concept of clique which is a fully connected subgraph and clique potential are used for computing P(X|Y). Considering one potential function for each clique of the graph, the probability of a variable configuration corresponds to the product of a series of non-negative potential function. The value computed by each potential function is equivalent to the probability of the variables in its corresponding clique taken on a particular configuration.\n\n\n.. image:: docs/pic/CRF.png\n\n\nExample from `Here \u003chttp://sklearn-crfsuite.readthedocs.io/en/latest/tutorial.html\u003e`__\nLet’s use CoNLL 2002 data to build a NER system\nCoNLL2002 corpus is available in NLTK. We use Spanish data.\n\n\n.. code:: python\n\n import nltk\n import sklearn_crfsuite\n from sklearn_crfsuite import metrics\n nltk.corpus.conll2002.fileids()\n train_sents = list(nltk.corpus.conll2002.iob_sents('esp.train'))\n test_sents = list(nltk.corpus.conll2002.iob_sents('esp.testb'))\n \n \nsklearn-crfsuite (and python-crfsuite) supports several feature formats; here we use feature dicts.\n\n.. code:: python\n\n def word2features(sent, i):\n word = sent[i][0]\n postag = sent[i][1]\n\n features = {\n 'bias': 1.0,\n 'word.lower()': word.lower(),\n 'word[-3:]': word[-3:],\n 'word[-2:]': word[-2:],\n 'word.isupper()': word.isupper(),\n 'word.istitle()': word.istitle(),\n 'word.isdigit()': word.isdigit(),\n 'postag': postag,\n 'postag[:2]': postag[:2],\n }\n if i \u003e 0:\n word1 = sent[i-1][0]\n postag1 = sent[i-1][1]\n features.update({\n '-1:word.lower()': word1.lower(),\n '-1:word.istitle()': word1.istitle(),\n '-1:word.isupper()': word1.isupper(),\n '-1:postag': postag1,\n '-1:postag[:2]': postag1[:2],\n })\n else:\n features['BOS'] = True\n\n if i \u003c len(sent)-1:\n word1 = sent[i+1][0]\n postag1 = sent[i+1][1]\n features.update({\n '+1:word.lower()': word1.lower(),\n '+1:word.istitle()': word1.istitle(),\n '+1:word.isupper()': word1.isupper(),\n '+1:postag': postag1,\n '+1:postag[:2]': postag1[:2],\n })\n else:\n features['EOS'] = True\n\n return features\n\n\n def sent2features(sent):\n return [word2features(sent, i) for i in range(len(sent))]\n\n def sent2labels(sent):\n return [label for token, postag, label in sent]\n\n def sent2tokens(sent):\n return [token for token, postag, label in sent]\n\n X_train = [sent2features(s) for s in train_sents]\n y_train = [sent2labels(s) for s in train_sents]\n\n X_test = [sent2features(s) for s in test_sents]\n y_test = [sent2labels(s) for s in test_sents]\n\n\nTo see all possible CRF parameters check its docstring. Here we are useing L-BFGS training algorithm (it is default) with Elastic Net (L1 + L2) regularization.\n\n\n\n.. code:: python\n\n crf = sklearn_crfsuite.CRF(\n algorithm='lbfgs',\n c1=0.1,\n c2=0.1,\n max_iterations=100,\n all_possible_transitions=True\n )\n crf.fit(X_train, y_train)\n\n\nEvaluation\n\n\n.. code:: python\n\n y_pred = crf.predict(X_test)\n print(metrics.flat_classification_report(\n y_test, y_pred, digits=3\n ))\n\n\nOutput:\n\n.. code:: python\n\n precision recall f1-score support\n\n B-LOC 0.810 0.784 0.797 1084\n B-MISC 0.731 0.569 0.640 339\n B-ORG 0.807 0.832 0.820 1400\n B-PER 0.850 0.884 0.867 735\n I-LOC 0.690 0.637 0.662 325\n I-MISC 0.699 0.589 0.639 557\n I-ORG 0.852 0.786 0.818 1104\n I-PER 0.893 0.943 0.917 634\n O 0.992 0.997 0.994 45355\n\n avg / total 0.970 0.971 0.971 51533\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nDeep Learning\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n-----------------------------------------\nDeep Neural Networks\n-----------------------------------------\n\nDeep Neural Networks architectures are designed to learn through multiple connection of layers where each single layer only receives connection from previous and provides connections only to the next layer in hidden part. The input is a connection of feature space (As discussed in Section Feature_extraction with first hidden layer. For Deep Neural Networks (DNN), input layer could be tf-ifd, word embedding, or etc. as shown in standard DNN in Figure. The output layer houses neurons equal to the number of classes for multi-class classification and only one neuron for binary classification. But our main contribution in this paper is that we have many trained DNNs to serve different purposes. Here, we have multi-class DNNs where each learning model is generated randomly (number of nodes in each layer as well as the number of layers are randomly assigned). Our implementation of Deep Neural Network (DNN) is basically a discriminatively trained model that uses standard back-propagation algorithm and sigmoid or ReLU as activation functions. The output layer for multi-class classification should use Softmax.\n\n\n.. image:: docs/pic/DNN.png\n\nimport packages:\n\n.. code:: python\n\n from sklearn.datasets import fetch_20newsgroups\n from keras.layers import Dropout, Dense\n from keras.models import Sequential\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n from sklearn import metrics\n\n\nconvert text to TF-IDF:\n\n.. code:: python\n\n def TFIDF(X_train, X_test,MAX_NB_WORDS=75000):\n vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)\n X_train = vectorizer_x.fit_transform(X_train).toarray()\n X_test = vectorizer_x.transform(X_test).toarray()\n print(\"tf-idf with\",str(np.array(X_train).shape[1]),\"features\")\n return (X_train,X_test)\n\n\nBuild a DNN Model for Text:\n\n.. code:: python\n\n def Build_Model_DNN_Text(shape, nClasses, dropout=0.5):\n \"\"\"\n buildModel_DNN_Tex(shape, nClasses,dropout)\n Build Deep neural networks Model for text classification\n Shape is input feature space\n nClasses is number of classes\n \"\"\"\n model = Sequential()\n node = 512 # number of nodes\n nLayers = 4 # number of hidden layer\n\n model.add(Dense(node,input_dim=shape,activation='relu'))\n model.add(Dropout(dropout))\n for i in range(0,nLayers):\n model.add(Dense(node,input_dim=node,activation='relu'))\n model.add(Dropout(dropout))\n model.add(Dense(nClasses, activation='softmax'))\n\n model.compile(loss='sparse_categorical_crossentropy',\n optimizer='adam',\n metrics=['accuracy'])\n\n return model\n\n\n\nLoad text dataset (20newsgroups):\n\n.. code:: python\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n\n\nrun DNN and see our result:\n\n\n.. code:: python\n\n X_train_tfidf,X_test_tfidf = TFIDF(X_train,X_test)\n model_DNN = Build_Model_DNN_Text(X_train_tfidf.shape[1], 20)\n model_DNN.fit(X_train_tfidf, y_train,\n validation_data=(X_test_tfidf, y_test),\n epochs=10,\n batch_size=128,\n verbose=2)\n\n predicted = model_DNN.predict_class(X_test_tfidf)\n\n print(metrics.classification_report(y_test, predicted))\n\n\nModel summary:\n\n.. code:: python \n\n _________________________________________________________________\n Layer (type) Output Shape Param # \n =================================================================\n dense_1 (Dense) (None, 512) 38400512 \n _________________________________________________________________\n dropout_1 (Dropout) (None, 512) 0 \n _________________________________________________________________\n dense_2 (Dense) (None, 512) 262656 \n _________________________________________________________________\n dropout_2 (Dropout) (None, 512) 0 \n _________________________________________________________________\n dense_3 (Dense) (None, 512) 262656 \n _________________________________________________________________\n dropout_3 (Dropout) (None, 512) 0 \n _________________________________________________________________\n dense_4 (Dense) (None, 512) 262656 \n _________________________________________________________________\n dropout_4 (Dropout) (None, 512) 0 \n _________________________________________________________________\n dense_5 (Dense) (None, 512) 262656 \n _________________________________________________________________\n dropout_5 (Dropout) (None, 512) 0 \n _________________________________________________________________\n dense_6 (Dense) (None, 20) 10260 \n =================================================================\n Total params: 39,461,396\n Trainable params: 39,461,396\n Non-trainable params: 0\n _________________________________________________________________\n\n\n\nOutput:\n\n.. code:: python \n\n Train on 11314 samples, validate on 7532 samples\n Epoch 1/10\n - 16s - loss: 2.7553 - acc: 0.1090 - val_loss: 1.9330 - val_acc: 0.3184\n Epoch 2/10\n - 15s - loss: 1.5330 - acc: 0.4222 - val_loss: 1.1546 - val_acc: 0.6204\n Epoch 3/10\n - 15s - loss: 0.7438 - acc: 0.7257 - val_loss: 0.8405 - val_acc: 0.7499\n Epoch 4/10\n - 15s - loss: 0.2967 - acc: 0.9020 - val_loss: 0.9214 - val_acc: 0.7767\n Epoch 5/10\n - 15s - loss: 0.1557 - acc: 0.9543 - val_loss: 0.8965 - val_acc: 0.7917\n Epoch 6/10\n - 15s - loss: 0.1015 - acc: 0.9705 - val_loss: 0.9427 - val_acc: 0.7949\n Epoch 7/10\n - 15s - loss: 0.0595 - acc: 0.9835 - val_loss: 0.9893 - val_acc: 0.7995\n Epoch 8/10\n - 15s - loss: 0.0495 - acc: 0.9866 - val_loss: 0.9512 - val_acc: 0.8079\n Epoch 9/10\n - 15s - loss: 0.0437 - acc: 0.9867 - val_loss: 0.9690 - val_acc: 0.8117\n Epoch 10/10\n - 15s - loss: 0.0443 - acc: 0.9880 - val_loss: 1.0004 - val_acc: 0.8070\n\n\n precision recall f1-score support\n\n 0 0.76 0.78 0.77 319\n 1 0.67 0.80 0.73 389\n 2 0.82 0.63 0.71 394\n 3 0.76 0.69 0.72 392\n 4 0.65 0.86 0.74 385\n 5 0.84 0.75 0.79 395\n 6 0.82 0.87 0.84 390\n 7 0.86 0.90 0.88 396\n 8 0.95 0.91 0.93 398\n 9 0.91 0.92 0.92 397\n 10 0.98 0.92 0.95 399\n 11 0.96 0.85 0.90 396\n 12 0.71 0.69 0.70 393\n 13 0.95 0.70 0.81 396\n 14 0.86 0.91 0.88 394\n 15 0.85 0.90 0.87 398\n 16 0.79 0.84 0.81 364\n 17 0.99 0.77 0.87 376\n 18 0.58 0.75 0.65 310\n 19 0.52 0.60 0.55 251\n\n avg / total 0.82 0.81 0.81 7532\n\n\n-----------------------------------------\nRecurrent Neural Networks (RNN)\n-----------------------------------------\n\n.. image:: docs/pic/RNN.png\n\nAnother neural network architecture that is addressed by the researchers for text miming and classification is Recurrent Neural Networks (RNN). RNN assigns more weights to the previous data points of sequence. Therefore, this technique is a powerful method for text, string and sequential data classification. Moreover, this technique could be used for image classification as we did in this work. In RNN, the neural net considers the information of previous nodes in a very sophisticated method which allows for better semantic analysis of the structures in the dataset. \n\n\nGated Recurrent Unit (GRU)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nGated Recurrent Unit (GRU) is a gating mechanism for RNN which was introduced by `J. Chung et al. \u003chttps://arxiv.org/abs/1412.3555\u003e`__ and `K.Cho et al. \u003chttps://arxiv.org/abs/1406.1078\u003e`__. GRU is a simplified variant of the LSTM architecture, but there are differences as follows: GRU contains two gates and does not possess any internal memory (as shown in Figure; and finally, a second non-linearity is not applied (tanh in Figure).\n\n.. image:: docs/pic/LSTM.png\n\nLong Short-Term Memory (LSTM)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLong Short-Term Memory~(LSTM) was introduced by `S. Hochreiter and J. Schmidhuber \u003chttps://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735\u003e`__ and developed by many research scientists.\n\nTo deal with these problems Long Short-Term Memory (LSTM) is a special type of RNN that preserves long term dependency in a more effective way compared to the basic RNNs. This is particularly useful to overcome vanishing gradient problem. Although LSTM has a chain-like structure similar to RNN, LSTM uses multiple gates to carefully regulate the amount of information that will be allowed into each node state. Figure shows the basic cell of a LSTM model.\n\n\n\nimport packages:\n\n.. code:: python\n\n\n from keras.layers import Dropout, Dense, GRU, Embedding\n from keras.models import Sequential\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n from sklearn import metrics\n from keras.preprocessing.text import Tokenizer\n from keras.preprocessing.sequence import pad_sequences\n from sklearn.datasets import fetch_20newsgroups\n\nconvert text to word embedding (Using GloVe):\n\n.. code:: python\n\n def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):\n np.random.seed(7)\n text = np.concatenate((X_train, X_test), axis=0)\n text = np.array(text)\n tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n tokenizer.fit_on_texts(text)\n sequences = tokenizer.texts_to_sequences(text)\n word_index = tokenizer.word_index\n text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n print('Found %s unique tokens.' % len(word_index))\n indices = np.arange(text.shape[0])\n # np.random.shuffle(indices)\n text = text[indices]\n print(text.shape)\n X_train = text[0:len(X_train), ]\n X_test = text[len(X_train):, ]\n embeddings_index = {}\n f = open(\".\\\\Glove\\\\glove.6B.50d.txt\", encoding=\"utf8\")\n for line in f:\n\n values = line.split()\n word = values[0]\n try:\n coefs = np.asarray(values[1:], dtype='float32')\n except:\n pass\n embeddings_index[word] = coefs\n f.close()\n print('Total %s word vectors.' % len(embeddings_index))\n return (X_train, X_test, word_index,embeddings_index)\n\nBuild a RNN Model for Text:\n\n.. code:: python\n\n\n def Build_Model_RNN_Text(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):\n \"\"\"\n def buildModel_RNN(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):\n word_index in word index ,\n embeddings_index is embeddings index, look at data_helper.py\n nClasses is number of classes,\n MAX_SEQUENCE_LENGTH is maximum lenght of text sequences\n \"\"\"\n\n model = Sequential()\n hidden_layer = 3\n gru_node = 32\n\n embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))\n for word, i in word_index.items():\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None:\n # words not found in embedding index will be all-zeros.\n if len(embedding_matrix[i]) != len(embedding_vector):\n print(\"could not broadcast input array from shape\", str(len(embedding_matrix[i])),\n \"into shape\", str(len(embedding_vector)), \" Please make sure your\"\n \" EMBEDDING_DIM is equal to embedding_vector file ,GloVe,\")\n exit(1)\n embedding_matrix[i] = embedding_vector\n model.add(Embedding(len(word_index) + 1,\n EMBEDDING_DIM,\n weights=[embedding_matrix],\n input_length=MAX_SEQUENCE_LENGTH,\n trainable=True))\n\n\n print(gru_node)\n for i in range(0,hidden_layer):\n model.add(GRU(gru_node,return_sequences=True, recurrent_dropout=0.2))\n model.add(Dropout(dropout))\n model.add(GRU(gru_node, recurrent_dropout=0.2))\n model.add(Dropout(dropout))\n model.add(Dense(256, activation='relu'))\n model.add(Dense(nclasses, activation='softmax'))\n\n\n model.compile(loss='sparse_categorical_crossentropy',\n optimizer='adam',\n metrics=['accuracy'])\n return model\n\n\n\n\nrun RNN and see our result:\n\n\n.. code:: python\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)\n\n\n model_RNN = Build_Model_RNN_Text(word_index,embeddings_index, 20)\n\n model_RNN.fit(X_train_Glove, y_train,\n validation_data=(X_test_Glove, y_test),\n epochs=10,\n batch_size=128,\n verbose=2)\n\n predicted = model_RNN.predict_classes(X_test_Glove)\n\n print(metrics.classification_report(y_test, predicted))\n\n\nModel summary:\n\n.. code:: python \n\n _________________________________________________________________\n Layer (type) Output Shape Param # \n =================================================================\n embedding_1 (Embedding) (None, 500, 50) 8960500 \n _________________________________________________________________\n gru_1 (GRU) (None, 500, 256) 235776 \n _________________________________________________________________\n dropout_1 (Dropout) (None, 500, 256) 0 \n _________________________________________________________________\n gru_2 (GRU) (None, 500, 256) 393984 \n _________________________________________________________________\n dropout_2 (Dropout) (None, 500, 256) 0 \n _________________________________________________________________\n gru_3 (GRU) (None, 500, 256) 393984 \n _________________________________________________________________\n dropout_3 (Dropout) (None, 500, 256) 0 \n _________________________________________________________________\n gru_4 (GRU) (None, 256) 393984 \n _________________________________________________________________\n dense_1 (Dense) (None, 20) 5140 \n =================================================================\n Total params: 10,383,368\n Trainable params: 10,383,368\n Non-trainable params: 0\n _________________________________________________________________\n\n\n\nOutput:\n\n.. code:: python \n\n Train on 11314 samples, validate on 7532 samples\n Epoch 1/20\n - 268s - loss: 2.5347 - acc: 0.1792 - val_loss: 2.2857 - val_acc: 0.2460\n Epoch 2/20\n - 271s - loss: 1.6751 - acc: 0.3999 - val_loss: 1.4972 - val_acc: 0.4660\n Epoch 3/20\n - 270s - loss: 1.0945 - acc: 0.6072 - val_loss: 1.3232 - val_acc: 0.5483\n Epoch 4/20\n - 269s - loss: 0.7761 - acc: 0.7312 - val_loss: 1.1009 - val_acc: 0.6452\n Epoch 5/20\n - 269s - loss: 0.5513 - acc: 0.8112 - val_loss: 1.0395 - val_acc: 0.6832\n Epoch 6/20\n - 269s - loss: 0.3765 - acc: 0.8754 - val_loss: 0.9977 - val_acc: 0.7086\n Epoch 7/20\n - 270s - loss: 0.2481 - acc: 0.9202 - val_loss: 1.0485 - val_acc: 0.7270\n Epoch 8/20\n - 269s - loss: 0.1717 - acc: 0.9463 - val_loss: 1.0269 - val_acc: 0.7394\n Epoch 9/20\n - 269s - loss: 0.1130 - acc: 0.9644 - val_loss: 1.1498 - val_acc: 0.7369\n Epoch 10/20\n - 269s - loss: 0.0640 - acc: 0.9808 - val_loss: 1.1442 - val_acc: 0.7508\n Epoch 11/20\n - 269s - loss: 0.0567 - acc: 0.9828 - val_loss: 1.2318 - val_acc: 0.7414\n Epoch 12/20\n - 268s - loss: 0.0472 - acc: 0.9858 - val_loss: 1.2204 - val_acc: 0.7496\n Epoch 13/20\n - 269s - loss: 0.0319 - acc: 0.9910 - val_loss: 1.1895 - val_acc: 0.7657\n Epoch 14/20\n - 268s - loss: 0.0466 - acc: 0.9853 - val_loss: 1.2821 - val_acc: 0.7517\n Epoch 15/20\n - 271s - loss: 0.0269 - acc: 0.9917 - val_loss: 1.2869 - val_acc: 0.7557\n Epoch 16/20\n - 271s - loss: 0.0187 - acc: 0.9950 - val_loss: 1.3037 - val_acc: 0.7598\n Epoch 17/20\n - 268s - loss: 0.0157 - acc: 0.9959 - val_loss: 1.2974 - val_acc: 0.7638\n Epoch 18/20\n - 270s - loss: 0.0121 - acc: 0.9966 - val_loss: 1.3526 - val_acc: 0.7602\n Epoch 19/20\n - 269s - loss: 0.0262 - acc: 0.9926 - val_loss: 1.4182 - val_acc: 0.7517\n Epoch 20/20\n - 269s - loss: 0.0249 - acc: 0.9918 - val_loss: 1.3453 - val_acc: 0.7638\n\n\n precision recall f1-score support\n\n 0 0.71 0.71 0.71 319\n 1 0.72 0.68 0.70 389\n 2 0.76 0.62 0.69 394\n 3 0.67 0.58 0.62 392\n 4 0.68 0.67 0.68 385\n 5 0.75 0.73 0.74 395\n 6 0.82 0.74 0.78 390\n 7 0.83 0.83 0.83 396\n 8 0.81 0.90 0.86 398\n 9 0.92 0.90 0.91 397\n 10 0.91 0.94 0.93 399\n 11 0.87 0.76 0.81 396\n 12 0.57 0.70 0.63 393\n 13 0.81 0.85 0.83 396\n 14 0.74 0.93 0.82 394\n 15 0.82 0.83 0.83 398\n 16 0.74 0.78 0.76 364\n 17 0.96 0.83 0.89 376\n 18 0.64 0.60 0.62 310\n 19 0.48 0.56 0.52 251\n\n avg / total 0.77 0.76 0.76 7532\n\n-----------------------------------------\nConvolutional Neural Networks (CNN)\n-----------------------------------------\n\nAnother deep learning architecture that is employed for hierarchical document classification is Convolutional Neural Networks (CNN) . Although originally built for image processing with architecture similar to the visual cortex, CNNs have also been effectively used for text classification. In a basic CNN for image processing, an image tensor is convolved with a set of kernels of size *d by d*. These convolution layers are called feature maps and can be stacked to provide multiple filters on the input. To reduce the computational complexity, CNNs use pooling which reduces the size of the output from one layer to the next in the network. Different pooling techniques are used to reduce outputs while preserving important features.\n\nThe most common pooling method is max pooling where the maximum element is selected from the pooling window. In order to feed the pooled output from stacked featured maps to the next layer, the maps are flattened into one column. The final layers in a CNN are typically fully connected dense layers.\nIn general, during the back-propagation step of a convolutional neural network not only the weights are adjusted but also the feature detector filters. A potential problem of CNN used for text is the number of 'channels', *Sigma* (size of the feature space). This might be very large (e.g. 50K), for text but for images this is less of a problem (e.g. only 3 channels of RGB). This means the dimensionality of the CNN for text is very high.\n\n\n.. image:: docs/pic/CNN.png\n\nimport packages:\n\n.. code:: python\n\n\n from keras.layers import Dropout, Dense,Input,Embedding,Flatten, MaxPooling1D, Conv1D\n from keras.models import Sequential,Model\n from sklearn.feature_extraction.text import TfidfVectorizer\n import numpy as np\n from sklearn import metrics\n from keras.preprocessing.text import Tokenizer\n from keras.preprocessing.sequence import pad_sequences\n from sklearn.datasets import fetch_20newsgroups\n from keras.layers.merge import Concatenate\n\n\n\nconvert text to word embedding (Using GloVe):\n\n.. code:: python\n\n def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):\n np.random.seed(7)\n text = np.concatenate((X_train, X_test), axis=0)\n text = np.array(text)\n tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n tokenizer.fit_on_texts(text)\n sequences = tokenizer.texts_to_sequences(text)\n word_index = tokenizer.word_index\n text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n print('Found %s unique tokens.' % len(word_index))\n indices = np.arange(text.shape[0])\n # np.random.shuffle(indices)\n text = text[indices]\n print(text.shape)\n X_train = text[0:len(X_train), ]\n X_test = text[len(X_train):, ]\n embeddings_index = {}\n f = open(\".\\\\Glove\\\\glove.6B.50d.txt\", encoding=\"utf8\")\n for line in f:\n values = line.split()\n word = values[0]\n try:\n coefs = np.asarray(values[1:], dtype='float32')\n except:\n pass\n embeddings_index[word] = coefs\n f.close()\n print('Total %s word vectors.' % len(embeddings_index))\n return (X_train, X_test, word_index,embeddings_index)\n\n\nBuild a CNN Model for Text:\n\n.. code:: python\n\n def Build_Model_CNN_Text(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):\n\n \"\"\"\n def buildModel_CNN(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):\n word_index in word index ,\n embeddings_index is embeddings index, look at data_helper.py\n nClasses is number of classes,\n MAX_SEQUENCE_LENGTH is maximum lenght of text sequences,\n EMBEDDING_DIM is an int value for dimention of word embedding look at data_helper.py\n \"\"\"\n\n model = Sequential()\n embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))\n for word, i in word_index.items():\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None:\n # words not found in embedding index will be all-zeros.\n if len(embedding_matrix[i]) !=len(embedding_vector):\n print(\"could not broadcast input array from shape\",str(len(embedding_matrix[i])),\n \"into shape\",str(len(embedding_vector)),\" Please make sure your\"\n \" EMBEDDING_DIM is equal to embedding_vector file ,GloVe,\")\n exit(1)\n\n embedding_matrix[i] = embedding_vector\n\n embedding_layer = Embedding(len(word_index) + 1,\n EMBEDDING_DIM,\n weights=[embedding_matrix],\n input_length=MAX_SEQUENCE_LENGTH,\n trainable=True)\n\n # applying a more complex convolutional approach\n convs = []\n filter_sizes = []\n layer = 5\n print(\"Filter \",layer)\n for fl in range(0,layer):\n filter_sizes.append((fl+2))\n\n node = 128\n sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n embedded_sequences = embedding_layer(sequence_input)\n\n for fsz in filter_sizes:\n l_conv = Conv1D(node, kernel_size=fsz, activation='relu')(embedded_sequences)\n l_pool = MaxPooling1D(5)(l_conv)\n #l_pool = Dropout(0.25)(l_pool)\n convs.append(l_pool)\n\n l_merge = Concatenate(axis=1)(convs)\n l_cov1 = Conv1D(node, 5, activation='relu')(l_merge)\n l_cov1 = Dropout(dropout)(l_cov1)\n l_pool1 = MaxPooling1D(5)(l_cov1)\n l_cov2 = Conv1D(node, 5, activation='relu')(l_pool1)\n l_cov2 = Dropout(dropout)(l_cov2)\n l_pool2 = MaxPooling1D(30)(l_cov2)\n l_flat = Flatten()(l_pool2)\n l_dense = Dense(1024, activation='relu')(l_flat)\n l_dense = Dropout(dropout)(l_dense)\n l_dense = Dense(512, activation='relu')(l_dense)\n l_dense = Dropout(dropout)(l_dense)\n preds = Dense(nclasses, activation='softmax')(l_dense)\n model = Model(sequence_input, preds)\n\n model.compile(loss='sparse_categorical_crossentropy',\n optimizer='adam',\n metrics=['accuracy'])\n\n\n\n return model\n\n\n\nrun CNN and see our result:\n\n\n.. code:: python\n\n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)\n\n\n model_CNN = Build_Model_CNN_Text(word_index,embeddings_index, 20)\n\n\n model_CNN.summary()\n\n model_CNN.fit(X_train_Glove, y_train,\n validation_data=(X_test_Glove, y_test),\n epochs=15,\n batch_size=128,\n verbose=2)\n\n predicted = model_CNN.predict(X_test_Glove)\n\n predicted = np.argmax(predicted, axis=1)\n\n\n print(metrics.classification_report(y_test, predicted))\n\n\nModel:\n\n.. code:: python \n\n __________________________________________________________________________________________________\n Layer (type) Output Shape Param # Connected to \n ==================================================================================================\n input_1 (InputLayer) (None, 500) 0 \n __________________________________________________________________________________________________\n embedding_1 (Embedding) (None, 500, 50) 8960500 input_1[0][0] \n __________________________________________________________________________________________________\n conv1d_1 (Conv1D) (None, 499, 128) 12928 embedding_1[0][0] \n __________________________________________________________________________________________________\n conv1d_2 (Conv1D) (None, 498, 128) 19328 embedding_1[0][0] \n __________________________________________________________________________________________________\n conv1d_3 (Conv1D) (None, 497, 128) 25728 embedding_1[0][0] \n __________________________________________________________________________________________________\n conv1d_4 (Conv1D) (None, 496, 128) 32128 embedding_1[0][0] \n __________________________________________________________________________________________________\n conv1d_5 (Conv1D) (None, 495, 128) 38528 embedding_1[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_1 (MaxPooling1D) (None, 99, 128) 0 conv1d_1[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_2 (MaxPooling1D) (None, 99, 128) 0 conv1d_2[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_3 (MaxPooling1D) (None, 99, 128) 0 conv1d_3[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_4 (MaxPooling1D) (None, 99, 128) 0 conv1d_4[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_5 (MaxPooling1D) (None, 99, 128) 0 conv1d_5[0][0] \n __________________________________________________________________________________________________\n concatenate_1 (Concatenate) (None, 495, 128) 0 max_pooling1d_1[0][0] \n max_pooling1d_2[0][0] \n max_pooling1d_3[0][0] \n max_pooling1d_4[0][0] \n max_pooling1d_5[0][0] \n __________________________________________________________________________________________________\n conv1d_6 (Conv1D) (None, 491, 128) 82048 concatenate_1[0][0] \n __________________________________________________________________________________________________\n dropout_1 (Dropout) (None, 491, 128) 0 conv1d_6[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_6 (MaxPooling1D) (None, 98, 128) 0 dropout_1[0][0] \n __________________________________________________________________________________________________\n conv1d_7 (Conv1D) (None, 94, 128) 82048 max_pooling1d_6[0][0] \n __________________________________________________________________________________________________\n dropout_2 (Dropout) (None, 94, 128) 0 conv1d_7[0][0] \n __________________________________________________________________________________________________\n max_pooling1d_7 (MaxPooling1D) (None, 3, 128) 0 dropout_2[0][0] \n __________________________________________________________________________________________________\n flatten_1 (Flatten) (None, 384) 0 max_pooling1d_7[0][0] \n __________________________________________________________________________________________________\n dense_1 (Dense) (None, 1024) 394240 flatten_1[0][0] \n __________________________________________________________________________________________________\n dropout_3 (Dropout) (None, 1024) 0 dense_1[0][0] \n __________________________________________________________________________________________________\n dense_2 (Dense) (None, 512) 524800 dropout_3[0][0] \n __________________________________________________________________________________________________\n dropout_4 (Dropout) (None, 512) 0 dense_2[0][0] \n __________________________________________________________________________________________________\n dense_3 (Dense) (None, 20) 10260 dropout_4[0][0] \n ==================================================================================================\n Total params: 10,182,536\n Trainable params: 10,182,536\n Non-trainable params: 0\n __________________________________________________________________________________________________\n\n\nOutput:\n\n\n.. code:: python \n\n Train on 11314 samples, validate on 7532 samples\n Epoch 1/15\n - 6s - loss: 2.9329 - acc: 0.0783 - val_loss: 2.7628 - val_acc: 0.1403\n Epoch 2/15\n - 4s - loss: 2.2534 - acc: 0.2249 - val_loss: 2.1715 - val_acc: 0.4007\n Epoch 3/15\n - 4s - loss: 1.5643 - acc: 0.4326 - val_loss: 1.7846 - val_acc: 0.5052\n Epoch 4/15\n - 4s - loss: 1.1771 - acc: 0.5662 - val_loss: 1.4949 - val_acc: 0.6131\n Epoch 5/15\n - 4s - loss: 0.8880 - acc: 0.6797 - val_loss: 1.3629 - val_acc: 0.6256\n Epoch 6/15\n - 4s - loss: 0.6990 - acc: 0.7569 - val_loss: 1.2013 - val_acc: 0.6624\n Epoch 7/15\n - 4s - loss: 0.5037 - acc: 0.8200 - val_loss: 1.0674 - val_acc: 0.6807\n Epoch 8/15\n - 4s - loss: 0.4050 - acc: 0.8626 - val_loss: 1.0223 - val_acc: 0.6863\n Epoch 9/15\n - 4s - loss: 0.2952 - acc: 0.8968 - val_loss: 0.9045 - val_acc: 0.7120\n Epoch 10/15\n - 4s - loss: 0.2314 - acc: 0.9217 - val_loss: 0.8574 - val_acc: 0.7326\n Epoch 11/15\n - 4s - loss: 0.1778 - acc: 0.9436 - val_loss: 0.8752 - val_acc: 0.7270\n Epoch 12/15\n - 4s - loss: 0.1475 - acc: 0.9524 - val_loss: 0.8299 - val_acc: 0.7355\n Epoch 13/15\n - 4s - loss: 0.1089 - acc: 0.9657 - val_loss: 0.8034 - val_acc: 0.7491\n Epoch 14/15\n - 4s - loss: 0.1047 - acc: 0.9666 - val_loss: 0.8172 - val_acc: 0.7463\n Epoch 15/15\n - 4s - loss: 0.0749 - acc: 0.9774 - val_loss: 0.8511 - val_acc: 0.7313\n \n \n precision recall f1-score support\n\n 0 0.75 0.61 0.67 319\n 1 0.63 0.74 0.68 389\n 2 0.74 0.54 0.62 394\n 3 0.49 0.76 0.60 392\n 4 0.60 0.70 0.64 385\n 5 0.79 0.57 0.66 395\n 6 0.73 0.76 0.74 390\n 7 0.83 0.74 0.78 396\n 8 0.86 0.88 0.87 398\n 9 0.95 0.78 0.86 397\n 10 0.93 0.93 0.93 399\n 11 0.92 0.77 0.84 396\n 12 0.55 0.72 0.62 393\n 13 0.76 0.85 0.80 396\n 14 0.86 0.83 0.84 394\n 15 0.91 0.73 0.81 398\n 16 0.75 0.65 0.70 364\n 17 0.95 0.86 0.90 376\n 18 0.60 0.49 0.54 310\n 19 0.37 0.60 0.46 251\n\n avg / total 0.76 0.73 0.74 7532\n\n\n\n\n-----------------------------------------\nHierarchical Attention Networks\n-----------------------------------------\n\n.. image:: docs/pic/HAN.png\n\n---------------------------------------------\nRecurrent Convolutional Neural Networks (RCNN)\n---------------------------------------------\n\nRecurrent Convolutional Neural Networks (RCNN) is also used for text classification. The main idea of this technique is capturing contextual information with the recurrent structure and constructing the representation of text using a convolutional neural network. This architecture is a combination of RNN and CNN to use advantages of both technique in a model.\n\n\n\nimport packages:\n\n.. code:: python \n\n from keras.preprocessing import sequence\n from keras.models import Sequential\n from keras.layers import Dense, Dropout, Activation\n from keras.layers import Embedding\n from keras.layers import GRU\n from keras.layers import Conv1D, MaxPooling1D\n from keras.datasets import imdb\n from sklearn.datasets import fetch_20newsgroups\n import numpy as np\n from sklearn import metrics\n from keras.preprocessing.text import Tokenizer\n from keras.preprocessing.sequence import pad_sequences\n\n\n\nConvert text to word embedding (Using GloVe):\n\n.. code:: python \n\n def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):\n np.random.seed(7)\n text = np.concatenate((X_train, X_test), axis=0)\n text = np.array(text)\n tokenizer = Tokenizer(num_words=MAX_NB_WORDS)\n tokenizer.fit_on_texts(text)\n sequences = tokenizer.texts_to_sequences(text)\n word_index = tokenizer.word_index\n text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)\n print('Found %s unique tokens.' % len(word_index))\n indices = np.arange(text.shape[0])\n # np.random.shuffle(indices)\n text = text[indices]\n print(text.shape)\n X_train = text[0:len(X_train), ]\n X_test = text[len(X_train):, ]\n embeddings_index = {}\n f = open(\"C:\\\\Users\\\\kamran\\\\Documents\\\\GitHub\\\\RMDL\\\\Examples\\\\Glove\\\\glove.6B.50d.txt\", encoding=\"utf8\")\n for line in f:\n values = line.split()\n word = values[0]\n try:\n coefs = np.asarray(values[1:], dtype='float32')\n except:\n pass\n embeddings_index[word] = coefs\n f.close()\n print('Total %s word vectors.' % len(embeddings_index))\n return (X_train, X_test, word_index,embeddings_index)\n\n\n.. code:: python \n\n def Build_Model_RCNN_Text(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50):\n\n kernel_size = 2\n filters = 256\n pool_size = 2\n gru_node = 256\n\n embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))\n for word, i in word_index.items():\n embedding_vector = embeddings_index.get(word)\n if embedding_vector is not None:\n # words not found in embedding index will be all-zeros.\n if len(embedding_matrix[i]) !=len(embedding_vector):\n print(\"could not broadcast input array from shape\",str(len(embedding_matrix[i])),\n \"into shape\",str(len(embedding_vector)),\" Please make sure your\"\n \" EMBEDDING_DIM is equal to embedding_vector file ,GloVe,\")\n exit(1)\n\n embedding_matrix[i] = embedding_vector\n\n\n\n model = Sequential()\n model.add(Embedding(len(word_index) + 1,\n EMBEDDING_DIM,\n weights=[embedding_matrix],\n input_length=MAX_SEQUENCE_LENGTH,\n trainable=True))\n model.add(Dropout(0.25))\n model.add(Conv1D(filters, kernel_size, activation='relu'))\n model.add(MaxPooling1D(pool_size=pool_size))\n model.add(Conv1D(filters, kernel_size, activation='relu'))\n model.add(MaxPooling1D(pool_size=pool_size))\n model.add(Conv1D(filters, kernel_size, activation='relu'))\n model.add(MaxPooling1D(pool_size=pool_size))\n model.add(Conv1D(filters, kernel_size, activation='relu'))\n model.add(MaxPooling1D(pool_size=pool_size))\n model.add(LSTM(gru_node, return_sequences=True, recurrent_dropout=0.2))\n model.add(LSTM(gru_node, return_sequences=True, recurrent_dropout=0.2))\n model.add(LSTM(gru_node, return_sequences=True, recurrent_dropout=0.2))\n model.add(LSTM(gru_node, recurrent_dropout=0.2))\n model.add(Dense(1024,activation='relu'))\n model.add(Dense(nclasses))\n model.add(Activation('softmax'))\n\n model.compile(loss='sparse_categorical_crossentropy',\n optimizer='adam',\n metrics=['accuracy'])\n\n return model\n\n\n.. code:: python \n\n newsgroups_train = fetch_20newsgroups(subset='train')\n newsgroups_test = fetch_20newsgroups(subset='test')\n X_train = newsgroups_train.data\n X_test = newsgroups_test.data\n y_train = newsgroups_train.target\n y_test = newsgroups_test.target\n\n X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)\n\n\nRun RCNN :\n\n\n.. code:: python \n\n\n model_RCNN = Build_Model_CNN_Text(word_index,embeddings_index, 20)\n\n\n model_RCNN.summary()\n\n model_RCNN.fit(X_train_Glove, y_train,\n validation_data=(X_test_Glove, y_test),\n epochs=15,\n batch_size=128,\n verbose=2)\n\n predicted = model_RCNN.predict(X_test_Glove)\n\n predicted = np.argmax(predicted, axis=1)\n print(metrics.classification_report(y_test, predicted))\n\n\nsummary of the model:\n\n\n.. code:: python \n\n _________________________________________________________________\n Layer (type) Output Shape Param # \n =================================================================\n embedding_1 (Embedding) (None, 500, 50) 8960500 \n _________________________________________________________________\n dropout_1 (Dropout) (None, 500, 50) 0 \n _________________________________________________________________\n conv1d_1 (Conv1D) (None, 499, 256) 25856 \n _________________________________________________________________\n max_pooling1d_1 (MaxPooling1 (None, 249, 256) 0 \n _________________________________________________________________\n conv1d_2 (Conv1D) (None, 248, 256) 131328 \n _________________________________________________________________\n max_pooling1d_2 (MaxPooling1 (None, 124, 256) 0 \n _________________________________________________________________\n conv1d_3 (Conv1D) (None, 123, 256) 131328 \n _________________________________________________________________\n max_pooling1d_3 (MaxPooling1 (None, 61, 256) 0 \n _________________________________________________________________\n conv1d_4 (Conv1D) (None, 60, 256) 131328 \n _________________________________________________________________\n max_pooling1d_4 (MaxPooling1 (None, 30, 256) 0 \n _________________________________________________________________\n lstm_1 (LSTM) (None, 30, 256) 525312 \n _________________________________________________________________\n lstm_2 (LSTM) (None, 30, 256) 525312 \n _________________________________________________________________\n lstm_3 (LSTM) (None, 30, 256) 525312 \n _________________________________________________________________\n lstm_4 (LSTM) (None, 256) 525312 \n _________________________________________________________________\n dense_1 (Dense) (None, 1024) 263168 \n _________________________________________________________________\n dense_2 (Dense) (None, 20) 20500 \n _________________________________________________________________\n activation_1 (Activation) (None, 20) 0 \n =================================================================\n Total params: 11,765,256\n Trainable params: 11,765,256\n Non-trainable params: 0\n _________________________________________________________________\n\n\n\nOutput:\n\n.. code:: python \n\n Train on 11314 samples, validate on 7532 samples\n Epoch 1/15\n - 28s - loss: 2.6624 - acc: 0.1081 - val_loss: 2.3012 - val_acc: 0.1753\n Epoch 2/15\n - 22s - loss: 2.1142 - acc: 0.2224 - val_loss: 1.9168 - val_acc: 0.2669\n Epoch 3/15\n - 22s - loss: 1.7465 - acc: 0.3290 - val_loss: 1.8257 - val_acc: 0.3412\n Epoch 4/15\n - 22s - loss: 1.4730 - acc: 0.4356 - val_loss: 1.5433 - val_acc: 0.4436\n Epoch 5/15\n - 22s - loss: 1.1800 - acc: 0.5556 - val_loss: 1.2973 - val_acc: 0.5467\n Epoch 6/15\n - 22s - loss: 0.9910 - acc: 0.6281 - val_loss: 1.2530 - val_acc: 0.5797\n Epoch 7/15\n - 22s - loss: 0.8581 - acc: 0.6854 - val_loss: 1.1522 - val_acc: 0.6281\n Epoch 8/15\n - 22s - loss: 0.7058 - acc: 0.7428 - val_loss: 1.2385 - val_acc: 0.6033\n Epoch 9/15\n - 22s - loss: 0.6792 - acc: 0.7515 - val_loss: 1.0200 - val_acc: 0.6775\n Epoch 10/15\n - 22s - loss: 0.5782 - acc: 0.7948 - val_loss: 1.0961 - val_acc: 0.6577\n Epoch 11/15\n - 23s - loss: 0.4674 - acc: 0.8341 - val_loss: 1.0866 - val_acc: 0.6924\n Epoch 12/15\n - 23s - loss: 0.4284 - acc: 0.8512 - val_loss: 0.9880 - val_acc: 0.7096\n Epoch 13/15\n - 22s - loss: 0.3883 - acc: 0.8670 - val_loss: 1.0190 - val_acc: 0.7151\n Epoch 14/15\n - 22s - loss: 0.3334 - acc: 0.8874 - val_loss: 1.0025 - val_acc: 0.7232\n Epoch 15/15\n - 22s - loss: 0.2857 - acc: 0.9038 - val_loss: 1.0123 - val_acc: 0.7331\n\n\n precision recall f1-score support\n\n 0 0.64 0.73 0.68 319\n 1 0.45 0.83 0.58 389\n 2 0.81 0.64 0.71 394\n 3 0.64 0.57 0.61 392\n 4 0.55 0.78 0.64 385\n 5 0.77 0.52 0.62 395\n 6 0.84 0.77 0.80 390\n 7 0.87 0.79 0.83 396\n 8 0.85 0.90 0.87 398\n 9 0.98 0.84 0.90 397\n 10 0.93 0.96 0.95 399\n 11 0.92 0.79 0.85 396\n 12 0.59 0.53 0.56 393\n 13 0.82 0.82 0.82 396\n 14 0.84 0.84 0.84 394\n 15 0.83 0.89 0.86 398\n 16 0.68 0.86 0.76 364\n 17 0.97 0.86 0.91 376\n 18 0.66 0.50 0.57 310\n 19 0.53 0.31 0.40 251\n\n avg / total 0.77 0.75 0.75 7532\n\n\n\n-----------------------------------------\nRandom Multimodel Deep Learning (RMDL)\n-----------------------------------------\n\n\nReferenced paper : `RMDL: Random Multimodel Deep Learning for\nClassification \u003chttps://www.researchgate.net/publication/324922651_RMDL_Random_Multimodel_Deep_Learning_for_Classification\u003e`__\n\n\nA new ensemble, deep learning approach for classification. Deep\nlearning models have achieved state-of-the-art results across many domains.\nRMDL solves the problem of finding the best deep learning structure\nand architecture while simultaneously improving robustness and accuracy\nthrough ensembles of different deep learning architectures. RDMLs can accept\na variety of data as input including text, video, images, and symbols.\n\n\n|RMDL|\n\nRandom Multimodel Deep Learning (RDML) architecture for classification.\nRMDL includes 3 Random models, oneDNN classifier at left, one Deep CNN\nclassifier at middle, and one Deep RNN classifier at right (each unit could be LSTMor GRU).\n\n\nInstallation\n\nThere are pip and git for RMDL installation:\n\nUsing pip\n\n\n.. code:: python\n\n pip install RMDL\n\nUsing git\n\n.. code:: bash\n\n git clone --recursive https://github.com/kk7nc/RMDL.git\n\nThe primary requirements for this package are Python 3 with Tensorflow. The requirements.txt file\ncontains a listing of the required `Python packages \u003chttps://www.scaler.com/topics/python/python-packages/\u003e`__ to install all requirements, run the following:\n\n.. code:: bash\n\n pip -r install requirements.txt\n\nOr\n\n.. code:: bash\n\n pip3 install -r requirements.txt\n\nOr:\n\n.. code:: bash\n\n conda install --file requirements.txt\n\nDocumentation:\n\n\nThe exponential growth in the number of complex datasets every year requires more enhancement in\nmachine learning methods to provide robust and accurate data classification. Lately, deep learning\napproaches are achieving better results compared to previous machine learning algorithms\non tasks like image classification, natural language processing, face recognition, and etc. The\nsuccess of these deep learning algorithms rely on their capacity to model complex and non-linear\nrelationships within the data. However, finding suitable structures for these models has been a challenge\nfor researchers. This paper introduces Random Multimodel Deep Learning (RMDL): a new ensemble, deep learning\napproach for classification. RMDL aims to solve the problem of finding the best deep learning architecture while simultaneously improving the robustness and accuracy through ensembles of multiple deep\nlearning architectures. In short, RMDL trains multiple models of Deep Neural Networks (DNN),\nConvolutional Neural Network (CNN) and Recurrent Neural Network (RNN) in parallel and combine\ntheir results to produce the better results of any of those models individually. To create these models,\neach deep learning model has been constructed in a random fashion regarding the number of layers and\nnodes in their neural network structure. The resulting RDML model can be used in various domains such\nas text, video, images, and symbolism. In this Project, we describe the RMDL model in depth and show the results\nfor image and text classification as well as face recognition. For image classification, we compared our\nmodel with some of the available baselines using MNIST and CIFAR-10 datasets. Similarly, we used four\ndatasets namely, WOS, Reuters, IMDB, and 20newsgroup, and compared our results with available baselines.\nWeb of Science (WOS) has been collected by authors and consists of three sets~(small, medium, and large sets).\nLastly, we used ORL dataset to compare the performance of our approach with other face recognition methods.\nThese test results show that the RDML model consistently outperforms standard methods over a broad range of\ndata types and classification problems.\n\n--------------------------------------------\nHierarchical Deep Learning for Text (HDLTex)\n--------------------------------------------\n\nRefrenced paper : `HDLTex: Hierarchical Deep Learning for Text\nClassification \u003chttps://arxiv.org/abs/1709.08267\u003e`__\n\n\n|HDLTex|\n\nDocumentation:\n\nIncreasingly large document collections require improved information processing methods for searching, retrieving, and organizing text documents. Central to these information processing methods is document classification, which has become an important task supervised learning aims to solve. Recently, the performance of traditional supervised classifiers has degraded as the number of documents has increased. This exponential growth of document volume has also increated the number of categories. This paper approaches this problem differently from current document classification methods that view the problem as multi-class classification. Instead we perform hierarchical classification using an approach we call Hierarchical Deep Learning for Text classification (HDLTex). HDLTex employs stacks of deep learning architectures to provide hierarchical understanding of the documents.\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nComparison Text Classification Algorithms\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Model** | **Advantages** | **Disadvantages** |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Rocchio Algorithm** | * Easy to implement | * The user can only retrieve a few relevant documents |\n| | | |\n| | * Computationally is very cheap | * Rocchio often misclassifies the type for multimodal class |\n| | | |\n| | * Relevance feedback mechanism (benefits to ranking documents as not relevant) | * This technique is not very robust |\n| | | |\n| | | * linear combination in this algorithm is not good for multi-class datasets |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Boosting and Bagging** | * Improves the stability and accuracy (takes the advantage of ensemble learning where in multiple weak learner outperform a single strong learner.) | * Computational complexity |\n| | | |\n| | * Reducing variance which helps to avoid overfitting problems. | * loss of interpretability (if the number of models is hight, understanding the model is very difficult) |\n| | | |\n| | | * Requires careful tuning of different hyper-parameters. |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Logistic Regression** | * Easy to implement | * it cannot solve non-linear problems |\n| | | |\n| | * does not require too many computational resources | * prediction requires that each data point be independent |\n| | | |\n| | * it does not require input features to be scaled (pre-processing) | * attempting to predict outcomes based on a set of independent variables |\n| | | |\n| | * It does not require any tuning | |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Naive Bayes Classifier** | * It works very well with text data | * A strong assumption about the shape of the data distribution |\n| | | |\n| | * Easy to implement | * limited by data scarcity for which any possible value in feature space, a likelihood value must be estimated by a frequentist |\n| | | |\n| | * Fast in comparing to other algorithms | |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **K-Nearest Neighbor** | * Effective for text datasets | * computational of this model is very expensive |\n| | | |\n| | * non-parametric | * diffcult to find optimal value of k |\n| | | |\n| | * More local characteristics of text or document are considered | * Constraint for large search problem to find nearest neighbors |\n| | | |\n| | * Naturally handles multi-class datasets | * Finding a meaningful distance function is difficult for text datasets |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Support Vector Machine (SVM)** | * SVM can model non-linear decision boundaries | * lack of transparency in results caused by a high number of dimensions (especially for text data). |\n| | | |\n| | * Performs similarly to logistic regression when linear separation | * Choosing an efficient kernel function is difficult (Susceptible to overfitting/training issues depending on kernel) |\n| | | |\n| | * Robust against overfitting problems~(especially for text dataset due to high-dimensional space) | * Memory complexity |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Decision Tree** | * Can easily handle qualitative (categorical) features | * Issues with diagonal decision boundaries |\n| | | |\n| | * Works well with decision boundaries parellel to the feature axis | * Can be easily overfit |\n| | | |\n| | * Decision tree is a very fast algorithm for both learning and prediction | * extremely sensitive to small perturbations in the data |\n| | | |\n| | | * Problems with out-of-sample prediction |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Conditional Random Field (CRF)** | * Its feature design is flexible | * High computational complexity of the training step |\n| | | |\n| | * Since CRF computes the conditional probability of global optimal output nodes, it overcomes the drawbacks of label bias | * this algorithm does not perform with unknown words |\n| | | |\n| | * Combining the advantages of classification and graphical modeling which combining the ability to compactly model multivariate data | * Problem about online learning (It makes it very difficult to re-train the model when newer data becomes available.) |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Random Forest** | * Ensembles of decision trees are very fast to train in comparison to other techniques | * Quite slow to create predictions once trained |\n| | | |\n| | * Reduced variance (relative to regular trees) | * more trees in forest increases time complexity in the prediction step |\n| | | |\n| | * Not require preparation and pre-processing of the input data | * Not as easy to visually interpret |\n| | | |\n| | | * Overfitting can easily occur |\n| | | |\n| | | * Need to choose the number of trees at forest |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n| **Deep Learning** | * Flexible with features design (Reduces the need for feature engineering, one of the most time-consuming parts of machine learning practice.) | * Requires a large amount of data (if you only have small sample text data, deep learning is unlikely to outperform other approaches. |\n| | | |\n| | * Architecture that can be adapted to new problems | * Is extremely computationally expensive to train. |\n| | | |\n| | * Can deal with complex input-output mappings | * Model Interpretability is most important problem of deep learning~(Deep learning in most of the time is black-box) |\n| | | |\n| | * Can easily handle online learning (It makes it very easy to re-train the model when newer data becomes available.) | * Finding an efficient architecture and structure is still the main challenge of this technique |\n| | | |\n| | * Parallel processing capability (It can perform more than one job at the same time) | |\n+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\n\n\n\n==========\nEvaluation\n==========\n\n----\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nF1 Score\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\n.. image:: docs/pic/F1.png\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nMatthew correlation coefficient (MCC)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\nCompute the Matthews correlation coefficient (MCC)\n\nThe Matthews correlation coefficient is used in machine learning as a measure of the quality of binary (two-class) classification problems. It takes into account of true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient. \n\n\n.. code:: python\n\n from sklearn.metrics import matthews_corrcoef\n y_true = [+1, +1, +1, -1]\n y_pred = [+1, -1, +1, +1]\n matthews_corrcoef(y_true, y_pred) \n\n\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nReceiver operating characteristics (ROC)\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nROC curves are typically used in binary classification to study the output of a classifier. In order to extend ROC curve and ROC area to multi-class or multi-label classification, it is necessary to binarize the output. One ROC curve can be drawn per label, but one can also draw a ROC curve by considering each element of the label indicator matrix as a binary prediction (micro-averaging).\n\nAnother evaluation measure for multi-class classification is macro-averaging, which gives equal weight to the classification of each label. [`sources \u003chttp://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html\u003e`__] \n\n.. code:: python\n\n import numpy as np\n import matplotlib.pyplot as plt\n from itertools import cycle\n\n from sklearn import svm, datasets\n from sklearn.metrics import roc_curve, auc\n from sklearn.model_selection import train_test_split\n from sklearn.preprocessing import label_binarize\n from sklearn.multiclass import OneVsRestClassifier\n from scipy import interp\n\n # Import some data to play with\n iris = datasets.load_iris()\n X = iris.data\n y = iris.target\n\n # Binarize the output\n y = label_binarize(y, classes=[0, 1, 2])\n n_classes = y.shape[1]\n\n # Add noisy features to make the problem harder\n random_state = np.random.RandomState(0)\n n_samples, n_features = X.shape\n X = np.c_[X, random_state.randn(n_samples, 200 * n_features)]\n\n # shuffle and split training and test sets\n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.5,\n random_state=0)\n\n # Learn to predict each class against the other\n classifier = OneVsRestClassifier(svm.SVC(kernel='linear', probability=True,\n random_state=random_state))\n y_score = classifier.fit(X_train, y_train).decision_function(X_test)\n\n # Compute ROC curve and ROC area for each class\n fpr = dict()\n tpr = dict()\n roc_auc = dict()\n for i in range(n_classes):\n fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_score[:, i])\n roc_auc[i] = auc(fpr[i], tpr[i])\n\n # Compute micro-average ROC curve and ROC area\n fpr[\"micro\"], tpr[\"micro\"], _ = roc_curve(y_test.ravel(), y_score.ravel())\n roc_auc[\"micro\"] = auc(fpr[\"micro\"], tpr[\"micro\"])\n \n\n\nPlot of a ROC curve for a specific class\n\n\n.. code:: python\n\n plt.figure()\n lw = 2\n plt.plot(fpr[2], tpr[2], color='darkorange',\n lw=lw, label='ROC curve (area = %0.2f)' % roc_auc[2])\n plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n plt.xlim([0.0, 1.0])\n plt.ylim([0.0, 1.05])\n plt.xlabel('False Positive Rate')\n plt.ylabel('True Positive Rate')\n plt.title('Receiver operating characteristic example')\n plt.legend(loc=\"lower right\")\n plt.show()\n\n\n.. image:: /docs/pic/sphx_glr_plot_roc_001.png\n\n\n~~~~~~~~~~~~~~~~~~~~~~~\nArea Under Curve (AUC)\n~~~~~~~~~~~~~~~~~~~~~~~\n\nArea under ROC curve (AUC) is a summary metric that measures the entire area underneath the ROC curve. AUC holds helpful properties, such as increased sensitivity in the analysis of variance (ANOVA) tests, independence of decision threshold, invariance to a priori class probability and the indication of how well negative and positive classes are regarding decision index.\n\n\n.. code:: python\n\n import numpy as np\n from sklearn import metrics\n fpr, tpr, thresholds = metrics.roc_curve(y, pred, pos_label=2)\n metrics.auc(fpr, tpr)\n\n\n\n\n==========================\nText and Document Datasets\n==========================\n\n----\n\n~~~~~\nIMDB\n~~~~~\n\n- `IMDB Dataset \u003chttp://ai.stanford.edu/~amaas/data/sentiment/\u003e`__\n\nDataset of 25,000 movies reviews from IMDB, labeled by sentiment (positive/negative). Reviews have been preprocessed, and each review is encoded as a sequence of word indexes (integers). For convenience, words are indexed by overall frequency in the dataset, so that for instance the integer \"3\" encodes the 3rd most frequent word in the data. This allows for quick filtering operations, such as \"only consider the top 10,000 most common words, but eliminate the top 20 most common words\".\n\nAs a convention, \"0\" does not stand for a specific word, but instead is used to encode any unknown word.\n\n\n.. code:: python\n\n\n from keras.datasets import imdb\n\n (x_train, y_train), (x_test, y_test) = imdb.load_data(path=\"imdb.npz\",\n num_words=None,\n skip_top=0,\n maxlen=None,\n seed=113,\n start_char=1,\n oov_char=2,\n index_from=3)\n\n~~~~~~~~~~~~~\nReuters-21578\n~~~~~~~~~~~~~\n\n- `Reters-21578 Dataset \u003chttps://keras.io/datasets/\u003e`__\n\n\nDataset of 11,228 newswires from Reuters, labeled over 46 topics. As with the IMDB dataset, each wire is encoded as a sequence of word indexes (same conventions).\n\n\n.. code:: python\n\n from keras.datasets import reuters\n\n (x_train, y_train), (x_test, y_test) = reuters.load_data(path=\"reuters.npz\",\n num_words=None,\n skip_top=0,\n maxlen=None,\n test_split=0.2,\n seed=113,\n start_char=1,\n oov_char=2,\n index_from=3)\n \n \n~~~~~~~~~~~~~\n20Newsgroups\n~~~~~~~~~~~~~\n\n- `20Newsgroups Dataset \u003chttps://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups\u003e`__\n\nThe 20 newsgroups dataset comprises around 18000 newsgroups posts on 20 topics split in two subsets: one for training (or development) and the other one for testing (or for performance evaluation). The split between the train and test set is based upon messages posted before and after a specific date.\n\nThis module contains two loaders. The first one, sklearn.datasets.fetch_20newsgroups, returns a list of the raw texts that can be fed to text feature extractors, such as sklearn.feature_extraction.text.CountVectorizer with custom parameters so as to extract feature vectors. The second one, sklearn.datasets.fetch_20newsgroups_vectorized, returns ready-to-use features, i.e., it is not necessary to use a feature extractor.\n\n\n.. code:: python\n\n from sklearn.datasets import fetch_20newsgroups\n newsgroups_train = fetch_20newsgroups(subset='train')\n\n from pprint import pprint\n pprint(list(newsgroups_train.target_names))\n \n ['alt.atheism',\n 'comp.graphics',\n 'comp.os.ms-windows.misc',\n 'comp.sys.ibm.pc.hardware',\n 'comp.sys.mac.hardware',\n 'comp.windows.x',\n 'misc.forsale',\n 'rec.autos',\n 'rec.motorcycles',\n 'rec.sport.baseball',\n 'rec.sport.hockey',\n 'sci.crypt',\n 'sci.electronics',\n 'sci.med',\n 'sci.space',\n 'soc.religion.christian',\n 'talk.politics.guns',\n 'talk.politics.mideast',\n 'talk.politics.misc',\n 'talk.religion.misc']\n \n \n~~~~~~~~~~~~~~~~~~~~~~\nWeb of Science Dataset\n~~~~~~~~~~~~~~~~~~~~~~\n\nDescription of Dataset:\n\nHere is three datasets which include WOS-11967 , WOS-46985, and WOS-5736\nEach folder contains:\n\n- X.txt\n- Y.txt\n- YL1.txt\n- YL2.txt\n\nX is input data that include text sequences\nY is target value\nYL1 is target value of level one (parent label)\nYL2 is target value of level one (child label)\n\nMeta-data:\nThis folder contain on data file as following attribute:\nY1 Y2 Y Domain area keywords Abstract\n\nAbstract is input data that include text sequences of 46,985 published paper\nY is target value\nYL1 is target value of level one (parent label)\nYL2 is target value of level one (child label)\nDomain is majaor domain which include 7 labales: {Computer Science,Electrical Engineering, Psychology, Mechanical Engineering,Civil Engineering, Medical Science, biochemistry}\narea is subdomain or area of the paper, such as CS-\u003e computer graphics which contain 134 labels.\nkeywords : is authors keyword of the papers\n\n- Web of Science Dataset `WOS-11967 \u003chttp://dx.doi.org/10.17632/9rw3vkcfy4.2\u003e`__\n..\n\n This dataset contains 11,967 documents with 35 categories which include 7 parents categories.\n\n- Web of Science Dataset `WOS-46985 \u003chttp://dx.doi.org/10.17632/9rw3vkcfy4.2\u003e`__\n \n..\n\n This dataset contains 46,985 documents with 134 categories which include 7 parents categories.\n\n- Web of Science Dataset `WOS-5736 \u003chttp://dx.doi.org/10.17632/9rw3vkcfy4.2\u003e`__\n\n..\n \n This dataset contains 5,736 documents with 11 categories which include 3 parents categories.\n\nReferenced paper: HDLTex: Hierarchical Deep Learning for Text Classification\n \n================================\nText Classification Applications\n================================\n\n\n----\n\n\n\n~~~~~~~~~~~~~~~~~~~~~~\nInformation Retrieval\n~~~~~~~~~~~~~~~~~~~~~~\nInformation retrieval is finding documents of an unstructured data that meet an information need from within large collections of documents. With the rapid growth of online information, particularly in text format, text classification has become a significant technique for managing this type of data. Some of the important methods used in this area are Naive Bayes, SVM, decision tree, J48, k-NN and IBK. One of the most challenging applications for document and text dataset processing is applying document categorization methods for information retrieval.\n\n- 🎓 `Introduction to information retrieval \u003chttp://eprints.bimcoordinator.co.uk/35/\u003e`__ Manning, C., Raghavan, P., \u0026 Schütze, H. (2010).\n \n- 🎓 `Web forum retrieval and text analytics: A survey \u003chttp://www.nowpublishers.com/article/Details/INR-062\u003e`__ Hoogeveen, Doris, et al.. (2018).\n\n- 🎓 `Automatic Text Classification in Information retrieval: A Survey \u003chttps://dl.acm.org/citation.cfm?id=2905191\u003e`__ Dwivedi, Sanjay K., and Chandrakala Arya.. (2016).\n\n~~~~~~~~~~~~~~~~~~~~~~\nInformation Filtering\n~~~~~~~~~~~~~~~~~~~~~~\nInformation filtering refers to selection of relevant information or rejection of irrelevant information from a stream of incoming data. Information filtering systems are typically used to measure and forecast users' long-term interests. Probabilistic models, such as Bayesian inference network, are commonly used in information filtering systems. Bayesian inference networks employ recursive inference to propagate values through the inference network and return documents with the highest ranking. Chris used vector space model with iterative refinement for filtering task.\n \n\n- 🎓 `Search engines: Information retrieval in practice \u003chttp://library.mpib-berlin.mpg.de/toc/z2009_2465.pdf/\u003e`__ Croft, W. B., Metzler, D., \u0026 Strohman, T. (2010).\n\n- 🎓 `Implementation of the SMART information retrieval system \u003chttps://ecommons.cornell.edu/bitstream/handle/1813/6526/85-686.pdf?sequence=1\u003e`__ Buckley, Chris\n\n~~~~~~~~~~~~~~~~~~~~~~\nSentiment Analysis\n~~~~~~~~~~~~~~~~~~~~~~\nSentiment analysis is a computational approach toward identifying opinion, sentiment, and subjectivity in text. Sentiment classification methods classify a document associated with an opinion to be positive or negative. The assumption is that document d is expressing an opinion on a single entity e and opinions are formed via a single opinion holder h. Naive Bayesian classification and SVM are some of the most popular supervised learning methods that have been used for sentiment classification. Features such as terms and their respective frequency, part of speech, opinion words and phrases, negations and syntactic dependency have been used in sentiment classification techniques.\n\n- 🎓 `Opinion mining and sentiment analysis \u003chttp://www.nowpublishers.com/article/Details/INR-011\u003e`__ Pang, Bo, and Lillian Lee. (2008).\n\n- 🎓 `A survey of opinion mining and sentiment analysis \u003chttps://link.springer.com/chapter/10.1007/978-1-4614-3223-4_13\u003e`__ Liu, Bing, and Lei Zhang. (2010).\n\n- 🎓 `Thumbs up?: sentiment classification using machine learning techniques \u003chttps://dl.acm.org/citation.cfm?id=1118704\u003e`__ Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. \n\n~~~~~~~~~~~~~~~~~~~~~~\nRecommender Systems\n~~~~~~~~~~~~~~~~~~~~~~\nContent-based recommender systems suggest items to users based on the description of an item and a profile of the user's interests. \nA user's profile can be learned from user feedback (history of the search queries or self reports) on items as well as self-explained features~(filter or conditions on the queries) in one's profile. \nIn this way, input to such recommender systems can be semi-structured such that some attributes are extracted from free-text field while others are directly specified. Many different types of text classification methods, such as decision trees, nearest neighbor methods, Rocchio's algorithm, linear classifiers, probabilistic methods, and Naive Bayes, have been used to model user's preference.\n\n- 🎓 `Content-based recommender systems \u003chttps://link.springer.com/chapter/10.1007/978-3-319-29659-3_4\u003e`__ Aggarwal, Charu C. (2016).\n\n- 🎓 `Content-based recommendation systems \u003chttps://link.springer.com/chapter/10.1007/978-3-540-72079-9_10\u003e`__ Pazzani, Michael J., and Daniel Billsus.\n\n~~~~~~~~~~~~~~~~~~~~~~\nKnowledge Management\n~~~~~~~~~~~~~~~~~~~~~~\nTextual databases are significant sources of information and knowledge. A large percentage of corporate information (nearly 80 %) exists in textual data formats (unstructured). In knowledge distillation, patterns or knowledge are inferred from immediate forms that can be semi-structured ( e.g.conceptual graph representation) or structured/relational data representation). A given intermediate form can be document-based such that each entity represents an object or concept of interest in a particular domain. Document categorization is one of the most common methods for mining document-based intermediate forms. In the other work, text classification has been used to find the relationship between railroad accidents' causes and their correspondent descriptions in reports.\n\n- 🎓 `Text mining: concepts, applications, tools and issues-an overview \u003chttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.403.2426\u0026rep=rep1\u0026type=pdf\u003e`__ Sumathy, K. L., and M. Chidambaram. (2013).\n\n- 🎓 `Analysis of Railway Accidents' Narratives Using Deep Learning \u003chttps://ieeexplore.ieee.org/abstract/document/8614260/\u003e`__ Heidarysafa, Mojtaba, et al. (2018).\n\n~~~~~~~~~~~~~~~~~~~~~~\nDocument Summarization\n~~~~~~~~~~~~~~~~~~~~~~\nText classification used for document summarizing which summary of a document may employ words or phrases which do not appear in the original document. Multi-document summarization also is necessitated due to increasing online information rapidly. So, many researchers focus on this task using text classification to extract important feature out of a document.\n\n- 🎓 `Advances in automatic text summarization \u003chttps://books.google.com/books?hl=en\u0026lr=\u0026id=YtUZQaKDmzEC\u0026oi=fnd\u0026pg=PA215\u0026dq=Advances+in+automatic+text+summarization\u0026ots=ZpvCsrG-dC\u0026sig=8ecTDTrQR4mMzDnKvI58sowh3Fg\u003e`__ Mani, Inderjeet. \n\n- 🎓 `Improving Multi-Document Summarization via Text Classification. \u003chttps://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/viewPaper/14525\u003e`__ Cao, Ziqiang, et al. (2017).\n\n================================\nText Classification Support\n================================\n\n~~~~~~~~~~~~~~~~~~~~~~\nHealth\n~~~~~~~~~~~~~~~~~~~~~~\nMost textual information in the medical domain is presented in an unstructured or narrative form with ambiguous terms and typographical errors. Such information needs to be available instantly throughout the patient-physicians encounters in different stages of diagnosis and treatment. Medical coding, which consists of assigning medical diagnoses to specific class values obtained from a large set of categories, is an area of healthcare applications where text classification techniques can be highly valuable. In the other research, J. Zhang et al. introduced Patient2Vec, to learn an interpretable deep representation of longitudinal electronic health record (EHR) data which is personalized for each patient. Patient2Vec is a novel technique of text dataset feature embedding that can learn a personalized interpretable deep representation of EHR data based on recurrent neural networks and the attention mechanism. Text classification has also been applied in the development of Medical Subject Headings (MeSH) and Gene Ontology (GO). \n\n\n- 🎓 `Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record \u003chttps://ieeexplore.ieee.org/abstract/document/8490816/\u003e`__ Zhang, Jinghe, et al. (2018)\n\n- 🎓 `Combining Bayesian text classification and shrinkage to automate healthcare coding: A data quality analysis \u003chttps://dl.acm.org/citation.cfm?id=2063506\u003e`__ Lauría, Eitel JM, and Alan D. March. (2011).\n\n- 🎓 `A \u003chttp://b/\u003e`__ c. (2010).\n\n- 🎓 `MeSH Up: effective MeSH text classification for improved document retrieval \u003chttps://academic.oup.com/bioinformatics/article-abstract/25/11/1412/333120\u003e`__ Trieschnigg, Dolf, et al.\n\n~~~~~~~~~~~~~~~~~~~~~~\nSocial Sciences\n~~~~~~~~~~~~~~~~~~~~~~\nText classification and document categorization has increasingly been applied to understanding human behavior in past decades. Recent data-driven efforts in human behavior research have focused on mining language contained in informal notes and text datasets, including short message service (SMS), clinical notes, social media, etc. These studies have mostly focused on using approaches based on frequencies of word occurrence (i.e. how often a word appears in a document) or features based on Linguistic Inquiry Word Count (LIWC), a well-validated lexicon of categories of words with psychological relevance.\n\n- 🎓 `Identification of imminent suicide risk among young adults using text messages \u003chttps://dl.acm.org/citation.cfm?id=3173987\u003e`__ Nobles, Alicia L., et al. (2018).\n\n- 🎓 `Textual Emotion Classification: An Interoperability Study on Cross-Genre Data Sets \u003chttps://link.springer.com/chapter/10.1007/978-3-319-63004-5_21\u003e`__ Ofoghi, Bahadorreza, and Karin Verspoor. (2017).\n\n- 🎓 `Social Monitoring for Public Health \u003chttps://www.morganclaypool.com/doi/abs/10.2200/S00791ED1V01Y201707ICR060\u003e`__ Paul, Michael J., and Mark Dredze (2017).\n\n~~~~~~~~~~~~~~~~~~~~~~\nBusiness and Marketing\n~~~~~~~~~~~~~~~~~~~~~~\nprofitable companies and organizations are progressively using social media for marketing purposes. Opening mining from social media such as Facebook, Twitter, and so on is main target of companies to rapidly increase their profits. Text and documents classification is a powerful tool for companies to find their customers easier than ever. \n\n- 🎓 `Opinion mining using ensemble text hidden Markov models for text classification \u003chttps://www.sciencedirect.com/science/article/pii/S0957417417304979\u003e`__ Kang, Mangi, Jaelim Ahn, and Kichun Lee. (2018).\n\n- 🎓 `Classifying business marketing messages on Facebook \u003chttps://www.researchgate.net/profile/Bei_Yu2/publication/236246670_Classifying_Business_Marketing_Messages_on_Facebook/links/56bcb34408ae6cc737c6335b.pdf\u003e`__ Yu, Bei, and Linchi Kwok.\n\n~~~~~~~~~~~~~~~~~~~~~~\nLaw\n~~~~~~~~~~~~~~~~~~~~~~\nHuge volumes of legal text information and documents have been generated by governmental institutions. Retrieving this information and automatically classifying it can not only help lawyers but also their clients.\nIn the United States, the law is derived from five sources: constitutional law, statutory law, treaties, administrative regulations, and the common law. Also, many new legal documents are created each year. Categorization of these documents is the main challenge of the lawyer community.\n\n- 🎓 `Represent yourself in court: How to prepare \u0026 try a winning case \u003chttps://books.google.com/books?hl=en\u0026lr=\u0026id=-lodDQAAQBAJ\u0026oi=fnd\u0026pg=PP1\u0026dq=Represent+yourself+in+court:+How+to+prepare+%5C%26+try+a+winning+case\u0026ots=tgJ8Q2MkH_\u0026sig=9o3ILDn3LfO30BZKsyI2Ou7Q8Qs\u003e`__ Bergman, Paul, and Sara J. Berman. (2016)\n\n- 🎓 `Text retrieval in the legal world \u003chttps://link.springer.com/article/10.1007/BF00877694\u003e`__ Turtle, Howard.\n\n==========\nCitations:\n==========\n\n----\n\n.. code::\n\n @ARTICLE{Kowsari2018Text_Classification,\n title={Text Classification Algorithms: A Survey},\n author={Kowsari, Kamran and Jafari Meimandi, Kiana and Heidarysafa, Mojtaba and Mendu, Sanjana and Barnes, Laura E. and Brown, Donald E.},\n journal={Information},\n VOLUME = {10}, \n YEAR = {2019},\n NUMBER = {4},\n ARTICLE-NUMBER = {150},\n URL = {http://www.mdpi.com/2078-2489/10/4/150},\n ISSN = {2078-2489},\n publisher={Multidisciplinary Digital Publishing Institute}\n }\n\n.. |RMDL| image:: docs/pic/RMDL.jpg\n.. |line| image:: docs/pic/line.png\n :alt: Foo\n.. |HDLTex| image:: docs/pic/HDLTex.png\n\n\n.. |twitter| image:: https://img.shields.io/twitter/url/http/shields.io.svg?style=social\n :target: https://twitter.com/intent/tweet?text=Text%20Classification%20Algorithms:%20A%20Survey%0aGitHub:\u0026url=https://github.com/kk7nc/Text_Classification\u0026hashtags=Text_Classification,classification,MachineLearning,Categorization,NLP,NATURAL,LANGUAGE,PROCESSING\n \n.. |contributions-welcome| image:: https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat\n :target: https://github.com/kk7nc/Text_Classification/pulls\n.. |ansicolortags| image:: https://img.shields.io/pypi/l/ansicolortags.svg\n :target: https://github.com/kk7nc/Text_Classification/blob/master/LICENSE\n.. |contributors| image:: https://img.shields.io/github/contributors/kk7nc/Text_Classification.svg\n :target: https://github.com/kk7nc/Text_Classification/graphs/contributors \n\n.. |arXiv| image:: https://img.shields.io/badge/arXiv-1904.08067-red.svg?style=flat\n :target: https://arxiv.org/abs/1904.08067\n \n.. |DOI| image:: https://img.shields.io/badge/DOI-10.3390/info10040150-blue.svg?style=flat\n :target: https://doi.org/10.3390/info10040150\n \n \n.. |medium| image:: https://img.shields.io/badge/Medium-Text%20Classification-blueviolet.svg\n :target: https://medium.com/text-classification-algorithms/text-classification-algorithms-a-survey-a215b7ab7e2d\n\n.. |UniversityCube| image:: https://img.shields.io/badge/UniversityCube-Follow%20us%20for%20the%20Latest%20News!-blue.svg\n :target: https://www.universitycube.net/news\n\n\n.. |mendeley| image:: https://img.shields.io/badge/Mendeley-Add%20to%20Library-critical.svg\n :target: https://www.mendeley.com/import/?url=https://doi.org/10.3390/info10040150\n \n.. |Best| image:: https://img.shields.io/badge/Award-Best%20Paper%20Award%202019-brightgreen\n :target: https://www.mdpi.com/journal/information/awards\n \n.. |BPW| image:: docs/pic/BPW.png\n :target: https://www.mdpi.com/journal/information/awards\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkk7nc%2Ftext_classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkk7nc%2Ftext_classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkk7nc%2Ftext_classification/lists"}