{"id":16194650,"url":"https://github.com/hironsan/sentiment-analysis-toolbox","last_synced_at":"2025-08-11T21:31:39.435Z","repository":{"id":78289753,"uuid":"88914480","full_name":"Hironsan/sentiment-analysis-toolbox","owner":"Hironsan","description":"Sentiment analysis toolbox for all NLPer.","archived":false,"fork":false,"pushed_at":"2018-11-21T22:26:54.000Z","size":526,"stargazers_count":11,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-02T02:04:46.751Z","etag":null,"topics":["machine-learning","natural-language-processing","sentiment-analysis"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Hironsan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-20T22:06:00.000Z","updated_at":"2024-05-21T14:20:33.000Z","dependencies_parsed_at":"2023-04-09T04:33:41.002Z","dependency_job_id":null,"html_url":"https://github.com/Hironsan/sentiment-analysis-toolbox","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hironsan%2Fsentiment-analysis-toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hironsan%2Fsentiment-analysis-toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hironsan%2Fsentiment-analysis-toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hironsan%2Fsentiment-analysis-toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Hironsan","download_url":"https://codeload.github.com/Hironsan/sentiment-analysis-toolbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229608528,"owners_count":18098039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","natural-language-processing","sentiment-analysis"],"created_at":"2024-10-10T08:24:09.921Z","updated_at":"2024-12-13T19:59:46.242Z","avatar_url":"https://github.com/Hironsan.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sentiment analysis toolbox\n\nIn this project, you can quickly perform sentiment analysis on some datasets such as Tweets, movie review and so on since all of the codes are on Jupyter Notebooks.\nAlso, there are various methods such as SVM, Multinomial Naive Bayes, Convolutional Neural Network and so on.\n\n## Table of contents\n\n1. Basic Text Pre-processing\n* Tokenization\n* Text Normalization\n  * Normalizing case\n  * Stemming\n  * Lemmatization\n  * Spelling correction\n  * Negation handling\n* Removing meaningless words\n  * Removing punctuation\n  * Stopwords removal\n  * Frequent words removal\n  * Rare words removal\n\n## Text Pre-processing\n\n### Tokenization\n\nTokenization is the process of taking a text or set of texts and breaking it up into its individual words. In this step, we will tokenize text with the help of splitting text by space or punctuation marks.\n\nA text can be split into words using the function `word_tokenize` in `nltk.tokenize` module:\n\n```python\n\u003e\u003e\u003e from nltk.tokenize import word_tokenize\n\u003e\u003e\u003e s = '''Good muffins cost $3.88\\nin New York.  Please buy me\n... two of them.\\n\\nThanks.'''\n\u003e\u003e\u003e word_tokenize(s)\n['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York', '.',\n'Please', 'buy', 'me', 'two', 'of', 'them', '.', 'Thanks', '.']\n```\n\nNLTK has a tokenizer for tweets:\n\n```python\n\u003e\u003e\u003e from nltk.tokenize import TweetTokenizer\n\u003e\u003e\u003e s = '@remy: This is waaaaayyyy too much for you!!!!!!'\n\u003e\u003e\u003e word_tokenize(s)\n['@', 'remy', ':', 'This', 'is', 'waaaaayyyy', 'too', 'much',\n 'for', 'you', '!', '!', '!', '!', '!', '!']\n\u003e\u003e\u003e t = TweetTokenizer(strip_handles=True, reduce_len=True)\n\u003e\u003e\u003e t.tokenize(s)\n[':', 'This', 'is', 'waaayyy', 'too', 'much', 'for',\n 'you', '!', '!', '!']\n```\n\nFor further detail, see [nltk.tokenize package](https://www.nltk.org/api/nltk.tokenize.html).\n\n### Text Normalization\n\nText normalization is the process of transforming text into a single canonical. Normalizing text before processing it ensure that these words are consistent. For example, converting character case or stemming words:\n\n* Normalizing case\n* Normalizing number\n* Stemming\n* Lemmatization\n* Spelling correctioin\n* Negation handling\n\nFor further details, see [CS506/606: Txt Nrmlztn](http://www.csee.ogi.edu/~sproatr/Courses/TextNorm/).\n\n#### Normalizing case\n\nNormalizing case is the process of transforming characters into the same case. This process ensures that the same words are recognised as the same. In this case we transformed into lowercase:\n\n```python\n\u003e\u003e\u003e s = 'The Da Vinci Code was REALLY good.'\n\u003e\u003e\u003e s.lower()\n'the da vinci code was really good.'\n```\n\nWe can also lower text after tokenization:\n\n```python\n\u003e\u003e\u003e from nltk.tokenize import word_tokenize\n\u003e\u003e\u003e s = 'The Da Vinci Code was REALLY good.'\n\u003e\u003e\u003e words = word_tokenize(s)\n\u003e\u003e\u003e words\n['The', 'Da', 'Vinci', 'Code', 'was', 'REALLY', 'good', '.']\n\u003e\u003e\u003e [w.lower() for w in words]\n['the', 'da', 'vinci', 'code', 'was', 'really', 'good', '.']\n```\n\n\u003c!--\n#### Normalizing number\n\nNormalizing number is the process of \n--\u003e\n\n#### Stemming\n\nStemming is the process of reducing inflected words to their word stem. For example, \"listen\", \"listened\", \"listening\" are reduced to the same stem \"listen\". Some application like sentiment analysis can benefit from stemming because it reduces vocabulary and increase the relevance of the concept.\n\nFor stemming, we can use `SnowballStemmer` in `nltk.stem.snowball`:\n\n```python\n\u003e\u003e\u003e from nltk.stem.snowball import SnowballStemmer\n\u003e\u003e\u003e st = SnowballStemmer('english')\n\u003e\u003e\u003e st.stem('running')\n'run'\n\u003e\u003e\u003e st.stem('greatly')\n'great'\n```\n\nThere are other stemming algorithms in NLTK. For further information, see [nltk.stem package](http://www.nltk.org/api/nltk.stem.html).\n\n#### Lemmatization\n\nLemmatization is the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word's lemma, or dictionary form. For example, \"am\", \"are\", \"is\" are lemmatized to the same form \"be\".\n\nFor stemming, we can use `WordNetLemmatizer` in `nltk.stem.wordnet`. Before we use the lemmatizer, we should download WordNet:\n\n```python\n\u003e\u003e\u003e import nltk\n\u003e\u003e\u003e nltk.download('wordnet')\n```\n\nNow, we are ready to lemmatize word:\n\n```python\n\u003e\u003e\u003e from nltk.stem.wordnet import WordNetLemmatizer\n\u003e\u003e\u003e lt = WordNetLemmatizer()\n\u003e\u003e\u003e lt.lemmatize('am', pos='v')\n'be'\n\u003e\u003e\u003e lt.lemmatize('are', pos='v')\n'be'\n\u003e\u003e\u003e lt.lemmatize('is', pos='v')\n'be'\n```\n\n#### Spelling correction\n\nSpelling correction is the process of correcting spelling mistakes. In text processing, spelling correction is a useful pre-processing step because this reduces the vocabulary size. For example, \"speling\" and \"spelling\" will be treated as the same word after spelling correction.\n\nFor spelling correction, we can use `TextBlob`:\n\n```python\n\u003e\u003e\u003e from textblob import TextBlob\n\u003e\u003e\u003e TextBlob('speling').correct()\nTextBlob(\"spelling\")\n\u003e\u003e\u003e TextBlob('korrectud').correct()\nTextBlob(\"corrected\")\n```\n\nIf you want to build your own spelling corrector, below is the best choice to read:\n\n* [How to Write a Spelling Corrector](https://norvig.com/spell-correct.html)\n\n#### Negation handling\n\nNegation handling is the process of converting negation abbreviation to a canonical format. For example, \"aren't\" is converted to \"are not\". It is helpful for sentiment analysis. \n\nFor negation handling, we will use a dictionary as follows:\n\n```python\n\u003e\u003e\u003e d = {\n\"aren't\" : \"are not\",\n\"can't\" : \"cannot\",\n\"couldn't\" : \"could not\",\n\"didn't\" : \"did not\",\n\"doesn't\" : \"does not\",\n\"don't\" : \"do not\"\n}\n\u003e\u003e\u003e s = \"We don't like this dish\"\n\u003e\u003e\u003e words = s.split()\n\u003e\u003e\u003e words\n['We', \"don't\", 'like', 'this', 'dish']\n\u003e\u003e\u003e ' '.join(d[w] if w in d else w for w in words)\n'We do not like this dish'\n```\n\nThis is a naive implementation, so inefficient.\n\nYou can download a dictionary from [here](https://drive.google.com/file/d/0B1yuv8YaUVlZZ1RzMFJmc1ZsQmM/view).\n\n### Removing meaningless words\n\n* Removing punctuation\n* Stopwords removal\n* Frequent words removal\n* Rare words removal\n\n#### Removing punctuation\n\nRemoving punctuation is the process of removing punctuation characters in text. As punctuation may not have usuful information, removing punctuation will help us reduce the size of the training data.\n\nPython provides a constant punctuation by `string.punctuation`. For example:\n\n```python\n\u003e\u003e\u003e import string\n\u003e\u003e\u003e string.punctuation\n'!\"#$%\u0026\\'()*+,-./:;\u003c=\u003e?@[\\\\]^_`{|}~'\n```\n\nWe can remove punctuation using `string.punctuation` and `translate` method:\n\n```python\n\u003e\u003e\u003e translate_dict = dict((c, ' ') for c in string.punctuation)\n\u003e\u003e\u003e translate_map = str.maketrans(translate_dict)\n\u003e\u003e\u003e text = 'state-of-the-art'\n\u003e\u003e\u003e text.translate(translate_map)\n'state of the art'\n```\n\n#### Stop words removal\n\nStop words removal is the process of eliminating stop words from text. Stop words are words which are filtered out before or after processing of text. For example, \"a\", \"the\", \"is\", \"of\", etc... are stop words. Stop words are meaningless and useless for the sentiment analysis. This process reduces the corpus size.\n\nStop words can be filtered by using `stopwords` in `nltk.corpus`:\n\n```python\n\u003e\u003e\u003e from nltk.corpus import stopwords\n\u003e\u003e\u003e from nltk.tokenize import word_tokenize\n\u003e\u003e\u003e \n\u003e\u003e\u003e s = 'This is a fantastic movie!'\n\u003e\u003e\u003e stop_words = stopwords.words('english')\n\u003e\u003e\u003e words = word_tokenize(s)\n\u003e\u003e\u003e words\n['This', 'is', 'a', 'fantastic', 'movie', '!']\n\u003e\u003e\u003e [w for w in words if w not in stop_words]\n['This', 'fantastic', 'movie', '!']\n```\n\n\u003c!--\nhttp://www.cs.cmu.edu/~mccallum/bow/rainbow/ \n--\u003e\n\n## Reference\n\n* [Sentiment analysis of reviews: Text Pre-processing](https://medium.com/@annabiancajones/sentiment-analysis-of-reviews-text-pre-processing-6359343784fb)\n* [Ultimate guide to deal with Text Data (using Python) – for Data Scientists \u0026 Engineers](https://www.analyticsvidhya.com/blog/2018/02/the-different-methods-deal-text-data-predictive-python/)\n* [How to Clean Text for Machine Learning with Python](https://machinelearningmastery.com/clean-text-machine-learning-python/)\n* [A Comparison between Preprocessing Techniques for Sentiment Analysis in Twitter](http://ceur-ws.org/Vol-1748/paper-06.pdf)\n* [Data Preprocessing, Sentiment Analysis \u0026 NER On Twitter Data.](http://www.iosrjournals.org/iosr-jce/papers/Conf.17014-2017/Volume-2/15.%2073-79.pdf?id=7557)\n* [The effect of preprocessing techniques on Twitter sentiment analysis](https://www.researchgate.net/publication/311755864_The_effect_of_preprocessing_techniques_on_Twitter_sentiment_analysis)\n* [Preprocessing text before use RNN](https://datascience.stackexchange.com/questions/11402/preprocessing-text-before-use-rnn)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhironsan%2Fsentiment-analysis-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhironsan%2Fsentiment-analysis-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhironsan%2Fsentiment-analysis-toolbox/lists"}