{"id":13857868,"url":"https://github.com/trinker/textstem","last_synced_at":"2025-03-16T13:31:18.044Z","repository":{"id":56937323,"uuid":"81699730","full_name":"trinker/textstem","owner":"trinker","description":"Tools for fast text stemming \u0026 lemmatization","archived":false,"fork":false,"pushed_at":"2018-06-07T20:27:43.000Z","size":182,"stargazers_count":45,"open_issues_count":6,"forks_count":8,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-12T06:45:29.204Z","etag":null,"topics":["lemmatization","r","stemming","text-mining"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trinker.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-12T03:14:52.000Z","updated_at":"2025-01-07T16:49:15.000Z","dependencies_parsed_at":"2022-08-21T06:50:13.026Z","dependency_job_id":null,"html_url":"https://github.com/trinker/textstem","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trinker%2Ftextstem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trinker%2Ftextstem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trinker%2Ftextstem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trinker%2Ftextstem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trinker","download_url":"https://codeload.github.com/trinker/textstem/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243817012,"owners_count":20352476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lemmatization","r","stemming","text-mining"],"created_at":"2024-08-05T03:01:49.245Z","updated_at":"2025-03-16T13:31:17.522Z","avatar_url":"https://github.com/trinker.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\ntitle: \"textstem\"\ndate: \"`r format(Sys.time(), '%d %B, %Y')`\"\noutput:\n  md_document:\n    toc: true      \n---\n\n```{r, echo=FALSE, warning=FALSE}\ndesc \u003c- suppressWarnings(readLines(\"DESCRIPTION\"))\nregex \u003c- \"(^Version:\\\\s+)(\\\\d+\\\\.\\\\d+\\\\.\\\\d+)\"\nloc \u003c- grep(regex, desc)\nver \u003c- gsub(regex, \"\\\\2\", desc[loc])\nverbadge \u003c- sprintf('\u003ca href=\"https://img.shields.io/badge/Version-%s-orange.svg\"\u003e\u003cimg src=\"https://img.shields.io/badge/Version-%s-orange.svg\" alt=\"Version\"/\u003e\u003c/a\u003e\u003c/p\u003e', ver, ver)\nverbadge \u003c- ''\n\npacman::p_load(textstem)\npacman::p_load_current_gh('trinker/numform')\nnr \u003c- numform::f_comma(length(presidential_debates_2012$dialogue))\nnw \u003c- numform::f_comma(sum(stringi::stri_count_words(presidential_debates_2012$dialogue), na.rm = TRUE))\n````\n\n[![Project Status: Active - The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/0.1.0/active.svg)](https://www.repostatus.org/#active)\n[![Build Status](https://travis-ci.org/trinker/textstem.svg?branch=master)](https://travis-ci.org/trinker/textstem)\n[![Coverage Status](https://coveralls.io/repos/trinker/textstem/badge.svg?branch=master)](https://coveralls.io/r/trinker/textstem?branch=master)\n[![](https://cranlogs.r-pkg.org/badges/textstem)](https://cran.r-project.org/package=textstem)\n`r verbadge`\n\n![](tools/textstem_logo/r_textstem.png)\n\n**textstem** is a tool-set for stemming and lemmatizing words.  Stemming is a process that removes affixes.   Lemmatization is the process of grouping inflected forms together as a single base form.  \n\n# Functions\n\nThe main functions, task category, \u0026 descriptions are summarized in the table below:\n\n\n| Function                      | Task        | Description                                | \n|-------------------------------|-------------|--------------------------------------------| \n| `stem_words`                  | stemming    | Stem words                                 | \n| `stem_strings`                | stemming    | Stem strings                               | \n| `lemmatize_words`             | lemmatizing | Lemmatize words                            | \n| `lemmatize_strings`           | lemmatizing | Lemmatize strings                          | \n| `make_lemma_dictionary_words` | lemmatizing | Generate a dictionary of lemmas for a text |     \n\n\n# Installation\n\nTo download the development version of **textstem**:\n\nDownload the [zip ball](https://github.com/trinker/textstem/zipball/master) or [tar ball](https://github.com/trinker/textstem/tarball/master), decompress and run `R CMD INSTALL` on it, or use the **pacman** package to install the development version:\n\n```r\nif (!require(\"pacman\")) install.packages(\"pacman\")\npacman::p_load_gh(\"trinker/textstem\")\n```\n\n# Contact\n\nYou are welcome to:    \n- submit suggestions and bug-reports at: \u003chttps://github.com/trinker/textstem/issues\u003e    \n- send a pull request on: \u003chttps://github.com/trinker/textstem/\u003e    \n- compose a friendly e-mail to: \u003ctyler.rinker@gmail.com\u003e    \n\n\n# Examples\n\nThe following examples demonstrate some of the functionality of **textstem**.\n\n## Load the Tools/Data\n\n```{r, message=FALSE, warning=FALSE}\nif (!require(\"pacman\")) install.packages(\"pacman\")\npacman::p_load(textstem, dplyr)\n\ndata(presidential_debates_2012)\n```\n\n## Stemming Versus Lemmatizing\n\nBefore moving into the meat these two examples let's highlight the difference between stemming and lemmatizing.\n\n### \"Drive\" Stemming vs. Lemmatizing\n\n```{r}\ndw \u003c- c('driver', 'drive', 'drove', 'driven', 'drives', 'driving')\n\nstem_words(dw)\nlemmatize_words(dw)\n```\n\n### \"Be\" Stemming vs. Lemmatizing\n\n```{r}\nbw \u003c- c('are', 'am', 'being', 'been', 'be')\n\nstem_words(bw)\nlemmatize_words(bw)\n```\n\n\n## Stemming\n\nStemming is the act of removing inflections from a word not necessarily [\"identical to the morphological root of the word\" (wikipedia)](https://en.wikipedia.org/wiki/Stemming).  Below I show stemming of several small strings.\n\n```{r}\ny \u003c- c(\n    'the dirtier dog has eaten the pies',\n    'that shameful pooch is tricky and sneaky',\n    \"He opened and then reopened the food bag\",\n    'There are skies of blue and red roses too!',\n    NA,\n    \"The doggies, well they aren't joyfully running.\",\n     \"The daddies are coming over...\",\n    \"This is 34.546 above\"\n)\nstem_strings(y)\n```\n\n## Lemmatizing\n### Default Lemma Dictionary\n\nLemmatizing is the [\"grouping together the inflected forms of a word so they can be analysed as a single item\" (wikipedia)](https://en.wikipedia.org/wiki/Lemmatisation).  In the example below I reduce the strings to their lemma form.  `lemmatize_strings` uses a lookup dictionary.  The default uses [Mechura's (2016)](http://www.lexiconista.com) English lemmatization list available from the [**lexicon**](https://cran.r-project.org/package=lexicon) package. The `make_lemma_dictionary` function contains two additional engines for generating a lemma lookup table for use in `lemmatize_strings`.\n\n\n```{r}\ny \u003c- c(\n    'the dirtier dog has eaten the pies',\n    'that shameful pooch is tricky and sneaky',\n    \"He opened and then reopened the food bag\",\n    'There are skies of blue and red roses too!',\n    NA,\n    \"The doggies, well they aren't joyfully running.\",\n     \"The daddies are coming over...\",\n    \"This is 34.546 above\"\n)\nlemmatize_strings(y)\n```\n### Hunspell Lemma Dictionary\n\nThis lemmatization uses the [**hunspell**](https://CRAN.R-project.org/package=hunspell) package to generate lemmas.\n\n\n```{r}\nlemma_dictionary_hs \u003c- make_lemma_dictionary(y, engine = 'hunspell')\nlemmatize_strings(y, dictionary = lemma_dictionary_hs)\n```\n\n### koRpus Lemma Dictionary\n\nThis lemmatization uses the [**koRpus**](https://CRAN.R-project.org/package=koRpus) package and the [TreeTagger](http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/) program to generate lemmas.  You'll have to get TreeTagger set up, preferably in your machine's root directory.\n\n```{r}\nlemma_dictionary_tt \u003c- make_lemma_dictionary(y, engine = 'treetagger')\nlemmatize_strings(y, lemma_dictionary_tt)\n```\n\n### Lemmatization Speed\n\nIt's pretty fast too.  Observe:\n\n```{r}\ntic \u003c- Sys.time()\n\npresidential_debates_2012$dialogue %\u003e%\n    lemmatize_strings() %\u003e%\n    head()\n\n(toc \u003c- Sys.time() - tic)\n```\n\n\nThat's `r nr` rows of text, or `r nw` words, in `r round(as.numeric(toc), 2)` seconds.\n\n\n## Combine With Other Text Tools\n\nThis example shows how stemming/lemmatizing might be complemented by other text tools such as `replace_contraction` from the **textclean** package.\n\n```{r}\nlibrary(textclean)\n\n'aren\\'t' %\u003e% \n    lemmatize_strings()\n\n'aren\\'t' %\u003e% \n    textclean::replace_contraction() %\u003e%\n    lemmatize_strings()\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrinker%2Ftextstem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrinker%2Ftextstem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrinker%2Ftextstem/lists"}