{"id":13423835,"url":"https://github.com/juliasilge/tidytext","last_synced_at":"2025-05-14T14:09:00.467Z","repository":{"id":6279548,"uuid":"55175084","full_name":"juliasilge/tidytext","owner":"juliasilge","description":"Text mining using tidy tools :sparkles::page_facing_up::sparkles:","archived":false,"fork":false,"pushed_at":"2024-04-10T13:44:30.000Z","size":135169,"stargazers_count":1185,"open_issues_count":12,"forks_count":182,"subscribers_count":63,"default_branch":"main","last_synced_at":"2025-05-03T06:02:57.029Z","etag":null,"topics":["natural-language-processing","r","text-mining","tidy-data","tidyverse"],"latest_commit_sha":null,"homepage":"https://juliasilge.github.io/tidytext/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/juliasilge.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-03-31T18:51:39.000Z","updated_at":"2025-04-30T09:23:43.000Z","dependencies_parsed_at":"2024-06-18T12:16:11.016Z","dependency_job_id":"cc856f94-aacc-471e-92c1-5f90619534cf","html_url":"https://github.com/juliasilge/tidytext","commit_stats":{"total_commits":710,"total_committers":34,"mean_commits":20.88235294117647,"dds":"0.26901408450704223","last_synced_commit":"87d90fc6edb4740e3706e2ec3852d7ee01d23fe3"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliasilge%2Ftidytext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliasilge%2Ftidytext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliasilge%2Ftidytext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juliasilge%2Ftidytext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/juliasilge","download_url":"https://codeload.github.com/juliasilge/tidytext/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254069757,"owners_count":22009558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["natural-language-processing","r","text-mining","tidy-data","tidyverse"],"created_at":"2024-07-31T00:00:43.512Z","updated_at":"2025-05-14T14:08:55.450Z","avatar_url":"https://github.com/juliasilge.png","language":"R","funding_links":[],"categories":["Text analysis","R","APIs and Libraries","Libraries","Packages"],"sub_categories":["hOCR","Knowledge Graphs","Books","Machine learning"],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n\n```{r}\n#| include = FALSE\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\",\n  message = FALSE\n)\nsuppressPackageStartupMessages(library(ggplot2))\ntheme_set(theme_light())\n```\n\n# tidytext: Text mining using tidy tools \u003cimg src=\"man/figures/tidytext.png\" align=\"right\" /\u003e\n\n**Authors:** [Julia Silge](https://juliasilge.com/), [David Robinson](http://varianceexplained.org/)\u003cbr/\u003e\n**License:** [MIT](https://opensource.org/licenses/MIT)\n\n\u003c!-- badges: start --\u003e\n[![R-CMD-check](https://github.com/juliasilge/tidytext/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/juliasilge/tidytext/actions/workflows/R-CMD-check.yaml)\n[![CRAN_Status_Badge](https://www.r-pkg.org/badges/version/tidytext)](https://cran.r-project.org/package=tidytext)\n[![Codecov test coverage](https://codecov.io/gh/juliasilge/tidytext/branch/main/graph/badge.svg)](https://app.codecov.io/gh/juliasilge/tidytext?branch=main)\n[![DOI](https://zenodo.org/badge/22224/juliasilge/tidytext.svg)](https://zenodo.org/badge/latestdoi/22224/juliasilge/tidytext)\n[![JOSS](https://joss.theoj.org/papers/10.21105/joss.00037/status.svg)](https://joss.theoj.org/papers/10.21105/joss.00037)\n[![Downloads](https://cranlogs.r-pkg.org/badges/tidytext)](https://CRAN.R-project.org/package=tidytext)\n[![Total Downloads](https://cranlogs.r-pkg.org/badges/grand-total/tidytext?color=orange)](https://CRAN.R-project.org/package=tidytext)\n\u003c!-- badges: end --\u003e\n\n\nUsing [tidy data principles](https://doi.org/10.18637/jss.v059.i10) can make many text mining tasks easier, more effective, and consistent with tools already in wide use. Much of the infrastructure needed for text mining with tidy data frames already exists in packages like [dplyr](https://cran.r-project.org/package=dplyr), [broom](https://cran.r-project.org/package=broom), [tidyr](https://cran.r-project.org/package=tidyr), and [ggplot2](https://cran.r-project.org/package=ggplot2). In this package, we provide functions and supporting data sets to allow conversion of text to and from tidy formats, and to switch seamlessly between tidy tools and existing text mining packages. Check out [our book](https://www.tidytextmining.com/) to learn more about text mining using tidy data principles.\n\n### Installation\n\nYou can install this package from CRAN:\n\n```{r}\n#| eval = FALSE\ninstall.packages(\"tidytext\")\n```\n\n\nOr you can install the development version from GitHub with [remotes](https://github.com/r-lib/remotes):\n\n```{r}\n#| eval = FALSE\nlibrary(remotes)\ninstall_github(\"juliasilge/tidytext\")\n```\n\n### Tidy text mining example: the `unnest_tokens` function\n\nThe novels of Jane Austen can be so tidy! Let's use the text of Jane Austen's 6 completed, published novels from the [janeaustenr](https://cran.r-project.org/package=janeaustenr) package, and transform them to a tidy format. janeaustenr provides them as a one-row-per-line format:\n\n```{r}\nlibrary(janeaustenr)\nlibrary(dplyr)\n\noriginal_books \u003c- austen_books() %\u003e%\n  group_by(book) %\u003e%\n  mutate(line = row_number()) %\u003e%\n  ungroup()\n\noriginal_books\n```\n\nTo work with this as a tidy dataset, we need to restructure it as **one-token-per-row** format. The `unnest_tokens()` function is a way to convert a dataframe with a text column to be one-token-per-row:\n\n```{r}\nlibrary(tidytext)\ntidy_books \u003c- original_books %\u003e%\n  unnest_tokens(word, text)\n\ntidy_books\n```\n\nThis function uses the [tokenizers](https://docs.ropensci.org/tokenizers/) package to separate each line into words. The default tokenizing is for words, but other options include characters, n-grams, sentences, lines, paragraphs, or separation around a regex pattern.\n\nNow that the data is in a one-word-per-row format, we can manipulate it with tidy tools like dplyr. We can remove stop words (available via the function `get_stopwords()`) with an `anti_join()`.\n\n```{r}\ntidy_books \u003c- tidy_books %\u003e%\n  anti_join(get_stopwords())\n```\n\nWe can also use `count()` to find the most common words in all the books as a whole.\n\n```{r}\ntidy_books %\u003e%\n  count(word, sort = TRUE) \n```\n\nSentiment analysis can be implemented as an inner join. Three sentiment lexicons are available via the `get_sentiments()` function. Let's examine how sentiment changes across each novel. Let's find a sentiment score for each word using the Bing lexicon, then count the number of positive and negative words in defined sections of each novel.\n\n```{r}\n#| fig.width = 8,\n#| fig.height = 10\nlibrary(tidyr)\nget_sentiments(\"bing\")\n\njaneaustensentiment \u003c- tidy_books %\u003e%\n  inner_join(get_sentiments(\"bing\"), by = \"word\", relationship = \"many-to-many\") %\u003e% \n  count(book, index = line %/% 80, sentiment) %\u003e% \n  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %\u003e% \n  mutate(sentiment = positive - negative)\n\njaneaustensentiment\n```\n\nNow we can plot these sentiment scores across the plot trajectory of each novel.\n\n```{r}\n#| fig.width = 7,\n#| fig.height = 7,\n#| fig.alt = \"Sentiment scores across the trajectories of Jane Austen's six published novels\",\n#| warning = FALSE\nlibrary(ggplot2)\n\nggplot(janeaustensentiment, aes(index, sentiment, fill = book)) +\n  geom_col(show.legend = FALSE) +\n  facet_wrap(vars(book), ncol = 2, scales = \"free_x\")\n```\n\nFor more examples of text mining using tidy data frames, see the tidytext vignette.\n\n### Tidying document term matrices\n\nSome existing text mining datasets are in the form of a DocumentTermMatrix class (from the tm package). For example, consider the corpus of 2246 Associated Press articles from the topicmodels dataset.\n\n```{r}\nlibrary(tm)\ndata(\"AssociatedPress\", package = \"topicmodels\")\nAssociatedPress\n```\n\nIf we want to analyze this with tidy tools, we need to transform it into a one-row-per-term data frame first with a `tidy()` function. (For more on the tidy verb, [see the broom package](https://broom.tidymodels.org/)).\n\n```{r}\ntidy(AssociatedPress)\n```\n\nWe could find the most negative documents:\n\n```{r}\nap_sentiments \u003c- tidy(AssociatedPress) %\u003e%\n  inner_join(get_sentiments(\"bing\"), by = c(term = \"word\")) %\u003e%\n  count(document, sentiment, wt = count) %\u003e%\n  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %\u003e%\n  mutate(sentiment = positive - negative) %\u003e%\n  arrange(sentiment)\n```\n\nOr we can join the Austen and AP datasets and compare the frequencies of each word:\n\n```{r}\n#| fig.height = 8,\n#| fig.width = 8,\n#| fig.alt = 'Scatterplot for word frequencies in Jane Austen vs. AP news articles. Some words like \"cried\" are only common in Jane Austen, some words like \"national\" are only common in AP articles, and some word like \"time\" are common in both.'\ncomparison \u003c- tidy(AssociatedPress) %\u003e%\n  count(word = term) %\u003e%\n  rename(AP = n) %\u003e%\n  inner_join(count(tidy_books, word)) %\u003e%\n  rename(Austen = n) %\u003e%\n  mutate(AP = AP / sum(AP),\n         Austen = Austen / sum(Austen))\n\n\ncomparison\n\nlibrary(scales)\nggplot(comparison, aes(AP, Austen)) +\n  geom_point(alpha = 0.5) +\n  geom_text(aes(label = word), check_overlap = TRUE,\n            vjust = 1, hjust = 1) +\n  scale_x_log10(labels = percent_format()) +\n  scale_y_log10(labels = percent_format()) +\n  geom_abline(color = \"red\")\n```\n\nFor more examples of working with objects from other text mining packages using tidy data principles, see the [vignette](https://juliasilge.github.io/tidytext/articles/tidying_casting.html) on converting to and from document term matrices.\n\n### Community Guidelines\n\nThis project is released with a [Contributor Code of Conduct](https://github.com/juliasilge/tidytext/blob/main/CONDUCT.md). By participating in this project you agree to abide by its terms. Feedback, bug reports (and fixes!), and feature requests are welcome; file issues or seek support [here](https://github.com/juliasilge/tidytext/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliasilge%2Ftidytext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuliasilge%2Ftidytext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliasilge%2Ftidytext/lists"}