Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/juliasilge/widyr

Widen, process, and re-tidy a dataset
https://github.com/juliasilge/widyr

Last synced: 12 days ago
JSON representation

Widen, process, and re-tidy a dataset

Awesome Lists containing this project

README 

        
---

output: github_document

---



```{r, include = FALSE}

knitr::opts_chunk$set(

  collapse = TRUE,

  comment = "#>",

  fig.path = "man/figures/README-",

  out.width = "100%",

  message = FALSE

)

suppressPackageStartupMessages(library(dplyr))

```



# widyr: Widen, process, and re-tidy a dataset



**Authors:** [Julia Silge](https://juliasilge.com/), [David Robinson](http://varianceexplained.org/)


**License:** [MIT](https://opensource.org/licenses/MIT)



[![R-CMD-check](https://github.com/juliasilge/widyr/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/juliasilge/widyr/actions/workflows/R-CMD-check.yaml)

[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/widyr)](https://cran.r-project.org/package=widyr)

[![Codecov test coverage](https://codecov.io/gh/juliasilge/widyr/branch/main/graph/badge.svg)](https://app.codecov.io/gh/juliasilge/widyr?branch=main)



This package wraps the pattern of un-tidying data into a wide matrix, performing some processing, then turning it back into a tidy form. This is useful for several mathematical operations such as co-occurrence counts, correlations, or clustering that are best done on a wide matrix.



## Installation



You can install the released version of widyr from [CRAN](https://CRAN.R-project.org) with:



``` r

install.packages("widyr")

```



And the development version from [GitHub](https://github.com/) with:



``` r

# install.packages("devtools")

devtools::install_github("juliasilge/widyr")

```



## Towards a precise definition of "wide" data



The term "wide data" has gone out of fashion as being "imprecise" [(Wickham 2014)](http://vita.had.co.nz/papers/tidy-data.pdf), but I think with a proper definition the term could be entirely meaningful and useful.



A **wide** dataset is one or more matrices where:



* Each row is one **item**

* Each column is one **feature**

* Each value is one **observation**

* Each matrix is one **variable**



When would you want data to be wide rather than tidy? Notable examples include classification, clustering, correlation, factorization, or other operations that can take advantage of a matrix structure. In general, when you want to **compare between pairs of items** rather than compare between variables or between groups of observations, this is a useful structure.



The widyr package is based on the observation that during a tidy data analysis, you often want data to be wide only *temporarily*, before returning to a tidy structure for visualization and further analysis. widyr makes this easy through a set of `pairwise_` functions.



## Example: gapminder



Consider the gapminder dataset in the [gapminder package](https://CRAN.R-project.org/package=gapminder).



```{r}

library(dplyr)

library(gapminder)



gapminder

```



This tidy format (one-row-per-country-per-year) is very useful for grouping, summarizing, and filtering operations. But if we want to *compare* countries (for example, to find countries that are similar to each other), we would have to reshape this dataset. Note that here, each country is an **item**, while each year is the **feature**.



#### Pairwise operations



The widyr package offers `pairwise_` functions that operate on pairs of items within data. An example is `pairwise_dist`:



```{r}

library(widyr)



gapminder %>%

  pairwise_dist(country, year, lifeExp)

```



This finds the Euclidean distance between the `lifeExp` value in each pair of countries. It knows which values to compare between countries with `year`, which is the feature column.



We could find the closest pairs of countries overall with `arrange()`:



```{r}

gapminder %>%

  pairwise_dist(country, year, lifeExp) %>%

  arrange(distance)

```



Notice that this includes duplicates (Germany/Belgium and Belgium/Germany). To avoid those (the upper triangle of the distance matrix), use `upper = FALSE`:



```{r}

gapminder %>%

  pairwise_dist(country, year, lifeExp, upper = FALSE) %>%

  arrange(distance)

```



In some analyses, we may be interested in correlation rather than distance of pairs. For this we would use `pairwise_cor`:



```{r}

gapminder %>%

  pairwise_cor(country, year, lifeExp, upper = FALSE)

```



### Code of Conduct



This project is released with a [Contributor Code of Conduct](https://www.contributor-covenant.org/version/2/1/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.