Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/luckinet/tabshiftr

Reshape disorganised messy data
https://github.com/luckinet/tabshiftr

data-management data-reshaping schemas

Last synced: 8 days ago
JSON representation

Reshape disorganised messy data

Awesome Lists containing this project

README 

        
---

output: github_document

---



```{r, include = FALSE}

knitr::opts_chunk$set(

  collapse = TRUE,

  comment = "#>",

  fig.path = "man/figures/README-",

  out.width = "100%"

)

options(knitr.kable.NA = '.')

```



# tabshiftr 



[![CRAN\_Status\_Badge](http://www.r-pkg.org/badges/version/tabshiftr)](https://cran.r-project.org/package=tabshiftr)

[![DOI](https://zenodo.org/badge/248468235.svg)](https://zenodo.org/badge/latestdoi/248468235)



[![R-CMD-check](https://github.com/luckinet/tabshiftr/workflows/R-CMD-check/badge.svg)](https://github.com/luckinet/tabshiftr/actions)

[![codecov](https://codecov.io/gh/luckinet/tabshiftr/branch/master/graph/badge.svg?token=hjppymcGr3)](https://codecov.io/gh/luckinet/tabshiftr)

[![Lifecycle:maturing](https://img.shields.io/badge/lifecycle-maturing-blue.svg)](https://lifecycle.r-lib.org/articles/stages.html)



[![](http://cranlogs.r-pkg.org/badges/grand-total/tabshiftr)](https://cran.r-project.org/package=tabshiftr)



## Overview



Data are stored in many different ways in tables or spreadsheets because no strict semantic or topographic standards for the organisation of tables are commonly accepted.

In the R environment the *tidy* paradigm is a first step towards interoperability of data, in that it requires a certain arrangement of tables, where variables are recorded in columns and observations in rows (see https://tidyr.tidyverse.org/).

Tables can be tidied (i.e., brought into a tidy arrangement) via packages such as `tidyr`, however, all functions that deal with reshaping tables to date require data that are already organised into topologically coherent, rectangular tables.

This is often violated in practice, especially in data that are scraped off of the internet.



`tabshiftr` fills this gap in the toolchain towards more interoperable data via `schema` descriptions that are built with setters and debugged with getters and a `reorganise()` function that ties everything together.



## Installation



1) Install the official version from CRAN:



```{r, eval=FALSE}

install.packages("tabshiftr")

```



or the latest development version from github:



```{r, eval=FALSE}

devtools::install_github("luckinet/tabshiftr")

```



2) The [vignette](https://luckinet.github.io/tabshiftr/articles/tabshiftr.html) 

 - gives an introduction of the nature of tabular data and of the dimensions of disorganization dealt with here, 

 - provides an instruction on how to set up a schema description

 - shows a wide range of table arrangements that can be reshaped with the tools provided here.



## Examples



A disorganized table may look like the following table:



```{r}

library(tabshiftr)

library(knitr)



# a rather disorganized table with messy clusters and a distinct variable

input <- tabs2shift$clusters_messy

kable(input)

```



If we were to transform this data into tidy data by merely using the functions in `tidyr` (or the extended `tidyverse` in general), we'd potentially end up with a massive algorithm, especially for such complicated table arrangements. For other tables that may or may not be as complicated, we'd have to set up yet more algorithms and while a pipeline of tidy functions is relatively easy to set up, it would still become very laborious to repeat this for the dozens of potential table arrangements. In `tabshiftr` we solve that by describing the schema of the input table and providing this schema description to the `reorganise()` function. This requires us to use a vastly smaller set of code and makes it thus a lot more efficient to bring multiple heterogeneous data into an interoperable format.



```{r}

# put together schema description by ...

# ... identifying cluster positions

schema <- setCluster(id = "territories", left = c(1, 1, 4), top = c(1, 8, 8))



# ... specifying the cluster ID as id variable (obligatory for when we deal with clusters)

schema <- schema %>%

   setIDVar(name = "territories", columns = c(1, 1, 4), rows = c(2, 9, 9))



# ... specifying a distinct variable (explicit position)

schema <- schema %>%

   setIDVar(name = "year", columns = 4, rows = c(3:6), distinct = TRUE)



# ... specifying a tidy identifying variable (by giving the column values)

schema <- schema %>%

   setIDVar(name = "commodities", columns = c(1, 1, 4))



# ... identifying the (tidy) observed variables

schema <- schema %>%

   setObsVar(name = "harvested", columns = c(2, 2, 5)) %>%

   setObsVar(name = "production", columns = c(3, 3, 6))



# to potentially debug the schema description, first validate the schema ...

schema_valid <- validateSchema(schema = schema, input = input)



# ... and extract parts of it per cluster (also check out the other getters in

# this package)

getIDVars(schema = schema_valid, input = input)

getObsVars(schema = schema_valid, input = input)



# alternatively, if the clusters are regular, relative values starting from the

# cluster origin could be set

schema_alt <- setCluster(id = "territories",

                         left = c(1, 1, 4), top = c(1, 8, 8)) %>%

  setIDVar(name = "territories", columns = 1, rows = .find(row = 2, relative = TRUE)) %>%

  setIDVar(name = "year", columns = 4, rows = c(3:6), distinct = TRUE) %>%

  setIDVar(name = "commodities", columns = .find(col = 1, relative = TRUE)) %>%

  setObsVar(name = "harvested", columns = .find(col = 2, relative = TRUE)) %>%

  setObsVar(name = "production", columns = .find(col = 3, relative = TRUE))

```



The `reorganise()` function carries out the steps of validating, extracting the variables, pivoting the tentative output and putting the final table together automatically, so it merely requires the finalised `schema` and the `input` table.



```{r}

schema # has a pretty print function



output <- reorganise(input = input, schema = schema)

kable(output)

```



# Contributions



* tabshiftr is still in development. So far it reliably reorganizes 20 different types of tables, but additional dimensions of disorganization might show themselves. If you encounter a table that can't be reorganized with the current infrastructure, we'd be more than happy to collaborate on advancing `tabshiftr`.

* Informative error management is work in process.

* Moreover, the resulting schema descriptions can be useful for data archiving or database building and `tabshiftr` should at some point support that those schemas can be exported into data-formats that are used by downstream applications (xml, json, ...), following proper (ISO) standards. In case you have experience with those standards and would like to collaborate on it, please get in touch!



# Acknowledgement



This work was supported by funding to Carsten Meyer through the Flexpool mechanism of the German Centre for Integrative Biodiversity Research (iDiv) (FZT-118, DFG).