{"id":13423843,"url":"https://github.com/nacnudus/tidyxl","last_synced_at":"2025-04-04T14:02:51.873Z","repository":{"id":11097933,"uuid":"61286119","full_name":"nacnudus/tidyxl","owner":"nacnudus","description":"Read untidy Excel files in R https://nacnudus.github.io/tidyxl/","archived":false,"fork":false,"pushed_at":"2024-06-13T21:36:06.000Z","size":9212,"stargazers_count":251,"open_issues_count":20,"forks_count":20,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-28T13:08:15.096Z","etag":null,"topics":["excelreader","r","rcpp","spreadsheet","tidy"],"latest_commit_sha":null,"homepage":"https://nacnudus.github.io/tidyxl/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nacnudus.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-06-16T11:05:29.000Z","updated_at":"2025-03-26T04:43:35.000Z","dependencies_parsed_at":"2023-12-21T14:32:44.670Z","dependency_job_id":"6b2eb25d-e6ac-41ea-ab4e-38b1bb9a82a1","html_url":"https://github.com/nacnudus/tidyxl","commit_stats":{"total_commits":682,"total_committers":5,"mean_commits":136.4,"dds":0.06744868035190621,"last_synced_commit":"816501f97cbe8fa3074267c819207cd78728c5ad"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nacnudus%2Ftidyxl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nacnudus%2Ftidyxl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nacnudus%2Ftidyxl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nacnudus%2Ftidyxl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nacnudus","download_url":"https://codeload.github.com/nacnudus/tidyxl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247190233,"owners_count":20898700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["excelreader","r","rcpp","spreadsheet","tidy"],"created_at":"2024-07-31T00:00:43.627Z","updated_at":"2025-04-04T14:02:51.853Z","avatar_url":"https://github.com/nacnudus.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"README-\"\n)\n```\n\n# tidyxl\n\n\u003c!-- badges: start --\u003e\n[![R-CMD-check](https://github.com/nacnudus/tidyxl/workflows/R-CMD-check/badge.svg)](https://github.com/nacnudus/tidyxl/actions)\n[![Cran Status](http://www.r-pkg.org/badges/version/tidyxl)](https://cran.r-project.org/package=tidyxl)\n[![Cran Downloads](https://cranlogs.r-pkg.org/badges/tidyxl)](https://www.r-pkg.org/pkg/tidyxl)\n[![codecov](https://app.codecov.io/gh/nacnudus/tidyxl/coverage.svg?branch=master)](https://app.codecov.io/gh/nacnudus/tidyxl)\n[![R-CMD-check](https://github.com/nacnudus/tidyxl/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/nacnudus/tidyxl/actions/workflows/R-CMD-check.yaml)\n\u003c!-- badges: end --\u003e\n\n[tidyxl](https://github.com/nacnudus/tidyxl) imports non-tabular data from Excel\nfiles into R.  It exposes cell content, position, formatting and comments in a\ntidy structure for further manipulation, especially by the\n[unpivotr](https://github.com/nacnudus/unpivotr) package.  It supports the\nxml-based file formats '.xlsx' and '.xlsm' via the embedded\n[RapidXML](https://rapidxml.sourceforge.net) C++ library.  It does not support\nthe binary file formats '.xlsb' or '.xls'.\n\nIt also provides a function `xlex()` for tokenizing formulas.  See the\n[vignette](https://nacnudus.github.io/tidyxl/articles/smells.html) for\ndetails.  It is useful for detecting 'spreadsheet smells' (poor practice such as\nembedding constants in formulas, or using deep levels of nesting), and for\nunderstanding the dependency structures within spreadsheets.\n\n## Make cells tidy\n\nTidyxl represents each cell as one row in a dataframe.  Like this.\n\n```{r, echo = FALSE, fig.alt = \"Gif of tidyxl converting cells into a tidy representation of one row per cell\"}\nknitr::include_graphics(\"man/figures/tidy_xlsx.gif\")\n```\n\nWhat can you do with tidy cells?  The best places to start are:\n\n* [Spreadsheet Munging\n  Strategies](https://nacnudus.github.io/spreadsheet-munging-strategies/), a\n  free, online cookbook using [tidyxl](https://github.com/nacnudus/tidyxl) and\n  [unpivotr](https://github.com/nacnudus/unpivotr)\n* [Screencasts](https://www.youtube.com/watch?v=1sinC7wsS5U) on YouTube.\n* [Worked examples](https://github.com/nacnudus/ukfarm) on GitHub.\n\nOtherwise see examples below.\n\n***Breaking changes***\n\nThe version 1.0.0 release has some breaking changes.  See `NEWS.md` for details.\nThe previous version can be installed as follows.\n\n```r\ndevtools::install_version(\"tidyxl\", version = \"0.2.3\", repos = \"http://cran.us.r-project.org\")\n```\n\n## Installation\n\n```{r, echo = TRUE, eval = FALSE}\ndevtools::install_github(\"nacnudus/tidyxl\")\n```\n\n## Examples\n\nThe package includes a spreadsheet, 'titanic.xlsx', which contains the following\npivot table:\n\n```{r, echo = TRUE}\nftable(Titanic, row.vars = 1:2)\n```\n\nThe multi-row column headers make this difficult to import.  A popular package\nfor importing spreadsheets coerces the pivot table into a dataframe. It treats\nthe second header row as though it were observations.\n\n```{r, echo = TRUE}\ntitanic \u003c- system.file(\"extdata/titanic.xlsx\", package = \"tidyxl\")\nreadxl::read_excel(titanic)\n```\n\n[tidyxl](https://github.com/nacnudus/tidyxl) doesn't coerce the pivot table into\na data frame.  Instead, it represents each cell in its own row, where it\ndescribes the cell's address, value and other properties.\n\n```{r, echo = TRUE}\nlibrary(tidyxl)\nx \u003c- xlsx_cells(titanic)\ndplyr::glimpse(x)\n```\n\nIn this structure, the cells can be found by filtering.\n\n```{r, echo = TRUE}\nx[x$data_type == \"character\", c(\"address\", \"character\")]\nx[x$row == 4, c(\"address\", \"character\", \"numeric\")]\n```\n\nSpecific sheets can be requested using `xlsx_cells(file, sheet)`, and the names\nof all sheets in a file are given by `xlsx_sheet_names()`.\n\n\n### Formatting\n\nThe original spreadsheet has formatting applied to the cells.  This can also be\nretrieved using [tidyxl](https://github.com/nacnudus/tidyxl), with the\n`xlsx_formats()` function.\n\n```{r, echo = FALSE, fig.alt = \"Screenshot of a spreadsheet with meaningful formatting\"}\nknitr::include_graphics(\"vignettes/titanic-screenshot.png\")\n```\n\nFormatting is available by using the columns `local_format_id` and\n`style_format` as indexes into a separate list-of-lists structure.  'Local'\nformatting is the most common kind, applied to individual cells.  'Style'\nformatting is usually applied to blocks of cells, and defines several formats at\nonce.  Here is a screenshot of the styles buttons in Excel.\n\n```{r, echo = FALSE, fig.alt = \"Screenshot of a formatting options available in Excel\"}\nknitr::include_graphics(\"vignettes/styles-screenshot.png\")\n```\n\nFormatting can be looked up as follows.\n\n```{r, echo = TRUE}\n# Bold\nformats \u003c- xlsx_formats(titanic)\nformats$local$font$bold\nx[x$local_format_id %in% which(formats$local$font$bold),\n  c(\"address\", \"character\")]\n\n# Yellow fill\nformats$local$fill$patternFill$fgColor$rgb\nx[x$local_format_id %in%\n  which(formats$local$fill$patternFill$fgColor$rgb == \"FFFFFF00\"),\n  c(\"address\", \"numeric\")]\n\n# Styles by name\nformats$style$font$name[\"Normal\"]\nhead(x[x$style_format == \"Normal\", c(\"address\", \"character\")])\n\n# In-cell formatting is available in the `character_formatted` column as a data\n# frame, one row per substring.\nexamples \u003c- system.file(\"/extdata/examples.xlsx\", package = \"tidyxl\")\nxlsx_cells(examples)$character_formatted[77]\n```\n\nTo see all the available kinds of formats, use `str(formats)`.\n\n### Comments\n\nComments are available alongside cell values.\n\n```{r, echo = TRUE}\nx[!is.na(x$comment), c(\"address\", \"comment\")]\n```\n\n### Formulas\n\nFormulas are available, but with a few quirks.\n\n```{r, echo = TRUE}\noptions(width = 120)\ny \u003c- xlsx_cells(examples, \"Sheet1\")\ny[!is.na(y$formula),\n  c(\"address\", \"formula\", \"is_array\", \"formula_ref\", \"formula_group\",\n    \"error\", \"logical\", \"numeric\", \"date\", \"character\")]\n```\n\nThe top five cells show that the results of formulas are available as usual in\nthe columns `error`, `logical`, `numeric`, `date`, and `character`.\n\n#### Shared formulas\n\nCells `A20` and `A21` illustrate how formulas are normalised before being\nwritten to file, using the `formula_ref` and `formula_group` columns.  When\nthere is a group of cells whose formulas only differ by cell reference (e.g.\n\"=A1+1\", \"=A2+1\", \"=A3+1\", etc.), only one formula in each group is written to\nthe file, so `tidyxl` infers what the formulas in the other cells in the group\nmust be, from their relative positions.\n\n#### Array formulas\n\nThere are two kinds of array formulas: ones that compute over arrays, and ones\nwhose output is an array (of cells).\n\nBoth kinds are distinguished in\nspreadsheet programs by curly braces, e.g. `{SUM(A19:A21*B19:B21)}`.  In\n`tidyxl`, the curly braces are ommitted (as they are from the file itself), and\ninstead the `is_array` column has the value `TRUE`.\n\nThe first kind (those that compute over arrays) is illustrated by cell `A22`.\n\nThe second kind (those whose value is spread across an array of cells) is\nillustrated by cells `A23` and `A24`.  The formula is only given in the top-left\ncell (`A23`), which is also the only cell that describes the range of cells\ncontaining the result, in the `formula-ref` column.  The results themselves are\nstored in all relevant cells (`A23` and `A24`).  Unlike shared formulas, there\nis no `formula_group` to associate the cells of an array formula's result.  If\nyou need to do identify those cells, use the\n[cellranger](https://github.com/rsheets/cellranger) package and the\n`formula_ref` column.\n\n#### Formulas referring to other files\n\nCell `A25` contains a formula that refers to another file.  The `[1]` is an\nindex into a table of files.  The roadmap for\n[tidyxl](https://github.com/nacnudus/tidyxl) includes de-referencing such\nnumbers.\n\n#### Tokenizing formulas\n\nThe function `xlex()` separates formulas into tokens of different types, and\ngives their depth within a nested formula.  Its name is a bad pun on 'Excel' and\n'lexer'.  Try the [online demo](https://duncan-garmonsway.shinyapps.io/xlex/),\nor install the more experimental [lexl](https://nacnudus.github.io/lexl/)\npackage to run `demo_lexl()` locally.\n\nIt is useful for detecting spreadsheet smells, which are poor practices in\nspreadsheet design, such as deep nests of functions, or embedding\nconstants in formulas.\n\n```{r, fig.width = 7, fig.height = 5}\nx \u003c- xlex(\"MIN(3,MAX(2,A1))\")\nx\n```\n\nSee the [vignette](https://nacnudus.github.io/tidyxl/articles/smells.html) for\nmore examples and details.\n\n### Named ranges\n\nNames are imported with `xlsx_names()`.  Also called 'named formulas' and 'defined\nnames', these are usually used to name particular cells or ranges, making\nformulas that refer to them more readable.  Ones that *are* ranges are identifed\nby the `is_range` column (using `is_range()`), making it easier to match the\nnames to the cells returned by `xlsx_cells()` -- e.g. by using the\n[`cellranger`](https://github.com/rsheets/cellranger) package.\n\nWhen the scope of the name is within a particular sheet, rather than global, the\nsheet name is given.\n\n```{r}\nxlsx_names(examples)\n```\n\n### Data validation rules\n\nData validation rules are imported with `xlsx_validation()`.  These rules\ncontrol what values may be entered into a cell, and are often used to create a\ndrop-down list in a cell.  Read the\n[vignette](https://nacnudus.github.io/tidyxl/articles/data-validation-rules.html)\nfor details.\n\n```{r}\nxlsx_validation(examples)\n```\n\n## Philosophy\n\nInformation in in many spreadsheets cannot be easily imported into R.  Why?\n\nMost R packages that import spreadsheets have difficulty unless the layout of\nthe spreadsheet conforms to a strict definition of a 'table', e.g.:\n\n* observations in rows\n* variables in columns\n* a single header row\n* all information represented by characters, whether textual, logical, or\n  numeric\n\nThese rules are designed to eliminate ambiguity in the interpretation of the\ninformation.  But most spreadsheeting software relaxes these rules in a trade of\nambiguity for expression via other media:\n\n* proximity (other than headers, i.e. other than being the first value at the\n  top of a column)\n* formatting (colours and borders)\n\nHumans can usually resolve the ambiguities with contextual knowledge, but\ncomputers are limited by their ignorance.  Programmers are hampered by:\n\n* their language's expressiveness\n* loss of information in transfer from spreadsheet to programming library\n\nInformation is lost when software discards it in order to force the data into\ntabular form.  Sometimes date formatting is retained, but mostly formatting\nis lost, and position has to be inferred again.\n\n[tidyxl](https://github.com/nacnudus/tidyxl) addresses the programmer's problems\nby not discarding information.  It imports the content, position and formatting\nof cells, leaving it up to the user to associate the different forms of\ninformation, and to re-encode them in tabular form without loss.  The\n[unpivotr](https://github.com/nacnudus/unpivotr) package has been developed to\nassist with that step.\n\n## Similar projects\n\n[tidyxl](https://github.com/nacnudus/tidyxl) was originally derived from\n[readxl](https://github.com/tidyverse/readxl) and still contains some of the same\ncode, hence it inherits the GPL-3 licence.\n[readxl](https://github.com/tidyverse/readxl) is intended for importing tabular\ndata with a single row of column headers, whereas\n[tidyxl](https://github.com/nacnudus/tidyxl) is more general, and less magic.\n\nThe [rsheets](https://github.com/rsheets) project of several R packages is in\nthe early stages of importing spreadsheet information from Excel and Google\nSheets into R, manipulating it, and potentially parsing and processing formulas\nand writing out to spreadsheet files.  In particular,\n[jailbreaker](https://github.com/rsheets/jailbreakr) attempts to extract\nnon-tabular data from spreadsheets into tabular structures automatically via\nsome clever algorithms.\n\n[tidyxl](https://github.com/nacnudus/tidyxl) differs from\n[rsheets](https://github.com/rsheets) in scope\n([tidyxl](https://github.com/nacnudus/tidyxl) will never import charts, for\nexample), and implementation ([tidyxl](https://github.com/nacnudus/tidyxl) is\nimplemented mainly in C++ and is quite fast, only a little slower than\n[readxl](https://github.com/tidyverse/readxl)).\n[unpivotr](https://github.com/nacnudus/unpivotr) is a package related to\n[tidyxl](https://github.com/nacnudus/tidyxl) that provides tools for unpivoting\ncomplex and non-tabular data layouts using I not AI (intelligence, not\nartificial intelligence).  In this way it corresponds to\n[jailbreaker](https://github.com/rsheets/jailbreakr), but with a different\nphilosophy.\n\n## Mailing list\n\nFor bugs and/or issues, create a new issue on\n[GitHub](https://github.com/nacnudus/tidyxl/issues) For other questions or\ncomments, please subscribe to the [tidyxl-devel mailing\nlist](https://groups.google.com/forum/#!forum/tidyxl-devel). You must be a\nmember to post messages, but anyone can read the archived discussions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnacnudus%2Ftidyxl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnacnudus%2Ftidyxl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnacnudus%2Ftidyxl/lists"}