Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tidymodels/butcher
Reduce the size of model objects saved to disk
https://github.com/tidymodels/butcher
Last synced: 3 days ago
JSON representation
Reduce the size of model objects saved to disk
- Host: GitHub
- URL: https://github.com/tidymodels/butcher
- Owner: tidymodels
- License: other
- Created: 2019-06-06T19:45:18.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-08-26T22:00:18.000Z (3 months ago)
- Last Synced: 2024-10-31T22:03:14.656Z (12 days ago)
- Language: R
- Homepage: https://butcher.tidymodels.org/
- Size: 23.1 MB
- Stars: 131
- Watchers: 9
- Forks: 12
- Open Issues: 7
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
Awesome Lists containing this project
- jimsghstars - tidymodels/butcher - Reduce the size of model objects saved to disk (R)
README
---
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# butcher 
[](https://github.com/tidymodels/butcher/actions/workflows/R-CMD-check.yaml)
[](https://CRAN.R-project.org/package=butcher)
[](https://app.codecov.io/gh/tidymodels/butcher?branch=main)
[](https://lifecycle.r-lib.org/articles/stages.html#stable)
## Overview
Modeling or machine learning in R can result in fitted model objects that take up too much memory. There are two main culprits:
1. Heavy usage of formulas and closures that capture the enclosing environment in model training
2. Lack of selectivity in the construction of the model object itself
As a result, fitted model objects contain components that are often redundant and not required for post-fit estimation activities. The butcher package provides tooling to "axe" parts of the fitted output that are no longer needed, without sacrificing prediction functionality from the original model object.
## Installation
Install the released version from CRAN:
```{r, eval = FALSE}
install.packages("butcher")
```
Or install the development version from [GitHub](https://github.com/):
```{r, eval = FALSE}
# install.packages("pak")
pak::pak("tidymodels/butcher")
```
## Butchering
As an example, let's wrap an `lm` model so it contains a lot of unnecessary stuff:
```{r example}
library(butcher)
our_model <- function() {
some_junk_in_the_environment <- runif(1e6) # we didn't know about
lm(mpg ~ ., data = mtcars)
}
```
This object is unnecessarily large:
```{r}
library(lobstr)
obj_size(our_model())
```
When, in fact, it should only be:
```{r}
small_lm <- lm(mpg ~ ., data = mtcars)
obj_size(small_lm)
```
To understand which part of our original model object is taking up the most memory, we leverage the `weigh()` function:
```{r}
big_lm <- our_model()
weigh(big_lm)
```
The problem here is in the `terms` component of our `big_lm`. Because of how `lm()` is implemented in the `stats` package, the environment in which our model was made is carried along in the fitted output. To remove the (mostly) extraneous component, we can use `butcher()`:
```{r}
cleaned_lm <- butcher(big_lm, verbose = TRUE)
```
Comparing it against our `small_lm`, we find:
```{r}
weigh(cleaned_lm)
```
And now it will take up about the same memory on disk as `small_lm`:
```{r}
weigh(small_lm)
```
To make the most of your memory available, this package provides five S3 generics for you to remove parts of a model object:
- `axe_call()`: To remove the call object.
- `axe_ctrl()`: To remove controls associated with training.
- `axe_data()`: To remove the original training data.
- `axe_env()`: To remove environments.
- `axe_fitted()`: To remove fitted values.
When you run `butcher()`, you execute all of these axing functions at once. Any kind of axing on the object will append a butchered class to the current model object class(es) as well as a new attribute named `butcher_disabled` that lists any post-fit estimation functions that are disabled as a result.
## Model Object Coverage
Check out the `vignette("available-axe-methods")` to see butcher's current coverage. If you are working with a new model object that could benefit from any kind of axing, we would love for you to make a pull request! You can visit the `vignette("adding-models-to-butcher")` for more guidelines, but in short, to contribute a set of axe methods:
1. Run `new_model_butcher(model_class = "your_object", package_name = "your_package")`
2. Use butcher helper functions `weigh()` and `locate()` to decide what to axe
3. Finalize edits to `R/your_object.R` and `tests/testthat/test-your_object.R`
4. Make a pull request!
## Contributing
This project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.
- For questions and discussions about tidymodels packages, modeling, and machine learning, please [post on RStudio Community](https://forum.posit.co/new-topic?category_id=15&tags=tidymodels,question).
- If you think you have encountered a bug, please [submit an issue](https://github.com/tidymodels/butcher/issues).
- Either way, learn how to create and share a [reprex](https://reprex.tidyverse.org/articles/articles/learn-reprex.html) (a minimal, reproducible example), to clearly communicate about your code.
- Check out further details on [contributing guidelines for tidymodels packages](https://www.tidymodels.org/contribute/) and [how to get help](https://www.tidymodels.org/help/).