{"id":19253592,"url":"https://github.com/tidymodels/tidyposterior","last_synced_at":"2025-04-12T19:46:44.439Z","repository":{"id":38900441,"uuid":"107034275","full_name":"tidymodels/tidyposterior","owner":"tidymodels","description":"Bayesian comparisons of models using resampled statistics","archived":false,"fork":false,"pushed_at":"2024-10-17T15:29:44.000Z","size":33908,"stargazers_count":102,"open_issues_count":6,"forks_count":10,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-03T22:11:17.501Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://tidyposterior.tidymodels.org","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tidymodels.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-15T17:39:33.000Z","updated_at":"2024-12-30T22:23:16.000Z","dependencies_parsed_at":"2024-11-09T18:33:47.079Z","dependency_job_id":"81fa93bc-b0de-4f5a-a2dd-cdda1e7b30fb","html_url":"https://github.com/tidymodels/tidyposterior","commit_stats":{"total_commits":190,"total_committers":10,"mean_commits":19.0,"dds":"0.13684210526315788","last_synced_commit":"04d0aa85cbd9a7bf0d3c4664aa42927933301508"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidymodels%2Ftidyposterior","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidymodels%2Ftidyposterior/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidymodels%2Ftidyposterior/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tidymodels%2Ftidyposterior/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tidymodels","download_url":"https://codeload.github.com/tidymodels/tidyposterior/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625497,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T18:31:53.191Z","updated_at":"2025-04-12T19:46:44.403Z","avatar_url":"https://github.com/tidymodels.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n# tidyposterior \u003ca href='https://tidyposterior.tidymodels.org/'\u003e\u003cimg src='man/figures/logo.png' align=\"right\" height=\"139\" /\u003e\u003c/a\u003e\n\n\u003c!-- badges: start --\u003e\n[![R-CMD-check](https://github.com/tidymodels/tidyposterior/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/tidymodels/tidyposterior/actions/workflows/R-CMD-check.yaml)\n[![Codecov test coverage](https://codecov.io/gh/tidymodels/tidyposterior/branch/main/graph/badge.svg)](https://app.codecov.io/gh/tidymodels/tidyposterior?branch=main)\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/tidyposterior)](https://CRAN.r-project.org/package=tidyposterior)\n[![Downloads](http://cranlogs.r-pkg.org/badges/tidyposterior)](https://CRAN.r-project.org/package=tidyposterior)\n![](https://img.shields.io/badge/lifecycle-maturing-blue.svg)\n\n\u003c!-- badges: end --\u003e\n\nThis package can be used to conduct _post hoc_ analyses of resampling results generated by models. \n\nFor example, if two models are evaluated with the root mean squared error (RMSE) using 10-fold cross-validation, there are 10 paired statistics. These can be used to make comparisons between models without involving a test set. \n\nThere is a rich literature on the analysis of model resampling results such as McLachlan's [_Discriminant Analysis and Statistical Pattern Recognition_](https://books.google.com/books?id=O_qHDLaWpDUC\u0026lpg=PR7\u0026ots=6GJnIREXZM\u0026dq=%22Discriminant%20Analysis%20and%20Statistical%20Pattern%20Recognition%22\u0026lr\u0026pg=PR7#v=onepage\u0026q=%22Discriminant%20Analysis%20and%20Statistical%20Pattern%20Recognition%22\u0026f=false) and the references therein. This package follows _the spirit_ of [Benavoli _et al_ (2017)](https://people.idsia.ch//~marco/papers/2017jmlr-tests.pdf). \n\ntidyposterior uses Bayesian generalized linear models for this purpose and can be considered an upgraded version of the [`caret::resamples()`](https://topepo.github.io/caret/model-training-and-tuning.html#exploring-and-comparing-resampling-distributions) function. The package works with [rsample](https://rsample.tidymodels.org/) objects natively but any results in a data frame can be used. \n\nSee [Chapter 11](https://www.tmwr.org/compare.html) of [_Tidy Models with R_](https://www.tmwr.org) for examples and more details. \n\n## Installation\n\nYou can install the released version of tidyposterior from [CRAN](https://CRAN.R-project.org) with:\n\n``` r\ninstall.packages(\"tidyposterior\")\n```\n\nAnd the development version from [GitHub](https://github.com/) with:\n\n``` r\n# install.packages(\"pak\")\npak::pak(\"tidymodels/tidyposterior\")\n```\n## Example\n\nTo illustrate, here are some example objects using 10-fold cross-validation for a simple two-class problem: \n\n```{r setup, results = \"hide\"}\nlibrary(tidymodels)\nlibrary(tidyposterior)\n\ndata(two_class_dat, package = \"modeldata\")\n\nset.seed(100)\nfolds \u003c- vfold_cv(two_class_dat)\n```\n\nWe can define two different models (for simplicity, with no tuning parameters).\n\n```{r model-specs}\nlogistic_reg_glm_spec \u003c-\n  logistic_reg() %\u003e%\n  set_engine('glm')\n\nmars_earth_spec \u003c-\n  mars(prod_degree = 1) %\u003e%\n  set_engine('earth') %\u003e%\n  set_mode('classification')\n```\n\nFor tidymodels, the [tune::fit_resamples()] function can be used to estimate performance for each model/resample:\n\n```{r tm-resamples}\nrs_ctrl \u003c- control_resamples(save_workflow = TRUE)\n\nlogistic_reg_glm_res \u003c- \n  logistic_reg_glm_spec %\u003e% \n  fit_resamples(Class ~ ., resamples = folds, control = rs_ctrl)\n\nmars_earth_res \u003c- \n  mars_earth_spec %\u003e% \n  fit_resamples(Class ~ ., resamples = folds, control = rs_ctrl)\n```\n\nFrom these, there are several ways to pass the results to the `perf_mod()` function. The most general approach is to have a data frame with the resampling labels (i.e., one or more id columns) as well as columns for each model that you would like to compare. \n\nFor the model results above, [tune::collect_metrics()] can be used along with some basic data manipulation steps: \n\n```{r df-results}\nlogistic_roc \u003c- \n  collect_metrics(logistic_reg_glm_res, summarize = FALSE) %\u003e% \n  dplyr::filter(.metric == \"roc_auc\") %\u003e% \n  dplyr::select(id, logistic = .estimate)\n\nmars_roc \u003c- \n  collect_metrics(mars_earth_res, summarize = FALSE) %\u003e% \n  dplyr::filter(.metric == \"roc_auc\") %\u003e% \n  dplyr::select(id, mars = .estimate)\n\nresamples_df \u003c- full_join(logistic_roc, mars_roc, by = \"id\")\nresamples_df\n```\n\nWe can then give this directly to `perf_mod()`: \n\n```{r df-mod}\nset.seed(101)\nroc_model_via_df \u003c- perf_mod(resamples_df, iter = 2000)\n```\n\nFrom this, the posterior distributions for each model can be obtained from the `tidy()` method: \n\n```{r post}\n#| fig-alt: \"Faceted histogram chart. Area Under the ROC Curve along the x-axis, count along the y-axis. The two facets are logistic and mars. Both histogram looks fairly normally distributed, with a mean of 0.89 for logistic and 0.88 for mars. The full range is 0.84 to 0.93.\"\nroc_model_via_df %\u003e% \n  tidy() %\u003e% \n  ggplot(aes(x = posterior)) + \n  geom_histogram(bins = 40, col = \"blue\", fill = \"blue\", alpha = .4) + \n  facet_wrap(~ model, ncol = 1) + \n  xlab(\"Area Under the ROC Curve\")\n```\n\n\nSee `contrast_models()` for how to analyze these distributions \n\n\n## Contributing\n\nThis project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/1/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.\n\n- For questions and discussions about tidymodels packages, modeling, and machine learning, please [post on RStudio Community](https://community.rstudio.com/new-topic?category_id=15\u0026tags=tidymodels,question).\n\n- If you think you have encountered a bug, please [submit an issue](https://github.com/tidymodels/tidyposterior/issues).\n\n- Either way, learn how to create and share a [reprex](https://reprex.tidyverse.org/articles/articles/learn-reprex.html) (a minimal, reproducible example), to clearly communicate about your code.\n\n- Check out further details on [contributing guidelines for tidymodels packages](https://www.tidymodels.org/contribute/) and [how to get help](https://www.tidymodels.org/help/).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftidymodels%2Ftidyposterior","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftidymodels%2Ftidyposterior","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftidymodels%2Ftidyposterior/lists"}