{"id":13665815,"url":"https://github.com/easystats/performance","last_synced_at":"2025-04-29T18:59:56.327Z","repository":{"id":34541208,"uuid":"169959804","full_name":"easystats/performance","owner":"easystats","description":":muscle: Models' quality and performance metrics (R2, ICC, LOO, AIC, BF, ...)","archived":false,"fork":false,"pushed_at":"2025-04-22T10:48:48.000Z","size":723753,"stargazers_count":1069,"open_issues_count":105,"forks_count":97,"subscribers_count":24,"default_branch":"main","last_synced_at":"2025-04-29T18:59:50.706Z","etag":null,"topics":["aic","easystats","hacktoberfest","loo","machine-learning","mixed-models","models","performance","r","r2","statistics"],"latest_commit_sha":null,"homepage":"https://easystats.github.io/performance/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/easystats.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":".github/CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":".github/SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"easystats"}},"created_at":"2019-02-10T09:01:04.000Z","updated_at":"2025-04-28T05:16:56.000Z","dependencies_parsed_at":"2023-01-16T23:15:32.223Z","dependency_job_id":"aecf62f3-eccb-4503-99d7-34f636033368","html_url":"https://github.com/easystats/performance","commit_stats":{"total_commits":2131,"total_committers":26,"mean_commits":81.96153846153847,"dds":"0.23510089160018766","last_synced_commit":"4faff5aabcd5802aae04da30b81e10d5d8c3f220"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easystats%2Fperformance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easystats%2Fperformance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easystats%2Fperformance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/easystats%2Fperformance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/easystats","download_url":"https://codeload.github.com/easystats/performance/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251565802,"owners_count":21609979,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aic","easystats","hacktoberfest","loo","machine-learning","mixed-models","models","performance","r","r2","statistics"],"created_at":"2024-08-02T06:00:51.213Z","updated_at":"2025-04-29T18:59:56.304Z","avatar_url":"https://github.com/easystats.png","language":"R","funding_links":["https://github.com/sponsors/easystats"],"categories":["R","Tools"],"sub_categories":["Performance (\u0026 Automated ML)"],"readme":"---\noutput: github_document\nbibliography: paper.bib\n---\n\n# performance \u003cimg src='man/figures/logo.png' align=\"right\" height=\"139\" /\u003e\n\n```{r, echo = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  warning = FALSE,\n  message = FALSE,\n  out.width = \"100%\",\n  dpi = 150,\n  fig.path = \"man/figures/\",\n  comment = \"#\u003e\"\n)\n\noptions(\n  knitr.kable.NA = \"\",\n  digits = 4,\n  width = 100\n)\n\nlibrary(performance)\n```\n\n[![DOI](https://joss.theoj.org/papers/10.21105/joss.03139/status.svg)](https://doi.org/10.21105/joss.03139)\n[![downloads](https://cranlogs.r-pkg.org/badges/performance)](https://cran.r-project.org/package=performance) [![total](https://cranlogs.r-pkg.org/badges/grand-total/performance)](https://cranlogs.r-pkg.org/)\n\n***Test if your model is a good model!***\n\nA crucial aspect when building regression models is to evaluate the quality of modelfit. It is important to investigate how well models fit to the data and which fit indices to report. Functions to create diagnostic plots or to compute fit measures do exist, however, mostly spread over different packages. There is no unique and consistent approach to assess the model quality for different kind of models.\n\nThe primary goal of the **performance** package is to fill this gap and to provide utilities for computing **indices of model quality** and **goodness of fit**. These include measures like r-squared (R2), root mean squared error (RMSE) or intraclass correlation coefficient (ICC) , but also functions to check (mixed) models for overdispersion, zero-inflation, convergence or singularity.\n\n## Installation\n\n[![CRAN](https://www.r-pkg.org/badges/version/performance)](https://cran.r-project.org/package=performance) [![performance status badge](https://easystats.r-universe.dev/badges/performance)](https://easystats.r-universe.dev) [![codecov](https://codecov.io/gh/easystats/performance/branch/main/graph/badge.svg)](https://app.codecov.io/gh/easystats/performance)\n\nThe *performance* package is available on CRAN, while its latest development version is available on R-universe (from _rOpenSci_).\n\nType | Source | Command\n---|---|---\nRelease | CRAN | `install.packages(\"performance\")`\nDevelopment | R-universe | `install.packages(\"performance\", repos = \"https://easystats.r-universe.dev\")`\n\nOnce you have downloaded the package, you can then load it using:\n\n```{r, eval=FALSE}\nlibrary(\"performance\")\n```\n\n\u003e **Tip**\n\u003e\n\u003e Instead of `library(performance)`, use `library(easystats)`. This will make all features of the easystats-ecosystem available.\n\u003e\n\u003e To stay updated, use `easystats::install_latest()`.\n\n## Citation\n\nTo cite performance in publications use:\n\n```{r}\ncitation(\"performance\")\n```\n\n## Documentation\n\n[![Documentation](https://img.shields.io/badge/documentation-performance-orange.svg?colorB=E91E63)](https://easystats.github.io/performance/)\n[![Blog](https://img.shields.io/badge/blog-easystats-orange.svg?colorB=FF9800)](https://easystats.github.io/blog/posts/)\n[![Features](https://img.shields.io/badge/features-performance-orange.svg?colorB=2196F3)](https://easystats.github.io/performance/reference/index.html)\n\n\nThere is a nice introduction into the package on [youtube](https://www.youtube.com/watch?v=EPIxQ5i5oxs).\n\n## The *performance* workflow\n\n```{r workflow, echo=FALSE, out.width=\"75%\"}\nknitr::include_graphics(\"man/figures/figure_workflow.png\")\n```\n\n### Assessing model quality\n\n#### R-squared\n\n**performance** has a generic `r2()` function, which computes the r-squared for\nmany different models, including mixed effects and Bayesian regression models.\n\n`r2()` returns a list containing values related to the \"most appropriate\"\nr-squared for the given model.\n\n```{r}\nmodel \u003c- lm(mpg ~ wt + cyl, data = mtcars)\nr2(model)\n\nmodel \u003c- glm(am ~ wt + cyl, data = mtcars, family = binomial)\nr2(model)\n\nlibrary(MASS)\ndata(housing)\nmodel \u003c- polr(Sat ~ Infl + Type + Cont, weights = Freq, data = housing)\nr2(model)\n```\n\nThe different R-squared measures can also be accessed directly via functions like `r2_bayes()`, `r2_coxsnell()` or `r2_nagelkerke()` (see a full list of functions [here](https://easystats.github.io/performance/reference/index.html#section-r-functions)).\n\nFor mixed models, the _conditional_ and _marginal_ R-squared are returned. The\n_marginal R-squared_ considers only the variance of the fixed effects and\nindicates how much of the model's variance is explained by the fixed effects\npart only. The _conditional R-squared_ takes both the fixed and random effects\ninto account and indicates how much of the model's variance is explained by the\n\"complete\" model.\n\nFor frequentist mixed models, `r2()` (resp. `r2_nakagawa()`) computes the _mean_\nrandom effect variances, thus `r2()` is also appropriate for mixed models with\nmore complex random effects structures, like random slopes or nested random\neffects [@johnson_extension_2014; @nakagawa_coefficient_2017].\n\n```{r}\nset.seed(123)\nlibrary(rstanarm)\n\nmodel \u003c- stan_glmer(\n  Petal.Length ~ Petal.Width + (1 | Species),\n  data = iris,\n  cores = 4\n)\n\nr2(model)\n\nlibrary(lme4)\nmodel \u003c- lmer(Reaction ~ Days + (1 + Days | Subject), data = sleepstudy)\nr2(model)\n```\n\n#### Intraclass Correlation Coefficient (ICC)\n\nSimilar to R-squared, the ICC provides information on the explained variance and\ncan be interpreted as \"the proportion of the variance explained by the grouping\nstructure in the population\" [@hox_multilevel_2010].\n\n`icc()` calculates the ICC for various mixed model objects, including `stanreg`\nmodels.\n\n```{r}\nlibrary(lme4)\nmodel \u003c- lmer(Reaction ~ Days + (1 + Days | Subject), data = sleepstudy)\nicc(model)\n```\n\n...and models of class `brmsfit`.\n\n```{r, echo=FALSE, eval=curl::has_internet()}\nmodel \u003c- insight::download_model(\"brms_mixed_1\")\n```\n```{r, eval=FALSE}\nlibrary(brms)\nset.seed(123)\nmodel \u003c- brm(mpg ~ wt + (1 | cyl) + (1 + wt | gear), data = mtcars)\n```\n```{r}\nicc(model)\n```\n\n### Model diagnostics\n\n#### Check for overdispersion\n\nOverdispersion occurs when the observed variance in the data is higher than the\nexpected variance from the model assumption (for Poisson, variance roughly\nequals the mean of an outcome). `check_overdispersion()` checks if a count model\n(including mixed models) is overdispersed or not.\n\n```{r}\nlibrary(glmmTMB)\ndata(Salamanders)\nmodel \u003c- glm(count ~ spp + mined, family = poisson, data = Salamanders)\ncheck_overdispersion(model)\n```\n\nOverdispersion can be fixed by either modelling the dispersion parameter (not\npossible with all packages), or by choosing a different distributional family\n(like Quasi-Poisson, or negative binomial, see [@gelman_data_2007]).\n\n#### Check for zero-inflation\n\nZero-inflation (in (Quasi-)Poisson models) is indicated when the amount of\nobserved zeros is larger than the amount of predicted zeros, so the model is\n_underfitting_ zeros. In such cases, it is recommended to use negative binomial\nor zero-inflated models.\n\nUse `check_zeroinflation()` to check if zero-inflation is present in the fitted model.\n\n```{r}\nmodel \u003c- glm(count ~ spp + mined, family = poisson, data = Salamanders)\ncheck_zeroinflation(model)\n```\n\n#### Check for singular model fits\n\nA \"singular\" model fit means that some dimensions of the variance-covariance\nmatrix have been estimated as exactly zero. This often occurs for mixed models\nwith overly complex random effects structures.\n\n`check_singularity()` checks mixed models (of class `lme`, `merMod`, `glmmTMB`\nor `MixMod`) for singularity, and returns `TRUE` if the model fit is singular.\n\n```{r}\nlibrary(lme4)\ndata(sleepstudy)\n\n# prepare data\nset.seed(123)\nsleepstudy$mygrp \u003c- sample(1:5, size = 180, replace = TRUE)\nsleepstudy$mysubgrp \u003c- NA\nfor (i in 1:5) {\n  filter_group \u003c- sleepstudy$mygrp == i\n  sleepstudy$mysubgrp[filter_group] \u003c-\n    sample(1:30, size = sum(filter_group), replace = TRUE)\n}\n\n# fit strange model\nmodel \u003c- lmer(\n  Reaction ~ Days + (1 | mygrp / mysubgrp) + (1 | Subject),\n  data = sleepstudy\n)\n\ncheck_singularity(model)\n```\n\nRemedies to cure issues with singular fits can be found [here](https://easystats.github.io/performance/reference/check_singularity.html).\n\n#### Check for heteroskedasticity\n\nLinear models assume constant error variance (homoskedasticity).\n\nThe `check_heteroscedasticity()` functions assess if this assumption has been\nviolated:\n\n```{r}\ndata(cars)\nmodel \u003c- lm(dist ~ speed, data = cars)\n\ncheck_heteroscedasticity(model)\n```\n\n#### Comprehensive visualization of model checks\n\n**performance** provides many functions to check model assumptions, like\n`check_collinearity()`, `check_normality()` or `check_heteroscedasticity()`. To\nget a comprehensive check, use `check_model()`.\n\n```{r, fig.height=12, fig.width=10, out.width=\"80%\"}\n# defining a model\nmodel \u003c- lm(mpg ~ wt + am + gear + vs * cyl, data = mtcars)\n\n# checking model assumptions\ncheck_model(model)\n```\n\n### Model performance summaries\n\n`model_performance()` computes indices of model performance for regression\nmodels. Depending on the model object, typical indices might be r-squared, AIC,\nBIC, RMSE, ICC or LOOIC.\n\n#### Linear model\n\n```{r}\nm1 \u003c- lm(mpg ~ wt + cyl, data = mtcars)\nmodel_performance(m1)\n```\n\n#### Logistic regression\n\n```{r}\nm2 \u003c- glm(vs ~ wt + mpg, data = mtcars, family = \"binomial\")\nmodel_performance(m2)\n```\n\n#### Linear mixed model\n\n```{r}\nlibrary(lme4)\nm3 \u003c- lmer(Reaction ~ Days + (1 + Days | Subject), data = sleepstudy)\nmodel_performance(m3)\n```\n\n### Models comparison\n\nThe `compare_performance()` function can be used to compare the performance and\nquality of several models (including models of different types).\n\n```{r}\ncounts \u003c- c(18, 17, 15, 20, 10, 20, 25, 13, 12)\noutcome \u003c- gl(3, 1, 9)\ntreatment \u003c- gl(3, 3)\nm4 \u003c- glm(counts ~ outcome + treatment, family = poisson())\n\ncompare_performance(m1, m2, m3, m4, verbose = FALSE)\n```\n\n#### General index of model performance\n\nOne can also easily compute and a [**composite index**](https://easystats.github.io/performance/reference/compare_performance.html#details) of model performance and sort the models from the best one to the worse.\n\n```{r}\ncompare_performance(m1, m2, m3, m4, rank = TRUE, verbose = FALSE)\n```\n\n#### Visualisation of indices of models' performance\n\nFinally, we provide convenient visualisation (the `see` package must be\ninstalled).\n\n```{r}\nplot(compare_performance(m1, m2, m4, rank = TRUE, verbose = FALSE))\n```\n\n### Testing models\n\n`test_performance()` (and `test_bf`, its Bayesian sister) carries out the most\nrelevant and appropriate tests based on the input (for instance, whether the\nmodels are nested or not).\n\n```{r}\nset.seed(123)\ndata(iris)\n\nlm1 \u003c- lm(Sepal.Length ~ Species, data = iris)\nlm2 \u003c- lm(Sepal.Length ~ Species + Petal.Length, data = iris)\nlm3 \u003c- lm(Sepal.Length ~ Species * Sepal.Width, data = iris)\nlm4 \u003c- lm(Sepal.Length ~ Species * Sepal.Width + Petal.Length + Petal.Width, data = iris)\n\ntest_performance(lm1, lm2, lm3, lm4)\n\ntest_bf(lm1, lm2, lm3, lm4)\n```\n\n### Plotting Functions\n\nPlotting functions are available through the [**see** package](https://easystats.github.io/see/articles/performance.html).\n\n\n# Code of Conduct\n\nPlease note that the performance project is released with a [Contributor Code of Conduct](https://easystats.github.io/performance/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.\n\n\n# Contributing\n\nWe are happy to receive bug reports, suggestions, questions, and (most of all)\ncontributions to fix problems and add features.\n\nPlease follow contributing guidelines mentioned here:\n\n\u003chttps://easystats.github.io/performance/CONTRIBUTING.html\u003e\n\n## References\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feasystats%2Fperformance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feasystats%2Fperformance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feasystats%2Fperformance/lists"}