{"id":18520169,"url":"https://github.com/mlr-org/mlr3inferr","last_synced_at":"2025-07-24T12:34:36.839Z","repository":{"id":248548899,"uuid":"825742779","full_name":"mlr-org/mlr3inferr","owner":"mlr-org","description":"Statistical methods for inference on the generalization error","archived":false,"fork":false,"pushed_at":"2025-04-23T08:58:54.000Z","size":2701,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-18T10:52:59.011Z","etag":null,"topics":["machine-learning","r","statistics"],"latest_commit_sha":null,"homepage":"https://mlr3inferr.mlr-org.com/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlr-org.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"mlr-org"}},"created_at":"2024-07-08T12:07:28.000Z","updated_at":"2025-04-23T08:57:01.000Z","dependencies_parsed_at":"2025-01-28T08:22:05.497Z","dependency_job_id":"e9dc955c-0fc6-46b8-9feb-e39d82cd2721","html_url":"https://github.com/mlr-org/mlr3inferr","commit_stats":null,"previous_names":["mlr-org/mlr3inference","mlr-org/mlr3inferr"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mlr-org/mlr3inferr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3inferr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3inferr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3inferr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3inferr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlr-org","download_url":"https://codeload.github.com/mlr-org/mlr3inferr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3inferr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266844176,"owners_count":23993963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","r","statistics"],"created_at":"2024-11-06T17:18:50.480Z","updated_at":"2025-07-24T12:34:36.800Z","avatar_url":"https://github.com/mlr-org.png","language":"R","funding_links":["https://github.com/sponsors/mlr-org"],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  cache = FALSE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\nlibrary(mlr3)\nlibrary(mlr3misc)\nset.seed(1)\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\n```\n\n# mlr3inferr \u003ca href=\"https://mlr3inferr.mlr-org.com\"\u003e\u003cimg src=\"man/figures/logo.png\" align=\"right\" height=\"138\" alt=\"mlr3inferr website\" /\u003e\u003c/a\u003e\n\nMethods for statistical **inf**erence on the generalization **err**or.\n\nPackage website: [release](https://mlr3inferr.mlr-org.com/) | [dev](https://mlr3inferr.mlr-org.com/dev/)\n\n\u003c!-- badges: start --\u003e\n[![RCMD Check](https://github.com/mlr-org/mlr3inferr/actions/workflows/r-cmd-check.yml/badge.svg)](https://github.com/mlr-org/mlr3inferr/actions/workflows/r-cmd-check.yml)\n[![CRAN status](https://www.r-pkg.org/badges/version/mlr3inferr)](https://CRAN.R-project.org/package=mlr3inferr)\n[![StackOverflow](https://img.shields.io/badge/stackoverflow-mlr3-orange.svg)](https://stackoverflow.com/questions/tagged/mlr3)\n[![Mattermost](https://img.shields.io/badge/chat-mattermost-orange.svg)](https://lmmisld-lmu-stats-slds.srv.mwn.de/mlr_invite/)\n\u003c!-- badges: end --\u003e\n\n\n\n## Installation\n\n```{r, eval = FALSE}\n# Install release from CRAN\ninstall.packages(\"mlr3inferr\")\n# Install development version from GitHub\npak::pkg_install(\"mlr-org/mlr3inferr\")\n```\n\n\n## What is `mlr3inferr`?\n\nThe main purpose of the package is to allow to obtain confidence intervals for the generalization error for a number of resampling methods.\nBelow, we evaluate a decision tree on the sonar task using a holdout resampling and obtain a confidence interval for the generalization error.\nThis is achieved using the `msr(\"ci.holdout\")` measure, to which we pass another `mlr3::Measure` that determines the loss function.\n\n\n```{r}\nlibrary(mlr3inferr)\n\nrr = resample(tsk(\"sonar\"), lrn(\"classif.rpart\"), rsmp(\"holdout\"))\n# 0.05 is also the default\nci = msr(\"ci.holdout\", \"classif.acc\", alpha = 0.05)\nrr$aggregate(ci)\n```\n\nIt is also possible to select the default inference method for a certain `Resampling` method using `msr(\"ci\")`\n\n```{r}\nci_default = msr(\"ci\", \"classif.acc\")\nrr$aggregate(ci_default)\n```\n\nWith [`mlr3viz`](https://mlr3viz.mlr-org.com), it is also possible to visualize multiple confidence intervals.\nBelow, we compare a random forest with a decision tree and a featureless learner:\n\n\n```{r, dpi = 300, out.width = \"70%\", fig.align = \"center\"}\nlibrary(mlr3learners)\nlibrary(mlr3viz)\n\nbmr = benchmark(benchmark_grid(\n  tsks(c(\"sonar\", \"german_credit\")),\n  lrns(c(\"classif.rpart\", \"classif.ranger\", \"classif.featureless\")),\n  rsmp(\"subsampling\")\n))\n\nautoplot(bmr, \"ci\", msr(\"ci\", \"classif.ce\"))\n```\n\nNote that:\n\n* Some methods require pointwise loss functions, i.e. have an `$obs_loss` field.\n* Not for every resampling method exists an inference method.\n* There are combinations of datasets and learners, where inference methods can fail.\n\n## Features\n\n* Additional Resampling Methods\n* Confidence Intervals for the Generalization Error for some resampling methods\n\n\n## Inference Methods\n\n```{r, echo = FALSE}\ncontent = as.data.table(mlr3::mlr_measures, objects = TRUE)[startsWith(get(\"key\"), \"ci.\"),]\ncontent$resamplings = map(content$object, function(x) paste0(gsub(\"Resampling\", \"\", x$resamplings), collapse = \", \"))\ncontent[[\"only pointwise loss\"]] = map_chr(content$object, function(object) {\n  if (get_private(object)$.requires_obs_loss) \"yes\" else \"false\"\n})\ncontent = content[, c(\"key\", \"label\", \"resamplings\", \"only pointwise loss\")]\nknitr::kable(content, format = \"markdown\", col.names = tools::toTitleCase(names(content)))\n```\n\n## Citing mlr3\n\nIf you use mlr3inferr, please cite our paper:\n\n    @misc{kuempelfischer2024ciforge,\n          title={Constructing Confidence Intervals for 'the' Generalization Error -- a Comprehensive Benchmark Study},\n          author={Hannah Schulz-Kümpel and Sebastian Fischer and Thomas Nagler and Anne-Laure Boulesteix and Bernd Bischl and Roman Hornung},\n          year={2024},\n          eprint={2409.18836},\n          archivePrefix={arXiv},\n          primaryClass={stat.ML},\n          url={https://arxiv.org/abs/2409.18836},\n    }\n\n\n## Acknowledgements\n\nThis R package is developed as part of the [Mathematical Research Data Initiative](https://www.mardi4nfdi.de/about/mission).\n\n## Bugs, Questions, Feedback\n\n*mlr3inferr* is a free and open source software project that\nencourages participation and feedback. If you have any issues,\nquestions, suggestions or feedback, please do not hesitate to open an\n“issue” about it on the GitHub page\\!\n\nIn case of problems / bugs, it is often helpful if you provide a\n“minimum working example” that showcases the behaviour (but don’t\nworry about this if the bug is obvious).\n\nPlease understand that the resources of the project are limited:\nresponse may sometimes be delayed by a few days, and some feature\nsuggestions may be rejected if they are deemed too tangential to the\nvision behind the project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlr-org%2Fmlr3inferr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlr-org%2Fmlr3inferr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlr-org%2Fmlr3inferr/lists"}