{"id":18520132,"url":"https://github.com/mlr-org/mlr3fairness","last_synced_at":"2025-10-07T06:55:56.336Z","repository":{"id":37686275,"uuid":"368615363","full_name":"mlr-org/mlr3fairness","owner":"mlr-org","description":"mlr3 extension for Fairness in Machine Learning","archived":false,"fork":false,"pushed_at":"2025-06-24T19:55:39.000Z","size":164839,"stargazers_count":14,"open_issues_count":7,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-09-08T15:28:47.798Z","etag":null,"topics":["fairness","machine-learning","mlr3","r","r-package"],"latest_commit_sha":null,"homepage":"https://mlr3fairness.mlr-org.com","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlr-org.png","metadata":{"funding":{"github":"mlr-org"},"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-05-18T17:32:02.000Z","updated_at":"2025-06-24T19:50:59.000Z","dependencies_parsed_at":"2023-02-16T06:31:01.991Z","dependency_job_id":"ffafd917-b4d3-45cf-bfbf-b35cabbd1b9f","html_url":"https://github.com/mlr-org/mlr3fairness","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/mlr-org/mlr3fairness","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3fairness","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3fairness/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3fairness/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3fairness/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlr-org","download_url":"https://codeload.github.com/mlr-org/mlr3fairness/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlr-org%2Fmlr3fairness/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278734410,"owners_count":26036404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fairness","machine-learning","mlr3","r","r-package"],"created_at":"2024-11-06T17:18:41.494Z","updated_at":"2025-10-07T06:55:56.329Z","avatar_url":"https://github.com/mlr-org.png","language":"HTML","funding_links":["https://github.com/sponsors/mlr-org"],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n```{r, echo = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/\"\n)\n```\n\n```{r, include = FALSE}\nlgr::get_logger(\"mlr3\")$set_threshold(\"warn\")\nset.seed(1)\noptions(datatable.print.class = FALSE, datatable.print.keys = FALSE)\n```\n\n# [mlr3fairness](https://github.com/mlr-org/mlr3fairness)  \u003cimg src=\"man/figures/scale_mlr3.png\" align=\"right\" /\u003e\n\nMachine Learning Fairness Extension for [mlr3](https://github.com/mlr-org/mlr3).\n\n[![r-cmd-check](https://github.com/mlr-org/mlr3fairness/actions/workflows/r-cmd-check.yml/badge.svg)](https://github.com/mlr-org/mlr3fairness/actions/workflows/r-cmd-check.yml)\n[![CRAN status](https://www.r-pkg.org/badges/version/mlr3fairness)](https://CRAN.R-project.org/package=mlr3fairness)\n[![Mattermost](https://img.shields.io/badge/chat-mattermost-orange.svg)](https://lmmisld-lmu-stats-slds.srv.mwn.de/mlr_invite/)\n\n##  Installation\n\nInstall the development version from github:\n\n```r\nremotes::install_github(\"mlr-org/mlr3fairness\")\n```\n\n## Why should I care about fairness in machine learning?\n\nMachine Learning model predictions can be skewed by a range of factors and thus might be considered unfair towards certain groups or individuals.\nAn example would be the COMPAS algorithm, which is a popular commercial algorithm used by judges and parole officers for scoring criminal defendant's\nlikelihood of reoffending (recidivism).\n[Studies](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) have shown, that the algorithm might be biased in favor of white defendants.\nBiases can occur in a large variety of situations where algorithms automate or support human decision making e.g. credit checks, automatic HR tools along with a variety of other domains.\n\n\nThe **goal of `mlr3fairness`** is to allow for auditing of `mlr3` learners, visualization and subsequently trying to improve fairness using debiasing strategies.\n\n\u003e :warning: **Note**\n\u003e Bias auditing and debiasing solely based on observational data **can not** guarantee fairness of a decision making system.\n\u003e Several biases, for example comming from the data can not be detected using the approaches implemented in `mlr3fairness`.\n\u003e This goal of this software is thus to allow for a better understanding and first hints at possible fairness problems in a\n\u003e studied model.\n\n\n## Feature Overview\n\n* [**Fairness Measures:**](#fairness-metrics) Audit algorithmms for fairness using a variety of fairness criteria. This also allows for designing custom criteria.\n\n* [**Fairness Visualizations:**](#fairness-visualizations) Diagnose fairness problems through visualizations.\n\n* [**Debiasing Methods:**](#debiasing-methods) Correct fairness problems in three lines of code.\n\n* [**Fairness Report:**](#model-cards--datasheets) Obtain a report regarding an algorithm's fairness. (Under development)\n\n**More Information**\n\n- [Debiasing](https://mlr3fairness.mlr-org.com/articles/debiasing-vignette.html)\n- [Fairness Metrics](https://mlr3fairness.mlr-org.com/articles/measures-vignette.html)\n- [Visualizations](https://mlr3fairness.mlr-org.com/articles/visualization-vignette.html)\n- [Reports](https://mlr3fairness.mlr-org.com/articles/reports-vignette.html)\n\n### Protected Attribute\n\n`mlr3fairness` requires information about the protected attribute wrt. which we want to assess fairness. This can be set via the `col_role` \"pta\" (protected attribute).\n\n```{r, eval = FALSE}\ntask$col_roles$pta = \"variable_name\"\n```\n\nIn case a non-categorical or more complex protected attribute is required, it can be manually computed and added to the task. `mlr3fairness` does not require specific types for `pta`, but will compute one metric for every unique value in the `pta` column.\n\n### Fairness Metrics {#metrics}\n\n`mlr3fairness` offers a variety of fairness metrics.\nMetrics are prefixed with `fairness.` and can be found in the `msr()` dictionary.\nMost fairness metrics are based on a difference between two protected groups (e.g. male and female) for a given metric (e.g. the false positive rate: `fpr`).\nSee [the vignette](https://textbook.coleridgeinitiative.org/chap-bias.html) for a more in-depth introduction to fairness metrics and how to choose them.\n\n```{r}\nlibrary(mlr3)\nlibrary(mlr3fairness)\n```\n```{r, echo = FALSE}\nknitr::kable(mlr3fairness:::mlr_measures_fairness)\n```\n\nAdditional **custom fairness metrics** can be easily constructed, [the vignette](https://textbook.coleridgeinitiative.org/chap-bias.html) contains more details.\nThe `fairness_tensor()` function can be used with a `Prediction` in order to print group-wise confusion matrices for each protected attribute group.\nWe can furthermore measure fairrness in each group separately using `MeasureSubgroup` and `groupwise_metrics`.\n\n### Fairness Visualizations {#viz}\n\nVisualizations can be used with either a `Prediction`, `ResampleResult` or a `BenchmarkResult`.\nFor more information regarding those objects, refer to the [mlr3 book](https://mlr3book.mlr-org.com/).\n\n\n- **fairness_accuracy_tradeoff**: Plot available trade-offs between fairness and model performance.\n\n- **compare_metrics**: Compare fairness across models and cross-validation folds.\n\n- **fairness_prediction_density**: Density plots for each protected attribute.\n\n\n```{r, echo = FALSE, fig.width = 12}\nlibrary(mlr3learners)\nt = tsk(\"adult_train\")$filter(1:5000)\ndesign = benchmark_grid(\n  tasks = t,\n  learners = lrns(c(\"classif.ranger\", \"classif.rpart\"), predict_type = \"prob\"),\n  resamplings = rsmps(\"cv\", folds = 3)\n)\n\nbmr = benchmark(design)\n\nlibrary(ggplot2)\nlibrary(patchwork)\np1 = fairness_accuracy_tradeoff(bmr, msr(\"fairness.fpr\")) + ggtitle(\"fairness_accuracy_tradeoff\")\np2 = compare_metrics(bmr, msrs(c(\"fairness.fpr\", \"fairness.tpr\"))) + ggtitle(\"compare_metrics\")\np3 = fairness_prediction_density(bmr) + ggtitle(\"fairness_prediction_density\")\np1 / (p2 | p3)\n```\n\n### Debiasing Methods {#debiasing}\n\nDebiasing methods can be used to improve the fairness of a given model.\n`mlr3fairness` includes several methods that can be used together with\n`mlr3pipelines` to obtain fair(er) models:\n\n\n```{r, eval = FALSE}\nlibrary(mlr3pipelines)\nlrn = as_learner(po(\"reweighing_wts\") %\u003e\u003e% lrn(\"classif.rpart\"))\nrs = resample(lrn, task = tsk(\"compas\")$filter(1:500), rsmp(\"cv\"))\nrs$score(msr(\"fairness.acc\"))\n```\n\n**Overview:**\n\n```{r, echo = FALSE}\nlibrary(mlr3misc)\nlibrary(mlr3pipelines)\ndt = as.data.table(mlr_pipeops)\nknitr::kable(dt[map_lgl(dt$tags, function(x) \"fairness\" %in% x)][, c(1,7,8,9,10)])\n```\n\n### Fair Learners\n\n`mlr3fairness` furthermore contains several learners that can be used to directly learn fair models:\n\n```{r, echo = FALSE}\ntab = mlr_learners_fairness\nknitr::kable(tab)\n```\n\n### Datasets\n\n`mlr3fairness` includes two fairness datasets: `adult` and `compas`.\nSee `?adult` and `?compas` for additional information regarding columns.\n\nYou can load them using `tsk(\u003ckey\u003e)`.\n\n### Model Cards \u0026 Datasheets\n\nAn important step towards achieving more equitable outcomes for ML models is adequate documentation\nfor datasets and models in machine learning. `mlr3fairness` comes with reporting aides for `models`\nand `datasets`. This provides empty templates that can be used to create interactive reports through\n`RMarkdown`.\n\n\n| Report             |  Description             | Reference                   | Example                                                                    |\n|--------------------|--------------------------|-----------------------------|----------------------------------------------------------------------------|\n| `report_modelcard` | Modelcard for ML models  | Mitchell et al., 2018       | [link](https://mlr3fairness.mlr-org.com/articles/modelcard/modelcard.html) |\n| `report_datasheet` | Datasheet for data sets  | Gebru et al., 2018          | [link](https://mlr3fairness.mlr-org.com/articles/datasheet/datasheet.html) |\n| `report_fairness`  | Fairness Report          | -[^1]                       | [link](https://mlr3fairness.mlr-org.com/articles/fairness/fairness.html)   |\n\n**Usage:**\n\nThe `report_*` functions instantiate a new `.Rmd` template that contains a set of pre-defined\nquestions which can be used for reporting as well as initial graphics.\nThe goal is that a user extends this `.Rmd` file to create comprehensive documentation for\ndatasets, ML models or to document a model's fairness.\nIt can later be converted into a `html` report using`rmarkdown`'s `render`.\n\n```{r, eval= FALSE}\nrmdfile = report_datasheet()\nrmarkdown::render(rmdfile)\n```\n\n\n### Demo for Adult Dataset\n\nWe provide a short example detailing how `mlr3fairness` integrates with the `mlr3` ecosystem.\n\n```r\nlibrary(mlr3fairness)\n\n#Initialize Fairness Measure\nfairness_measure = msr(\"fairness.fpr\")\n#Initialize tasks\ntask_train = tsk(\"adult_train\")\ntask_test = tsk(\"adult_test\")\n#Initialize model\nlearner = lrn(\"classif.rpart\", predict_type = \"prob\")\n\n#Verify fairness metrics\nlearner$train(task_train)\npredictions = learner$predict(task_test)\npredictions$score(fairness_measure, task = task_test)\n\n#Visualize the predicted probability score based on protected attribute.\nfairness_prediction_density(predictions, task_test)\n```\n\n### Extensions\n\n- The [mcboost](https://github.com/mlr-org/mcboost) package integrates with **mlr3** and offers\n  additional debiasing post-processing functionality for **classification**, **regression** and **survival**.\n\n### Other Fairness Toolkits in R\n\n- [fairmodels](https://github.com/ModelOriented/fairmodels/) integrates with the [DALEX](https://github.com/ModelOriented/DALEX) R-packages and similarly allows for bias auditing, visualization and mitigation.\n- The [fairness](https://github.com/kozodoi/fairness) package allows for bias auditing in R.\n- The [fairml](https://cran.r-project.org/package=fairml) package contains methods for learning de-biased regression and classification models. Learners from `fairml` are included as learners in `mlr3fairness`.\n\n### Other Fairness Toolkits\n\n- [Aequitas](http://aequitas.dssg.io/) Allows for constructing a fairness report for different fairness metrics along with visualization in Python.\n- [fairlearn](https://fairlearn.org/) Allows for model auditing and debiasing as well as visualization in Python.\n- [AI Fairness 360](https://github.com/Trusted-AI/AIF360) Allows for model auditing and debiasing as well as visualization in R and Python.\n\n\n### Future Development\n\nSeveral future developments are currently planned. Contributions are highly welcome!\n\n- Visualizations:\n  Improvement on visualizations, like anchor points and others. See issues.\n- Debiasing Methods:\n  More debiasing methods, post-processing and in-processing.\n\n## Bugs, Feedback and Questions\n\n`mlr3fairness` is a free and open source software project that encourages participation and feedback.\nIf you have any issues, questions, suggestions or feedback, please do not hesitate to open an \"issue\" about it on the GitHub page!\nIn case of problems / bugs, it is often helpful if you provide a \"minimum working example\" that showcases the behaviour.\n\n[^1]: The fairness report is inspired by the [Aequitas Bias report](http://aequitas.dssg.io/example.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlr-org%2Fmlr3fairness","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlr-org%2Fmlr3fairness","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlr-org%2Fmlr3fairness/lists"}