{"id":13400674,"url":"https://github.com/ModelOriented/DrWhy","last_synced_at":"2025-03-14T06:31:53.354Z","repository":{"id":45508930,"uuid":"153565978","full_name":"ModelOriented/DrWhy","owner":"ModelOriented","description":"DrWhy is the collection of tools for eXplainable AI (XAI). It's based on shared principles and simple grammar for exploration, explanation and visualisation of predictive models.","archived":false,"fork":false,"pushed_at":"2023-02-21T16:34:36.000Z","size":14810,"stargazers_count":680,"open_issues_count":1,"forks_count":85,"subscribers_count":35,"default_branch":"master","last_synced_at":"2024-07-31T19:25:39.951Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://ModelOriented.github.io/DrWhy/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ModelOriented.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-10-18T05:01:20.000Z","updated_at":"2024-07-22T19:27:11.000Z","dependencies_parsed_at":"2022-07-18T18:42:10.164Z","dependency_job_id":"1371a476-e027-49a9-bfa3-0cf798bd97ae","html_url":"https://github.com/ModelOriented/DrWhy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelOriented%2FDrWhy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelOriented%2FDrWhy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelOriented%2FDrWhy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ModelOriented%2FDrWhy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ModelOriented","download_url":"https://codeload.github.com/ModelOriented/DrWhy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243537833,"owners_count":20307098,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T19:00:54.515Z","updated_at":"2025-03-14T06:31:53.347Z","avatar_url":"https://github.com/ModelOriented.png","language":"R","funding_links":[],"categories":["R","Tools","Technical Resources"],"sub_categories":["(RAI) Toolkit","Open Source/Access Responsible AI Software Packages"],"readme":"# Responsible Machine Learning\n\n*With Great Power Comes Great Responsibility*. \nVoltaire (well, maybe)\n\nHow to develop machine learning models in a responsible manner? There are several topics worth considering:\n\n* **Effective**. Is the model good enough? Models with low performance should not be used because they can do more harm than good. Communicate the performance of the model in a language that the user understands. Remember that the models will work on a different dataset than the training one. Make sure to assess the performance on the target dataset.\n\n* **Transparent**. Does the user know what influences model predictions? Interpretability and explainability is important. If the model decisions affect us directly or indirectly, we should know where these decisions come from and how they can be changed.\n\n* **Fair**. Does the model discriminate on the basis of gender, age, race or other sensitive attribute? Direct or indirect? It should not! Discrimination can come in many faces. The model may give lower scores, may have lower performance, or may be based on different variables for the protected population.\n\n* **Secure**. Do not let your model be hacked. Every complex system has its vulnerabilities. Seek them out and fix them. Some users may use various tricks to pull model predictions onto their site.\n\n* **Confidential**. Models are often built on sensitive data. Make sure that the data does not leak, so that sensitive attributes are not shared with unauthorized persons. Also beware of model leaks. \n\n* **Reproducible**. Usually the model development process consists of many steps. Make sure that they are completely reproducible and thus can be verified one by one.\n\n\n\n\n\n\n\n# Collection of tools for Visual Exploration, Explanation and Debugging of Predictive Models\n\n*It takes a village to raise a \u003cdel\u003echild\u003c/del\u003e model*.\n\nThe way how we do predictive modeling is very ineffective. We spend way too much time on manual time-consuming and easy to automate activities like data cleaning and exploration, crisp modeling, model validation. We should be focusing more on model understanding, productisation and communication.\n\nHere are gathered tools that can be used to make out work more efficient through the whole model lifecycle.\nThe unified grammar beyond DrWhy.AI universe is described in the [Explanatory Model Analysis: Explore, Explain and Examine Predictive Models](https://pbiecek.github.io/ema/) book.\n\n## Lifecycle for Predictive Models\n\nThe DrWhy is based on an unified [Model Development Process](https://github.com/ModelOriented/DrWhy/blob/master/images/ModelDevelopmentProcess.pdf) inspired by RUP. Find an overview in the diagram below.\n\n[![images/DALEXverse.png](images/DALEXverse.png)]( https://modeloriented.github.io/ModelDevelopmentProcess/ )\n\n## The DrWhy.AI family\n\nPackages in the `DrWhy.AI` family of models may be divided into four classes.\n\n* **Model adapters**. Predictive models created with different tools have different structures, and different interfaces. Model adapters create uniform wrappers. This way other packages may operate on models in an unified way. `DALEX` is a lightweight package with generic interface. `DALEXtra` is a package with extensions for heavyweight interfaces like `scikitlearn`, `h2o`, `mlr`.\n\n* **Model agnostic explainers**. These packages implement specific methods for model exploration. They can be applied to a single model or they can compare different models. `ingredients` implements variable specific techniques like Ceteris Paribus, Partial Dependency, Permutation based Feature Importance. `iBreakDown` implements techniques for variable attribution, like Break Down or SHAPley values. `auditor` implements techniques for model validation, residual diagnostic and performance diagnostic.\n\n* **Model specific explainers**. These packages implement model specific techniques. `randomForestExplainer` implements techniques for exploration of `randomForest` models. `EIX` implements techniques for exploration of gbm and xgboost models. `cr19` implements techniques for exploration of survival models. \n\n* **Automated exploration**. These packages combine series of model exploration techniques and produce an automated report of website for model exploration. `modelStudio` implements a dashboard generator for local and global interactive model exploration. `modelDown` implements a HTML website generator for global model cross comparison. \n\n[![images/grammar_of_explanations.png](images/grammar_of_explanations.png)]( https://drwhy.ai/ )\n\nHere is a more detailed overview.\n\n--------------------\n\n### [DALEX](http://github.com/ModelOriented/DALEX) \u003cimg src=\"https://modeloriented.github.io/DALEX/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/DALEX)](https://cran.r-project.org/package=DALEX) [![Build Status](https://api.travis-ci.org/ModelOriented/DALEX.png)](https://travis-ci.org/ModelOriented/DALEX) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/DALEX/master.svg)](https://codecov.io/github/ModelOriented/DALEX?branch=master)[![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-BackBone-373589)](http://drwhy.ai/#BackBone) \n\nThe DALEX package (Descriptive mAchine Learning EXplanations) helps to understand how complex models are working. The main function [explain](https://modeloriented.github.io/DALEX/reference/explain.html) creates a wrapper around a predictive model. Wrapped models may then be explored and compared with a collection of local and global explainers. Recent developments from the area of Interpretable Machine Learning/eXplainable Artificial Intelligence.\n\nDALEX wraps methods from other packages, i.e. 'pdp' (Greenwell 2017) \u003cdoi:10.32614/RJ-2017-016\u003e, 'ALEPlot' (Apley 2018) \u003carXiv:1612.08468\u003e, 'factorMerger' (Sitko and Biecek 2017) \u003carXiv:1709.04412\u003e,  'breakDown' package (Staniak and Biecek 2018) \u003cdoi:10.32614/RJ-2018-072\u003e, (Fisher at al. 2018) \u003carXiv:1801.01489\u003e.\n\nVignettes: \n\n* [General introduction: Survival on the RMS Titanic](https://modeloriented.github.io/DALEX/articles/vignette_titanic.html)\n\n--------------------\n\n### [DALEXtra](http://github.com/ModelOriented/DALEXtra)  \u003cimg src=\"https://github.com/ModelOriented/DALEXtra/blob/master/man/figures/logo.png?raw=true\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/DALEXtra)](https://cran.r-project.org/package=DALEXtra) [![Build Status](https://api.travis-ci.org/ModelOriented/DALEXtra.png)](https://travis-ci.org/ModelOriented/DALEXtra) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/DALEXtra/master.svg)](https://codecov.io/github/ModelOriented/DALEXtra?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-BackBone-373589)](http://drwhy.ai/#BackBone) \n\nThe `DALEXtra` package is an extension pack for [DALEX](https://modeloriented.github.io/DALEX) package.\nThis package provides easy to use connectors for models created with scikitlearn, keras, H2O, mljar and mlr.\n\nVignettes: \n\n* [General introduction: DALEX with scikitlearn models](https://raw.githack.com/pbiecek/DALEX_docs/master/vignettes/How_to_use_DALEXtra_to_explain_and_visualize_scikitlearn_models.html)\n\n--------------------\n\n### [survex](http://github.com/ModelOriented/survex)  \u003cimg src=\"https://raw.githubusercontent.com/ModelOriented/survex/main/man/figures/survex.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN status](https://www.r-pkg.org/badges/version/survex)](https://cran.r-project.org/package=survex) [![R-CMD-check](https://github.com/ModelOriented/survex/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/ModelOriented/survex/actions/workflows/R-CMD-check.yaml) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/survex/master.svg)](https://codecov.io/github/ModelOriented/survex?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-BackBone-373589)](http://drwhy.ai/#BackBone) \n\nThe survex package provides model-agnostic explanations for machine learning survival models. It is based on the DALEX package.\n\nDue to a functional type of prediction, either in the form of survival function or cumulative hazard function, standard model-agnostic explanations cannot be applied directly to survival analysis machine learning models. The survex package contains implementations of explanation methods specific to survival analysis, as well as extensions of existing ones for classification or regression.\n\nVignettes:\n\n* [Package usage](https://modeloriented.github.io/survex/articles/survex-usage.html)\n* [Creating custom explainers](https://modeloriented.github.io/survex/articles/custom-explainers.html)\n* [Integration with `mlr3proba`](https://modeloriented.github.io/survex/articles/mlr3proba-usage.html)\n\n--------------------\n\n### [ingredients](http://github.com/ModelOriented/ingredients)  \u003cimg src=\"https://modeloriented.github.io/ingredients/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/ingredients)](https://cran.r-project.org/package=ingredients) [![Build Status](https://api.travis-ci.org/ModelOriented/ingredients.svg?branch=master)](https://travis-ci.org/ModelOriented/ingredients) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/ingredients/master.svg)](https://codecov.io/github/ModelOriented/ingredients?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `ingredients` package is a collection of tools for assessment of feature importance and feature effects.\n\nKey functions: `feature_importance()` for assessment of global level feature importance, `ceteris_paribus()` for calculation of the Ceteris Paribus / What-If Profiles, `partial_dependency()` for Partial Dependency Plots, `conditional_dependency()` for Conditional Dependency Plots also called M Plots, `accumulated_dependency()` for Accumulated Local Effects Plots,  `cluster_profiles()` for aggregation of Ceteris Paribus Profiles, generic `print()` and `plot()` for better usability of selected explainers, generic `plotD3()` for interactive D3 based explanations, and generic `describe()` for explanations in natural language.\n\n\nVignettes: \n\n* [General introduction: Survival on the RMS Titanic](https://modeloriented.github.io/ingredients/articles/vignette_titanic.html),\n* [Aspects importance](https://modeloriented.github.io/ingredients/articles/vignette_aspect_importance.html), \n* [Explanations in natural language](https://modeloriented.github.io/ingredients/articles/Describing-Explanations.html)\n\n--------------------\n\n### [iBreakDown](http://github.com/ModelOriented/iBreakDown)   \u003cimg src=\"https://modeloriented.github.io/iBreakDown/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/iBreakDown)](https://cran.r-project.org/package=iBreakDown) [![Build Status](https://api.travis-ci.org/ModelOriented/iBreakDown.svg?branch=master)](https://travis-ci.org/ModelOriented/iBreakDown) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/iBreakDown/master.svg)](https://codecov.io/github/ModelOriented/iBreakDown?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `iBreakDown` package is a model agnostic tool for explanation of predictions from black boxes ML models.\nBreak Down Table shows contributions of every variable to a final prediction. \nBreak Down Plot presents variable contributions in a concise graphical way. \nSHAP (Shapley Additive Attributions) values are calculated as average from random Break Down profiles.\nThis package works for binary classifiers as well as regression models. \n\n`iBreakDown` is a successor of the [breakDown](https://github.com/pbiecek/breakDown) package. It is faster (complexity O(p) instead of O(p^2)). It supports interactions and interactive explainers with D3.js plots.\n\nVignettes: \n\n* [General introduction: Survival on the RMS Titanic](https://modeloriented.github.io/iBreakDown/articles/vignette_iBreakDown_titanic.html),\n* [Explanations in natural language](https://modeloriented.github.io/iBreakDown/articles/vignette_iBreakDown_description.html)\n\n--------------------\n\n### [auditor](http://github.com/ModelOriented/auditor)   \u003cimg src=\"https://modeloriented.github.io/auditor/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/auditor)](https://cran.r-project.org/package=auditor) [![Build Status](https://api.travis-ci.org/ModelOriented/auditor.svg?branch=master)](https://travis-ci.org/ModelOriented/auditor) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/auditor/master.svg)](https://codecov.io/github/ModelOriented/auditor?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `auditor` package is a tool for model-agnostic validation. Implemented techniques facilitate assessing and comparing the goodness of fit and performance of models. In addition, they may be used for the analysis of the similarity of residuals and for the identification of outliers and influential observations. The examination is carried out by diagnostic scores and visual verification. Due to the flexible and consistent grammar, it is simple to validate models of any classes.\n\nLearn more: \n\n* Preprint: [auditor: an R Package for Model-Agnostic Visual Validation and Diagnostic](https://arxiv.org/abs/1809.07763),\n* [List of implemented audits](https://github.com/ModelOriented/auditor#a-short-overview-of-plots)\n\n\n--------------------\n\n### [fairmodels](http://github.com/ModelOriented/fairmodels)   \u003cimg src=\"https://modeloriented.github.io/fairmodels/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/fairmodels)](https://cran.r-project.org/package=fairmodels) [![R build status](https://github.com/ModelOriented/fairmodels/workflows/R-CMD-check/badge.svg)](https://github.com/ModelOriented/fairmodels/actions) [![Codecov test coverage](https://codecov.io/gh/ModelOriented/fairmodels/branch/master/graph/badge.svg)](https://codecov.io/gh/ModelOriented/fairmodels?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI)\n\nFlexible tool for bias detection, visualization, and mitigation. Use models explained with [DALEX](https://modeloriented.github.io/DALEX/) and calculate fairness classification metrics based on confusion matrices using `fairness_check()` or try newly developed module for regression models using `fairness_check_regression()`. R package fairmodels allows to compare and gain information about various machine learning models. Mitigate bias with various pre-processing and post-processing techniques. *Make sure your models are classifying protected groups similarly*.\n\nLearn more:\n\n* [fairmodels website](https://fairmodels.drwhy.ai/)\n* Compas recidivism data use case: [Basic tutorial](https://modeloriented.github.io/fairmodels/articles/Basic_tutorial.html)   \n* Bias mitigation techniques on Adult data: [Advanced tutorial](https://modeloriented.github.io/fairmodels/articles/Advanced_tutorial.html)\n\n\n--------------------\n\n### [vivo](http://github.com/ModelOriented/vivo)   \u003cimg src=\"https://modeloriented.github.io/vivo/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/vivo)](https://cran.r-project.org/package=vivo) [![Build Status](https://api.travis-ci.org/ModelOriented/vivo.svg?branch=master)](https://travis-ci.org/ModelOriented/vivo) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/vivo/master.svg)](https://codecov.io/github/ModelOriented/vivo?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `vivo` package helps to calculate instance level variable importance (measure of local sensitivity). The importance measure is based on Ceteris Paribus profiles and can be calculated in eight variants. Select the variant that suits your needs by setting parameters: `absolute_deviation`, `point` and `density`.\n\nLearn more:\n\n* [Intuition for Ceteris Paribus Oscillations](https://github.com/ModelOriented/vivo#intuition)\n\n--------------------\n\n### [randomForestExplainer](http://github.com/ModelOriented/randomForestExplainer)   \u003cimg src=\"https://modeloriented.github.io/randomForestExplainer/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/randomForestExplainer)](https://cran.r-project.org/package=randomForestExplainer) [![Build Status](https://api.travis-ci.org/ModelOriented/randomForestExplainer.svg?branch=master)](https://travis-ci.org/ModelOriented/randomForestExplainer) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/randomForestExplainer/master.svg)](https://codecov.io/github/ModelOriented/randomForestExplainer?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `randomForestExplainer` package helps to understand what is happening inside a Random Forest model. This package helps to explore main effects and pairwise interactions, depth distribution, conditional responses and feature importance.\n\nLearn more:\n\n* Vignettes: [Boston data: Understanding random forests with randomForestExplainer](https://modeloriented.github.io/randomForestExplainer/articles/randomForestExplainer.html), [Glioblastoma data: Understanding random forests with randomForestExplainer](https://cdn.staticaly.com/gh/geneticsMiNIng/BlackBoxOpener/master/randomForestExplainer/inst/doc/randomForestExplainer.html)\n* [Cheatsheet](https://github.com/ModelOriented/randomForestExplainer/blob/master/materials/cheatsheet.pdf)\n\n--------------------\n\n### [xspliner](http://github.com/ModelOriented/xspliner)   \u003c!-- \u003cimg src=\"https://modeloriented.github.io/xspliner/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e --\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/xspliner)](https://cran.r-project.org/package=xspliner) [![Build Status](https://api.travis-ci.org/ModelOriented/xspliner.svg?branch=master)](https://travis-ci.org/ModelOriented/xspliner) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/vivo/master.svg)](https://codecov.io/github/ModelOriented/xspliner?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-AutoMat-ae2c87)](http://drwhy.ai/#AutoMat) \n\nThe `xspliner` package is a collection of tools for training interpretable surrogate ML models. The package helps to build simple, interpretable models that inherits informations provided by more complicated ones - resulting model may be treated as explanation of provided black box, that was supplied prior to the algorithm. Provided functionality offers graphical and statistical evaluation both for overall model and its components.\n\n--------------------\n\n### [shapper](http://github.com/ModelOriented/shapper)  \u003cimg src=\"https://github.com/ModelOriented/shapper/blob/master/man/figures/logo.png?raw=true\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/shapper)](https://cran.r-project.org/package=shapper) [![Build Status](https://api.travis-ci.org/ModelOriented/shapper.svg?branch=master)](https://travis-ci.org/ModelOriented/shapper) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/shapper/master.svg)](https://codecov.io/github/ModelOriented/shapper?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `shapper` is an R wrapper of SHAP python library.\nIt accesses python implementation through `reticulate` connector.\n\n\n--------------------\n\n### [drifter](http://github.com/ModelOriented/drifter) \n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/drifter)](https://cran.r-project.org/package=drifter) [![Build Status](https://api.travis-ci.org/ModelOriented/drifter.svg?branch=master)](https://travis-ci.org/ModelOriented/drifter) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/drifter/master.svg)](https://codecov.io/github/ModelOriented/drifter?branch=master)[![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `drifter` is an R package that identifies concept drift in model structure or in data structure.\n\nMachine learning models are often fitted and validated on historical data under an assumption that data are stationary. The most popular techniques for validation (k-fold cross-validation, repeated cross-validation, and so on) test models on data with the same distribution as training data.\n\nYet, in many practical applications, deployed models are working in a changing environment. After some time, due to changes in the environment, model performance may degenerate, as model may be less reliable.\n\nConcept drift refers to the change in the data distribution or in the relationships between variables over time. Think about model for energy consumption for a school, over time the school may be equipped with larger number of devices of with more power-efficient devices that may affect the model performance.\n\n\n--------------------\n\n### [EIX](http://github.com/ModelOriented/EIX) \n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/EIX)](https://cran.r-project.org/package=EIX) [![Build Status](https://api.travis-ci.org/ModelOriented/EIX.svg?branch=master)](https://travis-ci.org/ModelOriented/EIX) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/EIX/master.svg)](https://codecov.io/github/ModelOriented/EIX?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-eXtrAI-4378bf)](http://drwhy.ai/#eXtraAI) \n\nThe `EIX` package implements set of techniques to explore and explain `XGBoost` and `LightGBM` models. Main functions of this package cover various variable importance measures and well as functions for identification of interactions between variables.\n\nLearn more:\n\n* [Cheatsheet](https://github.com/ModelOriented/EIX#cheatsheets)\n\n--------------------\n\n### [modelStudio](http://github.com/ModelOriented/modelStudio)   \u003cimg src=\"https://modeloriented.github.io/modelStudio/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/modelStudio)](https://cran.r-project.org/package=modelStudio) [![Build Status](https://api.travis-ci.org/ModelOriented/modelStudio.svg?branch=master)](https://travis-ci.org/ModelOriented/modelStudio) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/modelStudio/master.svg)](https://codecov.io/github/ModelOriented/modelStudio?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-AutoMat-ae2c87)](http://drwhy.ai/#AutoMat) \n\nThe `modelStudio` package **automates the explanatory analysis of machine learning predictive models**. It generates advanced interactive model explanations in the form of a **serverless HTML site** with only one line of code. This tool is model-agnostic, therefore compatible with most of the black-box predictive models and frameworks (e.g.\u0026nbsp;`mlr/mlr3`, `xgboost`, `caret`, `h2o`, `parsnip`, `tidymodels`, `scikit-learn`, `lightgbm`, `keras/tensorflow`).\n\nThe main `modelStudio()` function computes various (instance and model-level) explanations and produces a\u0026nbsp;**customisable dashboard**, which consists of multiple panels for plots with their short descriptions. Easily\u0026nbsp;**save**\u0026nbsp; the dashboard and\u0026nbsp;**share** it with others. Tools for [Explanatory Model Analysis](https://ema.drwhy.ai/) unite with tools for Exploratory Data Analysis to give a broad overview of the model behavior.\n\nLearn more: \n\n* [Getting started](https://modeloriented.github.io/modelStudio/#demo)\n* [Vignette: perks and features](https://modelstudio.drwhy.ai/articles/ms-perks-features.html)\n* [JOSS paper: Interactive Studio with Explanations for ML Predictive Models](https://joss.theoj.org/papers/10.21105/joss.01798)\n\n--------------------\n\n### [arenar](https://github.com/ModelOriented/arenar) \n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/arenar)](https://cran.r-project.org/package=arenar) [![R build status](https://github.com/ModelOriented/ArenaR/workflows/R-CMD-check/badge.svg)](https://github.com/ModelOriented/ArenaR/actions) [![Codecov test coverage](https://codecov.io/gh/ModelOriented/ArenaR/branch/master/graph/badge.svg)](https://codecov.io/gh/ModelOriented/ArenaR?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-AutoMat-ae2c87)](http://drwhy.ai/#AutoMat) \n\nArena is an interactive tool that allows you to explore and compare any model regardless of its internal structure. \n\nThe `arenar` package can be run in two modes - live (R runs in the background and calculates all necessary explanations) and serverless (all necessary explanations are calculated earlier).\n\nUsing the Arena is trivially simple. Examples with different levels of advancement are available:\n\n* [Arena website](http://arenar.drwhy.ai/)\n* [arenar documentation](https://arenar.drwhy.ai/)\n\n--------------------\n\n### [modelDown](http://github.com/ModelOriented/modelDown) \n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/modelDown)](https://cran.r-project.org/package=modelDown) [![Build Status](https://api.travis-ci.org/ModelOriented/modelDown.svg?branch=master)](https://travis-ci.org/ModelOriented/modelDown) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/modelDown/master.svg)](https://codecov.io/github/ModelOriented/modelDown?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-AutoMat-ae2c87)](http://drwhy.ai/#AutoMat) \n\nThe`modelDown` package generates a website with HTML summaries for predictive models. Is uses `DALEX` explainers to compute and plot summaries of how given models behave. We can see how well models behave (*Model Performance*, *Auditor*), how much each variable contributes to predictions (*Variable Response*) and which variables are the most important for a given model (*Variable Importance*). We can also compare Concept Drift for pairs of models (*Drifter*). Additionally, data available on the website can be easily recreated in current R session (using `archivist` package).\n\nLearn more: \n\n* [Getting started](https://modeloriented.github.io/modelDown/getting-started)\n* [JOSS paper: modelDown: automated website generator with interpretable documentation for predictive machine learning models](https://joss.theoj.org/papers/10.21105/joss.01444)\n\n--------------------\n\n### [rSAFE](http://github.com/ModelOriented/rSAFE)   \u003cimg src=\"https://modeloriented.github.io/rSAFE/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/rSAFE)](https://cran.r-project.org/package=rSAFE) [![Build Status](https://api.travis-ci.org/ModelOriented/rSAFE.svg?branch=master)](https://travis-ci.org/ModelOriented/rSAFE) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/rSAFE/master.svg)](https://codecov.io/github/ModelOriented/rSAFE?branch=master)[![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-AutoMat-ae2c87)](http://drwhy.ai/#AutoMat) \n\nThe `rSAFE` package is a model agnostic tool for making an interpretable white-box model more accurate using alternative black-box model called surrogate model. Based on the complicated model, such as neural network or random forest, new features are being extracted and then used in the process of fitting a simpler interpretable model, improving its overall performance.\n\nLearn more:\n\n* article [Simpler is better: Lifting interpretability-performance trade-off via automated feature engineering](https://www.sciencedirect.com/science/article/pii/S016792362100066X)\n* package [website](https://modeloriented.github.io/rSAFE/)\n\n--------------------\n\n### [EloML](http://github.com/ModelOriented/EloML) \u003cimg src=\"https://modeloriented.github.io/EloML/reference/figures/logo.png\" align=\"right\" width=\"100\"/\u003e\n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/EloML)](https://cran.r-project.org/package=EloML) [![Build Status](https://api.travis-ci.org/ModelOriented/EloML.svg?branch=master)](https://travis-ci.org/ModelOriented/EloML) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/ModelOriented/EloML/master.svg)](https://codecov.io/github/ModelOriented/EloML?branch=master) [![DrWhy-AutoMat](https://img.shields.io/badge/DrWhy-AutoMat-373589)](http://drwhy.ai/#AutoMat) \n\nThe `EloML` package provides Elo rating system for machine learning models. Elo Predictive Power (EPP) score helps to assess model performance based Elo ranking system.\n\nLearn more: \n\n* [arxiv](https://arxiv.org/abs/1908.09213)\n\n\n--------------------\n\n### [archivist](http://github.com/pbiecek/archivist) \n\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/archivist)](https://cran.r-project.org/package=archivist) [![Build Status](https://api.travis-ci.org/pbiecek/archivist.svg?branch=master)](https://travis-ci.org/pbiecek/archivist) [![Coverage\nStatus](https://img.shields.io/codecov/c/github/pbiecek/archivist/master.svg)](https://codecov.io/github/pbiecek/archivist?branch=master) [![DrWhy-eXtrAI](https://img.shields.io/badge/DrWhy-BackBone-373589)](http://drwhy.ai/#BackBone) \n\nThe `archivist` package automate serialization and deserialization of R objects. Objects are stored with additional metadata to facilitate reproducibility and governance of data science projects. \n\nEverything that exists in R is an object. `archivist` is an R package that stores copies of all objects along with their metadata. It helps to manage and recreate objects with final or partial results from data analysis. Use `archivist` to record every result, to share these results with future you or with others, to search through repository of objects created in the past but needed now.\n\nLearn more: \n\n* [Cheatsheet](https://github.com/pbiecek/archivist/raw/master/cheatsheets/archivistCheatsheet.png)\n* [JSS article](http://doi.org/10.18637/jss.v082.i11)\n\n\n--------------------\n\n\nTools that are useful during the model lifetime. ![MI2](images/mi2.svg) stands for our internal tools.\n\n### 1. Data preparation\n\n* [dataMaid](https://cran.r-project.org/web/packages/dataMaid/index.html); A Suite of Checks for Identification of Potential Errors in a Data Frame as Part of the Data Screening Process\n* [inspectdf](https://cran.r-project.org/web/packages/inspectdf/index.html); A collection of utilities for columnwise summary, comparison and visualisation of data frames. \n* [validate](https://github.com/data-cleaning/validate); Professional data validation for the R environment \n* [errorlocate](https://github.com/data-cleaning/errorlocate); Find and replace erroneous fields in data using validation rules \n* [ggplot2](https://ggplot2.tidyverse.org/); System for declaratively creating graphics, based on The Grammar of Graphics.\n\n### 2. Data understanding\n\n* Model Agnostic Variable Importance Scores. Surrogate learning = Train an elastic model and measure feature importance in such model. See [DALEX](https://github.com/pbiecek/DALEX/), [Model Class Reliance MCR](https://arxiv.org/abs/1801.01489) \n* [vip](https://github.com/koalaverse/vip) Variable importance plots \n* [SAFE](https://github.com/MI2DataLab/SAFE) ![MI2](images/mi2.svg) Surrogate learning = Train an elastic model and extract feature transformations. \n* [xspliner](https://github.com/ModelOriented/xspliner) ![MI2](images/mi2.svg) Using surrogate black-boxes to train interpretable spline based additive models \n* [factorMerger](https://github.com/MI2DataLab/factorMerger) ![MI2](images/mi2.svg) Set of tools for factors merging [paper](https://arxiv.org/abs/1709.04412)\n* [ingredients](https://github.com/ModelOriented/ingredients) ![MI2](images/mi2.svg) Set of tools for model level feature effects and feature importance.\n\n### 4. Model assembly\n\n* [mlr](https://github.com/mlr-org/mlr) Machine Learning in R [paper](http://jmlr.org/papers/v17/15-066.html)\n* [caret](https://github.com/topepo/caret) Classification And Regression Training [paper](https://www.jstatsoft.org/article/view/v028i05)\n\n### 5. Model audit\n\n* [auditor](https://github.com/MI2DataLab/auditor) ![MI2](images/mi2.svg) model verification, validation, and error analysis [vigniette](https://mi2datalab.github.io/auditor/articles/model_performance_audit.html)\n* [DALEX](https://github.com/pbiecek/DALEX/) ![MI2](images/mi2.svg) Descriptive mAchine Learning EXplanations\n* [iml](https://github.com/christophM/iml); interpretable machine learning R package\n* [randomForestExplainer](https://github.com/MI2DataLab/randomForestExplainer) ![MI2](images/mi2.svg) A set of tools to understand what is happening inside a Random Forest\n* [survxai](https://github.com/MI2DataLab/survxai) ![MI2](images/mi2.svg) Explanations for survival models [paper](http://joss.theoj.org/papers/dcc9d53e8a1b1f613d59b9658b113fff)\n\n### 6. Model delivery\n\n* [iBreakDown](https://github.com/ModelOriented/iBreakDown), [pyBreakDown](https://github.com/MI2DataLab/pyBreakDown)  ![MI2](images/mi2.svg) Model Agnostic Explainers for Individual Predictions (with interactions)\n* [ceterisParibus](https://github.com/pbiecek/ceterisParibus), [pyCeterisParibus](https://github.com/ModelOriented/pyCeterisParibus), [ceterisParibusD3](https://github.com/MI2DataLab/ceterisParibusExt/tree/master/ceterisParibusD3) and [ingredients](https://github.com/ModelOriented/ingredients) ![MI2](images/mi2.svg) Ceteris Paribus Plots (What-If plots) for explanations of a single observation\n* [localModel](https://github.com/ModelOriented/localModel) and [live](https://github.com/MI2DataLab/live/) ![MI2](images/mi2.svg) \nLIME-like explanations with interpretable features based on Ceteris Paribus curves. \n* [lime](https://github.com/thomasp85/lime); Local Interpretable Model-Agnostic Explanations (R port of original Python package)\n* [shapper](https://github.com/ModelOriented/shapper) ![MI2](images/mi2.svg) An R wrapper of SHAP python library\n* [modelDown](https://github.com/ModelOriented/modelDown) ![MI2](images/mi2.svg) modelDown generates a website with HTML summaries for predictive models\n* [modelStudio](https://github.com/ModelOriented/modelStudio) ![MI2](images/mi2.svg) modelStudio generates an interactive dashboard with D3 based summaries for predictive models\n* [drifter](https://github.com/ModelOriented/drifter) ![MI2](images/mi2.svg) Concept Drift and Concept Shift Detection for Predictive Models\n* [archivist](https://github.com/pbiecek/archivist) ![MI2](images/mi2.svg) A set of tools for datasets and plots archiving [paper](http://doi.org/10.18637/jss.v082.i11)\n\n\n## Family of Model Explainers\n\n![images/DrWhyAI.png](images/DrWhyAI.png)\n\n\n## Architecture of DrWhy\n\n`DrWhy` works on fully trained predictive models. Models can be created with any tool. \n\n`DrWhy` uses `DALEX` package to wrap model with additional metadata required for explanations, like validation data, predict function etc.\n\nExplainers for predictive models can be created with model agnostic or model specific functions implemented in various packages.\n\n![Hype_Cycle](images/Hype_Cycle.svg)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FModelOriented%2FDrWhy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FModelOriented%2FDrWhy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FModelOriented%2FDrWhy/lists"}