https://github.com/vathymut/dsos
Dataset shift with outlier scores
https://github.com/vathymut/dsos
data-drift data-validation dataset-shifts drift-detection machine-learning mlops model-monitoring model-validation performance-monitoring r statistical-process-control statistical-tests
Last synced: 2 months ago
JSON representation
Dataset shift with outlier scores
- Host: GitHub
- URL: https://github.com/vathymut/dsos
- Owner: vathymut
- License: gpl-3.0
- Fork: true (rbc-research/dsos)
- Created: 2021-11-22T17:17:27.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2023-02-19T07:40:07.000Z (almost 3 years ago)
- Last Synced: 2025-10-22T04:04:56.798Z (3 months ago)
- Topics: data-drift, data-validation, dataset-shifts, drift-detection, machine-learning, mlops, model-monitoring, model-validation, performance-monitoring, r, statistical-process-control, statistical-tests
- Language: R
- Homepage: https://vathymut.github.io/dsos/
- Size: 1.45 MB
- Stars: 2
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- License: LICENSE.md
Awesome Lists containing this project
README
---
output: github_document
---
# `D-SOS`: Dataset shift with outlier scores
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
[](https://www.tidyverse.org/lifecycle)
[](https://www.gnu.org/licenses/gpl-3.0.en.html)
[](https://cran.r-project.org/package=dsos)
[](https://openreview.net/forum?id=S5UG2BLi9xc)
[](https://cran.r-project.org/package=dsos)
[](https://cran.r-project.org/package=dsos)
[](https://youtu.be/TALE9JUir8Q?t=26)
## Overview
`dsos` tests for no adverse shift based on outlier scores. Colloquially,
these tests check whether the new sample is not substantively worse than the
old sample, not if the two are equal as tests of equal distributions do.
`dsos` implements a family of two-sample comparison which assumes that
we have both a training set, the reference distribution, and a test set.
## Installation
The package is under active development.
From GitHub (which includes recent improvements), install with:
```{r github, eval=FALSE}
# install.packages("remotes")
remotes::install_github("vathymut/dsos")
```
The package is also on [CRAN](https://CRAN.R-project.org), although the
CRAN release may lag behind GitHub updates. From CRAN, install the
package with:
```{r cran, eval=FALSE}
install.packages("dsos")
```
## Quick Start
Simulate outlier scores to test for no adverse shift when the null (no
shift) holds. First, we use the frequentist permutation test:
```{r null_pt, eval=TRUE}
library(dsos)
set.seed(12345)
n <- 6e2
os_train <- rnorm(n = n)
os_test <- rnorm(n = n)
null_pt <- pt_from_os(os_train, os_test)
plot(null_pt)
```
We can also use the (faster) asymptotic test:
```{r null_at, eval=FALSE}
null_at <- at_from_os(os_train, os_test)
plot(null_at)
```
Doing the same exercise the Bayesian way (with Bayes factors):
```{r null_bf, eval=TRUE}
null_bf <- bf_from_os(os_train, os_test)
# plot(null_bf)
as_pvalue(null_bf$bayes_factor)
```
In all cases, we fail to reject the null of no adverse shift. Note how we
can convert a Bayes factor into a $p$-value.
We can repeat this exercise when there is an adverse shift. Again, with
the permutation test:
```{r shift_pt, eval=TRUE}
os_shift <- rnorm(n = n, mean = 0.2)
shift_pt <- pt_from_os(os_train, os_shift)
plot(shift_pt)
```
Once more, with the asymptotic test:
```{r shift_at, eval=FALSE}
shift_at <- at_from_os(os_train, os_shift)
plot(shift_at)
```
Doing it the Bayesian way (with Bayes factors):
```{r shift_bf, eval=TRUE}
shift_bf <- bf_from_os(os_train, os_shift)
# plot(shift_bf)
as_pvalue(shift_bf$bayes_factor)
```
We would reject the null of no adverse shift in all cases: the test set
is worse off relative to the reference (training) scores.
The function `bf_compare` is handy: it computes and contrasts Bayes
factors for the frequentist and Bayesian approach.
```{r shift_all, eval=TRUE}
shift_all <- bf_compare(os_train, os_shift)
shift_all
```
## Reference
To cite this work, please refer to the
[paper](https://openreview.net/forum?id=S5UG2BLi9xc). Sample Bibtex is below:
```bibtex
@inproceedings{kamulete2022test,
title = {Test for non-negligible adverse shifts},
author = {Vathy M. Kamulete},
booktitle = {The 38th Conference on Uncertainty in Artificial Intelligence},
year = {2022},
url = {https://openreview.net/forum?id=S5UG2BLi9xc}
}
```
I gave a talk introducing the `dsos` R package at
[useR! 2022](https://youtu.be/TALE9JUir8Q?t=26) during the
'Unique Applications and Methods' track. It is a 15-minute crash course,
focused on interpretation. I also wrote a
[blog post](https://vathymut.org/posts/2023-01-03-are-you-ok/)
to motivate the need for tests of adverse shift.