https://github.com/spotify/confidence
https://github.com/spotify/confidence
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/spotify/confidence
- Owner: spotify
- License: apache-2.0
- Created: 2021-03-15T20:56:40.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2024-11-26T15:46:56.000Z (over 1 year ago)
- Last Synced: 2025-05-11T10:57:00.655Z (about 1 year ago)
- Language: Python
- Size: 9.25 MB
- Stars: 271
- Watchers: 16
- Forks: 32
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- Changelog: HISTORY.rst
- Contributing: CONTRIBUTING.rst
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Authors: AUTHORS.md
Awesome Lists containing this project
README
Spotify Confidence
========
Python library for AB test analysis.
Why use Spotify Confidence?
-----------------
Spotify Confidence provides convenience wrappers around statsmodel's various functions for computing p-values and confidence intervalls.
With Spotify Confidence it's easy to compute several p-values and confidence bounds in one go, e.g. one for each country or for each date.
Each function comes in two versions:
- one that return a pandas dataframe,
- one that returns a [Chartify](https://github.com/spotify/chartify) chart.
Spotify Confidence has support calculating p-values and confidence intervals using Z-statistics, Student's T-statistics
(or more exactly [Welch's T-test](https://en.wikipedia.org/wiki/Welch%27s_t-test)), as well as Chi-squared statistics.
It also supports a variance reduction technique based on using pre-exposure data to fit a linear model.
There is also a Bayesian alternative in the BetaBinomial class.
Basic Example
--------
```
import spotify_confidence as confidence
import pandas as pd
data = pd.DataFrame(
{'variation_name': ['treatment1', 'control', 'treatment2', 'treatment3'],
'success': [50, 40, 10, 20],
'total': [100, 100, 50, 60]
}
)
test = confidence.ZTest(
data,
numerator_column='success',
numerator_sum_squares_column=None,
denominator_column='total',
categorical_group_columns='variation_name',
correction_method='bonferroni')
test.summary()
test.difference(level_1='control', level_2='treatment1')
test.multiple_difference(level='control', level_as_reference=True)
test.summary_plot().show()
test.difference_plot(level_1='control', level_2='treatment1').show()
test.multiple_difference_plot(level='control', level_as_reference=True).show()
```
There's a lot more you can do:
- Segment results by one or more dimensions
- Use non-inferiority margins
- Group sequential tests
- Sample size and power calculations
- etc
See jupyter notebooks in `examples` folder for more complete examples.
Installation
------------
Spotify Confidence can be installed via pip:
```pip install spotify-confidence```
[Find the latest release version here](https://github.com/spotify/confidence/releases)
### Code of Conduct
This project adheres to the [Open Code of Conduct](https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md) By participating, you are expected to honor this code.