https://github.com/niisaban/ab-test-lift-sim
A/B test lift simulation & analysis in Python: CUPED variance reduction, power analysis, and guardrail metrics with reproducible pytest CI.
https://github.com/niisaban/ab-test-lift-sim
ab-testing causal-inference cuped experimentation github-actions pandas power-analysis pytest python scipy srm statistics
Last synced: about 1 month ago
JSON representation
A/B test lift simulation & analysis in Python: CUPED variance reduction, power analysis, and guardrail metrics with reproducible pytest CI.
- Host: GitHub
- URL: https://github.com/niisaban/ab-test-lift-sim
- Owner: niisaban
- Created: 2025-10-05T15:27:21.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-11-02T19:24:32.000Z (7 months ago)
- Last Synced: 2025-11-02T21:13:46.789Z (7 months ago)
- Topics: ab-testing, causal-inference, cuped, experimentation, github-actions, pandas, power-analysis, pytest, python, scipy, srm, statistics
- Language: Python
- Homepage:
- Size: 41 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[](https://github.com/niisaban/ab-test-lift-sim/actions/workflows/ci.yml)
# A/B Test Lift — Simulation & Analysis
**A/B = "A versus B", aka controlled online experiment**.
This mini‑project lets you **simulate** experiments, run **power analyses**, apply **CUPED** (Controlled Pre‑Experiment Data) variance reduction, and compute **lift** with statistical tests.
## Why this repo
- Practice end‑to‑end: simulate → analyze → interpret → write up.
- Show **reproducible** DS craft (tests, CI, clean `src/`).
- Prep for interviews: power, guardrails, CUPED, heterogeneity.
## Quickstart
```bash
# (Recommended) Python 3.11+
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
# Run a small demo (simulate + analyze)
python scripts/run_demo.py
# Run tests
pytest -q
```
### Run tests locally
#### Windows (PowerShell, current session)
```powershell
# Expose repo root to Python for this shell session
$env:PYTHONPATH = (Get-Location).Path
pytest -q --maxfail=1 -ra
```
#### macOS / Linux (bash/zsh, current session)
```bash
# Expose repo root to Python for this shell session
export PYTHONPATH="$PWD"
pytest -q --maxfail=1 -ra
```
#### (Optional) Windows CMD
```bat
:: Expose repo root to Python for this CMD session
set PYTHONPATH=%CD%
:: Run tests (quiet; stop early; show failure summary)
pytest -q --maxfail=1 -ra
```
## Project Structure
```
ab-test-lift-sim/
├─ src/
│ ├─ simulate.py # traffic & conversions generator (+ covariate for CUPED)
│ ├─ metrics.py # lift, SE, z-test / Welch t-test
│ ├─ cuped.py # CUPED theta, adjusted outcome
│ └─ power.py # sample-size & detectable-effect calculators
├─ scripts/
│ └─ run_demo.py # CLI demo tying it all together
├─ tests/
│ ├─ test_metrics.py
│ └─ test_cuped.py
├─ data/
│ └─ README.md
├─ requirements.txt
├─ .github/workflows/ci.yml
└─ README.md
```
## Concepts (expanded)
- **Lift**: difference in mean outcome between Treatment and Control (absolute or relative).
- **Power analysis**: probability of detecting a true effect; trades off effect size, variance, alpha, and sample size.
- **CUPED**: variance reduction using a pre‑experiment covariate `X_pre`:
θ = Cov(Y, X_pre) / Var(X_pre), and `Y_adj = Y - θ(X_pre - mean(X_pre))`.
- **Guardrail metrics**: KPIs you ensure don’t degrade (e.g., latency, cancellation rate).
## What to practice / document
- State a **clear hypothesis** and primary metric.
- Check **randomization balance**; monitor **SRM** (sample‑ratio mismatch).
- Report **CIs**, **p‑values**, and practical impact (e.g., $$ ROI).
- Call out **assumptions** (independence, normal approx) & alternatives (bootstrap).
## Interview talking points
- When to **stop** an experiment; **sequential** peeking pitfalls.
- Dealing with **novelty effects** and **heterogeneous treatment effects**.
- Tradeoffs: **CUPED** vs. **stratification** vs. **reweighting**.
---
### Get help / share results
- ❓ Ask a question → [New Q&A](https://github.com/niisaban/ab-test-lift-sim/discussions/new?category=Q%26A)
- ✨ Share a result → [New Show & Tell](https://github.com/niisaban/ab-test-lift-sim/discussions/new?category=Show%20and%20tell)
- 🧰 **Quick fixes / FAQ** → [FAQ thread](https://github.com/niisaban/ab-test-lift-sim/discussions/21)
© Use freely for interviews/portfolio.