https://github.com/ibrahimasifali94/sql_ab_test_metrics
SQL A/B test metrics with DuckDB – CTR, lift, p-value, and retention
https://github.com/ibrahimasifali94/sql_ab_test_metrics
ab-testing analytics duckdb sql
Last synced: about 1 month ago
JSON representation
SQL A/B test metrics with DuckDB – CTR, lift, p-value, and retention
- Host: GitHub
- URL: https://github.com/ibrahimasifali94/sql_ab_test_metrics
- Owner: ibrahimasifali94
- Created: 2025-09-16T15:12:35.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2025-09-16T15:50:37.000Z (9 months ago)
- Last Synced: 2026-05-17T02:04:50.509Z (about 1 month ago)
- Topics: ab-testing, analytics, duckdb, sql
- Language: Jupyter Notebook
- Homepage:
- Size: 15.6 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# SQL A/B Test Metrics
**TL;DR:** DuckDB SQL to compute CTR lift, daily CTR, and D1 retention for a synthetic A/B test — with a z-test and p-value.
## What’s inside
- Problem: Measure whether variant **B** improves click-through rate (CTR) vs **A** and impact on D1 retention.
- Approach: Pure SQL over CSVs (DuckDB). Includes sanity checks, lift calc, and a two-sided z-test.
- Data: Synthetic CSVs in `data/` (generated by `generate_data.py`).
- Result: On this seed, B shows 1.30% absolute lift (28.14% relative lift) with a p-value of 0.039 (statistically significant)
## How to run
```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python generate_data.py # optional if you want fresh data
duckdb -c ".read queries.sql"