{"id":28203453,"url":"https://github.com/david26694/cluster-experiments","last_synced_at":"2026-02-18T06:00:39.618Z","repository":{"id":60429577,"uuid":"521667892","full_name":"david26694/cluster-experiments","owner":"david26694","description":"Power analysis and AB test analysis library","archived":false,"fork":false,"pushed_at":"2026-02-02T16:55:38.000Z","size":6348,"stargazers_count":47,"open_issues_count":26,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-02-03T06:08:10.787Z","etag":null,"topics":["abtesting","data-analysis","mde","pandas","power-analysis","python","statistics"],"latest_commit_sha":null,"homepage":"https://david26694.github.io/cluster-experiments/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/david26694.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-08-05T14:26:01.000Z","updated_at":"2026-02-02T16:54:59.000Z","dependencies_parsed_at":"2023-12-27T23:57:14.850Z","dependency_job_id":"5e0903a5-879f-4258-9e0a-6a764e326a9e","html_url":"https://github.com/david26694/cluster-experiments","commit_stats":null,"previous_names":["david26694/ab-lab","david26694/cluster-experiments"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/david26694/cluster-experiments","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/david26694%2Fcluster-experiments","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/david26694%2Fcluster-experiments/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/david26694%2Fcluster-experiments/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/david26694%2Fcluster-experiments/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/david26694","download_url":"https://codeload.github.com/david26694/cluster-experiments/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/david26694%2Fcluster-experiments/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29569994,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T04:18:28.490Z","status":"ssl_error","status_checked_at":"2026-02-18T04:13:49.018Z","response_time":162,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abtesting","data-analysis","mde","pandas","power-analysis","python","statistics"],"created_at":"2025-05-17T01:16:20.015Z","updated_at":"2026-02-18T06:00:39.612Z","avatar_url":"https://github.com/david26694.png","language":"Python","readme":"\u003cimg src=\"theme/icon-cluster.png\" width=200 height=200 align=\"right\"\u003e\n\n# cluster-experiments\n\n[![Downloads](https://static.pepy.tech/badge/cluster-experiments)](https://pepy.tech/project/cluster-experiments)\n[![PyPI](https://img.shields.io/pypi/v/cluster-experiments)](https://pypi.org/project/cluster-experiments/)\n[![Unit tests](https://github.com/david26694/cluster-experiments/workflows/Release%20unit%20Tests/badge.svg)](https://github.com/david26694/cluster-experiments/actions)\n[![CodeCov](https://codecov.io/gh/david26694/cluster-experiments/branch/main/graph/badge.svg)](https://app.codecov.io/gh/david26694/cluster-experiments/)\n![License](https://img.shields.io/github/license/david26694/cluster-experiments)\n[![Pypi version](https://img.shields.io/pypi/pyversions/cluster-experiments.svg)](https://pypi.python.org/pypi/cluster-experiments)\n\n**`cluster-experiments`** is a comprehensive Python library for **end-to-end A/B testing workflows**, from experiment design to statistical analysis.\n\n## 📖 What is cluster-experiments?\n\n`cluster-experiments` provides a complete toolkit for designing, running, and analyzing experiments, with particular strength in handling **clustered randomization** and complex experimental designs. Originally developed to address challenges in **switchback experiments** and scenarios with **network effects** where standard randomization isn't feasible, it has evolved into a general-purpose experimentation framework supporting both simple A/B tests and other randomization designs.\n\n### Why \"cluster\"?\n\nThe name reflects the library's origins in handling **cluster-randomized experiments**, where randomization happens at a group level (e.g., stores, cities, time periods) rather than at the individual level. This is critical when:\n\n- **Spillover/Network Effects**: Treatment of one unit affects others (e.g., testing driver incentives in ride-sharing)\n- **Operational Constraints**: You can't randomize individuals (e.g., testing restaurant menu changes)\n- **Switchback Designs**: Treatment alternates over time periods within the same unit\n\nWhile the library is aimed at these scenarios, it's equally capable of handling standard A/B tests with individual-level randomization.\n\n---\n\n## Key Features\n\n### Experiment Design\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003ePower Analysis \u0026 Sample Size Calculation\u003c/summary\u003e\n\n- Simulation-based (Monte Carlo) for any design complexity\n- Analytical (CLT-based) for standard designs\n- Minimum Detectable Effect (MDE) estimation\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eMultiple Experimental Designs\u003c/summary\u003e\n\n- Standard A/B tests with individual randomization\n- Cluster-randomized experiments\n- Switchback/crossover experiments\n- Stratified randomization\n- Observational studies with Synthetic Control\n\u003c/details\u003e\n\n### Statistical Methods\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eMultiple Analysis Methods\u003c/summary\u003e\n\n- OLS and Clustered OLS regression\n- GEE (Generalized Estimating Equations)\n- Mixed Linear Models (MLM)\n- Delta Method for ratio metrics\n- Synthetic Control for observational data\n\u003c/details\u003e\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eVariance Reduction Techniques\u003c/summary\u003e\n\n- CUPED (Controlled-experiment Using Pre-Experiment Data)\n- CUPAC (Control Using Predictions As Covariates)\n- Covariate adjustment\n\u003c/details\u003e\n\n### Analysis Workflow\n\n\u003cdetails markdown=\"1\"\u003e\n\u003csummary\u003eScorecard \u0026 Multi-dimensional Analysis\u003c/summary\u003e\n\n- **Scorecard Generation**: Analyze multiple metrics simultaneously\n- **Multi-dimensional Slicing**: Break down results by segments\n- **Multiple Treatment Arms**: Compare several treatments at once\n- **Ratio Metrics**: Built-in support for conversion rates, averages, etc.\n- **Relative Lift**: Analyze effects as percentage changes rather than absolute differences\n\u003c/details\u003e\n\n---\n\n## 📦 Installation\n\n```bash\npip install cluster-experiments\n```\n\n---\n\n## ⚡ Quick Example\n\nHere's how to run an analysis in just a few lines:\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom cluster_experiments import AnalysisPlan, Variant\n\nnp.random.seed(42)\n\n# 0. Create simple data\nN = 1_000\ndf = pd.DataFrame({\n    \"variant\": np.random.choice([\"control\", \"treatment\"], N),\n    \"orders\": np.random.poisson(10, N),\n    \"visits\": np.random.poisson(100, N),\n})\ndf[\"converted\"] = (df[\"orders\"] \u003e 0).astype(int)\n\n\n# 1. Define your analysis plan\nplan = AnalysisPlan.from_metrics_dict({\n    \"metrics\": [\n        {\"name\": \"orders\", \"alias\": \"revenue\", \"metric_type\": \"simple\"},\n        {\"name\": \"converted\", \"alias\": \"conversion\", \"metric_type\": \"ratio\", \"numerator\": \"converted\", \"denominator\": \"visits\"}\n    ],\n    \"variants\": [\n        {\"name\": \"control\", \"is_control\": True},\n        {\"name\": \"treatment\", \"is_control\": False}\n    ],\n    \"variant_col\": \"variant\",\n    \"analysis_type\": \"ols\"\n})\n\n# 2. Run analysis on your dataframe\nresults = plan.analyze(df)\nprint(results.to_dataframe().head())\n```\n\n**Output Example**:\n\n```\n  metric_alias control_variant_name treatment_variant_name  control_variant_mean  treatment_variant_mean analysis_type           ate  ate_ci_lower  ate_ci_upper   p_value     std_error     dimension_name dimension_value  alpha\n0      revenue              control              treatment              10.08554                9.941061           ols -1.444788e-01 -5.446603e-01  2.557026e-01  0.479186  2.041780e-01  __total_dimension           total   0.05\n1   conversion              control              treatment               1.00000                1.000000           ols  1.110223e-16 -1.096504e-16  3.316950e-16  0.324097  1.125902e-16  __total_dimension           total   0.05\n```\n\n---\n\n## Power Analysis\n\nDesign your experiment by estimating required sample size and detectable effects. Here's a complete example using **analytical (CLT-based) power analysis**:\n\n```python\nimport numpy as np\nimport pandas as pd\nfrom cluster_experiments import NormalPowerAnalysis\n\n# Create sample historical data\nnp.random.seed(42)\nN = 500\n\nhistorical_data = pd.DataFrame({\n    'user_id': range(N),\n    'metric': np.random.normal(100, 20, N),\n    'date': pd.to_datetime('2025-10-01') + pd.to_timedelta(np.random.randint(0, 30, N), unit='d')\n})\n\n# Initialize analytical power analysis (fast, CLT-based)\npower_analysis = NormalPowerAnalysis.from_dict({\n    'analysis': 'ols',\n    'splitter': 'non_clustered',\n    'target_col': 'metric',\n    'time_col': 'date'  # Required for mde_time_line\n})\n\n# 1. Calculate power for a given effect size\npower = power_analysis.power_analysis(historical_data, average_effect=5.0)\nprint(f\"Power for detecting +5 unit effect: {power:.1%}\")\n\n# 2. Calculate Minimum Detectable Effect (MDE) for desired power\nmde = power_analysis.mde(historical_data, power=0.8)\nprint(f\"Minimum detectable effect at 80% power: {mde:.2f}\")\n\n# 3. Power curve: How power changes with effect size\npower_curve = power_analysis.power_line(\n    historical_data,\n    average_effects=[2.0, 4.0, 6.0, 8.0, 10.0]\n)\nprint(power_curve)\n\n# 4. MDE timeline: How MDE changes with experiment length\nmde_timeline = power_analysis.mde_time_line(\n    historical_data,\n    powers=[0.8],\n    experiment_length=[7, 14, 21, 30]\n)\n```\n\n**Output:**\n\n```\nPower for detecting +5 unit effect: 72.7%\nMinimum detectable effect at 80% power: 5.46\n{2.0: 0.18, 4.0: 0.54, 6.0: 0.87, 8.0: 0.98, 10.0: 1.00}\n```\n\n**Key methods:**\n\n- `power_analysis()`: Calculate power for a given effect\n- `mde()`: Calculate minimum detectable effect\n- `power_line()`: Generate power curves across effect sizes\n- `mde_time_line()`: Calculate MDE for different experiment lengths\n\nFor simulation-based power analysis (for complex designs), see the [Power Analysis Guide](https://david26694.github.io/cluster-experiments/normal_power_lines.html).\n\n---\n\n## 📚 Documentation\n\nFor detailed guides, API references, and advanced examples, visit our [**documentation**](https://david26694.github.io/cluster-experiments/).\n\n### Core Concepts\n\nThe library is built around three main components:\n\n#### 1. **Splitter** - Define how to randomize\n\nChoose how to split your data into control and treatment groups:\n\n- `NonClusteredSplitter`: Standard individual-level randomization\n- `ClusteredSplitter`: Cluster-level randomization\n- `SwitchbackSplitter`: Time-based alternating treatments\n- `StratifiedClusteredSplitter`: Balance randomization across strata\n\n#### 2. **Analysis** - Measure the impact\n\nSelect the appropriate statistical method for your design:\n\n- `OLSAnalysis`: Standard regression for A/B tests\n- `ClusteredOLSAnalysis`: Clustered standard errors for cluster-randomized designs\n- `TTestClusteredAnalysis`: T-tests on cluster-aggregated data\n- `GeeExperimentAnalysis`: GEE for correlated observations\n- `SyntheticControlAnalysis`: Observational studies with synthetic controls\n\n#### 3. **AnalysisPlan** - Orchestrate your analysis\n\nDefine your complete analysis workflow:\n\n- Specify metrics (simple and ratio)\n- Define variants and dimensions\n- Configure hypothesis tests\n- Generate comprehensive scorecards\n\nFor **power analysis**, combine these with:\n\n- **Perturbator**: Simulate treatment effects for power calculations\n- **PowerAnalysis**: Estimate statistical power and sample sizes\n\n---\n\n## 🛠️ Advanced Features\n\n### Variance Reduction (CUPED/CUPAC)\n\nReduce variance and detect smaller effects by leveraging pre-experiment data. Use historical metrics as covariates to control for pre-existing differences between groups.\n\n**Use cases:**\n\n- Have pre-experiment metrics for your users/clusters\n- Want to detect smaller treatment effects\n- Need more sensitive tests with same sample size\n\nSee the [CUPAC Example](https://david26694.github.io/cluster-experiments/cupac_example.html) for detailed implementation.\n\n### Cluster Randomization\n\nHandle experiments where randomization occurs at group level (stores, cities, regions) rather than individual level. Essential for managing spillover effects and operational constraints.\n\nSee the [Cluster Randomization Guide](https://david26694.github.io/cluster-experiments/examples/cluster_randomization.html) for details.\n\n### Switchback Experiments\n\nDesign and analyze time-based crossover experiments where the same units receive both control and treatment at different times.\n\nSee the [Switchback Example](https://david26694.github.io/cluster-experiments/switchback.html) for implementation.\n\n---\n\n## 🌟 Support\n\n- ⭐ Star us on [GitHub](https://github.com/david26694/cluster-experiments)\n- 📝 Read the [documentation](https://david26694.github.io/cluster-experiments/)\n- 🐛 Report issues on our [issue tracker](https://github.com/david26694/cluster-experiments/issues)\n- 💬 Join discussions in [GitHub Discussions](https://github.com/david26694/cluster-experiments/discussions)\n\n---\n\n## 📚 Citation\n\nIf you use cluster-experiments in your research, please cite:\n\n```bibtex\n@software{cluster_experiments,\n  author = {David Masip and contributors},\n  title = {cluster-experiments: A Python library for designing and analyzing experiments},\n  url = {https://github.com/david26694/cluster-experiments},\n  year = {2022}\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavid26694%2Fcluster-experiments","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavid26694%2Fcluster-experiments","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavid26694%2Fcluster-experiments/lists"}