{"id":28622024,"url":"https://github.com/welpo/stop-early-stopping","last_synced_at":"2026-01-31T02:31:00.371Z","repository":{"id":294069609,"uuid":"985897918","full_name":"welpo/stop-early-stopping","owner":"welpo","description":"Why you shouldn't peek at significance levels to decide when to stop an experiment","archived":false,"fork":false,"pushed_at":"2025-08-17T17:59:13.000Z","size":172,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-13T09:57:54.478Z","etag":null,"topics":["ab-testing","bad-practices","early-stopping","experimental-design","experimentation","good-practices","p-hacking"],"latest_commit_sha":null,"homepage":"https://stop-early-stopping.osc.garden","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/welpo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"COPYING","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"welpo"}},"created_at":"2025-05-18T18:43:00.000Z","updated_at":"2025-08-17T17:59:16.000Z","dependencies_parsed_at":"2025-05-18T19:46:23.725Z","dependency_job_id":null,"html_url":"https://github.com/welpo/stop-early-stopping","commit_stats":null,"previous_names":["welpo/stop-early-stopping"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/welpo/stop-early-stopping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fstop-early-stopping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fstop-early-stopping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fstop-early-stopping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fstop-early-stopping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/welpo","download_url":"https://codeload.github.com/welpo/stop-early-stopping/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fstop-early-stopping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28927128,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T22:32:35.345Z","status":"online","status_checked_at":"2026-01-31T02:00:09.179Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","bad-practices","early-stopping","experimental-design","experimentation","good-practices","p-hacking"],"created_at":"2025-06-12T05:10:18.139Z","updated_at":"2026-01-31T02:31:00.356Z","avatar_url":"https://github.com/welpo.png","language":"HTML","funding_links":["https://github.com/sponsors/welpo"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca href=\"#contributing\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/PRs-welcome-0?style=flat-square\u0026labelcolor=202b2d\u0026color=4a5568\" alt=\"PRs welcome\"\u003e\u003c/a\u003e\n    \u003ca href=\"#license\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/license-AGPL-0?style=flat-square\u0026labelcolor=202b2d\u0026color=4a5568\" alt=\"License\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/welpo/git-sumi\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/clean_commits-git--sumi-0?style=flat-square\u0026labelcolor=202b2d\u0026color=4a5568\" alt=\"clean commits\"\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://stop-early-stopping.osc.garden\"\u003e\n        \u003cimg src=\"screenshot.png\" alt=\"Screenshot of the simulation\" width=\"600\"\u003e\n        \u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://stop-early-stopping.osc.garden\"\u003eTry it yourself!\u003c/a\u003e\n    \u003cbr\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003eWhy you shouldn't peek at significance in A/B tests\u003c/h3\u003e\n\n## What is Early Stopping?\n\nEarly stopping occurs when you repeatedly check your A/B test results and stop the experiment as soon as you see a \"statistically significant\" result. This is a form of [p-hacking](https://en.wikipedia.org/wiki/Data_dredging#Optional_stopping) that dramatically increases false positive rates.\n\nFor example, with a significance level of 0.05 (5%), you expect a 5% false positive rate when checking just once at the end. But if you check results daily during a 30-day test, you could see false positive rates as high as 30%, meaning many of your \"winning\" tests are actually detecting nothing at all!\n\n## Features\n\n- **Interactive simulation**: See how early stopping affects false positive rates in real-time\n- **Adjustable parameters**: Test different experiment durations, checking frequencies, and significance levels\n- **Visual comparison**: Directly compare the false positive rates of proper testing vs early stopping\n\n## How it works\n\nThis tool simulates A/B tests where both variants have identical performance (a true \"null hypothesis\" scenario):\n\n1. Each simulation generates two identical variants with:\n   - 250 visitors per day per variant\n   - 10% conversion rate for both variants\n   - No actual difference between variants\n\n2. The tool runs 625 simulated tests with two different stopping criteria:\n   - **Left grid**: Tests that only check for significance once, after the full duration\n   - **Right grid**: Tests that check at your specified frequency and stop when significance is found\n\n3. Results are color-coded:\n   - 🟢: tests correctly showing no significant difference\n   - 🟥: tests incorrectly showing a significant difference (false positives)\n\nStatistical significance is determined using a two-proportion Z-test, mathematically equivalent to a chi-squared test for this 2×2 case.\n\n## Contributing\n\nPlease do! I'd appreciate bug reports, improvements (however minor), suggestions…\n\nThe simulator uses vanilla JavaScript, HTML, and CSS. To run locally:\n\n1. Clone the repository: `git clone https://github.com/welpo/stop-early-stopping.git`\n2. Navigate to the project directory: `cd stop-early-stopping`\n3. Start a local server: `python3 -m http.server`\n4. Visit `http://localhost:8000` in your browser\n\nThe important files are:\n\n- `index.html`: Basic structure\n- `styles.css`: Styles\n- `app.js`: Main UI logic\n- `simulationWorker.js`: Web worker that runs simulations in background\n\n## Need help?\n\nSomething not working? Have an idea? Let me know!\n\n- Questions or ideas → [Start a discussion](https://github.com/welpo/stop-early-stopping/discussions)\n- Found a bug? → [Report it here](https://github.com/welpo/stop-early-stopping/issues/new?\u0026labels=bug)\n- Feature request? → [Let me know](https://github.com/welpo/stop-early-stopping/issues/new?\u0026labels=feature)\n\n## License\n\nThis simulator is free software: you can redistribute it and/or modify it under the terms of the [GNU Affero General Public License as published by the Free Software Foundation](./COPYING), either version 3 of the License, or (at your option) any later version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwelpo%2Fstop-early-stopping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwelpo%2Fstop-early-stopping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwelpo%2Fstop-early-stopping/lists"}