{"id":28622025,"url":"https://github.com/welpo/srm","last_synced_at":"2025-10-25T10:45:26.947Z","repository":{"id":294064898,"uuid":"985885051","full_name":"welpo/srm","owner":"welpo","description":"Precise Sample Ratio Mismatch calculator.","archived":false,"fork":false,"pushed_at":"2025-05-24T20:10:52.000Z","size":142,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-08T10:05:27.966Z","etag":null,"topics":["ab-testing","experimentation","sample-ratio-mismatch","srm","statistics"],"latest_commit_sha":null,"homepage":"https://srm.osc.garden/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/welpo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"COPYING","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"welpo"}},"created_at":"2025-05-18T18:10:34.000Z","updated_at":"2025-05-24T20:10:55.000Z","dependencies_parsed_at":"2025-05-18T19:35:13.817Z","dependency_job_id":null,"html_url":"https://github.com/welpo/srm","commit_stats":null,"previous_names":["welpo/srm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/welpo/srm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fsrm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fsrm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fsrm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fsrm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/welpo","download_url":"https://codeload.github.com/welpo/srm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/welpo%2Fsrm/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259402147,"owners_count":22851880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","experimentation","sample-ratio-mismatch","srm","statistics"],"created_at":"2025-06-12T05:10:18.531Z","updated_at":"2025-10-25T10:45:26.941Z","avatar_url":"https://github.com/welpo.png","language":"JavaScript","funding_links":["https://github.com/sponsors/welpo"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003ca href=\"#contributing\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/PRs-welcome-0?style=flat-square\u0026labelcolor=202b2d\u0026color=4a5568\" alt=\"PRs welcome\"\u003e\u003c/a\u003e\n    \u003ca href=\"#license\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/license-AGPL-0?style=flat-square\u0026labelcolor=202b2d\u0026color=4a5568\" alt=\"License\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/welpo/git-sumi\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/clean_commits-git--sumi-0?style=flat-square\u0026labelcolor=202b2d\u0026color=4a5568\" alt=\"clean commits\"\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://srm.osc.garden\"\u003e\n        \u003cimg src=\"screenshot.png\" alt=\"Sample Ratio Mismatch Calculator Screenshot\" width=\"600\"\u003e\n        \u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://srm.osc.garden\"\u003eTry it out!\u003c/a\u003e\n    \u003cbr\u003e\n\u003c/p\u003e\n\n## What is Sample Ratio Mismatch?\n\nPicture this: you set up your experiment to have 50% of users on variant A and 50% on variant B. The experiment is over and you have 44,000 users in A, and 45,000 in B. Is this a problem?\n\nMost likely. If your assignment worked properly, the probability of seeing this imbalance (or a larger one) is less than 0.1%.\n\nWhen the difference between the ratios is significant, you have a Sample Ratio Mismatch (SRM). Unless you understand why it happened, you should not analyse the results of the experiment; your setup may be flawed, invalidating any conclusions.\n\n## Features\n\n- **Precise**: less than 0.01% deviation from Python's [scipy's](https://scipy.org/) Chi Square calculations\n- **Unlimited variants**: can be used for A/B/n tests\n- **Custom ratios**: you didn't run a 50-50 split? No problem!\n- **Share** your results with colleagues through custom links\n- **Private**: no accounts, no tracking, works offline, and no data is sent to any server\n\n## How it works\n\nThis tool uses the **[Chi-Squared Goodness-of-Fit test](https://www.jmp.com/en/statistics-knowledge-portal/chi-square-test/chi-square-goodness-of-fit-test)** to detect Sample Ratio Mismatch (SRM). Here's the process:\n\n1. **You provide**:\n    - **Observed counts**: The actual number of users in each experiment variant.\n    - **Expected distribution**: How you intended to split traffic (e.g., 50/50, or a custom ratio like 60/40).\n\n2. **The calculator then**:\n    - Determines the **expected user counts** for each variant based on your input.\n    - Calculates the **Chi-Square statistic ($\\chi^2$)**. This value measures how much your observed counts ($O$) deviate from the expected ones ($E$). The formula is: $\\chi^2 = \\sum \\frac{(O - E)^2}{E}$.\n        - Derives a **p-value** using the [regularised incomplete gamma function](https://en.wikipedia.org/wiki/Incomplete_gamma_function#Regularized_gamma_functions). The degrees of freedom ($df$) are calculated as (number of variants - 1). The p-value is then: $p\\text{-value} = 1 - P(\\frac{df}{2}, \\frac{\\chi^2}{2})$\n        - For smaller chi-square values: [series expansion method](https://dlmf.nist.gov/8.7)\n        - For larger values: [continued fraction method](https://dlmf.nist.gov/8.9)\n\n3. **Interpretation**:\n    - The p-value tells you the probability of seeing your observed user distribution (or one even more imbalanced) if the traffic splitting was actually working as intended.\n        - **`p \u003c 0.01`**: Strong evidence of SRM.\n        - **`0.01 ≤ p \u003c 0.05`**: Possible SRM.\n        - **`p ≥ 0.05`**: No significant evidence of SRM.\n\n## Contributing\n\nPlease do! I'd appreciate bug reports, improvements (however minor), suggestions…\n\nThe calculator uses vanilla JavaScript, HTML, and CSS. To run locally:\n\n1. Clone the repository: `git clone https://github.com/welpo/srm.git`\n2. Navigate to the project directory: `cd srm`\n3. Start a local server: `python3 -m http.server`\n4. Visit `http://localhost:8000` in your browser\n\nThe important files are:\n\n- `index.html`: Basic structure\n- `styles.css`: Styles\n- `app.js`: Logic\n- `tests.js`: Tests, generated with `srm/srm_test_generator.py` (add `?test` to the URL to run validation tests)\n\n## Need help?\n\nSomething not working? Have an idea? Let me know!\n\n- Questions or ideas → [Start a discussion](https://github.com/welpo/srm/discussions)\n- Found a bug? → [Report it here](https://github.com/welpo/srm/issues/new?\u0026labels=bug)\n- Feature request? → [Let me know](https://github.com/welpo/srm/issues/new?\u0026labels=feature)\n\n## License\n\nThis SRM calculator is free software: you can redistribute it and/or modify it under the terms of the [GNU Affero General Public License as published by the Free Software Foundation](./COPYING), either version 3 of the License, or (at your option) any later version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwelpo%2Fsrm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwelpo%2Fsrm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwelpo%2Fsrm/lists"}