{"id":37082946,"url":"https://github.com/adivarma27/pyab","last_synced_at":"2026-01-14T10:02:05.612Z","repository":{"id":37648695,"uuid":"272311547","full_name":"AdiVarma27/pyAB","owner":"AdiVarma27","description":"Python package for Bayesian \u0026 Frequentist A/B Testing","archived":false,"fork":false,"pushed_at":"2023-07-06T21:32:45.000Z","size":11218,"stargazers_count":12,"open_issues_count":3,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-10T00:41:24.293Z","etag":null,"topics":["ab-testing","bayesian-statistics","data-science","frequentist-statistics","hypothesis-testing","marketing","statistical-methods","statistical-tests"],"latest_commit_sha":null,"homepage":"https://pyab.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdiVarma27.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"docs/contributing.rst","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-15T01:04:41.000Z","updated_at":"2024-10-02T13:20:43.000Z","dependencies_parsed_at":"2022-09-04T15:12:26.109Z","dependency_job_id":null,"html_url":"https://github.com/AdiVarma27/pyAB","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/AdiVarma27/pyAB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdiVarma27%2FpyAB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdiVarma27%2FpyAB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdiVarma27%2FpyAB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdiVarma27%2FpyAB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdiVarma27","download_url":"https://codeload.github.com/AdiVarma27/pyAB/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdiVarma27%2FpyAB/sbom","scorecard":{"id":8739,"data":{"date":"2025-08-11","repo":{"name":"github.com/AdiVarma27/pyAB","commit":"4cb3f693ce63ae48b42d3b8353057a00e5dceb6b"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.3,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/25 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":4,"reason":"6 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: GHSA-6p56-wp2h-9hxr","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: PYSEC-2023-102","Warn: Project is vulnerable to: PYSEC-2023-114"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 6 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-14T14:07:10.180Z","repository_id":37648695,"created_at":"2025-08-14T14:07:10.180Z","updated_at":"2025-08-14T14:07:10.180Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28416497,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","bayesian-statistics","data-science","frequentist-statistics","hypothesis-testing","marketing","statistical-methods","statistical-tests"],"created_at":"2026-01-14T10:02:04.625Z","updated_at":"2026-01-14T10:02:05.598Z","avatar_url":"https://github.com/AdiVarma27.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: https://readthedocs.org/projects/pyab/badge/?version=latest\n  :target: https://pyab.readthedocs.io/en/latest/?badge=latest\n\n.. image:: https://img.shields.io/pypi/v/pyAB\n  :target: https://github.com/AdiVarma27/pyAB\n\n.. image:: https://img.shields.io/github/license/AdiVarma27/pyab\n  :target: https://github.com/AdiVarma27/pyAB/blob/master/LICENSE\n\n.. image:: https://img.shields.io/github/stars/AdiVarma27/pyAB?logo=Github \n  :target: https://github.com/AdiVarma27/pyAB\n \n========\n**pyAB**\n========\npyAB is a Python package for Bayesian \u0026 Frequentist A/B Testing.\n\n========\nFeatures:\n========\nBayesian A/B Test:\n##################\n- Conduct quick experiments to check for winning variant with additional prior information (Beta Distribution parameters).\n- Try different evaluation metrics (Uplift Ratio, Uplift Difference \u0026 Uplift Percent Gain) \u0026 vary number of mcmc simulations.\n- Visualize \u0026 inspect Uplift Density \u0026 Cumulative Density distributions.\n\nFrequentist A/B Test:\n#####################\n- Conduct quick experiments to check for winning variant using two sample proportion test (Statistical significance).\n- Estimate required sample size per variant to reach provided Type-II error rate.\n- Visualize \u0026 inspect power curve for varying alternative proportions.\n\n============\nInstallation:\n============\nBest way to install pyAB is through pip\n\n.. code:: python\n\n   pip install pyAB\n\nTo install from source, use the following Github link\n\n.. code:: python\n\n   git clone https://github.com/AdiVarma27/pyAB.git\n   cd pyAB\n   python setup.py install\n\n============\nDependencies:\n============\n\npyAB has the following dependencies:\n\n- numpy\n- matplotlib\n- seaborn\n- scipy\n- statsmodels\n\n=============\nDocumentation:\n=============\n\npyAB documentation is available at `pyab.readthedocs.io \u003chttps://pyab.readthedocs.io/en/latest/\u003e`_ \u0026 `pyab.rtfd.io \u003chttps://pyab.rtfd.io/en/latest/\u003e`_.\n\nUsage:\n######\n\n\n=================\nBayesian A/B Test\n=================\n\nLet us assume we have two Banner Ads and want to run an AB Test to decide on the final version. We run the test and collect 1000 samples per version. We observe 100 and 120 clicks for version-A \u0026 Version-B respectively **(10 % \u0026 12.5 % Click-through-rates)**. From our previous experience, we know that the average Click-through-rate for our previous Ads was around 12 %. \n\nWe first need to import  ``ABTestBayesian`` class and provide prior clicks ``success_prior`` and prior impressions ``trials_prior``. Then, call the ``conduct_experiment`` method with successful clicks and impressions per version.\n\nFor ``uplift_method``, there are three metrics to choose from are ``'uplift_ratio'``, ``'uplift_percent'`` \u0026 ``'uplift_difference'``. We also choose mcmc ``num_simulations``, which samples from Uplift Probability Density function.\n\n\n.. code:: python\n\n   # import Bayesian class\n   from pyab.experiments import ABTestBayesian\n\n   # provide beta priors\n   ad_experiment_bayesian = ABTestBayesian(success_prior=120, trials_prior=1000)\n\n   # conduct experiment with two variants successes and trials, along with uplift method and number of simulations\n   ad_experiment_bayesian.conduct_experiment(success_null=100, trials_null=1000, \n                                             success_alt=125, trials_alt=1000, \n                                             uplift_method='uplift_ratio', num_simulations=1000)\n\nBayesian A/B test results can extremely useful to **understand \u0026 communicate test results** with other stakeholders and answers the main business question: **Which version works the best ?**\n\n**Output:**\n\n\n.. code::\n\n   pyAB Summary\n   ============\n\n   Test Parameters\n   _______________\n\n   Variant A: Successful Trials 100, Sample Size 1000\n   Variant B: Successful Trials 125, Sample Size 1000\n   Prior: Successful Trials 120, Sample Size 1000\n\n   Test Results\n   ____________\n\n   Evaluation Metric: uplift_ratio\n   Number of mcmc simulations: 1000\n\n   90.33 % simulations show Uplift Ratio above 1.\n\n.. image:: img/fig_2.png\n\n\n====================\nFrequentist A/B Test\n====================\n\nLet us now run a Frequentist A/B Test and verify if there is a significant difference between two proportions provided the sample sizes and Type-I Error rate. From above, we know the performance of version-A \u0026 version-B **(10 % \u0026 12.5 % Click-through-rates)**, for 1000 impressions of each version.\n\nWe first need to import  ``ABTestFrequentist`` class and provide type of alternative hypothesis ``alt_hypothesis``, ``'one_tailed'`` or ``'two_tailed'`` \u0026 Type-I error rate ``alpha`` (default = 0.05). Then, we call the ``conduct_experiment`` method with successful clicks and impressions per version.\n\nThis traditional methodology might be **slightly tricky to communicate**, and **Type-I \u0026 Type-II error rates** need to be accounted for, unlike Bayesian methods.\n\n\n.. code:: python\n\n   # import Frequentist class\n   from pyab.experiments import ABTestFrequentist\n\n   # provide significance rate and type of test\n   ad_experiment_freq = ABTestFrequentist(alpha=0.05, alt_hypothesis='one_tailed')\n\n   # conduct experiment with two variants successes and trials, returns stat \u0026 pvalue\n   stat, pvalue = ad_experiment_freq.conduct_experiment(success_null=100, trials_null=1000, \n                                    success_alt=125, trials_alt=1000)\n\n**Output:**\n\n\n.. code::\n\n   pyAB Summary\n   ============\n\n\n   Test Parameters\n   _______________\n\n   Variant A: Success Rate 0.1, Sample Size 1000\n   Variant B: Success Rate 0.125, Sample Size 1000\n   Type-I Error: 0.05, one_tailed test\n\n\n   Test Results\n   ____________\n\n   Test Stat: 1.769\n   p-value: 0.038\n   Type-II Error: 0.451\n   Power: 0.549\n\n   There is a statistically significant difference in proportions of two variants.\n\n.. image:: img/fig_1.png\n\n\nGiven that the current Type-II error is 0.451 at 1000 samples per variant, we can find out **required sample size per variant** to reach Type-II error of 0.1.\n\n\n.. code:: python\n   \n   # required sample size per variant for given beta\n   ad_experiment.get_sample_size(beta=0.1)\n\n**Output:**\n\n\n.. code::\n\n   2729\n\n===============================\nNever misinterpret your Results !\n===============================\n\n|ImageLink|_\n\n.. |ImageLink| image:: https://imgs.xkcd.com/comics/significant.png\n.. _ImageLink: https://imgs.xkcd.com/comics/significant.png\n\n\n\n=======\nLicense:\n=======\n\n`MIT License Copyright (c) 2020 \u003chttps://github.com/AdiVarma27/pyAB/blob/master/LICENSE\u003e`_\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadivarma27%2Fpyab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadivarma27%2Fpyab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadivarma27%2Fpyab/lists"}