{"id":20325426,"url":"https://github.com/getyourguide/basemath","last_synced_at":"2025-04-11T19:50:42.422Z","repository":{"id":208823222,"uuid":"722517084","full_name":"getyourguide/basemath","owner":"getyourguide","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-24T07:51:01.000Z","size":231,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-24T08:38:55.720Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getyourguide.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-23T10:16:53.000Z","updated_at":"2025-03-24T07:51:03.000Z","dependencies_parsed_at":"2023-11-23T13:47:36.200Z","dependency_job_id":"3f26b5d2-7e0a-44bf-99d8-c45f71dd9dfd","html_url":"https://github.com/getyourguide/basemath","commit_stats":null,"previous_names":["getyourguide/basemath"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getyourguide%2Fbasemath","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getyourguide%2Fbasemath/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getyourguide%2Fbasemath/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getyourguide%2Fbasemath/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getyourguide","download_url":"https://codeload.github.com/getyourguide/basemath/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248471358,"owners_count":21109416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T19:39:47.429Z","updated_at":"2025-04-11T19:50:42.384Z","avatar_url":"https://github.com/getyourguide.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Basemath\n\nWelcome to Basemath, an open-source implementation of the statistical test bearing the same name designed for analyzing\nAB experiments.\n\nBasemath employs a one-sided testing approach, where the null hypothesis posits that the treatment performs either\nequally or worse than the control concerning the target metric. The test has a predetermined maximum sample size determined\nby the input parameters. Additionally, it ensures that both type I and type II errors remain below specified error\nthresholds, denoted as α and β, respectively.  Basemath assesses the experiment in batches and terminates prematurely\nif it can not reject the null hypothesis. The β-spending function employed is O’Brien-Fleming-like.\nGiven that the majority of experiments yield either flat or negative results, stopping early in this scenario saves more\nrunning time compared to stopping in the less common case of a significant uplift.\n\nWhat sets Basemath apart is its avoidance of recurrent numerical integration resulting in a straightforward and fast\nimplementation. For a detailed exploration of Basemath's mathematical foundations, refer to our article\n[Basemath’s Test: Group Sequential Testing Without Recurrent Numerical Integration](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4599695).\n\n## Installation\nSimply install the library using your package manger of choice.\n\nFor instance, with pip:\n\n```pip install basemath-analysis```\n\n## Usage\n### The Binary Case\n\nAssume you are conducting an experiment on your platform. You have modified the user experience (UX) and aim to\ndemonstrate that this change improves the conversion of visitors to customers. To achieve this, you present the\ncurrent UX (control group) to half of your visitors and the new UX (treatment group) to the other half. Data on the\nnumber of people visiting your platform and the number of visitors converting to customers are processed on a daily\nbasis. Your expectation is that the UX change will lead to a relative increase in the conversion rate (number of\ncustomers / number of visitors) by at least 1%.\n\nYou initialize Basemath using the following Python code:\n\n```python\nimport basemath_analysis.basemath as bm\nbm_test = bm.BaseMathsTest(cr_A, mde, alpha, beta, seed=\"experiment_name\")\n```\nThe parameters are as follows:\n\n* __cr_A__: The estimated conversion rate of your control group.\n* __mde__: The minimal relative uplift you are aiming for (1% in our example).\n* __alpha__: The maximal type I error you are willing to tolerate (α is often set to 5%).\n* __beta__: The maximal type II error you are willing to tolerate (β is often set to 20%).\n* __seed__: As the algorithm contains a random element, set a seed to ensure consistent outcomes when running the\nalgorithm repeatedly on the same data. The seed is generated from a string such as the unique experiment name.\nAfter initialization, you can check the maximum number of required samples for each variation to estimate the running\ntime:\n\n```python\nprint(bm_test.required_samples)\n```\n\nEnter the most recent data daily into the instance to determine if there is a significant outcome:\n\n```python\nbm_test.evaluate_experiment(\n    previous_customer_delta,\n    customer_delta_since_yesterday,\n    previous_visitor_number,\n    visitors_since_yesterday\n)\n```\n\nThe parameters for this method are:\n\n* __previous_customer_delta__: The difference between the overall number of customers in the treatment and control groups\nas of the last check-in, i.e., as of yesterday in our example.\n* __customer_delta_since_yesterday__: The difference between the overall number of customers in the treatment and control\ngroups since the last check-in, i.e., since yesterday in our example.\n* __previous_visitor_number__: The number of visitors per variation as of the last check-in.\n* __visitors_since_yesterday__: The number of visitors per variation since the last check-in.\n\nThe outcome of this method is either 0, 1, or -1. If Basemath hasn't reached a significant conclusion yet, it returns\na 0. If the alternative hypothesis that the treatment is significantly better than the control can be rejected, it\nreturns a -1. This can occur in each evaluation step. If Basemath finds a significant uplift for the treatment\ngroup, it returns a 1. This can only happen once the maximum number of required samples has been reached.\n\nOnce the outcome is no longer 0, the test has concluded, and the experiment can be stopped. Further evaluation beyond\nthis point is futile.\n\n### The Continuous Case\nSome changes may not result in more visitors converting to customers. The number of customers might remain the same;\nhowever, customers in the treatment group might spend more money on your platform. In this case, change your target\nmetric from conversion rate to revenue per visitor (sum of revenue / number of visitors).\n\nFor a continuous target variable, the initialization of Basemath only slightly changes:\n```python\nbm_test = bm.BaseMathsTest(rpv_A, mde, alpha, beta, var_A=var_A, seed=\"experiment_name\")\n```\nInstead of the estimated conversion rate, enter the estimated average revenue per visitor (rpv_A) for the control\ngroup. Additionally, estimate the variance of the revenue per visitor, considering visitors who do not convert (i.e.,\nassuming a revenue of 0 for them). The other parameters remain the same as in the binary case.\n\nBasemath only works with two variations and a 50/50 split in traffic.\n\nMore detailed examples are available [here](./examples).\n\n## Contributing\n\nTo contribute, you'll need a working python 3.8+ installation. We also recommend setting up a virtual environment for the project. You'll also need to [install poetry](https://python-poetry.org/docs/) if it's not already present.\n\nOnce you have the dependencies installed (with `poetry install`), you can set up pre-commit with `pre-commit install`. We run pre-commit in our CI as well, but it's recommended to install it locally so that you get immediate feedback from our various linters.\n\nBeyond that, there's not really anything else you need to know to start contributing! Please create a pull request with whatever changes you'd like to propose and increment the version and update the changelog if necessary.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetyourguide%2Fbasemath","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetyourguide%2Fbasemath","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetyourguide%2Fbasemath/lists"}