{"id":18521577,"url":"https://github.com/transferwise/tw-experimentation","last_synced_at":"2025-04-09T09:33:20.584Z","repository":{"id":216481172,"uuid":"351008688","full_name":"transferwise/tw-experimentation","owner":"transferwise","description":"AB testing tool","archived":false,"fork":false,"pushed_at":"2024-11-26T14:30:02.000Z","size":15101,"stargazers_count":4,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T04:43:23.688Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/transferwise.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-24T08:51:22.000Z","updated_at":"2024-11-26T14:30:31.000Z","dependencies_parsed_at":"2024-01-12T22:30:52.911Z","dependency_job_id":"27154394-990c-4e58-a6dd-22a463a9a0ca","html_url":"https://github.com/transferwise/tw-experimentation","commit_stats":null,"previous_names":["transferwise/tw-experimentation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Ftw-experimentation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Ftw-experimentation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Ftw-experimentation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Ftw-experimentation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/transferwise","download_url":"https://codeload.github.com/transferwise/tw-experimentation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248012844,"owners_count":21033251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T17:26:36.432Z","updated_at":"2025-04-09T09:33:15.574Z","avatar_url":"https://github.com/transferwise.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TW Experimentation: A library for automated A/B testing\n\n\n**TW Experimentation** is a library to design experiments, check data, run statistical tests and make decisions\n\n\u003csummary\u003e\u003cstrong\u003e\u003cem\u003eTable of Contents\u003c/em\u003e\u003c/strong\u003e\u003c/summary\u003e\n\n- [TW Experimentation: A library for automated A/B testing and causal inference](#tw-experimentation-a-library-for-automated-ab-testing-and-causal-inference)\n  - [What can this do for you?](#what-can-this-do-for-you)\n    - [1. Designing experiments](#1-designing-experiments)\n    - [2. Evaluating results](#2-evaluating-results)\n  - [Installation](#installation)\n  - [Quick Start](#quick-start)\n    - [Notebooks](#notebooks)\n    - [Streamlit web app](#streamlit-web-app)\n  - [For Developers](#for-developers)\n    - [Testing](#testing)\n\n ## What can this do for you?\n\nThe experimentation library can help you with:\n- Sample Size Calculator  [link](https://github.com/transferwise/tw-experimentation/blob/main/notebooks/1_pre_experiment.ipynb) - here you can estimate how many observations you need to perform an experiment on the given parameters.\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/sample_size_notebook_1.png?raw=True)\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/sample_size_notebook_2.png?raw=True)\n\n- Integrity checks + Evaluation [link](https://github.com/transferwise/tw-experimentation/blob/main/notebooks/2_integrity_checks%20%2B%20evaluation.ipynb) - here you can check important plots to understand your data better and then run statistical evaluation. Our library will automatically detect if it is a binary metric or continuous and then apply specific method as well as multiple hypothesis correction if needed.\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/integrity_checks_notebook_1.png?raw=True)\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/integrity_checks_notebook_2.png?raw=True)\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/evaluation_notebook_1.png?raw=True)\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/evaluation_notebook_2.png?raw=True)\n\n- Evaluation (Bayesian A/B testing) [link](https://github.com/transferwise/tw-experimentation/blob/main/notebooks/2a_evaluation_bayesian.ipynb) - here you can apply bayesian evaluation on your dataset\n\n### 1. Designing experiments\nBy using **TW Experimentation** you can design your experiments, choose sample size, evaluate the experiment.\n\n\n### 2. Evaluating results\nYou can use various statistical tests for the metrics provided. (Frequentist / Bayesian)\nFor this goal you can use jupyter notebooks or streamlit app with user-friendly interface.\nUsing this repo you can:\n- Run frequentist evaluation\n- Run bayesian evaluation\n- Apply multiple hypothesis correction\n- Remove outliers\n- Check different plots\n- Run segmentation to check evaluation on specific segments as well as find unusual segments using our another tool: wise-pizza\n\n\n## Installation\n\n1. You can easily install this repo using \"pip\"\n```\npip install tw-experimentation\n```\n\nThen you can just use all functionality.\n\nFor running streamlit app please just open the terminal and run:\n```\nrun_tw_experimentation_streamlit\n```\n\n2. You can install the package  via the dependency manager poetry after cloning/git pull/download it as a zip from this repository.\n\nTo do so, clone the repository by running\n```\ngit clone git@github.com:transferwise/tw-experimentation.git\n```\nfrom terminal.\nTo set up poetry, run\n```\nmake set-up-poetry-mac\n```\nfor mac (or linux) and\n```\nmake set-up-poetry-windows\n```\nfor windows.\nThen, run\n```\nmake run-streamlit-poetry\n```\nfrom the root of the package folder.\n\n3. **Alernative:** TW Experimentation requires the following libraries to work which you can find in the .yml file. To install requirements please make sure you have installed the package manager Anaconda and then run the following commands in the terminal:\n\n```\nconda env create -n \u003cmy_env\u003e -f envs/environment.yml\nconda activate \u003cmy_env\u003e\n```\n\nIf you are using Windows, please do these additional steps:\n\n1. pick a jaxlib-0.3.7 wheel from here https://whls.blob.core.windows.net/unstable/index.html and install it manually (pip install \u003cwheel_url\u003e)\n2. Install jax==0.3.7\n\nIf you have any problems with jax on Mac, please do the following:\n```\npip uninstall jax jaxlib\nconda install -c conda-forge jaxlib\nconda install -c conda-forge jax\n```\n\n## Quick Start\n\nMake sure you have followed the installation instructions.\n\n### Notebooks\nYou can use the jupyter notebooks `1_pre_experiment.ipynb` or `2_integrity_checks + evaluation.ipynb` for experiments design and evaluation.\nThe tw experimentation package can be used for different things, for example for analyzing results:\n\n```Python\ndf = pd.read_csv('experiment.csv')\n\ned = ExperimentDataset(\n    data=df,\n    variant=\"T\",\n    targets=['conversion', 'revenue'],\n    date='trigger_dates',\n    pre_experiment_cols=None,\n    n_variants=2,\n)\ned.preprocess_dataset(remove_outliers=True)\n```\n\nThis code will generate the data model for experiment analysis\n\nAnd then you can run evaluation\n\n```Python\nevaluation = FrequentistEvaluation(experiment_dataset)\nevaluation.start()\n```\n\n### Streamlit web app\n\nFor running streamlit app please just open the terminal and run after \"pip install\" section:\n```\nrun_tw_experimentation_streamlit\n```\n\nOr:\n\nOpen terminal and navigate to the repository.\nThen navigate to the folder `./tw_experimentation/streamlit`.\n\nNow run the command `streamlit run Main.py` and the app should open in your browser.\n\nThen you can use or test dataset from data/test_data.csv\n\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/data_loading_streamlit.png?raw=True)\n![plot](https://github.com/transferwise/tw-experimentation/blob/main/docs/images/evaluation_streamlit.png?raw=True)\n\nTip on navigation:\n`ls` - show files in current directory\n`pwd` - print current directory address\n`cd` - change directory, e.g. `cd ./tw_experimentation/streamlit`\n\n## For Developers\n### Testing\nWe use [PyTest](https://docs.pytest.org/) for testing. If you want to contribute code, make sure that the tests in tests/ run without errors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftransferwise%2Ftw-experimentation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftransferwise%2Ftw-experimentation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftransferwise%2Ftw-experimentation/lists"}