{"id":30113851,"url":"https://github.com/dmitry-brazhenko/ab-test-advanced-toolkit","last_synced_at":"2026-03-09T20:06:21.481Z","repository":{"id":240846657,"uuid":"789385960","full_name":"dmitry-brazhenko/ab-test-advanced-toolkit","owner":"dmitry-brazhenko","description":"Advanced A/B Testing Toolkit: Powerful suite for reaching conclusions faster with fewer experiments using CUPED and Gradient Boosting. Includes metric calculations, statistical significance analysis, and visualization.","archived":false,"fork":false,"pushed_at":"2025-02-04T04:52:55.000Z","size":31572,"stargazers_count":32,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-02-01T02:22:22.671Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmitry-brazhenko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-20T11:58:22.000Z","updated_at":"2026-01-27T23:17:30.000Z","dependencies_parsed_at":"2024-05-28T19:05:49.669Z","dependency_job_id":"0f31847a-3737-4abe-87b6-17769a262803","html_url":"https://github.com/dmitry-brazhenko/ab-test-advanced-toolkit","commit_stats":null,"previous_names":["dmitry-brazhenko/ab-test-advanced-toolkit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dmitry-brazhenko/ab-test-advanced-toolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitry-brazhenko%2Fab-test-advanced-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitry-brazhenko%2Fab-test-advanced-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitry-brazhenko%2Fab-test-advanced-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitry-brazhenko%2Fab-test-advanced-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmitry-brazhenko","download_url":"https://codeload.github.com/dmitry-brazhenko/ab-test-advanced-toolkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmitry-brazhenko%2Fab-test-advanced-toolkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30310052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T20:05:46.299Z","status":"ssl_error","status_checked_at":"2026-03-09T19:57:04.425Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-10T07:31:05.301Z","updated_at":"2026-03-09T20:06:21.464Z","avatar_url":"https://github.com/dmitry-brazhenko.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A/B Test Advanced Toolkit: CUPED and Gradient Boosting\n\n[![PyPI version](https://img.shields.io/pypi/v/variatio.svg)](https://pypi.org/project/variatio/)\n[![Python](https://github.com/dmitry-brazhenko/SharpToken/actions/workflows/build-test-and-publish.yml/badge.svg?branch=main)](https://github.com/dmitry-brazhenko/SharpToken/actions/workflows/build-test-and-publish.yml)\n[![GitHub Issues](https://img.shields.io/github/issues/dmitry-brazhenko/Variatio.svg)](https://github.com/dmitry-brazhenko/Variatio/issues)\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)\n\nA/B Test Advanced Toolkit is an experimental Python library designed for advanced A/B testing analysis. It leverages statistical techniques and machine learning, including variance reduction through CUPED and integration with XGBoost for predictive insights. A/B Test Advanced Toolkit is ideal for data scientists and researchers looking to obtain deeper insights from their A/B testing efforts.\n\n## Features\n\nThis tool streamlines A/B testing by automatically calculating all the classic metrics behind the scenes. Simply provide your data, and it will compute the metrics you request, delivering them in a visually appealing report. Here are some of the key features:\n\n- **Statistical Significance Testing:** Easily compare metrics between control and test groups, with all necessary calculations done for you.\n- **CUPED Adjustments:** This experimental extension utilizes Controlled-experiment Using Pre-Experiment Data (CUPED) techniques for variance reduction. It innovatively employs user properties as covariates and leverages XGBoost regression, moving beyond traditional linear approaches to enhance A/B test sensitivity.\n\n## Installation\n\n```bash\npip install ab_test_advanced_toolkit\n```\n\n### Development Setup\n\nIf you want to contribute or install from source with development dependencies:\n\n1. Clone the repository and navigate to its directory:\n```bash\ngit clone https://github.com/dmitry-brazhenko/ab-test-advanced-toolkit.git\ncd ab_test-advanced-toolkit\n```\n\n2. Create and activate a virtual environment:\n```bash\npython -m venv venv\nsource venv/bin/activate  # On Unix/macOS\n# or\n.\\venv\\Scripts\\activate  # On Windows\n```\n\n3. Install development dependencies:\n```bash\npip install -e \".[dev]\"\n```\n\n4. Run tests to verify everything is working:\n```bash\npytest tests/\n```\n\n## Data Preparation\n\nTo use A/B Test Advanced Toolkit effectively, you need to prepare three main types of data:\n\n### 1. Event Data\n\nThe `event_data` is crucial for analysis in A/B Test Advanced Toolkit. It tracks user interactions and should include the following mandatory columns:\n\n- `timestamp`: The date and time when the event occurred.\n- `userid`: A unique identifier for the user who triggered the event.\n- `event_name`: The name of the event (e.g., 'login', 'purchase').\n\nIn addition to these mandatory columns, you can include optional attributes that provide additional details about each event. For example, `purchase_value` could be used to track the value of purchase events. These optional attributes can vary based on the event type and what you aim to analyze.\n\n**Event Data Example:**\n\n| timestamp           | userid | event_name | purchase_value |\n|---------------------|--------|------------|----------------|\n| 2023-01-01 00:00:00 | 7      | purchase   | 274            |\n| 2023-01-01 01:00:00 | 4      | purchase   | 175            |\n| 2023-01-01 02:00:00 | 5      | purchase   | 179            |\n| 2023-01-01 03:00:00 | 7      | purchase   | 102            |\n| 2023-01-01 04:00:00 | 3      | login      | 0              |\n\n### 2. User Allocations Data Sample\n\nThe `ab_test_allocations` dataset is essential for understanding the distribution of users across different A/B test groups. It includes the following columns:\n\n- `timestamp`: The date and time when the user was attributed to a specific A/B test group. This helps track when each user started experiencing the variant they were allocated to.\n- `userid`: A unique identifier for the user.\n- `abgroup`: The A/B test group the user was allocated to. This typically represents the control group (e.g., 'A') and one or more test groups (e.g., 'B', 'C').\n\n**Example:**\n\n| timestamp           | userid | abgroup |\n|---------------------|--------|---------|\n| 2022-12-15 00:00:00 | 1      | A       |\n| 2022-12-15 01:00:00 | 2      | B       |\n| 2022-12-15 02:00:00 | 3      | B       |\n| 2022-12-15 03:00:00 | 4      | B       |\n| 2022-12-15 04:00:00 | 5      | A       |\n\n### 3. User Properties Data Sample (Optional)\n\nThis optional `user_properties` dataset can enhance the analysis with user demographic or behavioral data. The `userid` column must match the `event_data` and `ab_test_allocations` datasets.\n\n| userid | age | gender | country    | device_type | membership_status |\n|--------|-----|--------|------------|-------------|-------------------|\n| 1      | 56  | Male   | USA        | Tablet      | Free              |\n| 2      | 69  | Female | India      | Mobile      | Free              |\n| 3      | 46  | Male   | Australia  | Tablet      | Free              |\n| 4      | 32  | Female | UK         | Tablet      | Free              |\n| 5      | 60  | Male   | Germany    | Tablet      | Free              |\n\n## Using ABTestAnalyzer\n\nAfter preparing your datasets, you can use `ABTestAnalyzer` to perform A/B testing analysis. Initialize the analyzer with your datasets and specify the control group:\n\n```python\nfrom ab_test_advanced_toolkit.analyzer import ABTestAnalyzer\nimport pandas as pd\n\nevent_data = pd.DataFrame({\n    \"timestamp\": pd.to_datetime([\"2023-01-01 00:00:00\", \"2023-01-01 01:00:00\", \n                                 \"2023-01-01 02:00:00\",\n                  \"2023-01-01 03:00:00\", \"2023-01-01 04:00:00\"]),\n    \"userid\": [7, 4, 5, 7, 3],\n    \"event_name\": [\"purchase\", \"purchase\", \"purchase\", \"purchase\", \"login\"],\n    \"purchase_value\": [274, 175, 179, 102, 0]\n})\n\n# Creating the user_allocations DataFrame\nuser_allocations = pd.DataFrame({\n    \"timestamp\": pd.to_datetime([\"2022-12-15 00:00:00\", \"2022-12-15 01:00:00\", \n                                 \"2022-12-15 02:00:00\",\n                  \"2022-12-15 03:00:00\", \"2022-12-15 04:00:00\"]),\n    \"userid\": [1, 2, 3, 4, 5],\n    \"abgroup\": [\"A\", \"B\", \"B\", \"B\", \"A\"]\n})\n\n# Creating the user_properties DataFrame\nuser_properties = pd.DataFrame({\n    \"userid\": [1, 2, 3, 4, 5],\n    \"age\": [56, 69, 46, 32, 60],\n    \"gender\": [\"Male\", \"Female\", \"Male\", \"Female\", \"Male\"],\n    \"country\": [\"USA\", \"India\", \"Australia\", \"UK\", \"Germany\"],\n    \"device_type\": [\"Tablet\", \"Mobile\", \"Tablet\", \"Tablet\", \"Tablet\"],\n    \"membership_status\": [\"Free\", \"Free\", \"Free\", \"Free\", \"Free\"]\n})\n\nanalyzer = ABTestAnalyzer(event_data, user_allocations, \"A\", user_properties, mode=\"gboost_cuped\")\n```\n\nYou can then calculate various metrics such as event count per user, attribute sum per user (useful for calculating metrics like ARPU), or conversion rates to specific events:\n\n```python\n# Calculate the count of 'purchase' events per user\nanalyzer.calculate_event_count_per_user('purchase')\n\n# Calculate the sum of 'purchase_value' for 'purchase' events per user\nanalyzer.calculate_event_attribute_sum_per_user('purchase', 'purchase_value')\n\n# Calculate the conversion rate to 'login' events\nanalyzer.calculate_conversion('login')\n```\n\n### Generating Reports\n\nAfter calculating the desired metrics, you can save them to an HTML file for easy viewing:\n\n```python\nanalyzer.save_report(\"abtest_report.html\")\n```\n\nHere's a sample report showcasing meticulously calculated metrics, organized in an easy-to-analyze table format.\n\n![Metrics Example](examples/metrics_example.png)\n\n## Contribution Guidelines\n\nWe welcome contributions to AB Test Advanced Toolkit! If you'd like to contribute, please follow these guidelines:\n\n- Fork the repository and create your feature branch.\n- Make sure your code adheres to the project's coding standards.\n- Submit a pull request with a detailed description of your changes.\n\n## License\n\nAB Test Advanced Toolkit is licensed under the MIT License. See [LICENSE](LICENSE) for more details.\n\n## Disclaimer\n\nAB Test Advanced Toolkit is shared with the community as an experimental library, offered \"as-is\" and without any warranties. Your explorations and tests with it are encouraged, but please proceed with awareness of its experimental nature. We're keen to hear about your experiences and eager to tackle any challenges you encounter. Should you have questions or face any issues, don't hesitate to open an issue in our repository. We welcome your feedback and contributions to make AB Test Advanced Toolkit even better!\n\n## Tests\n\n```bash\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmitry-brazhenko%2Fab-test-advanced-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmitry-brazhenko%2Fab-test-advanced-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmitry-brazhenko%2Fab-test-advanced-toolkit/lists"}