{"id":47944191,"url":"https://github.com/dataiku/dss-plugin-ab-testing","last_synced_at":"2026-04-04T08:20:45.618Z","repository":{"id":54249314,"uuid":"278628231","full_name":"dataiku/dss-plugin-ab-testing","owner":"dataiku","description":"Dataiku DSS plugin for A/B testing","archived":false,"fork":false,"pushed_at":"2025-02-07T10:17:15.000Z","size":242,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-02-07T10:27:29.106Z","etag":null,"topics":["ab-testing","dataiku","dss-plugin"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dataiku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-10T12:29:42.000Z","updated_at":"2021-03-01T17:14:27.000Z","dependencies_parsed_at":"2022-08-13T10:00:46.979Z","dependency_job_id":null,"html_url":"https://github.com/dataiku/dss-plugin-ab-testing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dataiku/dss-plugin-ab-testing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataiku%2Fdss-plugin-ab-testing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataiku%2Fdss-plugin-ab-testing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataiku%2Fdss-plugin-ab-testing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataiku%2Fdss-plugin-ab-testing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dataiku","download_url":"https://codeload.github.com/dataiku/dss-plugin-ab-testing/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dataiku%2Fdss-plugin-ab-testing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31392729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T04:26:24.776Z","status":"ssl_error","status_checked_at":"2026-04-04T04:23:34.147Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","dataiku","dss-plugin"],"created_at":"2026-04-04T08:20:44.905Z","updated_at":"2026-04-04T08:20:45.610Z","avatar_url":"https://github.com/dataiku.png","language":"HTML","readme":"# AB Testing Plugin\n\nThis plugin provides features to design AB testing and analyses their outcome inside DSS. \n\n## Design\n\nA web app computes the minimum sample sizes needed in the experiment while providing insightful visualisations of the distributions (Z test). A custom recipe can then use these estimated figures to split the input dataset containing the email addresses of the experiment population into two groups, A and B. \n\nHere is a more detailled description of these two main components : \n\n### 1- A/B test sample size calculator (custom web app)\n\nThis visual web app computes the required sample size to conduct the experiment. \n\n#### Input\n\n`Parameters` folder where the parameters and sample sizes are stored.\n\n#### Experiment parameters\n\nInside the web app, you may input your different parameters to compute the sample size : \n\n* **Baseline success rate (%)** : current success rate of the variant\n* **Minimal detectable effect (%)**: the minimal variation of the baseline conversion rate that you need to detect a statistically significant change.\n* **Daily number of people exposed**\n* **Percentage of the traffic affected**\n\nFrom these values, a minimum sample size is computed and illustrated thanks to the chart of the distributions. \n\n#### Output\n\nThere is no output, but when you click on the button save parameters, the parameters and the samples sizes are saved in the folder `Parameters` .\n\n### 2- Population split (custom recipe)\n\nThis recipe splits the users enrolled in the experiment into two groups, usually based on the sample sizes which were previously computed in the `AB testing design` web app. \n\n#### Input\n\n* `Population dataset` : Dataset with the reference of the users involved in the experiment(ids, emails...)\n* `Parameters folder` (optional) : Folder containing the parameters computed in the `AB testing design` web app, previously introduced. \n\n#### Parameters\n\n* **User reference** : Column containing user reference (user Id , email...). Each user should have a unique reference.\n* **Sample size definition**: do you want to retrieve the sample sizes from the web app or edit them manually? \n* **Parameters (computed in the web app)**: if you want to retrieve the sample sizes from the `parameters folder` , choose which json file contains the right parameters and sample sizes. \n* **Sample size for variation A** : Minimum sample size for the A group\n* **Sample size for variation B** : Minimum sample size for the B group\n* **Deal with leftover users** :  If the population is greater than the sample size, this field specifies in which group the leftover users should go.\n\n#### Output\n\n* `Experiment dataset` : Input dataset with an extra column containing the group indicators used for the AB test (A or B)\n\n## Analysis of the results\n\nOnce the experiment is complete, the user may upload the results back to DSS. With a custom recipe, she computes the resulting statistics (conversion rate per group). With the second web app, she can analyse these results and determine the outcome of the statistical test. \n\n### 3- Experiment summary (custom recipe) \n\nFrom the results of your experiment, this recipe computes the statistics required to analyse the outcome of the statistical test.\n\n#### Input\n\n* `experiment_results` : This dataset should contain the experiment's results at a user level. There should be group column and a conversion column. \n\n#### Parameters\n\n* **User reference** : Column containing user reference (user Id , email...). Each user should have a unique reference.\n* **Conversion column** : Column indicating if a user converted or not (Binary values)\n* **AB group column** : Column indicating to which group a user belongs. This column should contain binary values (O-1, A-B, group_A-group_B)\n\n#### Output\n\n* `AB testing statistics` : Statistics required to answer the statistical test\n\n### 4- Results analysis (custom web app)\n\nFrom the `AB testing statistics` dataset, this web app gives a clear answer to the statistical test. Make sure to **refresh** the settings page when you open it.  \n\n#### Input\n\n* `AB testing statistics` : Statistics required to answer the statistical test\n\n#### Parameters\n\n* **AB statistics entry from** : do you want to retrieve statistics from the `AB testing statistics` dataset or just enter the values manually?\n* **Dataset** : It should be the output of the recipe AB statistics of the AB testing plugin. Otherwise, use the manual mode\n* **AB group column** : Column indicating to which group a user belongs (A or B)\n* **Output folder for results** :  Where do you want to save the results of the experiment?\n\n#### Output\n\nThere is no output, but when you click on the button save results, the results are saved in the output folder. \n\n# Release notes\n## Release 1.0.1\n- Fixed a bug : the duration of the experiment was not always properly updated\n\n## Release 1.0.0 \n### AB test size calculator web app\n- Include A/B tests for binomial metrics such as click through rate\n- Compute minimum sample sizes for two different variants \n- Visualisation of the statistical test in a dynamic chart\n- Save the parameters of the experiment in a json, stored in a managed folder\n- Confusion matrix\n- Mathematical derivation of the sample size computation\n\n\n### Population split recipe\n- Split an input dataset into two groups\n- Use the json computed in the `AB test size calculator web app` to set the sizes of each group\n- Add the leftover users to group A, group B or leave blank\n\n### Summary recipe\n- Compute statistics for binomial metrics (success rates)\n- Conversion column should only contain 0 or 1\n- AB group column should only contain 2 unique values\n\n### AB test results analysis web app\n- Analyse results from the output dataset of the `summary recipe`\n- Manually edit sizes\n- Results are phrased in a text box\n- Visualisation of the results using the reject zone and the confidence interval\n- Save results in a json, stored in a managed folder\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataiku%2Fdss-plugin-ab-testing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdataiku%2Fdss-plugin-ab-testing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdataiku%2Fdss-plugin-ab-testing/lists"}