{"id":19725357,"url":"https://github.com/bodacious/optimisationtestcomparisons","last_synced_at":"2025-07-12T15:03:21.911Z","repository":{"id":66871697,"uuid":"56640980","full_name":"Bodacious/OptimisationTestComparisons","owner":"Bodacious","description":"Testing the effectiveness of split A/B testing vs. Epsilon Greedy testing","archived":false,"fork":false,"pushed_at":"2016-04-22T15:33:31.000Z","size":19,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-10T16:54:44.606Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bodacious.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-20T00:03:15.000Z","updated_at":"2016-05-08T21:56:04.000Z","dependencies_parsed_at":"2023-02-23T10:45:59.686Z","dependency_job_id":null,"html_url":"https://github.com/Bodacious/OptimisationTestComparisons","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodacious%2FOptimisationTestComparisons","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodacious%2FOptimisationTestComparisons/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodacious%2FOptimisationTestComparisons/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodacious%2FOptimisationTestComparisons/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bodacious","download_url":"https://codeload.github.com/Bodacious/OptimisationTestComparisons/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241049320,"owners_count":19900426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T23:29:29.387Z","updated_at":"2025-02-27T19:19:56.512Z","avatar_url":"https://github.com/Bodacious.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Comparing Split A/B Tests VS Epsilon Greedy\n\nThis is still a bit of a work in progress, but you should be able to download it and start trying it out yourself.\n\n\n## Background\n\nThe basic idea behind this test was to create a hypothetical optimization test with pre-defined results. I would then take a sample of pre-defined users and feed them through each type of test, to see how the test results compared to the final results (which were defined FIRST in this case.)\n\n\n### Results so far\n\nFrom initial testing, it confirms the expectations of Split A/B and Epsilon Greedy tests.\n\n**Split A/B** tests give more objective, accurate results, but more opportunities are _wasted_ on inferior variants.\n\n**Epsilon greedy** tests require much larger sample sizes for accurate results, but far fewer opportunities are _wasted_ on inferior variants.\n\n```\nTest: Split test\nSample size: 10000\n\nExpected results:\n  Variant A: 5%\n  Variant B: 7%\n\nActual results:\n  Variant A: 5.0% 248/4997\n  Variant B: 7.0% 348/5003\n\nTest: Epsilon Greedy\nSample size: 10000\n\nExpected results:\n  Variant A: 5%\n  Variant B: 7%\n\nActual results:\n  Variant A: 5.2% 495/9476\n  Variant B: 7.3% 38/524\n```\n\n## Prerequisites\n\n1. SQLite3\n2. Bundler\n3. Ruby 2.3.0 (or remove ruby version from Gemfile)\n\n## Installation\n\nDownload the repo\n\n    $ git clone https://github.com/Bodacious/OptmisationTestComparisons.git\n\nInstall the gems\n\n    $ bundle install\n\n## Making sure it works\n\nThere are a few basic specs in place. Run these with:\n\n    $ rspec spec\n\n## Running the tests\n\nYou can run each test with a simple\n\n    $ rake\n\nTo run a specific test, use\n\n    $ rake run:split\n\nor\n\n    $ rake run:epsilon_greedy\n\n\n### Setting the expected final results\n\nBy default, the tests will run with an A and B variant, both within the 5%—15% range. To pass in your own variant names, and their expected conversion rates, set the `RESULTS` environment variable, passing in a JSON object with the variants and conversion rates.\n\n\n    $ rake RESULTS='{\"A\":9.32,\"B\":11.65}'\n\n\n### Setting the number of samples\n\nBy default, the tests will run with 5000 samples each test. If you'd like to change this, simply set an env variable like so:\n\n    $ rake run:split SAMPLES=40000\n\n### Setting the exploration rate on Epsilon Greedy tests\n\nThe **Epsilon Greedy** test has a random element to it. The algoritm will assign a random variant to the sample user, a small percentage of the time. You can set the rate of randomness with the `RANDOM` environment variable.\n\n    # Run with 1000 samples, and allocate a random variant 20% of the time.\n    $ rake run:epsilon_greedy_test SAMPLES=1000 RANDOM=20\n\n\n## Want to run a test on your website?\n\n[Katana Code](http://katanacode.com/?utm_campaign=OptimisationTestComparisons\u0026utm_source=GitHub\u0026utm_medium=README) is a software development based in Edinburgh Scotland. We work with startups on 4 different continents, helping them design, build, and optimise their digital products.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbodacious%2Foptimisationtestcomparisons","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbodacious%2Foptimisationtestcomparisons","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbodacious%2Foptimisationtestcomparisons/lists"}