{"id":18941465,"url":"https://github.com/mchmarny/ab-test-demo","last_synced_at":"2026-04-26T22:31:10.028Z","repository":{"id":77051191,"uuid":"204096725","full_name":"mchmarny/ab-test-demo","owner":"mchmarny","description":"Simple A/B testing on Cloud Run","archived":false,"fork":false,"pushed_at":"2019-11-18T08:58:03.000Z","size":3873,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-31T22:42:54.414Z","etag":null,"topics":["ab-testing","cloudr","gcp","golang","knative","metrics","revision","stackdriver","traffic"],"latest_commit_sha":null,"homepage":"","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchmarny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-24T02:18:00.000Z","updated_at":"2019-11-18T08:58:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"35363155-975f-4fef-bc78-39e55d10e39d","html_url":"https://github.com/mchmarny/ab-test-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fab-test-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fab-test-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fab-test-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fab-test-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchmarny","download_url":"https://codeload.github.com/mchmarny/ab-test-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239942598,"owners_count":19722328,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","cloudr","gcp","golang","knative","metrics","revision","stackdriver","traffic"],"created_at":"2024-11-08T12:28:09.778Z","updated_at":"2026-03-23T21:30:18.560Z","avatar_url":"https://github.com/mchmarny.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ab-test-demo\n\nIn simplest terms of application development, `A/B Testing` implies an ability to compare two versions of the same application in a controlled environment to measure some behavior (e.g. use response) or properties (e.g. performance) of each version.\n\nWhile proper A/B testing will require more specialized platform and tooling, in this example I will overview a very simple experiment on Cloud Run.\n\n## Scenario\n\nImagine you published a corporate site aiming to collect users' sign ups for some marketing campaign. After a couple of days, the marketing department tells you that they expected more sign ups. And, that they think that the low number of user registrations is related to the location of the sign up button (very bottom of the page). Naturally, they recommend you move the button to the top of the page to get more sign ups.\n\n![](img/signup.png)\n\nNaturally, you as a data-driven developer, want to put together an experiment wherein subset of the applications users will be seeing the new application layout and the rest will continue seeing the current version. The number of sign ups form each version will indicate optimal location for the sign up button.\n\n\u003e I know, this is simplistic scenario but for the purposes of this sample this is all we need.\n\n## Pre-requirements\n\nIf you don't have one already, start by creating new project and configuring [Google Cloud SDK](https://cloud.google.com/sdk/docs/). To get access to the `traffic management` feature of Cloud Run you will also need to configure the `alpha` version of `gcloud`:\n\n```shell\ngcloud components install alpha\n```\n\n## Setup\n\n\u003e Throughout this sample I'm going to be using pre-build image of a demo application. This image is publicly accessible (`gcr.io/cloudylabs-public/ab-test-demo` with `a` and `b` versions). If you prefer, you can build your own images from this repository using the [bin/build-images](bin/build-images) script. As with any script, review it before executing it.\n\n## Version \"A\"\n\nTo mimic the above scenario, I'm first going to deploy the original version of the application (\"A\").\n\n```shell\ngcloud beta run deploy signup \\\n\t--image gcr.io/cloudylabs-public/ab-test-demo:a\n```\n\n\u003e Note, depending on how you configured your `gcloud`, you may also have to append the `--cluster` and `--cluster-location` flags or run. More information on how to configure these flags by default see [Quickstart: Deploy to Cloud Run on GKE](https://cloud.google.com/run/docs/quickstarts/prebuilt-deploy-gke)\n\nThe `gcloud` command will return a URL to the active revision which is serving traffic. This should look something like this:\n\n```shell\nDeploying container to Cloud Run on GKE service [signup] in namespace [demo] of cluster [cr-demo]\n✓ Deploying... Done.\n  ✓ Creating Revision...\n  ✓ Routing traffic...\nDone.\nService [signup] revision [signup-2zpw4] has been deployed and is serving traffic at http://signup.demo.cloudylabs.dev\n```\n\nIn the above example, my `gcloud` configuration was already configured with defaults for name of the cluster (`cr-demo`), its location (`us-east1` region), and the target namespace (`demo`). The same deployment with all these flags in line would look like this:\n\n```shell\ngcloud beta run deploy signup \\\n    --image gcr.io/cloudylabs-public/ab-test-demo:a \\\n    --cluster cr-demo \\\n    --cluster-location us-east1 \\\n    --namespace demo\n```\n\nTo see how the deployed version of the application should look like see https://signup-a.demo.cloudylabs.dev\n\n## Version \"B\"\n\nCloud Run does a lot of things automatically (in this case revision management). So, to implement my experiment I will first take my application out of the \"auto-pilot\" mode so I can manage the revisions manually. To do that, I'll first list the revisions:\n\n```shell\ngcloud alpha run revisions list --service signup\n```\n\nThere should be only one revision at this point, but, in case you have run the deployment a few times just copy the revision ID from the top most row\n\n```shell\nFor cluster [cr-prod] in [us-east1]:\n   REVISION      ACTIVE  SERVICE  DEPLOYED\n✔  signup-9p644  yes     signup   2019-08-26 22:44:10 UTC\n✔  signup-2zpw4          signup   2019-08-26 22:12:13 UTC\n✔  signup-m89nj          signup   2019-08-26 22:10:57 UTC\n```\n\nOnce you have captured the revision ID, you can tell Cloud Run to that you want to manage the revisions manually by setting traffic 100% explicitly to that revision.\n\n\u003e Make sure to replace the revision ID in the following command before running it\n\n```shell\ngcloud alpha run services set-traffic signup \\\n    --to-revision signup-9p644=100\n```\n\nThe result should look something like this:\n\n```shell\nDone.\nTraffic set to signup-9p644=100.\n```\n\nNow we can deploy new revisions of this application to Cloud Run and they will NOT take any traffic. 100% of the traffic will contuse to be routed to the revision you set above. Now I'm ready to deploy the \"B\" version.\n\n```shell\ngcloud beta run deploy signup \\\n\t--image gcr.io/cloudylabs-public/ab-test-demo:b\n```\n\n\u003e The confirmation message will make it sound like the traffic actually is sent to the new version right now but that's just a CLI bug and should be fixed soon\n\nNow if you list the service revisions you should see new one\n\n```shell\ngcloud alpha run revisions list --service signup\n```\n\nOur version \"B\" will be the top most on that returned list and the version \"A\" will be the third one from the top.\n\n\u003e There is a little \"side-effect\" revision created right now. THis will be removed in subsequent releases.\n\n```shell\n   REVISION      ACTIVE  SERVICE  DEPLOYED\n✔  signup-xf95v          signup   2019-08-26 23:00:55 UTC\n✔  signup-7cbsz          signup   2019-08-26 22:58:28 UTC\n✔  signup-9p644          signup   2019-08-26 22:44:10 UTC\n✔  signup-2zpw4          signup   2019-08-26 22:12:13 UTC\n```\n\nNow we are ready to execute our experiment. We will send 90% of the traffic to the original version (\"A\") and 10% to the update (\"B\") version.\n\n```shell\ngcloud alpha run services set-traffic signup \\\n    --to-revision signup-9p644=90,signup-xf95v=10\n```\n\nThe response should be\n\n```shell\nDone.\nTraffic set to signup-9p644=90, signup-xf95v=10.\n```\n\nTo compare how the version \"A\" and \"B\" should look like take a look at these already deployed versions:\n\n* http://signup-a.demo.cloudylabs.dev\n* http://signup-b.demo.cloudylabs.dev\n\n\n## Monitoring\n\nTo view the revision diagram, go to Stackdriver and select \"Metric Explorer\"\n\n![](img/metricexp.png)\n\nIn Metric Explorer type \"Revision Count\" and select the \"knative.dev/serving/revision/request_count\". This will show use the number of requests reaching the revision in Cloud Run on GKE\n\n![](img/metric.png)\n\nNow filter on \"Service Name\" and select \"signup\" and group by \"revision name\" and use \"count\" aggregation. Assuming there is actual traffic to your service, you should see now time series chart for each one of your revisions.\n\n![](img/custom-metrics.png)\n\nIn addition to the built-in Knative metrics that are already available on Revision-level in Stackdriver, this example is also instrumented with custom metrics tracking the visits and the number of user sessions which resulted in sign up.\n\n## Disclaimer\n\nThis is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fab-test-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchmarny%2Fab-test-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fab-test-demo/lists"}