{"id":25452291,"url":"https://github.com/aicorsair/python-case-study-ab-testing-for-lunartech-homepage-cta-button","last_synced_at":"2026-05-07T03:31:48.586Z","repository":{"id":275905553,"uuid":"927565253","full_name":"AiCorsair/Python-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button","owner":"AiCorsair","description":"This repository contains a detailed case study on an A/B test of LunarTech's homepage CTA button, using proxy data structured similarly to the company's real data.","archived":false,"fork":false,"pushed_at":"2025-02-05T09:06:06.000Z","size":161,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-11T03:03:41.585Z","etag":null,"topics":["ab-testing","click-through-rate","confidence-intervals","data-analysis","data-analytics","data-exploration","data-science","data-visualization","hypothesis-testing","matplotlib","normal-distribution","numpy","pandas","practical-significance","python","statistical-analysis","statistical-significance","z-critical","z-statistic","z-test"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AiCorsair.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-05T06:59:08.000Z","updated_at":"2025-02-05T09:16:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"6d29b30d-b2f1-43a7-9521-872805123ca0","html_url":"https://github.com/AiCorsair/Python-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button","commit_stats":null,"previous_names":["aicorsair/python-case-study-ab-testing-for-lunartech-homepage-cta-button"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AiCorsair/Python-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AiCorsair%2FPython-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AiCorsair%2FPython-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AiCorsair%2FPython-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AiCorsair%2FPython-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AiCorsair","download_url":"https://codeload.github.com/AiCorsair/Python-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AiCorsair%2FPython-Case-Study-AB-Testing-for-LunarTech-Homepage-CTA-Button/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006041,"owners_count":26084024,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","click-through-rate","confidence-intervals","data-analysis","data-analytics","data-exploration","data-science","data-visualization","hypothesis-testing","matplotlib","normal-distribution","numpy","pandas","practical-significance","python","statistical-analysis","statistical-significance","z-critical","z-statistic","z-test"],"created_at":"2025-02-17T23:05:16.604Z","updated_at":"2025-10-11T03:05:03.053Z","avatar_url":"https://github.com/AiCorsair.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Python Case Study: A/B Testing for LunarTech Homepage's CTA Button\n\n\n## Table of Contents\n- [I. Introduction](#I.-Introduction)\n- [II. Hypotheses \u0026 Dataset Overview](#II.-Hypotheses-\u0026-Dataset-Overview)\n- [III. Key Steps](#III.-Key-Steps)\n- [IV. Key Visualizations](#IV.-Key-Visualizations)\n- [V. Key Explanations](#V.-Key-Explanations)\n- [VI. Conclusion](#VI.-Conclusion)\n\n\n## I. Introduction\n\nIn this project, we conducted an A/B test for LunarTech using proxy data structured similarly to the company's real data. LunarTech is a platform offering courses, bootcamps, and career support to help students land their ideal data role.\n\nA/B testing is a widely used statistical method for comparing two versions of a variable. In this case, we aimed to identify which version of LunarTech homepage’s CTA button performs better based on the click-through rate (CTR) metric:\n\n- **Control Group:** Exposed to the current CTA button (\"Secure Free Trial\").\n\n- **Experimental Group:** Exposed to the new, generalized CTA button (\"Enroll Now\").\n\nClick-Through Rate (CTR) measures the percentage of users who click a button or link after viewing it. For LunarTech, CTR is important because it reflects user engagement with the platform and helps assess the effectiveness of call-to-action buttons in driving sign-ups and conversions.\n\nFinally, the results of this test will help decide whether to implement the new button on LunarTech’s homepage.\n\n\n## II. Hypotheses \u0026 Dataset Overview\n\nHere are the statistical hypotheses we made:\n\n- **Null Hypothesis (H₀):** There exists no statistically significant difference in CTR between the experimental (\"Enroll Now\") and control (\"Secure Free Trial\") buttons on the homepage.\n\n- **Alternative Hypothesis (H₁):** There exists a statistically significant difference in CTR between the experimental and control buttons on the homepage.\n\nWe used a sufficiently large, random sample to ensure the results represent the entire user population, enabling confident business decisions. Below are the columns in the dataset:\n\n- **user_id:** Unique identifier for users (`1` to `20,000`).\n\n- **click:** `1` if the user clicked the CTA button, `0` if not.\n\n- **group:** Either \"con\" (control) or \"exp\" (experimental), with an even split.\n\n- **timestamp:** Click date and time for the \"exp\" group (Jan `1-7`, `2024`) at minute-level precision.\n\n\n## III. Key Steps\n\n- We imported the necessary libraries and loaded the dataset from a CSV file.\n\n- We explored the data using summary statistics, plotted total clicks and non-clicks for each group, and annotated bars with click and non-click percentages.\n\n- We set the significance level at `α = 0.05` to control Type I errors (false positives) and the minimum detectable effect at `δ = 0.1`, as the business required at least a `10%` CTR increase to justify implementation.\n\n- We calculated total users and clicks per group, estimated click probabilities, pooled click probability, and pooled click variance.\n\n- We determined the standard error, Z-statistic, and Z-critical value, then assessed statistical significance using the p-value and a standard normal distribution plot.\n\n- Finally, we checked practical significance using a `95%` confidence interval.\n\n\n## IV. Key Visualizations\n\n![Total Clicks](https://github.com/user-attachments/assets/bda8fc17-008b-4684-a16d-c1d36bbdef7d)\n\nAs shown, `61.16%` of users clicked in the experimental group, compared to only `19.89%` in the control group. Hypothesis testing confirmed that this difference is statistically significant and not due to chance.\n\n![Statistical Significance](https://github.com/user-attachments/assets/afef3345-1460-4eac-ac7e-5fc637fc0cf2)\n\nThe graph shows the standard normal distribution with a mean of `0` and a standard deviation of `1`. The rejection regions are located before and after the Z-critical values of `-1.96` and `1.96`, respectively. Since the Z-statistic is `-59.44`, well beyond the critical value of `-1.96`, we rejected the null hypothesis.\n\n\n## V. Key Explanations\n\n- A two-sample Z-test was appropriate for comparing the click-through rates (CTRs) between the control and experimental groups. The large sample sizes (`10,000` per group) ensure the sampling distribution of the sample proportion approximates a normal distribution, regardless of the population distribution's shape. This justifies the use of the Z-statistic in hypothesis testing.\n\n- We found a very low p-value close to `0`, indicating strong evidence against the null hypothesis. At all common significance levels, we rejected the null hypothesis. In contrast, a high p-value (`0.05` or more) indicates weak evidence against the null hypothesis.\n\n- The `95%` CI of `0.399` to `0.426` gives a range of values within which the true difference between the experimental and control group click-through rates (CTRs) is likely to lie with `95%` confidence. A narrower interval indicates higher precision.\n\n\n## VI. Conclusion\n\n- We found a statistically significant difference in CTR between the experimental (\"Enroll Now\") and control (\"Secure Free Trial\") buttons at the `5%` significance level, meaning the observed difference is unlikely due to chance.\n\n- We also found a practically significant difference in CTR between the experimental and control versions at the `10%` minimum detectable effect (MDE).\n\n- Since the click probability estimate in the experimental group is higher than in the control group, we conclude that the experimental button resulted in a statistically significantly higher CTR.\n\n- From a business perspective, this statistically significant difference is large enough to justify changing the button for all users, expecting an increase in user engagement.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faicorsair%2Fpython-case-study-ab-testing-for-lunartech-homepage-cta-button","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faicorsair%2Fpython-case-study-ab-testing-for-lunartech-homepage-cta-button","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faicorsair%2Fpython-case-study-ab-testing-for-lunartech-homepage-cta-button/lists"}