{"id":32204295,"url":"https://github.com/hoxo-m/deltatest","last_synced_at":"2025-10-22T04:53:29.296Z","repository":{"id":265609547,"uuid":"859229127","full_name":"hoxo-m/deltatest","owner":"hoxo-m","description":"R Package for Statistical Hypothesis Testing Using the Delta Method for Online A/B Testing","archived":false,"fork":false,"pushed_at":"2025-06-16T19:39:47.000Z","size":2336,"stargazers_count":8,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-22T04:53:28.174Z","etag":null,"topics":["ab-testing","data-science","statistics"],"latest_commit_sha":null,"homepage":"https://hoxo-m.github.io/deltatest/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hoxo-m.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-09-18T10:01:52.000Z","updated_at":"2025-08-30T06:22:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"77d763e7-0f13-4c86-8ade-63fd160b435a","html_url":"https://github.com/hoxo-m/deltatest","commit_stats":null,"previous_names":["hoxo-m/deltatest"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/hoxo-m/deltatest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoxo-m%2Fdeltatest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoxo-m%2Fdeltatest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoxo-m%2Fdeltatest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoxo-m%2Fdeltatest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hoxo-m","download_url":"https://codeload.github.com/hoxo-m/deltatest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hoxo-m%2Fdeltatest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280382997,"owners_count":26321423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-22T02:00:06.515Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","data-science","statistics"],"created_at":"2025-10-22T04:53:27.121Z","updated_at":"2025-10-22T04:53:29.288Z","avatar_url":"https://github.com/hoxo-m.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r setup, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = 400,\n  message = FALSE\n)\n```\n\n# deltatest: Statistical Hypothesis Testing Using the Delta Method for Online A/B Testing\n\n\u003c!-- badges: start --\u003e\n[![CRAN-version](https://www.r-pkg.org/badges/version/deltatest)](https://cran.r-project.org/package=deltatest)\n[![CRAN-downloads](https://cranlogs.r-pkg.org/badges/deltatest)](https://cran.r-project.org/package=deltatest)\n[![R-CMD-check](https://github.com/hoxo-m/deltatest/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/hoxo-m/deltatest/actions/workflows/R-CMD-check.yaml)\n\u003c!-- badges: end --\u003e\n\n## 1. Overview\n\nIn online A/B testing, we often face a significant practical challenge: the randomization unit differs from the analysis unit. Typically, control and treatment groups are randomly assigned at the user level, while metrics—such as click-through rate—are measured at a more granular level (e.g., per page-view). In this case, the randomization unit is user, but the analysis unit is page-view.\n\nThis discrepancy raises concerns for statistical hypothesis testing, which assumes that data points are independent and identically distributed (i.i.d.). Specifically, a single user can generate multiple page-views, and each user may have a different probability of clicking. Consequently, the data may exhibit within-user correlation, thereby violating the i.i.d. assumption.\n\nWhen the standard Z-test is applied to such correlated data, the resulting  p-values do not follow the expected uniform distribution under the null hypothesis. As a result, smaller p-values tend to occur more frequently even when there is no true difference, increasing the risk of falsely detecting a significant difference.\n\n```{r p-values-from-z-test, echo=FALSE, fig.height=3, fig.width=4, fig.alt=\"p-values from standard Z-test on correlated data\"}\nlibrary(dplyr)\nlibrary(ggplot2)\n\nfile \u003c- \"data-raw/p_values_from_standard_Z_test.rds\"\n\np_values \u003c- \n  if (file.exists(file)) {\n    readRDS(file)\n  } else {\n    source(\"data-raw/compute_p_values.R\")\n    readRDS(file)\n  }\n\ndf \u003c- data.frame(p_value = p_values) |\u003e\n  mutate(range = cut(p_value, breaks = seq(0, 1, by = 0.05))) |\u003e\n  group_by(range) |\u003e\n  summarise(p = factor(ceiling(max(p_value) * 20) / 20), n = n()) |\u003e\n  mutate(prop = n / sum(n))\n\nggplot(df, aes(p, prop)) +\n  geom_col() +\n  geom_hline(yintercept = 0.05, color = \"red\") +\n  scale_y_continuous(breaks = seq(0, 1, by = 0.05)) +\n  xlab(\"p-value\") + ylab(\"proportion\") +\n  ggtitle(\"p-values from standard Z-test on correlated data\") +\n  theme(axis.text.x = element_text(angle = 60, hjust = 1), \n        plot.title = element_text(size = 11))\n```\n\nTo address this problem, Deng et al. (2018) proposed a modified statistical hypothesis testing method. Their approach replaces the standard variance estimation formula in the Z-test with an approximate formula derived via the Delta method, which accounts for within-user correlation. To simplify the application of this method, the **deltatest** package has been developed.\n\nTo illustrate how to use this package, we prepare a data frame that includes columns for the number of clicks and page-views aggregated for each user. This data frame also contains a column indicating whether each user was assigned to the control or treatment group.\n\n```{r prepare_data}\nlibrary(dplyr)\n\nn_user \u003c- 2000\n\nset.seed(314)\ndata \u003c- deltatest::generate_dummy_data(n_user) |\u003e \n  mutate(group = if_else(group == 0, \"control\", \"treatment\")) |\u003e\n  group_by(user_id, group) |\u003e \n  summarise(clicks = sum(metric), pageviews = n(), .groups = \"drop\")\n\ndata\n```\n\nThe statistical hypothesis test using the Delta method can then be performed on this data as follows:\n\n```{r execute}\nlibrary(deltatest)\n\ndeltatest(data, clicks / pageviews, by = group)\n```\n\nThis version of the Z-test yields p-values that follow the expected uniform distribution under the null hypothesis, even when within-user correlation is present.\n\n```{r p-values-from-delta-method, echo=FALSE, fig.height=3, fig.width=4, fig.alt=\"p-values from Z-test with Delta method on correlated data\"}\np_values \u003c- readRDS(\"data-raw/p_values_from_Delta_meethod.rds\")\n\ndf \u003c- data.frame(p_value = p_values) |\u003e\n  mutate(range = cut(p_value, breaks = seq(0, 1, by = 0.05))) |\u003e\n  group_by(range) |\u003e\n  summarise(p = factor(ceiling(max(p_value) * 20) / 20), n = n()) |\u003e\n  mutate(prop = n / sum(n))\n\nggplot(df, aes(p, prop)) +\n  geom_col() +\n  geom_hline(yintercept = 0.05, color = \"red\") +\n  scale_y_continuous(breaks = seq(0, 1, by = 0.01)) +\n  xlab(\"p-value\") + ylab(\"proportion\") +\n  ggtitle(\"p-values from Z-test with Delta method on correlated data\") +\n  theme(axis.text.x = element_text(angle = 60, hjust = 1), \n        plot.title = element_text(size = 10))\n```\n\n## 2. Installation\n\nYou can install the **deltatest** package from [CRAN](https://cran.r-project.org/package=deltatest).\n\n```{r install_cran, eval=FALSE}\ninstall.packages(\"deltatest\")\n```\n\nYou can also install the development version from [GitHub](https://github.com/) with:\n\n```{r install_github, eval=FALSE}\n# install.packages(\"remotes\")\nremotes::install_github(\"hoxo-m/deltatest\")\n```\n\n## 3. Details\n\nThe **deltatest** package provides the `deltatest` function for performing statistical hypothesis tests using the Delta method as proposed by Deng et al. (2018). In this section, we explain the function's arguments and its return value.\n\n### 3.1 `data` Argument\n\nTo run `deltatest`, you need to prepare an appropriately aggregated data frame. This data frame must include columns for the numerator and denominator of your metric, aggregated for each randomization unit (typically, each user). For example:\n\n- If your metric is click-through rate per page-view, the numerator is the number of clicks, and the denominator is the number of page-views.\n- If your metric is conversion rate per session, the numerator is the number of conversions (or converted sessions), and the denominator is the number of sessions.\n\nNote that the denominator should match the analysis unit.\n\nThe **deltatest** package provides the `generate_dummy_data` function to create dummy data. It generates metric values per page-view, so you need to aggregate the data by user.\n\n```{r generate_dummy_data}\nlibrary(dplyr)\n\nn_user \u003c- 2000\n\nset.seed(314)\ndata \u003c- deltatest::generate_dummy_data(n_user) |\u003e\n  group_by(user_id, group) |\u003e\n  summarise(clicks = sum(metric), pageviews = n(), .groups = \"drop\")\n\ndata\n```\n\nThis data frame includes the `user_id` column, but this column is not required to run `deltatest`.\n\n### 3.2 `formula` and `by` Arguments\n\nThe second argument, `formula`, and the third argument, `by`, specify which columns in the data frame represent the numerator, denominator, and group. There are three input styles available for the `formula` argument.\n\n#### (1) Standard Formula\n\nThis is the common formula format, where the left-hand side represents the target variable, and the right-hand side specifies the explanatory variable. In this case, the left-hand side should be of the form `numerator / denominator`, and the right-hand side should be the group column name. When using this style, you do not need to specify the `by` argument.\n\n```{r standard_formula, eval=FALSE}\ndeltatest(data, clicks / pageviews ~ group)\n```\n\n#### (2) Lambda Formula\n\nThis is a relatively new way to express functions within a formula, where the function is written on the right-hand side of the formula. Specifically, you can write the function as `~ numerator / denominator`. In this style, you must specify the group column using the `by` argument.\n\n```{r lambda_formula, eval=FALSE}\ndeltatest(data, ~ clicks / pageviews, by = group)\n```\n\n#### (3) NSE (Non-Standard Evaluation)\n\nIn this style, you can simply write `numerator / denominator`. The input is parsed using R's non-standard evaluation (NSE) feature, and you must specify the group column using the `by` argument.\n\n```{r NSE, eval=FALSE}\ndeltatest(data, clicks / pageviews, by = group)\n```\n\n#### With Calculation (Applicable to All Styles)\n\nAll styles accept calculations. For example, if your data frame contains only columns for the positive count and negative count, you can express the metric as follows:\n\n```{r with_calculation, eval=FALSE}\ndeltatest(data, pos / (pos + neg), by = group)\n```\n\n### 3.3 Other Arguments\n\n#### `group_names`\n\nFor this argument, list the two types of elements in the group column in the order of control and treatment. By default, the function assumes that the types are specified in dictionary order for this argument and will display a message to that effect. To suppress the message, set the `quiet` argument to `TRUE`.\n\n#### `type`\n\nBy default, `deltatest` tests the difference between two groups. If you specify `type = 'relative_change'`, it tests the rate of change, i.e., $(\\mu_{t} - \\mu_{c}) / \\mu_{c}$ where $\\mu_c$ and $\\mu_t$ represent the mean values of the control group and the treatment group, respectively.\n\n### 3.4 Return Value\n\nThe return value of `deltatest` is an object of class `htest`.\n\n```{r return_value}\nresult \u003c- deltatest(data, clicks / pageviews, by = group)\nresult\n```\n\nThis object contains the estimates, the p-value, the confidence interval, and more. \n\n```{r return_value_detail}\nresult$estimate\n\nresult$p.value\n\nresult$conf.int\n```\n\nYou can also tidy the results by applying the `tidy` function from the **broom** package.\n\n```{r}\nbroom::tidy(result)\n```\n\nFor more details, refer to `help(deltatest)`.\n\n## 4. Related Work\n\n- [tidydelta: Estimation of Standard Errors using Delta Method](https://cran.r-project.org/package=tidydelta)\n\n## 5. References\n\n- Deng, A., Knoblich, U., \u0026 Lu, J. (2018). Applying the Delta Method in Metric\nAnalytics: A Practical Guide with Novel Ideas. *Proceedings of the 24th ACM\nSIGKDD International Conference on Knowledge Discovery \u0026 Data Mining.*\n[doi:10.1145/3219819.3219919](https://doi.org/10.1145/3219819.3219919)\n- Deng, A., Lu, J., \u0026 Litz, J. (2017). Trustworthy Analysis of Online A/B Tests:\nPitfalls, challenges and solutions. *Proceedings of the Tenth ACM International\nConference on Web Search and Data Mining.*\n[doi:10.1145/3018661.3018677](https://doi.org/10.1145/3018661.3018677)\n- id:sz_dr (2018). Calculating the Mean and Variance of the Ratio of Random\nVariables Using the Delta Method [in Japanese]. *If You're Human, Think More\nNow.* https://www.szdrblog.info/entry/2018/11/18/154952\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhoxo-m%2Fdeltatest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhoxo-m%2Fdeltatest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhoxo-m%2Fdeltatest/lists"}