https://github.com/ChaWM/pwrAB

Power Calculations for AB Testing
https://github.com/ChaWM/pwrAB

Last synced: 3 months ago
JSON representation

Power Calculations for AB Testing

• Host: GitHub
• URL: https://github.com/ChaWM/pwrAB
• Owner: ChaWM
• Created: 2017-04-18T06:09:56.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-06-12T00:48:33.000Z (over 7 years ago)
• Last Synced: 2024-05-21T02:54:29.163Z (6 months ago)
• Language: R
• Size: 42 KB
• Stars: 5
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# pwrAB

**pwrAB** is a simple package to perform power analysis for AB testing. There are a number of R packages for power calculations, such as the widely used **pwr** package, but to my knowledge none of the t-test power functions allow the variance and sample size to simultaneously vary across the two groups. Instead, they are based on the pooled t-test, and will provide biased results when the variance and sample size are not equal across groups. In commercial AB testing, a treatment effect will frequently also shift the variance in the treatment group, making the pooled t-test a poor choice for analysis unless the two groups have equal sample sizes (50/50 split).

The two functions in the **pwrAB** package use the Welch's unequal variance t-test instead, and will provide valid results even when sample sizes are unequal. [Delacre et al. (2017)](http://www.rips-irsp.com/articles/10.5334/irsp.82/), or a [blog post](http://daniellakens.blogspot.com/2015/01/always-use-welchs-t-test-instead-of.html) by one of the authors of the paper, expands on this topic in more detail for those that are interested. The function **AB_t2n_prop** can be used when the response variable is binary, while the **AB_t2n** can be used in all other situations. An implicit assumption underlying the package is that the sample sizes are large enough for the central limit theorem to be relevant (and thus making the t-test an appropriate choice regardless of normality assumptions). In cases where the sample sizes are not sufficiently large relative to the departure to normality, using another package/function for sample sizing is strongly encouraged.

The package is [available on CRAN](https://CRAN.R-project.org/package=pwrAB), and can be easily installed by typing:

```

install.packages('pwrAB')

```

Much of the code/syntax of this package is borrowed from the [**pwr** package by Champely et al.](https://cran.r-project.org/web/packages/pwr/index.html). Many thanks to the authors for making their code publicly available.