https://github.com/fbartos/publicationbiasbenchmark
https://github.com/fbartos/publicationbiasbenchmark
benchmark meta-analysis publication-bias simulation
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/fbartos/publicationbiasbenchmark
- Owner: FBartos
- Created: 2025-07-22T15:39:52.000Z (6 months ago)
- Default Branch: master
- Last Pushed: 2025-10-23T08:54:53.000Z (3 months ago)
- Last Synced: 2025-10-23T10:31:55.491Z (3 months ago)
- Topics: benchmark, meta-analysis, publication-bias, simulation
- Language: R
- Homepage: https://fbartos.github.io/PublicationBiasBenchmark/
- Size: 488 MB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.Rmd
- Changelog: NEWS.md
Awesome Lists containing this project
README
---
title: "README"
bibliography: inst/REFERENCES.bib
csl: inst/apa.csl
output: github_document
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# PublicationBiasBenchmark
**PublicationBiasBenchmark** is an R package for benchmarking publication bias correction methods through simulation studies. It provides:
- Predefined data-generating mechanisms from the literature
- Functions for running meta-analytic methods on simulated data
- Pre-simulated datasets and pre-computed results for reproducible benchmarks
- Tools for visualizing and comparing method performance
All datasets and results are hosted on OSF:
For the methodology of living synthetic benchmarks please cite:
> Bartoš, F., Pawel, S., & Siepe, B. S. (2025). Living synthetic benchmarks: A neutral and cumulative framework for simulation studies. _arXiv Preprint_.
For the publication bias benchmark R package please cite:
> Bartoš, F., Pawel, S., & Siepe, B. S. (2025). PublicationBiasBenchmark: Benchmark for publication bias correction methods (version 0.1.0).
Overviews of the benchmark results are available as articles on the package website:
- [Overall Results](https://fbartos.github.io/PublicationBiasBenchmark/articles/Results.html)
- [Stanley (2017)](https://fbartos.github.io/PublicationBiasBenchmark/articles/Results_Stanley2017.html)
- [Alinaghi (2018)](https://fbartos.github.io/PublicationBiasBenchmark/articles/Results_Alinaghi2018.html)
- [Bom (2019)](https://fbartos.github.io/PublicationBiasBenchmark/articles/Results_Bom2019.html)
- [Carter (2019)](https://fbartos.github.io/PublicationBiasBenchmark/articles/Results_Carter2019.html)
Contributor guidelines for extending the package with data-generating mechanisms, methods, and results are available at:
- [How to add a new data-generating mechanism](https://fbartos.github.io/PublicationBiasBenchmark/articles/Adding_New_DGMs.html)
- [How to add a new method](https://fbartos.github.io/PublicationBiasBenchmark/articles/Adding_New_Methods.html)
- [How to compute method results](https://fbartos.github.io/PublicationBiasBenchmark/articles/Computing_Method_Results.html)
- [How to compute method measures](https://fbartos.github.io/PublicationBiasBenchmark/articles/Computing_Method_Measures.html)
Illustrations of how to use the precomputed datasets, results, and measures are available at:
- [How to use presimulated datasets](https://fbartos.github.io/PublicationBiasBenchmark/articles/Using_Presimulated_Datasets.html)
- [How to use precomputed results](https://fbartos.github.io/PublicationBiasBenchmark/articles/Using_Precomputed_Results.html)
- [How to use precomputed measures](https://fbartos.github.io/PublicationBiasBenchmark/articles/Using_Precomputed_Measures.html)
The rest of this file overviews the main features of the package.
## Installation
```{r, eval = FALSE}
# Install from GitHub
remotes::install_github("FBartos/PublicationBiasBenchmark")
```
## Usage
```{r message=FALSE, warning=FALSE}
library(PublicationBiasBenchmark)
```
### Simulating From Existing Data-Generating Mechanisms
```r
# Obtain a data.frame with pre-defined conditions
dgm_conditions("Stanley2017")
# simulate the data from the second condition
df <- simulate_dgm("Stanley2017", 2)
# fit a method
run_method("RMA", df)
```
### Using Pre-Simulated Datasets
```r
# download the pre-simulated datasets
# the default settings downloads the datasets to the `resources` directory, use
# PublicationBiasBenchmark.options(simulation_directory = "/path/")
# to change the settings
download_dgm_datasets("no_bias")
# retrieve first repetition of first condition from the downloaded datasets
retrieve_dgm_dataset("no_bias", condition_id = 1, repetition_id = 1)
```
### Using Pre-Computed Results
```r
# download the pre-computed results
download_dgm_results("no_bias")
# retrieve results the first repetition of first condition of RMA from the downloaded results
retrieve_dgm_results("no_bias", method = "RMA", condition_id = 1, repetition_id = 1)
# retrieve all results across all conditions and repetitions
retrieve_dgm_results("no_bias")
```
### Using Pre-Computed Measures
```r
# download the pre-computed measures
download_dgm_measures("no_bias")
# retrieve measures of bias the first condition of RMA from the downloaded results
retrieve_dgm_measures("no_bias", measure = "bias", method = "RMA", condition_id = 1)
# retrieve all measures across all conditions and measures
retrieve_dgm_measures("no_bias")
```
### Simulating From an Existing DGM With Custom Settings
```r
# define sim setting
sim_settings <- list(
n_studies = 100,
mean_effect = 0.3,
heterogeneity = 0.1
)
# check whether it is feasible
# (defined outside of the function - not to decrease performance during simulation)
validate_dgm_setting("no_bias", sim_settings)
# simulate the data
df <- simulate_dgm("no_bias", sim_settings)
# fit a method
run_method("RMA", df)
```
### Key Functions
#### Data-Generating Mechanisms
- `simulate_dgm()`: Generates simulated data according to specified data-generating mechanism and settings.
- `dgm_conditions()`: Lists prespecified conditions of the data-generating mechanism.
- `validate_dgm_setting()`: Validates (custom) setting of the data-generating mechanism.
- `download_dgm_datasets()`: Downloads pre-simulated datasets from the OSF repository.
- `retrieve_dgm_dataset()`: Retrieves the pre-simulated dataset of a given condition and repetition from downloaded from the pre-downloaded OSF repository.
#### Method Estimation And Results
- `run_method()`: Estimates method on a supplied data according to the specified settings.
- `method_settings()`: Lists prespecified settings of the method.
- `download_dgm_results()`: Downloads pre-computed results from the OSF repository.
- `retrieve_dgm_results()`: Retrieves the pre-computed results of a given method, condition, and repetition from the pre-downloaded OSF repository.
#### Performance measures And Results
- `bias()`, `bias_mcse()`, etc.: Functions to compute performance measures and their Monte Carlo standard errors.
- `download_dgm_measures()`: Downloads pre-computed performance measures from the OSF repository.
- `retrieve_dgm_measures()`: Retrieves the pre-computed performance measures of a given method, condition, and repetition from the pre-downloaded OSF repository.
### Available Data-Generating Mechanisms
See `methods("dgm")` for the full list:
- `"no_bias"`: Generates data without publication bias (a test simulation)
- `"Stanley2017"`: @stanley2017finding
- `"Alinaghi2018"`: @alinaghi2018meta
- `"Bom2019"`: @bom2019kinked
- `"Carter2019"`: @carter2019correcting
### Available Methods
See `methods("method")` for the full list:
- `"mean"`: Mean effects size
- `"FMA"`: Fixed effects meta-analysis
- `"RMA"`: Random effects meta-analysis
- `"WLS"`: Weighted Least Squares
- `"trimfill"`: Trim-and-Fill [@duval2000trim]
- `"WAAPWLS"`: Weighted Least Squares - Weighted Average of Adequately Power Studies [@stanley2017finding]
- `"WILS"`: Weighted and Iterated Least Squares [@stanley2024harnessing]
- `"PET"`: Precision-Effect Test (PET) publication bias adjustment [@stanley2014meta]
- `"PEESE"`: Precision-Effect Estimate with Standard Errors (PEESE) publication bias adjustment [@stanley2014meta]
- `"PETPEESE"`: Precision-Effect Test and Precision-Effect Estimate with Standard Errors (PET-PEESE) publication bias adjustment [@stanley2014meta]
- `"EK"`: Endogenous Kink [@bom2019kinked]
- `"SM"`: Selection Models (3PSM, 4PSM) [@vevea1995general]
- `"pcurve"`: P-curve [@simonsohn2014pcurve]
- `"puniform"`: P-uniform [@vanassen2015meta] and P-uniform* [@vanaert2025puniform]
- `"AK"`: Andrews & Kasy selection models (AK1, AK2) [@andrews2019identification]
- `"RoBMA"`: Robust Bayesian Meta-Analysis [@bartos2023robust]
### Available Performance Measures
See `?measures` for the full list of performance measures and their Monte Carlo standard errors/
### DGM OSF Repositories
All DGMs are linked to the OSF repository () and contain the following elements:
- `data` : folder containing by-condition simulated datasets for all repetitions
- `results` : folder containing by-method results for all conditions * repetitions
- `measures` : folder containing by-measure performance for all methods * conditions
- `metadata` : folder containing the following information:
- `dgm-conditions.csv` : file mapping of all conditions and the corresponding settings
- `dgm-generation.R` : file with code for exact reproduction of the pre-simulated datasets
- `dgm-sessionInfo.txt`: file with reproducibility details for the pre-simulated datasets
- `dgm-session.log`: file with reproducibility details for the pre-simulated datasets (based on sessioninfo package)
- `results.R` : file with code for exact reproduction of the by method results (might be method / method groups specific)
- `results-sessionInfo.txt`: file with reproducibility details for the precomputed results (might be method / method groups specific)
- `pm-computation.R` : file with code for computation of performance measures
### References