https://github.com/pythonhealthdatascience/stars_wp1_summary
Summary of the eight computational reproducibility assessments conducted as part of STARS Work Package 1. These assessed discrete-event simulation papers with models in Python and R.
https://github.com/pythonhealthdatascience/stars_wp1_summary
Last synced: 4 months ago
JSON representation
Summary of the eight computational reproducibility assessments conducted as part of STARS Work Package 1. These assessed discrete-event simulation papers with models in Python and R.
- Host: GitHub
- URL: https://github.com/pythonhealthdatascience/stars_wp1_summary
- Owner: pythonhealthdatascience
- License: cc-by-4.0
- Created: 2024-08-14T10:16:54.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-10-17T15:32:36.000Z (8 months ago)
- Last Synced: 2024-10-20T11:48:09.296Z (8 months ago)
- Language: TeX
- Homepage: https://pythonhealthdatascience.github.io/stars_wp1_summary/
- Size: 2.75 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
Computational Reproducibility Assessments: Summary
Summary of the eight computational reproducibility assessments conducted as part of STARS Work Package 1.
## Table of contents
* [👋 About the repository](#-about-the-repository)
* [📍 Locating tables and figures from the article](#-locating-tables-and-figures-from-the-article)
* [📖 View book locally](#-view-book-locally)
* [📝 Citation](#-citation)
* [💰 Funding](#-funding)
## 👋 About the repository
In work package 1, we assessed the computational reproducibility of eight discrete-event simulation papers with models in Python and R. The reproductions and findings are summarised at: .
[](https://www.python.org/)
[](https://www.r-project.org/)
Relevant GitHub repositories:
| Repository | Description |
| --- | --- |
| [stars-reproduction-protocol](https://github.com/pythonhealthdatascience/stars_reproduction_protocol) | Latex files for reproduction protocol |
| [stars-reproduce-allen-2020](https://github.com/pythonhealthdatascience/stars-reproduce-allen-2020) |Test run of reproducibility protocol on Allen et al. 2020 |
| [stars-reproduction-template](https://github.com/pythonhealthdatascience/stars_reproduction_template) | Template for assessment of computational reproducibility |
| [stars-reproduce-shoaib-2022](https://github.com/pythonhealthdatascience/stars-reproduce-shoaib-2022) | Reproduction study 1: Shoaib and Ramamohan 2022 (Python) |
| [stars-reproduce-huang-2019](https://github.com/pythonhealthdatascience/stars-reproduce-huang-2019) | Reproduction study 2: Huang et al. 2019 (R) |
| [stars-reproduce-lim-2020](https://github.com/pythonhealthdatascience/stars-reproduce-lim-2020) | Reproduction study 3: Lim et al. 2020 (Python) |
| [stars-reproduce-kim-2021](https://github.com/pythonhealthdatascience/stars-reproduce-kim-2021) | Reproduction study 4: Kim et al. 2021 (R) |
| [stars-reproduce-anagnostou-2022](https://github.com/pythonhealthdatascience/stars-reproduce-anagnostou-2022) | Reproduction study 5: Anagnostou et al. 2022 (Python) |
| [stars-reproduce-johnson-2021](https://github.com/pythonhealthdatascience/stars-reproduce-johnson-2021) | Reproduction study 6: Johnson et al. 2021 (R) |
| [stars-reproduce-hernandez-2015](https://github.com/pythonhealthdatascience/stars-reproduce-hernandez-2015) | Reproduction study 7: Hernandez et al. 2015 (Python + R) |
| [stars-reproduce-wood-2021](https://github.com/pythonhealthdatascience/stars-reproduce-wood-2021) | Reproduction study 8: Wood et al. 2021 (R) |
Process followed for each study:
## 📍 Locating tables and figures from the article
| Figure/Table | Method | Location |
| - | - | - |
| **Figure 1.** Five standards that scientific code should strive to achieve, and the benefits of doing so | Inkscape | `images/5rs.svg` |
| **Figure 2.** Time to complete items in scope for each reproduction, inspired by figure in Krafczyk et al. 2021 | Matplotlib | Created within `pages/reproduction.qmd`, saved as `images/article_times.png` |
| **Figure 3.** Recommendations to support reproduction, with stars to highlight five recommendations considered to have the greatest potential impact. | Inkscape | `images/reproduction_wheel.svg`|
| **Figure 4.** Recommendations to support troubleshooting and reuse | Inkscape | `images/troubleshooting_wheel.svg` |
| **Figure 5.** Of the eight healthcare DES studies evaluated, proportion that met each recommendation in the current STARS framework. | Plotly express | Created within `pages/repo_evaluation.qmd`, saved as `images/stars_criteria.png` |
| **Figure 6.** Of the eight healthcare DES studies evaluated, proportion that met each item in the current STRESS-DES criteria. | Plotly express | Created within `pages/paper_evaluation.qmd`, saved as `stress_criteria.png` |
| **Figure 7.** Of the eight healthcare DES studies evaluated, proportion that met each criteria in the general reporting checklist for DES | Plotly express | Created within `pages/paper_evaluation.qmd`, saved as `ispor_criteria.png` |
| **Table 2.** Evaluation of repositories against ACM badge criteria. | - | Created within `pages/repo_evaluation.qmd`, saved as `data/badges_table.csv` (and Table 2 is an extract from that table) |
| **Table 3.** Proportion of applicable criteria that were fully met, from evaluation of repository or article, alongside the proportion of items reproduced from each study. | - | Combination of two tables: (1) `data/applicable_stars.csv` created within `pages/repo_evaluation.qmd`, and (2) `data/applicable_report.csv` created within `pages/paper_evaluation.qmd` |
| **Table D1.** Evaluation of studies against badge criteria - grouped into three themes, as defined by NISO. | - | Created within `pages/repo_evaluation.qmd`, saved as `data/badges_table.csv` |
The remaining tables were created directly in the Latex article, rather than in this repository, as they are not describing results from reproduction:
* **Table 1.** Description of the included studies.
* **Table 4.** Simple checklists to assist reviewers in assessing the openness, longevity, and reproducibility of DES models during peer review.
* **Table B1.** Links for reproduction and evaluation.
* **Table B2.** Links to original study repositories.
## 📖 View book locally
The [website](https://pythonhealthdatascience.github.io/stars_wp1_summary/) is a quarto book hosted with GitHub pages. If you want to view the book locally on your own machine you will need to:
1. Clone GitHub repository
```
git clone https://github.com/pythonhealthdatascience/stars_wp1_summary.git
```
2. Create the virtual environment
```
virtualenv stars_wp1_summary
source stars_wp1_summary/bin/activate
pip install -r requirements.txt
```
3. Create the book
```
quarto render
```
4. Open the book in your browser (open the `_book/index.html` file).
## 📝 Citation
This repository has been archived on Zenodo and can be cited as:
> Heather, A., Monks, T., & Harper, A. (2025). Computational Reproducibility Assessments: Summary. Zenodo. .
If you wish to cite this repository on GitHub, please refer to the citation file `CITATION.cff`, and the auto-generated alternatives `citation_apalike.apa` and `citation_bibtex.bib`. Authors:
| Member | ORCID | GitHub |
| --- | --- | --- |
| Amy Heather | [](https://orcid.org/0000-0002-6596-3479) | https://github.com/amyheather |
| Thomas Monks | [](https://orcid.org/0000-0003-2631-4481) | https://github.com/TomMonks |
| Alison Harper | [](https://orcid.org/0000-0001-5274-5037) | https://github.com/AliHarp |
## 💰 Funding
This project is supported by the Medical Research Council [grant number [MR/Z503915/1](https://gtr.ukri.org/projects?ref=MR%2FZ503915%2F1)].