https://github.com/epiforecasts/mme-review

Systematic review of multi-model ensemble evaluations in infectious disease
https://github.com/epiforecasts/mme-review

Last synced: 4 months ago
JSON representation

Systematic review of multi-model ensemble evaluations in infectious disease

• Host: GitHub
• URL: https://github.com/epiforecasts/mme-review
• Owner: epiforecasts
• License: mit
• Created: 2025-04-03T15:40:56.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2025-09-23T16:11:59.000Z (9 months ago)
• Last Synced: 2025-10-15T14:20:27.537Z (8 months ago)
• Language: TeX
• Size: 4.65 MB
• Stars: 4
• Watchers: 2
• Forks: 0
• Open Issues: 13
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Multi-model ensembles for infectious disease forecasting: A systematic review

- [Full protocol registration](https://osf.io/84zhn)

## Background

### Rationale

Infectious disease modelling is a useful tool for supporting outbreak control, offering to interpret the complex uncertainty of epidemiological dynamics. Modellers handle this uncertainty with a variety of approaches, choices, and interpretations during modelling work. Working in collaboration offers comparability across this diversity of modelling work. Modelling collaborations may aim to enable expert elicitation among modellers, clarify the extent and policy relevance of uncertainty, or provide a synthesis of modelling evidence (1–4). Specifically, collaboration among multiple independent and diverse modelling teams may create a stronger basis for evidence-informed policy support (5,6).

Outputs from such modelling collaborations often include a quantitative combination of numerical model results into an ensemble projection. A key benefit of model combination is the increased predictive accuracy of the combined result (7,8). Such findings underlie ensemble approaches in, for example, economics, logistics, or meteorology (9). This better performance partly comes from the independent information added by each projection, while reducing the variance in uncertainty across projections. Meanwhile, the unreliability of individual model performance may mean that there is no obvious best way to combine projections beyond a simple linear average (the “forecast combination puzzle” (10)). These results appear to hold in infectious disease settings, even with an increasing number and range of approaches and methods for model combination (e.g. (11–13)). 

This review aims to summarise existing evidence on the predictive accuracy of multi-model ensemble projections of infectious disease (forecasts). As a secondary aim, this review will capture some of the benefits and challenges involved in such collaborations among modellers. This will draw together an increasing literature analysing multi-model collaborations, and support their future design, communication, and evaluation.

### Objectives

To assess the accuracy and value of multi-model ensembles for forecasting infectious disease outbreaks.

RQ1: What is the predictive performance of multi-model ensemble projections from independent models in comparison to individual component models when forecasting infectious disease?

RQ2: What are the benefits and challenges of such multi-model ensembles?

---

### References

1.	Green LE, Medley GF. Mathematical modelling of the foot and mouth disease epidemic of 2001: strengths and weaknesses. Res Vet Sci. 2002 Dec;73(3):201–5.

2.	Hollingsworth TD, Medley GF. Learning from multi-model comparisons: Collaboration leads to insights, but limitations remain. Epidemics. 2017 Mar 1;18:1–3.

3.	Reich NG, Lessler J, Funk S, Viboud C, Vespignani A, Tibshirani RJ, et al. Collaborative Hubs: Making the Most of Predictive Epidemic Modeling. Am J Public Health. 2022 Jun;112(6):839–42.

4.	Teerawattananon Y, Kc S, Chi YL, Dabak S, Kazibwe J, Clapham H, et al. Recalibrating the notion of modelling for policymaking during pandemics. Epidemics. 2022 Mar;38:100552.

5.	Shea K, Runge MC, Pannell D, Probert WJM, Li SL, Tildesley M, et al. Harnessing multiple models for outbreak management. Science. 2020 May 8;368(6491):577–9.

6.	Medley GF. A consensus of evidence: The role of SPI-M-O in the UK COVID-19 response. Adv Biol Regul. 2022 Dec;86:100918.

7.	Bates JM, Granger CWJ. The Combination of Forecasts. OR. 1969;20(4):451–68.

8.	Makridakis S, Hyndman RJ, Petropoulos F. Forecasting in social settings: The state of the art. Int J Forecast. 2020 Jan 1;36(1):15–28.

9.	Chen L. A review of the applications of ensemble forecasting in fields other than meteorology. Weather. 2024;79(9):285–90.

10.	Claeskens G, Magnus JR, Vasnev AL, Wang W. The forecast combination puzzle: A simple theoretical explanation. Int J Forecast. 2016 Jul 1;32(3):754–62.

11.	Del Valle SY, McMahon BH, Asher J, Hatchett R, Lega JC, Brown HE, et al. Summary results of the 2014-2015 DARPA Chikungunya challenge. BMC Infect Dis. 2018 May 30;18(1):245.

12.	Ray E. Challenges in training ensembles to forecast COVID-19 cases and deaths in the United States [Internet]. International Institute of Forecasters. 2021 [cited 2021 Aug 5]. Available from: https://forecasters.org/blog/2021/04/09/challenges-in-training-ensembles-to-forecast-covid-19-cases-and-deaths-in-the-united-states/

13.	Fox SJ, Kim M, Meyers LA, Reich NG, Ray EL. Optimizing Disease Outbreak Forecast Ensembles. Emerg Infect Dis. 2024 Sep;30(9):1967–9.