Case Study: CovidSim

Case Study: Quantifying Uncertainty in Pandemic Predictions with EasyVVUQ

During the height of the COVID-19 pandemic, large-scale epidemiological models, such as the Imperial College London's CovidSim, became indispensable tools for policymakers. These models provided crucial projections on infection rates, peak hospital demand, and potential fatalities, directly informing national strategies. However, like all complex models, CovidSim relies on numerous input parameters – from initial infection numbers to contact rates – many of which inherently carry uncertainties. The critical question became: how do these input uncertainties translate into uncertainties in the model's vital predictions?

This groundbreaking research, led by Wouter Edeling from CWI, set out to systematically quantify this propagation of uncertainty. By robustly assessing the reliability of CovidSim's outputs, the study aimed to provide a more transparent and trustworthy basis for policy decisions, a goal perfectly aligned with the SEAVEA project's mission of providing verified, validated, and uncertainty-quantified applications.

The Challenge

Epidemiological models are intricate, involving millions of interacting "agents" (individuals) and complex disease progression rules. Running a single simulation is computationally intensive. To perform a thorough Uncertainty Quantification (UQ) and Sensitivity Analysis (SA), thousands of such simulations, each with slightly varied input parameters, are often required. Manually setting up, executing, monitoring, and analysing these vast computational campaigns presents a formidable challenge, risking errors and consuming enormous amounts of researcher time. Without such rigorous UQ, policymakers could be making decisions based on single-point predictions that might be highly uncertain, leading to potentially suboptimal or even detrimental outcomes.

SEAVEA's Solution: Powering Uncertainty Quantification with EasyVVUQ

To overcome these computational hurdles, the research team employed EasyVVUQ, a sophisticated Python package developed as a core component of the SEAVEA toolkit (and its predecessor, the VECMA project). EasyVVUQ is specifically designed to streamline and automate the process of performing UQ and SA for complex scientific simulations.

For the CovidSim study, EasyVVUQ was instrumental in:

• Automated Sampling: Efficiently generating the thousands of varied input parameter sets required for comprehensive Sobol's sensitivity analysis.

• Seamless Integration: Acting as a robust interface between the statistical sampling methods and the complex CovidSim epidemiological code, automating the execution of each simulation run.

• Scalable Workflow Management: Facilitating the submission and management of the vast number of simulation jobs on High-Performance Computing (HPC) systems, effectively harnessing significant computational power.

• Robust Data Analysis: Processing the extensive simulation outputs to perform a rigorous sensitivity analysis, identifying which input parameters contributed most significantly to the uncertainty in CovidSim's predictions.

EasyVVUQ's capabilities enabled the team to conduct an unprecedented, comprehensive UQ study on a model of CovidSim's scale and complexity, delivering crucial insights into the reliability of its forecasts.

Key Findings

The study's findings underscored the critical importance of UQ in real-world modelling:

• Large Predictive Uncertainty: Even small uncertainties in a few input parameters led to very large uncertainties (up to an order of magnitude) in critical predictions like the total number of deaths and peak ICU bed demand. This highlighted that a single-value prediction from such models can be misleading.

• Identified Key Drivers of Uncertainty: Sensitivity analysis revealed that a small subset of parameters disproportionately drove the uncertainty in outputs. These often included initial conditions (e.g., initial number of infected individuals in the population) and early contact patterns.

• Dynamic Parameter Importance: The relative importance of different parameters was not fixed but evolved over the course of the simulated epidemic, changing as the disease progressed and interventions were applied.

Wider Impact

This research, powered by EasyVVUQ from the SEAVEA toolkit, has had a profound impact:

• Informing Policy with Transparency: By explicitly quantifying the uncertainty in CovidSim's predictions, the study provided policymakers with a more honest and complete picture of the potential range of outcomes. This fostered more nuanced decision-making, acknowledging inherent limitations.

• Guiding Data Collection Efforts: Identifying the most influential parameters allowed public health bodies to focus their efforts on obtaining more accurate data for those specific inputs, where improvements in precision would have the greatest impact on reducing predictive uncertainty.

• Showcasing VVUQ's Value: The paper serves as a powerful demonstration of the absolute necessity of Verification, Validation, and Uncertainty Quantification (VVUQ) for high-stakes computational models. It validates the core mission of the SEAVEA project in making these advanced methodologies accessible and routinely applicable.

• Advancing Computational Science: The successful application of EasyVVUQ to a model of CovidSim's scale pushes the boundaries of computational epidemiology, setting a benchmark for future UQ studies in complex scientific domains.

The Edeling et al. paper, enabled by EasyVVUQ, stands as a testament to how the SEAVEA project is delivering the essential tools for ensuring that cutting-edge computational science provides not just answers, but answers we can truly trust.

Paper reference: https://www.nature.com/articles/s43588-021-00028-9 

Review of this work by WHO experts (in the same issue): https://www.nature.com/articles/s43588-021-00031-0