Assessing model error in counterfactual worlds

Counterfactual scenario modelling exercises that ask 'what would happen if?' are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision-making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision-making, but requires estimating model error in counterfactual worlds. Here, we present and contrast the theory underlying three approaches for estimating this error, which are possible when observations can inform alternative models of scenario assumptions. We use a simulation experiment to demonstrate the benefits and limitations of each under favourable conditions. Our results illustrate the conditions under which counterfactual errors can be estimated in order to evaluate scenario projections. We further outline how scenarios can be designed to maximize evaluability. This work can serve as the basis for further explorations into scenario evaluation under a variety of conditions using simulation studies and real-world application.