On the Need to Automate Support for Quality Assessment Studies of Space Weather Models

Model quality assessment (QA), and where possible model validation, are critical steps in the life cycle of models destined to play a role in Space Weather research and forecasting. A major goal of QA is to characterize the state of the art of models of a given class, for example models of the global corona and solar wind. This requires consistent assessment of all major models in the class. Commonly, QA has been done in what amounts to campaign mode, generating reports that capture the quality of a particular model or a limited set of models for specific events or time intervals. Inconsistencies between study designs, the limited scope of each study, and the intermittency with which the study reports appear in the literature mean that we never achieve a complete assessment of the state of the art. In addition, the timeframe in which new models appear or existing models are upgraded is comparable to the publication time of peer reviewed validation reports, which means that these reports are often out of date soon after they are published. The community's current QA strategy is inadequate and unsustainable. In this paper we show why it is unsustainable and advocate for the development of automated protocols which can, with minimal ongoing labor cost, support the community's efforts to maintain up to date results. We illustrate the concept with results from a pilot scheme developed in conjunction with the SHINE community.