Rethinking the assessment of photochemical modelin systems in air quality planning applications

The U.S. Environmental Protection Agency provides guidelines for demonstrating that future 8-hr ozone (O3) design values will be at or below the National Ambient Air Quality Standards on the basis of the application of photochemical modeling systems to simulate the effect of emission reductions. These guidelines also require assessment of the model simulation against observations. In this study, we examined the link between the simulated relative responses to emission reductions and model performance as measured by operational evaluation metrics, a part of the model evaluation required by the guidance, which often is the cornerstone of model evaluation in many practical applications. To this end, summertime O3 concentrations were simulated with two modeling systems for both 2002 and 2009 emission conditions. One of these two modeling systems was applied with two different parameterizations for vertical mixing. Comparison of the simulated base-case 8-hr daily maximum O3 concentrations showed marked model-to-model differences of up to 20 ppb, resulting in significant differences in operational model performance measures. In contrast, only relatively minor differences were detected in the relative response of O3 concentrations to emission reductions, resulting in differences of a few ppb or less in estimated future year design values. These findings imply that operational model evaluation metrics provide little insight into the reliability of the actual model application in the regulatory setting (i.e., the estimation of relative changes). In agreement with the guidance, it is argued that more emphasis should be placed on the diagnostic evaluation of O3-precursor relationships and on the development and application of dynamic and retrospective evaluation approaches in which the response of the model to changes in meteorology and emissions is compared with observed changes. As an example, simulated relative O3 changes between 1995 and 2007 are compared against observed changes. It is suggested that such retrospective studies can serve as the starting point for targeted diagnostic studies in which individual aspects of the modeling system are evaluated and refined to improve the characterization of observed changes.