An evaluation of the robustness of the visual air quality "preference study" method

In 2012, the U.S. Environmental Protection Agency (EPA) considered setting a secondary National Ambient Air Quality Standard (NAAQS) for particulate matter (PM) to maintain urban visual air quality (VAQ) above the level that EPA believes results in adverse effects on public welfare. EPA is relying on a type of survey it calls a "VAQ preference study" to determine this level. Individuals are shown photographs of the same vista under a range of different visibility conditions and asked to state whether the VAQ in each photograph is "acceptable" or "unacceptable." EPA considers the effect on public welfare to be adverse at the VAQ level that at least 50% ofrespondents deem unacceptable (the "VAQ cutpoint"). Given its central role in setting a NAAQS, the scientific validity of this method is an important question. This study tests the robustness of the VAQ preference study method by replicating the survey instrument from a prior VAQ preference study, and by applying two variants in which the only change was use of a different range of VAQ levels. Tested on split samples, these three variants produced statistically significantly different VAQ cutpoint estimates. In contrast, all three variants produced comparable results for a calibration task at the start of each survey in which respondents were asked to rate the VAQ in each photograph on a scale of 1-7 (without any opinion on the acceptability of each level). The significantly diferent estimates of VAQ cutpoints across survey variants cannot be attributed to inability on the part of respondents to discern whether they were being shown the entire range of actual visibility conditions. This suggests that VAQ preference surveys do not actually estimate individuals' enduring preferences regarding VAQ, because absolute preferences would not be influenced by the particular levels of VAQ over which their preferences are elicited.

In setting its particulate matter secondary National Ambient Air Quality Standard for urban visibility, the U.S. Environmental Protection Agency is relying on a survey method that has people rate a range of urban visibility levels as "acceptable" or "unacceptable." A test of that survey method using split samples finds its results are not robust to variations in the range of visibility it shows to respondents. This implies that the visibility preference survey method needs more scientific evaluation before it can be assumed to be measuring individual preferences for visibility in a valid manner.