Implications of excessive precision for risk comparisons: lessons from the past four decades

This article reviews five published "second-order" risk comparisons from the past four decades that implied precise understanding, and hence clear relationships or orderings, of the underlying risks. "Second order" here refers to efforts that extract information from original sources with the goal of relating diverse findings. All five of these publications have frequently been cited in the peer-reviewed literature and/or in risk regulatory debate in the United States. Each is associated with at least one contemporaneous critique that the findings were excessively precise. None of these critiques suggested that an alternative relationship or ordering of the risks evaluated was more appropriate. Instead, each critique concluded that alternative, contradictory relationships were at least as plausible given data and/or analytical limitations. In one case, the critique led to the withdrawal of the original publication. The original findings have been propagated or used uncritically in subsequent literature, including political support for cost-effectiveness analysis. In other cases, the critiques have been used to discredit quantitative risk analysis in general, especially in the cases of nuclear power and cost-benefit analysis. Both of these outcomes are undesirable. Future risk comparisons should avoid excessive precision, include explicit discussion of uncertainty, and differentiate between plausible estimates and expected values.