Out-of-sample validation for structured expert judgment of Asian carp establishment in Lake Erie

Structured expert judgment (SEJ) is used to quantify the uncertainty of nonindigenous fish (bighead carp [Hypophthalmichthys nobilis] and silver carp [H. molitrix]) establishment in Lake Erie. The classical model for structured expert judgment model is applied. Forming a weighted combination (called a decision maker) of experts' distributions, with weights derived from performance on a set of calibration variables from the experts' field, exhibits greater statistical accuracy and greater informativeness than simple averaging with equal weights. New methods of cross validation are applied and suggest that performance characteristics relative to equal weighting could be predicted with a small number (1-2) of calibration variables. The performance-based decision maker is somewhat degraded on out-of-sample prediction, but remained superior to the equal weight decision maker in terms of statistical accuracy and informativeness.