Risk Assessment in Peacekeeping: Are Visual Sparse Models Transparent?

Decision and risk analysis tools must be accurate and transparent. Classification trees, especially sparse ones, and other visual models, such as scorecards or risk tables, have been claimed to strike this balance. There is, however, little empirical evidence for the transparency of such models. We derive relevant hypotheses and test them in a controlled laboratory experiment with an ecologically valid, high-risk, critical task: threat classification in peacekeeping. Three classification models are studied: a complete tree, a sparse (specifically, fast-and-frugal) tree, and a risk table. To focus on transparency, all three models make identical classifications and thus have equal accuracy. We assess and score three aspects of transparency for each model: time required to learn to a strict criterion, accuracy of application under time pressure, and accuracy in a delayed surprise memory recall test. In a between-participants design, the fast-and-frugal tree is learned more quickly, applied more accurately, and recalled more accurately than the complete tree and the risk table; all statistical effect sizes are large. The recall accuracy of the fast-and-frugal tree is, in contrast to the other two models, robust to individual differences in statistical numeracy and risk literacy. In sum, the results of the experiment, together with reflection on limitations and challenges, plus theoretical arguments, suggest that sparse trees might serve as a reasonable benchmark and starting point for designing transparent support for risk assessment.