Validity of scoring 'dangerous answers' on a written certification examination

The American Board of Anesthesiology (ABA) investigated the possibility that selection of a disproportionately high number of incorrect "dangerous answers" by candidates who would otherwise pass the board's written certification examination might relate to dangerous clinical practice by the candidates. In that case, scoring the examination for dangerous answers would provide an alternate scoring method. Twenty-nine multiple-choice questions in the 1983 written examination were identified by a panel of judges as containing incorrect answers with "truly, not potentially" dangerous consequences. The 1,036 candidates who passed the 1983 examination selected an average of 1.6 dangerous answers, and the 1,413 who failed the examination selected an average of 3.4 (p less than .001). Ninety-two candidates who passed and who selected four or more dangerous answers were tracked through the certification process to learn whether labeling them as "smart but dangerous" was justified. Reports of their clinical competence from their residency programs did not relate dangerous behavior for any of these candidates. The 86 of these 92 candidates who subsequently took the ABA oral examination had pass rates that were identical to those of all candidates in 1983 and that were completely independent of the candidates' absolute and relative frequency of choosing dangerous answers. It is concluded that selection of dangerous answers in a multiple-choice test is the result of lack of information rather than a purposeful action and implementation of the scoring system for dangerous answers would likely be unjustified and unfairly punitive.