Valuing structured professional judgment: predictive validity, decision-making, and the clinical-actuarial conflict

Structured professional judgment (SPJ) has received considerable attention as an alternative to unstructured clinical judgment and actuarial assessment, and as a means of resolving their ongoing conflict. However, predictive validity studies have typically relied on receiver operating characteristic (ROC) analysis, the same technique commonly used to validate actuarial assessment tools. This paper presents SPJ as distinct from both unstructured clinical judgment and actuarial assessment. A key distinguishing feature of SPJ is the contribution of modifiable factors, either dynamic or protective, to summary risk ratings. With modifiable factors, the summary rating scheme serves as a prognostic model rather than a classification procedure. However, prognostic models require more extensive and thorough predictive validity testing than can be provided by ROC analysis. It is proposed that validation should include calibration and reclassification techniques, as well as additional measures of discrimination. Several techniques and measures are described and illustrated. The paper concludes by tracing the limitations of ROC analysis to its philosophical foundation and its origin as a statistical theory of decision-making. This foundation inhibits the performance of crucial tasks, such as determining the sufficiency of a risk assessment and examining the evidentiary value of statistical findings. The paper closes by noting a current effort to establish a viable and complementary relationship between SPJ and decision-making theory.