Valuing structured professional judgment: predictive validity, decision-making, and the clinical-actuarial conflict

Area of Science:

• Psychology
• Forensic Science
• Risk Assessment

Background:

• Structured professional judgment (SPJ) is gaining traction as an alternative to unstructured clinical judgment and actuarial methods.
• Existing validation studies for SPJ often use receiver operating characteristic (ROC) analysis, a method also used for actuarial tools.

Purpose of the Study:

• To differentiate SPJ from unstructured clinical judgment and actuarial assessment.
• To propose advanced validation techniques for SPJ's prognostic models.
• To address the limitations of ROC analysis in evaluating SPJ.

Main Methods:

• Presenting SPJ as a prognostic model utilizing modifiable factors.
• Advocating for validation methods including calibration, reclassification, and additional discrimination measures.
• Analyzing the philosophical and theoretical underpinnings of ROC analysis.

Main Results:

• SPJ's unique contribution lies in its use of modifiable factors for risk ratings.
• ROC analysis is insufficient for the thorough predictive validity testing required for SPJ prognostic models.
• Limitations of ROC analysis hinder the assessment of risk assessment sufficiency and evidentiary value.

Conclusions:

• SPJ requires validation techniques beyond ROC analysis, such as calibration and reclassification.
• The philosophical basis of ROC analysis limits its utility for complex risk assessment evaluations.
• Further development is needed to integrate SPJ with decision-making theory.