Evaluating risk assessments using receiver operating characteristic analysis: rationale, advantages, insights, and...

Area of Science:

• Forensic Psychology
• Psychometrics
• Clinical Psychiatry

Background:

• The past 20 years have seen a significant shift in how mental health professionals evaluate violence risk.
• There's a move from believing prediction is impossible to confidence in assessing risk with above-chance accuracy.

Purpose of the Study:

• To review Receiver Operating Characteristic (ROC) methods and their role in violence risk assessment.
• To explain the area under the ROC curve (AUC) and its relation to effect size d.
• To discuss the limitations of these indices in practical clinical decision-making.

Main Methods:

• Review of key concepts in Receiver Operating Characteristic (ROC) analysis.
• Explanation of the area under the ROC curve (AUC) as a measure of discrimination.
• Exploration of the relationship between AUC and effect size d in the context of violence risk.

Main Results:

• ROC analysis and associated indices (AUC, d) have contributed to increased confidence in violence risk assessment.
• AUC and d offer concise but incomplete measures of discrimination capacity.
• These global indices do not provide information on sensitivity-specificity trade-offs or error balancing.

Conclusions:

• Global accuracy indices like AUC and d are insufficient for determining the practical utility of violence risk assessment tools.
• Clinical practice justification requires a contextual evaluation of outcomes beyond summary accuracy metrics.
• A deeper analysis of sensitivity-specificity trade-offs and error management is necessary for informed clinical decisions.