How to interpret a small increase in AUC with an additional risk prediction marker: decision analysis comes through

Area of Science:

• Medical Informatics
• Decision Analysis
• Obstetrics

Background:

• Evaluating new risk prediction markers often involves assessing small changes in the area under the receiver operating characteristic curve (AUC).
• Researchers debate the utility of AUC changes, especially when markers have large odds ratios but yield only slight AUC increases.
• Statistical methods alone cannot definitively choose between odds ratios and AUC for evaluating markers.

Purpose of the Study:

• To determine the clinical utility of a small increase in AUC (0.02) from adding intrapartum markers to a risk prediction model for operative delivery.
• To apply decision analysis to quantify the value of improved risk prediction in obstetrics.

Main Methods:

• Utilized decision analysis incorporating costs and benefits to evaluate the addition of intrapartum markers.
• Calculated the risk threshold at which a patient is indifferent between elective cesarean section and usual care.
• Determined the number of women needing data collection for accurate risk prediction.

Main Results:

• An increase in AUC of 0.02 was observed when adding intrapartum markers to an antepartum model for predicting operative delivery.
• Decision analysis revealed that collecting intrapartum marker data for 68 to 124 women is required for each correct prediction.
• This data collection tradeoff was deemed clinically acceptable due to its non-invasive nature.

Conclusions:

• Adding intrapartum markers to risk prediction models provides clinically acceptable value, despite a small increase in AUC.
• Decision analysis offers a framework for interpreting the clinical significance of small AUC improvements in risk prediction.
• Intrapartum markers are valuable additions to antepartum models for predicting the risk of operative delivery.