Receiver operating characteristic plot and area under the curve with binary classifiers: pragmatic analysis of...

Area of Science:

• Cognitive assessment
• Psychometrics
• Medical diagnostics

Background:

• Receiver operating characteristic (ROC) plots and area under the curve (AUC) are common metrics for evaluating diagnostic tests.
• Assessing cognitive screening instruments requires careful consideration of how their data is treated (binary, categorical, or continuous).

Purpose of the Study:

• To investigate the potential for misleading AUC values when cognitive screening instruments are assessed as binary predictors versus categorical or continuous scales.
• To determine if dichotomizing cognitive screeners affects the accuracy of AUC calculations.

Main Methods:

• Calculated AUC using rank-sum and diagnostic odds ratio methods.
• Utilized data from test accuracy studies involving binary classifiers, a categorical screener (Codex), and a continuous scale screener (Mini-Addenbrooke's Cognitive Examination).

Main Results:

• AUC calculated via the diagnostic odds ratio method was consistently higher than via the rank-sum method across all screeners.
• When categorical (Codex) and continuous (Mini-Addenbrooke's Cognitive Examination) screeners were analyzed as binary tests, the rank-sum AUC was lower compared to their analysis as categorical or continuous scales, respectively.

Conclusions:

• Dichotomizing cognitive screeners that yield categorical or continuous data can lead to an underestimate of the calculated AUC.
• This underestimation potentially affects the accurate assessment of cognitive screening test performance.