What is test accuracy? Comparing unitary accuracy metrics for cognitive screening instruments

Area of Science:

• Neuroscience
• Gerontology
• Psychometrics

Background:

• Cognitive screening instruments are crucial for early detection of dementia and mild cognitive impairment.
• Accurate assessment of these instruments relies on appropriate statistical metrics.
• Existing metrics may have limitations in reflecting true diagnostic performance.

Purpose of the Study:

• To compare four common accuracy metrics: correct classification accuracy, area under the receiver operating characteristic curve (AUC), F-measure (F1 score), and Matthews Correlation Coefficient (MCC).
• To evaluate the performance of these metrics in assessing widely used cognitive screening instruments.
• To determine the most suitable metric for accuracy studies of cognitive tests.

Main Methods:

• Extracted raw data from accuracy studies of six cognitive screening instruments: Mini-Mental State Examination, Montreal Cognitive Assessment, Mini-Addenbrooke's Cognitive Examination, Six-item Cognitive Impairment Test, informant AD8, and Free-Cog.
• Calculated correct classification accuracy, AUC, F1 score, and MCC for each instrument.
• Analyzed the ordering of instruments based on each metric for diagnosing dementia and mild cognitive impairment.

Main Results:

• All four metrics produced a similar ranking of the cognitive screening instruments.
• Area under the ROC curve (AUC) provided the most optimistic accuracy values, while MCC yielded the most pessimistic.
• Correct classification accuracy and F1 score fell between AUC and MCC in terms of reported accuracy.

Conclusions:

• Each accuracy metric has inherent limitations and cannot be recommended as a sole definitive measure.
• The Matthews Correlation Coefficient (MCC) offers theoretical advantages and may be a more robust choice for future studies.
• Further research is needed to establish standardized outcome measures for cognitive screening test accuracy.