Statistical independence for the evaluation of classifier-based diagnosis.

Area of Science:

• Computer-aided diagnosis
• Machine learning in healthcare
• Statistical evaluation methods

Background:

• Standard evaluation metrics for machine learning classifiers can be unreliable with unbalanced datasets.
• Unbalanced data, common in medical diagnosis (e.g., more healthy controls than patients), hinders accurate classification assessment.
• Existing methods struggle to differentiate true class discrimination from chance, particularly in binary classification tasks.

Purpose of the Study:

• To propose a novel evaluation framework for machine learning classification results in computer-aided diagnosis.
• To address the limitations of traditional metrics when dealing with unbalanced diagnostic datasets.
• To enhance the reliability of evaluating diagnostic classifiers by treating evaluation as a statistical independence test.

Main Methods:

• Recasting classification evaluation as a statistical independence test between predicted and actual diagnostic groups.
• Employing a Bayesian hypothesis testing framework to assess this independence.
• Developing a method robust to data imbalance and sensitive to dataset size.

Main Results:

• The proposed Bayesian hypothesis testing method effectively evaluates classification performance on unbalanced data.
• Experimental results on simulated and real-world Attention Deficit Hyperactivity Disorder (ADHD) diagnostic data demonstrate the approach's efficacy.
• The method provides a more reliable assessment compared to standard metrics, especially when class distributions are uneven.

Conclusions:

• The novel statistical independence test offers a superior method for evaluating machine learning classifiers in computer-aided diagnosis, particularly with unbalanced datasets.
• This Bayesian approach enhances the trustworthiness of diagnostic tool evaluations.
• The method is validated on both synthetic and clinical data, showing significant improvements in assessment accuracy.