The data representativeness criterion: Predicting the performance of supervised classification based on data set...

Area of Science:

• Machine Learning
• Data Science
• Medical Imaging

Background:

• Supervised classification algorithms are often reused across different datasets.
• Algorithm generalization and performance depend heavily on the similarity between training and unseen data.
• Predicting algorithm performance on new data is a significant challenge, hindering deployment.

Purpose of the Study:

• To introduce and validate the Data Representativeness Criterion (DRC) for quantifying training data representativeness.
• To assess the DRC's ability to measure data set similarity and its correlation with classification algorithm performance.
• To provide a tool for predicting potential performance degradation of classifiers on new, unseen data.

Main Methods:

• Proposed the Data Representativeness Criterion (DRC) as a novel metric for data set similarity.
• Conducted a proof-of-principle study to evaluate the DRC's quantification capabilities.
• Compared multiple magnetic resonance imaging (MRI) data sets with varying acquisition parameter differences.

Main Results:

• The DRC effectively quantifies the similarity between training and unseen data sets.
• DRC scores correlate with the performance of supervised classification algorithms.
• The DRC can indicate when classifier performance is likely to decrease based on data set similarity.

Conclusions:

• The Data Representativeness Criterion (DRC) offers a valuable tool for assessing data set similarity in machine learning.
• DRC provides an indication of potential classifier underperformance on new data, enabling informed deployment decisions.
• The flexibility of the DRC allows users to adjust its strictness based on acceptable performance thresholds.