A-Test Method for Quantifying Structural Risk and Learning Capacity of Supervised Machine Learning Methods.

Area of Science:

• Biomedical Engineering
• Bioinformatics
• Machine Learning

Background:

• Supervised Machine Learning (ML) methods are widely used in bioinformatics and biomedical engineering.
• Overfitting is a common challenge in ML, particularly with small or medium-sized datasets.
• Existing validation methods like K-fold and random sub-sampling may not fully capture ML performance in these scenarios.

Purpose of the Study:

• To introduce and evaluate the A-Test method, an original approach for assessing supervised ML performance.
• To investigate the structural risk and learning capacity of ML methods quantitatively.
• To provide a robust validation technique for ML models trained on limited data.

Main Methods:

• Development of the A-Test method for quantitative analysis of ML performance.
• Application of A-Test to validate two ML methods using real-world heart sound signal datasets.
• Comparison of A-Test with K-fold validation and repeated random sub-sampling.

Main Results:

• The A-Test method was successfully applied to datasets of children's heart sound signals, including normal and pathological cases (aortic stenosis, ventricular septal defect, mitral regurgitation, pulmonary stenosis).
• A-Test provided more comprehensive and realistic performance information compared to traditional validation techniques.
• The study demonstrated the effectiveness of A-Test in scenarios prone to overfitting.

Conclusions:

• The A-Test method is a powerful and original validation tool for supervised ML, particularly effective for small to medium datasets.
• It offers superior insights into structural risk and learning capacity, addressing overfitting challenges.
• A-Test represents a significant advancement over existing validation methods in bioinformatics and biomedical applications.