Effect of finite sample size on feature selection and classification: a simulation study.

Area of Science:

• Computer-aided diagnosis (CAD)
• Machine learning in medical imaging
• Biomedical data analysis

Background:

• Limited training samples are a key challenge in developing optimal computer-aided diagnosis (CAD) systems.
• Small datasets introduce bias and variance, affecting CAD system performance.
• Understanding feature selection and classifier interactions is crucial for effective CAD development.

Purpose of the Study:

• To evaluate the performance of various classifier and feature selection technique combinations.
• To assess the impact of class distribution, dimensionality, and training sample size on CAD system performance.
• To guide the development of effective CAD systems under sample size constraints.

Main Methods:

• Investigated stepwise feature selection (SFS), sequential floating forward search (SFFS), and principal component analysis (PCA).
• Evaluated Fisher's linear discriminant analysis (LDA) and support vector machine (SVM) classifiers.
• Simulated data from Gaussian distributions and clinical datasets, varying sample sizes (15-100 per class) and feature counts (50-200).

Main Results:

• Classifier and feature selection performance varied with feature space distributions, dimensionality, and sample size.
• LDA and SVM (radial kernel) showed similar performance, with SVM slightly better in some hold-out conditions.
• PCA was comparable or superior to SFS/SFFS for LDA with small samples but inferior for SVM (polynomial kernel).

Conclusions:

• No single feature selection-classifier combination offered consistently superior performance across all conditions.
• SFFS was comparable to SFS; PCA showed potential for Gaussian spaces with unequal covariance matrices.
• SVM with a radial kernel generally outperformed or matched SVM with a polynomial kernel.