An efficient data preprocessing approach for large scale medical data mining.

Area of Science:

• Medical data mining
• Bioinformatics
• Machine learning

Background:

• Large medical datasets increase computational costs for data mining.
• Instance selection is a crucial preprocessing step to reduce data size and maintain mining quality.
• Existing instance selection methods are time-consuming for very large datasets.

Purpose of the Study:

• Introduce an efficient data preprocessing (EDP) approach for large-scale medical datasets.
• Reduce computational cost and time required for instance selection.
• Maintain high classification accuracy after data preprocessing.

Main Methods:

• The proposed EDP approach involves two steps: training a model on a small subset of data after instance selection, then using this model to identify noisy data in the larger set.
• Experiments utilized two large medical datasets (breast cancer, protein homology) with over 100,000 samples.
• Compared EDP with established instance selection algorithms (IB3, DROP3, genetic algorithms) and classification techniques (CART, k-NN, SVM).

Main Results:

• The EDP approach reduced computational costs by approximately two to three times compared to state-of-the-art algorithms.
• The method effectively maintained the final classification accuracy of the models.
• Demonstrated significant efficiency and effectiveness in large-scale instance selection.

Conclusions:

• Directly applying existing instance selection algorithms to large medical datasets incurs high computational costs.
• The proposed EDP approach effectively addresses this by training a model to distinguish between useful and noisy data.
• EDP offers an efficient and effective solution for large-scale instance selection, balancing computational complexity and classification accuracy.