Detecting representative data and generating synthetic samples to improve learning accuracy with imbalanced data...

Area of Science:

• Machine Learning
• Data Mining
• Artificial Intelligence

Background:

• Imbalanced datasets pose significant challenges for machine learning models, leading to poor classification performance.
• Classifiers are often biased towards majority classes, neglecting minority classes and hindering convergence.

Purpose of the Study:

• To develop an effective data pre-processing strategy for imbalanced datasets.
• To improve classification performance by balancing data distribution.

Main Methods:

• A novel data pre-processing (PPDP) method is proposed, combining majority data reduction and minority data synthesis.
• Majority data reduction involves outlier exclusion using box-and-whisker plots and representative data selection via Mega-Trend-Diffusion.
• Minority data synthesis is achieved by proposing a novel hypothesis to model minority data distribution and generate samples.

Main Results:

• The proposed PPDP method, integrated with the D3C method (PPDP+D3C), demonstrated superior classification performance.
• Paired t-tests showed significant improvements in Accuracy, G-mean, and F-measure compared to One-Sided Selection (OSS), SMOTEBoost (SB), and Normal Distribution-based Oversampling (NDO).

Conclusions:

• The proposed PPDP method effectively balances imbalanced datasets.
• This approach significantly enhances the classification performance of machine learning models on imbalanced data.