A comprehensive evaluation of oversampling techniques for enhancing text classification performance.

Area of Science:

• Natural Language Processing
• Machine Learning
• Data Science

Background:

• Class imbalance is a critical challenge in text classification, impairing model learning of minority classes.
• Skewed data distributions, following the 'garbage in, garbage out' principle, can lead to suboptimal performance even with advanced models.

Purpose of the Study:

• To systematically benchmark the effectiveness of the Synthetic Minority Over-sampling Technique (SMOTE) and its 30 variants.
• To evaluate these oversampling methods in the context of transformer-embedded text classification.
• To provide practical guidance for selecting appropriate oversampling techniques for imbalanced datasets.

Main Methods:

• Utilized two benchmark datasets: TREC and Emotions.
• Employed the MiniLMv2 transformer model for semantic text vectorization.
• Applied six distinct machine learning algorithms for classification tasks.
• Compared performance using F1-Score and Balanced Accuracy under balanced and imbalanced scenarios.
• Validated results using the Friedman test for statistical significance.

Main Results:

• Demonstrated significant performance variations among different SMOTE variants across datasets and classifiers.
• Identified specific SMOTE techniques that effectively mitigate the negative impact of class imbalance.
• Provided empirical evidence on the suitability of oversampling methods for transformer-based text classification.

Conclusions:

• The choice of SMOTE variant significantly impacts the performance of text classifiers on imbalanced data.
• Transformer embeddings combined with appropriate oversampling techniques can lead to more robust and fair NLP models.
• This large-scale benchmarking offers practical insights for practitioners dealing with imbalanced text classification challenges.