Improved feature-selection method considering the imbalance problem in text categorization.

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Filtering feature-selection algorithms are crucial for dimensionality reduction in text categorization.
• Existing algorithms often overlook dataset imbalance, negatively impacting performance.
• Dataset imbalance is a significant challenge in text classification.

Purpose of the Study:

• To propose a novel scheme that mitigates the adverse effects of dataset imbalance on feature selection.
• To enhance the performance of established feature selection methods in imbalanced text data.

Main Methods:

• Proposed a new scheme to address dataset imbalance in feature selection.
• Evaluated nine improved feature-selection methods: Information Gain, Chi statistic, Document Frequency, Orthogonal Centroid Feature Selection, DIA association factor, Comprehensive Measurement Feature Selection, Deviation from Poisson Feature Selection, improved Gini index, and Mutual Information.
• Utilized naïve Bayes and support vector machines classifiers on three benchmark datasets: 20-Newsgroups, Reuters-21578, and WebKB.

Main Results:

• The improved scheme significantly enhanced the performance of the evaluated feature-selection methods.
• The proposed approach effectively weakens the adverse impact of dataset imbalance.
• Demonstrated superior performance of enhanced algorithms on benchmark document collections.

Conclusions:

• The developed scheme offers a robust solution for handling imbalanced datasets in text categorization.
• The improved feature selection methods show substantial gains in text classification accuracy.
• This work contributes to more effective dimensionality reduction techniques for imbalanced text data.