A feature selection approach based on term distributions.

Area of Science:

• Computer Science
• Data Science
• Artificial Intelligence

Background:

• Feature selection significantly impacts text categorization accuracy.
• Existing methods often overlook the crucial role of term frequency and distribution.
• A gap exists in algorithms that holistically consider term distribution within and across classes.

Purpose of the Study:

• To propose a new feature selection algorithm, FSATD, for enhanced text categorization.
• To address the limitations of current document-level feature selection approaches.
• To integrate term frequency, inter-class, and intra-class distributions into a unified selection process.

Main Methods:

• Developed FSATD, a feature selection approach leveraging comprehensive term distribution analysis.
• Incorporated three key factors: term frequency, inter-class term distribution, and intra-class term distribution.
• Utilized the k-Nearest Neighbors (kNN) classifier for experimental evaluation.

Main Results:

• FSATD demonstrated superior performance compared to traditional DF and t-Test algorithms.
• Experiments conducted on the 20NewsGroup and SougouCS corpora validated the effectiveness of FSATD.
• The synthetic consideration of term distribution factors led to improved categorization outcomes.

Conclusions:

• FSATD offers a more effective feature selection strategy for text categorization.
• Considering term frequency and distribution patterns comprehensively enhances classification performance.
• The proposed method provides a valuable advancement for natural language processing and machine learning applications.