Generalized Term Similarity for Feature Selection in Text Classification Using Quadratic Programming.

Area of Science:

• Computer Science
• Information Science
• Data Science

Background:

• The proliferation of internet technologies has resulted in a massive increase in electronic documents.
• Text categorization is crucial for managing and organizing big data from unstructured documents.
• The bag-of-words model is a common, simple representation for text classification, but it leads to a large feature space.

Purpose of the Study:

• To propose a new feature selection method for text categorization.
• To address the issue of redundant terms in the bag-of-words model.
• To improve the accuracy of text classification by considering term similarity.

Main Methods:

• A novel feature selection method is proposed, incorporating term similarity alongside term ranking.
• Term similarity is quantified using methods like mutual information.
• A quadratic programming-based numerical optimization approach is used to balance term ranking and similarity.

Main Results:

• The proposed feature selection method effectively reduces redundant terms.
• Experimental results show higher accuracy compared to conventional feature selection methods.
• Considering term similarity in feature selection is proven to be effective.

Conclusions:

• The developed feature selection technique offers improved performance in text categorization.
• Balancing term ranking and term similarity is key to enhancing classification accuracy.
• This method provides a more efficient way to manage and classify large volumes of text data.