Improving the text classification using clustering and a novel HMM to reduce the dimensionality.

Area of Science:

• Computer Science
• Machine Learning
• Natural Language Processing

Background:

• High dimensionality in text representations burdens computational processes in machine learning.
• Effective document representation is crucial for successful text classification.
• Existing methods require optimization for large-scale real-world data.

Purpose of the Study:

• To propose a novel dimensionality reduction technique for text representation.
• To enhance the efficiency and performance of text classification systems.
• To evaluate the effectiveness of the proposed method against established techniques.

Main Methods:

• Document representation using a clustering technique and a Hidden Markov Model (HMM).
• Dimensionality reduction applied as a preprocessing step for text classification.
• Evaluation using k-Nearest Neighbors (k-NN) and Support Vector Machine (SVM) classifiers on OHSUMED and TREC corpora.

Main Results:

• The proposed dimensionality reduction technique yielded highly satisfactory experimental results.
• Performance was superior compared to commonly used methods like InfoGain.
• Statistical tests confirmed the technique's suitability for text classification preprocessing.

Conclusions:

• The novel clustering and HMM-based dimensionality reduction method is effective for text classification.
• The technique offers a significant improvement in efficiency and performance.
• This approach is well-suited for the preprocessing stage in text classification tasks.