Vector representation based on a supervised codebook for Nepali documents classification.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Machine Learning

Background:

• Document representation methods often struggle with outlier tokens, negatively impacting classification performance.
• Existing Nepali document representation techniques frequently lack effective outlier filtering strategies.
• Outlier tokens introduce uncertainty, hindering the accurate semantic understanding of documents.

Purpose of the Study:

• To propose a novel document representation method for Nepali text that effectively handles outlier tokens.
• To develop a supervised codebook containing only semantically relevant tokens for improved document representation.
• To enhance document classification performance for Nepali language data.

Main Methods:

• Developed a domain-specific supervised codebook based on token-class label similarity.
• Utilized probability-based word embedding for word representation within the codebook.
• Evaluated the method on four Nepali text datasets (A1, A2, A3, A4) using Support Vector Machine (SVM).
• Compared performance against established methods like Bag of Words, Latent Dirichlet Allocation, LSTM, Word2Vec, and BERT.

Main Results:

• Achieved state-of-the-art document classification accuracy across four Nepali datasets.
• Demonstrated superior performance with accuracies of 77.46% (A1), 67.53% (A2), 80.54% (A3), and 89.58% (A4).
• Outperformed widely used existing document representation methods on three datasets and showed comparable results on the fourth.
• Introduced the largest Nepali document dataset, NepaliLinguistic dataset (A4), to the research community.

Conclusions:

• The proposed supervised codebook method offers a robust approach to Nepali document representation by mitigating the impact of outlier tokens.
• This novel method significantly enhances document classification accuracy, setting a new benchmark for Nepali NLP tasks.
• The introduction of the NepaliLinguistic dataset facilitates future research in Nepali language processing and computational linguistics.