Identification of offensive language in Urdu using semantic and embedding models.

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Social Media Analysis

Background:

• Automatic detection of offensive language is crucial for online safety.
• Generalizing solutions across languages is challenging due to linguistic diversity.
• Prior research on offensive language detection has primarily focused on Western languages, with limited work on low-resource languages like Urdu.

Purpose of the Study:

• To develop and evaluate a robust system for offensive language detection in Urdu.
• To create a new, comprehensive dataset for Urdu offensive language identification.
• To explore and compare various feature engineering techniques for this task.

Main Methods:

• A new dataset of 7,500 Urdu Facebook posts was curated for offensive language detection.
• Four feature engineering models were employed: Term Frequency-Inverse Document Frequency (TF-IDF), Bag-of-Words, Word n-grams, and Word2Vec embeddings.
• Experiments utilized standalone models and ensemble methods, including stacking and wrapper-based feature selection.

Main Results:

• The Word2Vec embedding model, when used with a stacking ensemble, achieved 88.27% accuracy as a standalone model.
• A hybrid model combining TF-IDF, Bag-of-Words, and Word2Vec features reached 90% accuracy and 97% Area Under the Curve (AUC).
• The proposed hybrid approach significantly outperformed baseline methods across multiple performance metrics.

Conclusions:

• The developed hybrid feature model demonstrates superior performance for Urdu offensive language detection.
• The creation of a dedicated Urdu dataset addresses a gap in low-resource language NLP research.
• Findings have practical implications for developing content moderation tools and enhancing online safety in Urdu-speaking communities.