A normalization model for repeated letters in social media hate speech text based on rules and spelling correction

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Social Media Analysis

Background:

• Online hate speech is increasing with social media use.
• Words with repeated letters contribute to out-of-vocabulary (OOV) issues in hate speech detection.
• Existing models struggle to correctly normalize OOV words with repeated letters.

Purpose of the Study:

• To develop an improved unsupervised model for normalizing OOV words with repeated letters.
• To enhance the accuracy of hate speech detection by improving text normalization.
• To replace OOV words with correct in-vocabulary (IV) alternatives.

Main Methods:

• Combined rule-based patterns for repeated letters with the SymSpell algorithm.
• Developed rules based on letter repetition position (beginning, middle, end) and pattern.
• Utilized an unsupervised approach, avoiding special dictionaries or annotated data.

Main Results:

• Reduced the percentage of OOV words to 8%.
• Achieved F1 scores 9% and 13% higher than two benchmark studies.
• Demonstrated superior performance in replacing OOV words with correct IV replacements.

Conclusions:

• Rule-based patterns combined with spelling correction effectively normalize words with repeated letters.
• The proposed normalization model significantly improves hate speech detection performance.
• This unsupervised method offers a viable solution for enhancing text normalization in NLP tasks.