Compressed models for co-reference resolution: enhancing efficiency with debiased word embeddings

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Computational Linguistics

Background:

• Word embeddings capture semantic relationships but can inherit societal biases, such as gender bias.
• Existing debiasing methods may impact the utility of embeddings for downstream tasks.

Purpose of the Study:

• To develop and evaluate a comprehensive approach for reducing gender bias in GloVe word embeddings.
• To assess the impact of debiased embeddings on Natural Language Processing (NLP) tasks, including co-reference resolution and text classification.
• To investigate the effectiveness of debiasing techniques on resource-efficient, compressed NLP models.

Main Methods:

• Two GloVe embedding variations (840B and 50) were debiased by identifying and reducing the gender direction.
• Gender bias was quantified using the Word Embedding Association Test.
• Performance was evaluated on co-reference resolution and text classification tasks using accuracy metrics.
• Context preservation was analyzed using a Twitter misinformation dataset.

Main Results:

• Debiased embeddings demonstrated reduced gender bias.
• Models trained on debiased embeddings maintained or improved accuracy in co-reference resolution and text classification.
• Debiasing techniques proved effective even for compressed NLP models.
• Analysis of context preservation indicated the practical utility of debiased embeddings.

Conclusions:

• Comprehensive debiasing of word embeddings is feasible and beneficial for NLP tasks.
• Debiased embeddings retain semantic information crucial for model performance.
• This research pioneers the application of compression techniques to debiased NLP models, offering insights for real-world applications like person profiling.