From co-creation to technical bias detection methods: an interdisciplinary showcase from the BIAS project

Area of Science:

• Computational Linguistics
• Sociolinguistics
• Artificial Intelligence Ethics

Background:

• Machine learning models and word embeddings can reflect and amplify societal stereotypes.
• Existing bias detection methods predominantly focus on English, neglecting multilingual and cultural nuances.
• Direct translation of English bias benchmarks may fail to capture language-specific biases.

Purpose of the Study:

• To introduce novel metrics, BIAS-WEAT and BIAS-SEAT, for detecting biases in word embeddings and language models.
• To address the limitations of English-centric bias detection by developing methods for Dutch, German, Icelandic, Italian, Norwegian, and Turkish.
• To bridge the gap between real-world societal biases and technical bias evaluation in NLP.

Main Methods:

• Development of BIAS-WEAT and BIAS-SEAT metrics tailored for multiple languages.
• Utilizing insights from co-creation workshops with native speakers to identify context-specific biases (e.g., hiring situations).
• Application of metrics to general-purpose language resources to evaluate embedded biases.

Main Results:

• Demonstrated that language models embed and reproduce biases specific to their linguistic and geographic contexts.
• Identified language- and culture-specific biases in Dutch, German, Icelandic, Italian, Norwegian, and Turkish language models.
• Validated the effectiveness of BIAS-WEAT and BIAS-SEAT in capturing nuanced biases.

Conclusions:

• Culturally grounded approaches are essential for effective bias detection in NLP.
• Multilingual and context-aware bias evaluation is crucial for developing equitable AI systems.
• The developed metrics offer a valuable tool for researchers and practitioners working with non-English language models.