GECOBench: a gender-controlled text dataset and benchmark for quantifying biases in explanations

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence (AI)
• Machine Learning Bias

Background:

• Large pre-trained language models (LPLMs) are foundational for NLP tasks.
• LPLMs can inherit societal biases, like gender bias, from training data.
• The impact of these biases on feature attributions from explainable AI (XAI) techniques is not well understood.

Purpose of the Study:

• To systematically investigate how biases in LPLMs affect XAI feature attributions.
• To quantify the bias inherited by feature attributions during pre-training.
• To assess the effectiveness of fine-tuning in mitigating explanation bias.

Main Methods:

• Creation of GECO, a gender-controlled dataset with ground truth attributions for gender classification.
• Application of GECO to a pre-trained BERT model, fine-tuned to varying degrees.
• Development of GECOBench, a quantitative framework for evaluating XAI methods.

Main Results:

• Pre-training induces undesirable bias in feature attributions.
• Fine-tuning, especially of embedding layers, significantly mitigates explanation bias.
• Explanation performance shows a clear dependency on the number of fine-tuned layers.

Conclusions:

• Biases in LPLMs demonstrably affect XAI methods.
• Fine-tuning and retraining embedding layers are effective strategies to reduce explanation bias.
• GECO and GECOBench provide essential tools for evaluating and improving the fairness of XAI.