Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning Ethics

Background:

• Demographic biases in datasets, particularly statistical imbalances in group representation, are a key cause of unfairness and discrimination in Machine Learning (ML) models.
• Existing metrics for quantifying these biases are diverse and often borrowed from other disciplines, necessitating a structured approach for selection and application.

Purpose of the Study:

• To review and classify existing metrics for measuring demographic biases in datasets.
• To develop a practical taxonomy for selecting appropriate bias measurement metrics.
• To analyze the practical characteristics and redundancy of bias metrics through a case study.

Main Methods:

• Comprehensive literature review of metrics for quantifying demographic representation imbalances.
• Development of a classification taxonomy for these metrics.
• Empirical case study involving 20 Facial Emotion Recognition (FER) datasets to analyze bias measurement metrics.

Main Results:

• Identified and categorized a range of metrics for assessing dataset demographic biases.
• The case study demonstrated that many existing bias measurement metrics are redundant.
• A reduced subset of metrics can be sufficient for effectively measuring demographic bias in datasets.

Conclusions:

• The proposed framework and taxonomy aid researchers in selecting appropriate metrics for bias measurement.
• Findings suggest that a streamlined set of metrics can efficiently quantify dataset bias.
• Provides actionable insights for mitigating dataset bias and enhancing fairness and accuracy in AI models.