Unbiasing Fairness Evaluation of Radiology AI Model.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Fairness in AI and machine learning models is a significant concern, often stemming from imbalanced datasets.
• Existing efforts to minimize model bias may be insufficient due to potential biases in traditional fairness evaluation methods.
• Data uncertainty in minority groups within imbalanced datasets can compromise the accuracy of fairness judgments.

Purpose of the Study:

• To propose an innovative evaluation approach for assessing the fairness of AI models on imbalanced datasets.
• To address the limitations of traditional fairness evaluation methods that may themselves be biased.
• To develop a more objective statistical assessment by estimating data uncertainty in minority groups.

Main Methods:

• Introduced a novel evaluation approach incorporating bootstrapping from majority groups to estimate data uncertainty in minority groups.
• Conducted extensive experiments to compare the proposed method against traditional evaluation techniques.
• Utilized multiple evaluation metrics to account for varying results under different fairness criteria.

Main Results:

• Traditional evaluation methods may lead to inaccurate conclusions regarding model fairness.
• The proposed method provides an unbiased fairness assessment by effectively handling imbalanced datasets.
• The new approach enhances confidence in adopting AI models by providing a more reliable fairness evaluation.

Conclusions:

• A comprehensive evaluation scheme with multiple metrics is crucial for accurate fairness assessment.
• Estimating data uncertainty is vital for robust fairness judgments, especially with imbalanced data.
• The proposed bootstrapping-based method offers a more objective and reliable approach to evaluating AI model fairness.