Pitfalls and Best Practices in Evaluation of AI Algorithmic Biases in Radiology.

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Algorithmic Fairness and Bias

Background:

• Growing awareness of fairness issues in artificial intelligence (AI) models used in radiology.
• Challenges in evaluating AI biases due to incomplete demographic data, variable definitions, and inconsistent statistical measures.
• Need for standardized approaches to assess AI bias in medical imaging.

Purpose of the Study:

• To summarize underrecognized pitfalls in the evaluation and measurement of algorithmic biases in radiology AI.
• To guide the appropriate assessment of AI biases within medical imaging datasets.
• To provide actionable best practices for ensuring fairness in AI technologies.

Main Methods:

• Review and synthesis of potential pitfalls in AI bias evaluation across technical and social contexts.
• Categorization of pitfalls into three key areas: medical imaging datasets, demographic definitions, and statistical evaluations.
• Identification of actionable best practices and future directions to mitigate identified pitfalls.

Main Results:

• Identified complexities in AI bias evaluation, including incomplete demographic reporting and variable definitions.
• Highlighted technical pitfalls related to statistical definitions of bias and social context pitfalls concerning demographic conventions.
• Proposed best practices for medical imaging datasets, demographic definitions, and statistical bias evaluations.

Conclusions:

• Addressing underrecognized pitfalls in AI bias evaluation is crucial for the responsible implementation of AI in radiology.
• Actionable strategies across datasets, demographics, and statistical methods can help mitigate AI bias.
• Ensuring fairness in AI technologies is essential for equitable healthcare outcomes for all patients.