Putting explainable AI in context: institutional explanations for medical AI

Area of Science:

• Medical Informatics
• Artificial Intelligence
• Philosophy of Science

Background:

• The explainability of machine learning (ML) systems in medicine is debated.
• Arguments exist for post hoc explanations versus relying solely on system accuracy.
• Current approaches may not fully address medical professionals' epistemic concerns.

Purpose of the Study:

• To evaluate the necessity and nature of explainability for medical ML systems.
• To propose an alternative framework for justifying the use of these systems.
• To enhance the epistemic justification for medical professionals using ML tools.

Main Methods:

• Conceptual analysis of explainability in ML for healthcare.
• Examination of limitations in post hoc explanations and accuracy-based justifications.
• Development of the concept of 'institutional explanation' for ML systems.

Main Results:

• Both post hoc explanations and high accuracy have limitations in addressing user trust.
• Individual decision explanations do not fully resolve epistemic worries of medical professionals.
• An institutional explanation framework is proposed to build trust and ensure reliable use.

Conclusions:

• Medical ML systems require institutional explanations focusing on practical reliance.
• Transparency in system design, evaluation metrics, and bias auditing is crucial.
• Institutional explanations make accuracy and post hoc explanations more meaningful for end-users.