Physician understanding, explainability, and trust in a hypothetical machine learning risk calculator

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems

Background:

• Machine learning (ML) implementation in healthcare faces challenges due to patients' right to understand AI-driven decisions.
• Physician comprehension and trust are crucial for integrating ML into clinical practice and communicating outputs to patients.

Purpose of the Study:

• To investigate the associations between physician understanding of ML outputs, their ability to explain these to patients, and their trust in ML.
• To evaluate the impact of different ML explainability methods on physician understanding, explainability, and trust.

Main Methods:

• A survey was designed for physicians facing diagnostic dilemmas resolvable by an ML risk calculator.
• Physicians rated their understanding, explainability, and trust in three ML outputs: one without explanation (control) and two using model-agnostic explainability methods.
• Cochran-Mantel-Haenszel tests assessed relationships between understanding, explainability, and trust.

Main Results:

• Significant associations were found between physician understanding and explainability (P<.001), understanding and trust (P<.001), and explainability and trust (P<.001).
• 88% of physicians preferred ML outputs with model-agnostic explanations over those without any explanation.
• No specific ML explainability method demonstrated a greater influence on intended physician behavior.

Conclusions:

• Physician understanding, explainability, and trust in ML risk calculators are interconnected.
• Model-agnostic explanations enhance physician preference for ML outputs, but do not alter their intended clinical actions.