Integrating Explainable Machine Learning in Clinical Decision Support Systems: Study Involving a Modified Design...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Human-Computer Interaction

Background:

• Clinical implementation of machine learning (ML) in healthcare lags due to a lack of transparency.
• Explainable machine learning (XML) and design thinking approaches can improve ML tool adoption.

Purpose of the Study:

• Explore clinician engagement to address ML tool non-adoption in clinical settings.
• Investigate challenges in presenting explainability within decision support interfaces.

Main Methods:

• Utilized a design thinking approach with theoretical frameworks, including the NASSS framework for problem definition.
• Developed a prognostic tool predicting intensive care unit admission likelihood from chest X-rays.
• Incorporated a metric framework to assess physician trust in AI tools (domain representation, actionability, consistency).

Main Results:

• Physicians found the prototype design elegant and data representation domain-appropriate.
• A simplified explainability overlay highlighted key predictive areas, explaining 90% of the risk score.
• Physicians valued the ability to compare multiple X-rays and toggle explainability for consistent assessment.

Conclusions:

• The approach aligns AI with clinician trust through a theoretical framework and prototyping.
• End-user integration throughout the design process is crucial for AI alignment.
• Engaging end-users early is vital for creating trustworthy and usable XML-based clinical decision support tools.