Assessing Healthcare Stakeholder Understanding of Machine Learning Documentation

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Machine Learning Interpretability

Background:

• Machine Learning (ML) models are crucial for advanced clinical decision support systems.
• Current ML documentation presents technical barriers for healthcare stakeholders.
• Understanding ML documentation is vital for effective clinical integration.

Purpose of the Study:

• To assess healthcare stakeholder comprehension of ML model documentation for a sepsis prediction tool.
• To identify barriers hindering the understanding of technical ML documentation.
• To gather recommendations for improving ML documentation clarity.

Main Methods:

• An interdisciplinary workshop was conducted with 24 participants.
• Participants' understanding of sepsis prediction ML model documentation was evaluated.
• AI literacy levels and comprehension percentages were measured.

Main Results:

• Participants demonstrated moderate AI literacy (mean score: 40.13/65).
• On average, 65% of the ML model documentation was understood.
• Barriers included technical jargon, unexplained abbreviations, lack of context, and dense data presentation.

Conclusions:

• Technical documentation impedes the usability and integration of ML models in clinical workflows.
• Simplified language, clear explanations, and visual aids are essential for enhancing understanding.
• Improving documentation clarity will facilitate the adoption of AI tools in healthcare.