Developer Perspectives on Potential Harms of Machine Learning Predictive Analytics in Health Care: Qualitative...

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Regulatory Science

Background:

• Machine learning predictive analytics (MLPA) is increasingly used in healthcare, presenting both opportunities for improved efficacy and risks of patient harm and eroded trust.
• Existing guidelines for evaluating MLPA safety in healthcare lack universally accepted practices, with a trend towards self-regulation.
• Limited research exists on MLPA developers' perspectives regarding potential harms, despite their crucial role in bridging the gap between principles and practice.

Purpose of the Study:

• To understand how developers of MLPA tools for healthcare perceive potential harms.
• To investigate developers' responses to recognized harms associated with their MLPA products.

Main Methods:

• Qualitative analysis of interviews with 40 MLPA developers from 15 US-based organizations.
• Interviews explored developers' views on potential harms, influencing factors, and mitigation strategies.
• Participants included data scientists, software engineers, and management personnel across diverse organizational types and sizes.

Main Results:

• Developers acknowledged a spectrum of potential harms from MLPA, including privacy violations, bias, and system disruptions.
• Identified harm drivers linked to MLPA characteristics and the healthcare/commercial contexts.
• Developers proposed mitigation strategies like balancing performance with safety, integrating clinical expertise, and fostering shared values, though responsibility acceptance varied widely.

Conclusions:

• Despite recognizing potential harms, robust procedures for assessing and mitigating MLPA risks in healthcare are absent.
• Self-regulatory paradigms face challenges due to developers viewing certain harms as inherent to healthcare and business.
• Effective self-regulation necessitates developers' acceptance of safety and efficacy responsibilities, highlighting a need for substantial education to close the "principles-to-practice" gap.