Unseen Commercial Forces Could Undermine Artificial Intelligence Decision Support

AbstractArtificial intelligence (AI) is poised to transform health care, yet without robust safeguards, unseen commercial interests could distort care by prioritizing profit over patient well-being. The phenomenon of "biomarkup" underscores how subtle shifts in biomarker thresholds can fuel unwarranted testing, consultations, and treatments, ultimately undercutting cost-control efforts. It is easy to imagine a new generation of AI-based revenue cycle management model tools that achieve higher reimbursements by nudging clinicians toward more lucrative care pathways. AI-based decision support interventions are vulnerable across their entire development life cycle and could be manipulated to favor specific products or services. Furthermore, the growing consolidation of electronic health records and AI ecosystems restricts data access and risks stifling competition. Addressing these challenges requires a combination of technical and systemic solutions. Standardized application programming interfaces can foster electronic health record interoperability and mitigate vendor lock-in, while equitable and respectful data sharing and governance broadens participation in AI development. Robust oversight - including transparency mandates; rigorous auditing; addressing fraudulent, coercive, and monopolistic practices; and possibly new regulation - should curb exploitative practices and preserve AI's transformative potential for patient-centered care, cost containment, and societal benefit. (Funded by ARPA-H Biomedical Data Fabric Toolbox and others.).