Selecting, Scaling, and Measuring the Value of Ambient AI in a Nonacademic Health System: Multiphase Pilot Study

Area of Science:

• Healthcare Administration
• Medical Informatics
• Artificial Intelligence in Medicine

Background:

• US health systems face financial and staffing pressures, exacerbated by physician burnout and retention challenges.
• Ambient artificial intelligence (AI) documentation tools offer potential solutions but face hurdles in vendor selection due to biases and unvalidated claims.
• Limited real-world testing hinders objective evaluation of AI documentation solutions in healthcare.

Purpose of the Study:

• To develop and implement an objective, multiphase evaluation and adoption strategy for an ambient AI documentation solution across a multihospital system.
• To overcome challenges in vendor selection by minimizing cognitive biases and unvalidated marketing claims.
• To validate the effectiveness and value of an ambient AI solution through real-world clinical testing and system-wide implementation.

Main Methods:

• A 3-phase evaluation process involving live clinical simulations with 4 vendors and standardized patient scripts.
• Physician scoring of AI-generated notes for accuracy, billing quality, and readability, followed by Epic workflow integration demonstrations.
• A 90-day pilot across 5 ambulatory specialties, system-wide implementation, and tracking of key performance indicators including documentation time, coding, financial trends, and patient/provider satisfaction.

Main Results:

• A 35.4% decrease in "pajama time" and a 28.3% reduction in time spent in notes.
• Shift towards higher-complexity coding (3.8% increase in level 4 visits) and an 8.5% increase in patient volumes, projecting significant revenue gains.
• Significant improvements in patient satisfaction across multiple domains (listening, trust, communication, information), exceeding prior initiatives; 81% system-wide adoption achieved.

Conclusions:

• A structured, multiphase evaluation process effectively minimized bias and validated real-world clinical testing for ambient AI solutions.
• The implemented ambient AI solution demonstrated measurable value through reduced documentation time, improved coding, increased patient volume, and enhanced patient satisfaction.
• This approach provides a practical framework for nonacademic health systems to objectively assess, implement, and scale AI solutions transparently and effectively.