Implementation Evaluation of an AI Scribing Tool in the Ambulatory Setting: Single-Site Qualitative Implementation Evaluation Study

AI scribing tools to assist health care providers by generating clinical notes are rapidly spreading across health care settings. Although these tools may reduce documentation burden, successful implementation depends on the contextual factors influencing their integration into clinical practice. Implementation evaluations are needed to inform practice integration, guide future adoption, and enhance effective use of this novel technology.

This study aims to explore contextual factors influencing the implementation of an AI scribe for physicians in the ambulatory setting.

This study was a single-site qualitative implementation evaluation at a metropolitan university health system where the AI scribe was available across primary and specialty care clinics. The Practical, Robust Implementation and Sustainability Model (PRISM) and the Sociotechnical Systems framework guided the evaluation. Using maximum variation sampling, we recruited attending physicians with varied levels of AI scribe use. Semistructured interviews were conducted with attending physicians working in the ambulatory setting. Data collection and analysis occurred iteratively until thematic saturation was achieved. Interviews were analyzed using thematic analysis with a constant comparative method, incorporating both deductive and inductive coding. Resulting subthemes were mapped onto corresponding domains of the PRISM and Sociotechnical Systems frameworks.

A total of 25 physicians were interviewed, including 7 who had never used the AI scribe, 9 who used it minimally, and 9 who used it frequently. Participants represented a broad range of specialties. A total of 5 themes and 15 subthemes emerged. First, workflow integration influenced adoption. Limitations in customization were a barrier to adoption, while flexibility among users was critical to fit. Second, ease of tool usability facilitated initial adoption, whereby users tried the tool without paying close attention to training materials. However, this lack of training engagement hindered widespread sustained use due to limited knowledge of advanced functionalities. Third, attending physicians balanced trust and confidence in AI-generated documentation with mistrust and professional accountability. Many described a "trust but verify" approach. Also, some concerns about others' inadequate review of AI-generated content weakened physician-physician trust. Fourth, attending physicians viewed the ubiquitous presence of AI scribing as inevitable, driven by organizational priorities related to productivity, cost, and burnout. Interviewees disagreed about the appropriate use of AI scribes by trainees. Finally, identical mechanisms were viewed as both benefits and drawbacks. Increased efficiency from AI documentation was perceived by some as reducing burden and cognitive load, but perceived by others as potentially intensifying workload expectations. Similarly, some thought AI scribes facilitated more patient-centered communication, while others thought it had the opposite impact.

Implementation of AI scribes in ambulatory care is shaped by complex sociotechnical and contextual factors. Adoption is influenced by workflow alignment, trust, and preexisting beliefs about the tool's benefits and drawbacks, illustrating how clinical context affects AI scribe implementation.