Quality Assurance during the Rapid Implementation of an AI-Assisted Clinical Documentation Support Tool

AbstractWe describe a quality assurance evaluation process undertaken during the deployment of an artificial intelligence (AI)-assisted clinical documentation support tool throughout a large integrated delivery system. AI clinical documentation support tools have improved dramatically with the latest generation of large language models but have not been used in the variety of clinical specialties, geographic settings, and real-world situations available to large care delivery systems. In parallel with our deployment process, a small evaluation team assessed the level of risk for planned use cases, modified existing approaches, and leveraged the deployment teams and early adopters to provide a robust viewpoint on the technology through crowdsourced data and both quantitative and qualitative approaches. The results of the evaluation were used to modify training, provide feedback to the vendor, stage deployment, and transition to postdeployment monitoring.