Evaluating the Usability, Technical Performance, and Accuracy of Artificial Intelligence Scribes for Primary Care:...

Area of Science:

• Medical Informatics
• Human-Computer Interaction
• Primary Care Research

Background:

• Primary care providers (PCPs) experience burnout due to extensive administrative and documentation tasks.
• Artificial intelligence (AI) scribes offer a potential solution to automate clinical documentation and alleviate PCP administrative burden.
• Limited information exists on the usability, effectiveness, and accuracy of AI scribes in primary care settings.

Purpose of the Study:

• To develop and implement a framework for systematically evaluating AI scribes used in primary care.
• To assess the usability, technical performance, and accuracy of various AI scribes.
• To provide benchmark data for AI scribe research, policy, and practice.

Main Methods:

• A systematic comparison of 6 AI scribes using competitive analysis and an expert-developed evaluation framework.
• The framework assessed usability, effectiveness, technical performance, accuracy, and quality.
• Standardized patient encounter audio files were used to generate AI-generated SOAP notes, compared against benchmarks.

Main Results:

• AI scribes demonstrated varied performance; most could integrate with electronic medical records but had limited data exchange.
• SOAP notes were generated rapidly (approx. 1 minute for 15-min encounters), though time increased with complexity.
• All AI scribes produced good to excellent quality notes, but none were error-free, with common errors including omissions and structural issues.

Conclusions:

• This study provides a systematic evaluation of AI scribes in primary care, highlighting their potential and current limitations.
• Improvements in usability, technical performance, and accuracy are necessary for AI scribes to fully reduce PCP documentation burden.
• Future research should focus on real-world performance across diverse populations to ensure equitable and reliable AI scribe application.