The Great Scribe-Off: A Comparative Analysis of AI Scribes Versus Human Documentation in Simulated General Practice...

Clinical documentation burden remains a significant challenge in healthcare, particularly in primary care settings. Artificial intelligence (AI) scribes have emerged as potential solutions, but their effectiveness compared to human documentation lacks robust evidence, especially in community general practice environments. Documentation quality is compared between four commercial AI scribes and human-generated notes using four standardised clinical scenarios from the Royal Australian College of General Practitioners examination repository in simulated general practice consultations. Three experienced general practitioners, blinded to the source, assessed quality using a modified Physician Documentation Quality Instrument (PDQI-9). AI-generated notes outperformed human documentation across multiple quality domains. Top AI scribes scored a mean of 44.08/50 (SD = 3.32) vs. 37.42 (SD = 9.78) for humans, excelling in thoroughness (M = 4.92), accuracy (M = 4.67), and freedom from bias (M = 4.92). Inter-rater reliability was high for thoroughness (ICC = 0.879) and accuracy (ICC = 0.745), but lower for subjective areas like synthesis (ICC = 0.082). This study shows that AI scribes can outperform traditional documentation in simulated general practice. Successful implementation, however, depends on workflow integration and customisation. Standardised evaluation and balancing consistency with clinical context are key. Future research should explore real-world use, focusing on customisation and workflow impact.