A comparative analysis of AI scribes versus human documentation in simulated general practice consultations

Artificial intelligence (AI) scribes are emerging as transformative tools in healthcare to automatically generate clinical documentation from patient-clinician encounters. The aim of this study was to compare documentation quality between AI scribes and human-generated notes in simulated general practice consultations.

This was a cross-sectional study using The Royal Australian College of General Practitioners' clinical exam cases with four professional patient actors, two experienced general practitioners (GPs) and three blinded GP raters. Documentation quality was assessed using a modified Physician Documentation Quality Instrument (PDQI-9).

AI scribes demonstrated comparable or superior performance to human documentation using the modified PDQI-9, although the difference was not statistically significant (P = 0.071). Significant differences were found in the domains of accuracy (P = 0.022), thoroughness (P <0.001), succinctness (P <0.001) and freedom from hallucination (P = 0.025).

Commercially available AI scribes can potentially produce clinical documentation of comparable or superior quality to human documentation in simulated settings, particularly regarding accuracy, thoroughness and succinctness. The finding that both AI and human documentation contain 'hallucinations' challenges the assumption that human‑generated documentation represents the gold standard of clinical documentation quality. Further research is needed to evaluate performance in real-world settings.