A comparative analysis of AI scribes versus human documentation in simulated general practice consultations

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Artificial intelligence (AI) scribes are increasingly used for automated clinical documentation.
• Their performance compared to human-generated notes requires rigorous evaluation.

Purpose of the Study:

• To compare the documentation quality of AI scribes versus human-generated notes.
• To assess AI scribe performance in simulated general practice consultations.

Main Methods:

• Cross-sectional study utilizing simulated general practice consultations.
• Employed professional patient actors, experienced general practitioners (GPs), and blinded GP raters.
• Documentation quality assessed using a modified Physician Documentation Quality Instrument (PDQI-9).

Main Results:

• AI scribes demonstrated comparable or superior performance to human documentation (P = 0.071).
• Significant advantages for AI scribes in accuracy (P = 0.022), thoroughness (P <0.001), and succinctness (P <0.001).
• Both AI and human documentation exhibited 'hallucinations' (P = 0.025).

Conclusions:

• AI scribes can produce clinical documentation of comparable or superior quality in simulated settings.
• AI excels in accuracy, thoroughness, and succinctness compared to human documentation.
• The presence of 'hallucinations' in both AI and human notes necessitates further real-world research.