Artificial Intelligence-Assisted Error Detection in Complex Clinical Documentation: Leveraging Large Language Models...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Documentation Improvement

Background:

• Clinical documentation errors in high-risk specialties like oncology can lead to severe patient harm.
• There is a critical need for advanced safety checks to ensure accuracy in complex medical records.

Purpose of the Study:

• To evaluate the capability of frontier large language models (LLMs) in identifying and correcting errors within complex oncology clinical documentation.
• To compare LLM performance against human specialists in detecting documentation errors.

Main Methods:

• A two-phase evaluation using synthetic clinical vignettes and discharge summaries in hematology/oncology.
• Assessed LLMs (GPT-4-mini, Gemini 2.5 Pro, Gemma 3 27B) against human clinicians for error detection and localization.
• Benchmarked LLM performance against human expert data for accuracy and speed.

Main Results:

• LLMs, particularly Gemini 2.5 Pro, significantly outperformed human specialists in error detection and localization tasks.
• Gemini 2.5 Pro achieved high accuracies (0.928 for flagging, 0.915 for localization) and identified 97.8% of errors in discharge summaries.
• Advanced LLMs demonstrated superior speed and accuracy, with potential for synergistic human-AI collaboration.

Conclusions:

• Frontier LLMs offer superior error-detection capabilities and efficiency compared to human specialists and local models.
• LLMs can function as powerful assistants to reduce clinician workload and documentation errors.
• Integrating LLM-driven error flagging into EHRs can enhance oncology documentation accuracy, treatment quality, and patient safety.