Automated chain-of-thought evaluation framework for large language model-generated emergency department...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Emergency Medicine

Background:

• Automated assessment of emergency department (ED) documentation is crucial for efficiency.
• Large language models (LLMs) show promise but require robust evaluation frameworks.
• Aligning AI-generated clinical notes with expert judgment is a significant challenge.

Purpose of the Study:

• To develop and validate MEDIVAL, a progressive Chain-of-Thought (CoT) evaluation framework.
• To assess the alignment of LLM-generated ED documentation with expert clinical judgment.
• To enhance automated evaluation of clinical notes in acute care settings.

Main Methods:

• Developed a three-tier framework: Persona-based, Error-enhanced, and Insight-integrated CoT strategies.
• Tested on four LLMs (GPT-4o, GPT-4.1, Claude-3.5, Claude-3.7) using 33 ED records.
• Evaluated by four expert emergency physicians across five criteria: Appropriateness, Accuracy, Structure/Format, Conciseness, and Clinical Validity.

Main Results:

• All LLMs improved expert alignment with increased CoT complexity.
• Claude-3.7 (r=0.712) and GPT-4o (r=0.702) showed strongest correlations with the Insight-integrated strategy.
• GPT-4.1 demonstrated the largest relative improvement (43.3% increase).
• High reproducibility confirmed (ICC > 0.919), with Claude-3.5 showing highest consistency.

Conclusions:

• MEDIVAL effectively enhances automated evaluation of ED documentation using progressive CoT strategies.
• The framework maintains high reproducibility, offering a viable pre-screening tool.
• Supports reliable AI integration into emergency medicine workflows, reducing expert workload.